Data report generation method and device, computer device, and storage medium

Abstract

The application discloses a data report generation method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a target download task, wherein the target download task comprises task detail information; obtaining a target task name corresponding to the target download task; obtaining target download information corresponding to the target task name; obtaining a target data report corresponding to a data index from a big data set corresponding to a download path, and storing the target data report in a storage space corresponding to an output directory; and in response to a data query instruction containing the data index, reading the target data report corresponding to the data index from the storage space corresponding to the output directory, and visually displaying the target data report corresponding to the data index. The method automatically obtains the target data report in response to the data query instruction, does not need to separately develop a generation rule of each target data report, reduces human participation, can improve the target data report generation efficiency, and saves the development cost of the target data report.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data report generation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Big data refers to datasets with large volumes and diverse data types. An increasing number of fields require processing this data to extract reports that meet their specific needs, thereby achieving efficient workflows. For example, in the insurance industry, actuarial work requires generating numerous data reports each month, with each report potentially having different fields, data types, and dimensions. These reports all need to be generated from large datasets within the insurance sector. Current technologies, due to the different data dimensions of each report, require the separate development of report generation rules for each report, which not only reduces report generation efficiency but also wastes development resources.

[0003] In conclusion, how to quickly generate data reports based on big data is a technical problem that urgently needs to be solved in the existing technology. Summary of the Invention

[0004] This invention provides a data report generation method, apparatus, computer device, and storage medium to solve the problem of how to quickly generate data reports.

[0005] A method for generating data reports, comprising:

[0006] Obtain the target download task, which includes task details information;

[0007] Based on the task details, obtain the target task name corresponding to the target download task;

[0008] The target download information corresponding to the target task name is obtained by querying the preset configuration table according to the target task name. The target download information includes the download path, data index and output directory.

[0009] Obtain the target data report corresponding to the data index from the big data set corresponding to the download path, and store the target data report in the storage space corresponding to the output directory;

[0010] In response to a data query instruction containing the data index, the target data report corresponding to the data index is read from the storage space corresponding to the output directory, and the target data report corresponding to the data index is visualized.

[0011] Preferably, obtaining the target download task includes:

[0012] Periodically query the system's distributed tasks to obtain the target download task from the system's distributed tasks.

[0013] Preferably, after obtaining the target download task, the data report generation method further includes:

[0014] Display the task configuration interface and obtain the task type corresponding to the target download task determined based on the task configuration interface;

[0015] If the task type is a report download task, then the step of obtaining the target task name corresponding to the target download task based on the task details information is executed;

[0016] If the task type is a real-time data download task, then the target data corresponding to the data index of the real-time data download task is obtained from the big data set corresponding to the download path of the real-time data download task, and the target data corresponding to the data index is visualized.

[0017] Preferably, obtaining the target data report corresponding to the data index from the big data set corresponding to the download path includes:

[0018] Determine the target file header based on the output file type corresponding to the target data report;

[0019] Obtain the original data corresponding to the data index from the large data set corresponding to the download path;

[0020] The original data is filled into the data report corresponding to the header of the target file to obtain the target data report corresponding to the data index.

[0021] Preferably, the target download information further includes target download conditions;

[0022] The step of obtaining the original data corresponding to the data index from the big data set corresponding to the download path includes:

[0023] If the target download information contains a target partition field, then based on the target partition field and the target download conditions, the original data corresponding to the data index is obtained from the big data set corresponding to the download path;

[0024] If the target download information does not contain a target partition field, then based on the target download conditions, the original data corresponding to the data index is obtained from the big data set corresponding to the download path.

[0025] Preferably, the target download information further includes the target table name;

[0026] Before retrieving the original data corresponding to the data index from the large dataset corresponding to the download path based on the target partition field and the target download conditions, the data report generation method further includes:

[0027] Query the standard table based on the target table name to obtain the standard partition field;

[0028] Based on the standard partition field, the target partition field is validated to determine the target validation result;

[0029] If the target verification result is successful, then the original data corresponding to the data index is obtained from the big data set corresponding to the download path based on the target partition field and the target download conditions.

[0030] Preferably, after visually displaying the target data report corresponding to the data index, the data report generation method further includes:

[0031] The number of target download tasks corresponding to each target data report stored in the output directory is counted within a preset time period prior to the current time.

[0032] If the number of tasks in the target download task is not greater than a preset threshold, then the target data report corresponding to the target download task is deleted from the output directory.

[0033] A data report generation device, comprising:

[0034] The target download task acquisition module is used to acquire target download tasks, which include task details information.

[0035] The target task name acquisition module is used to obtain the target task name corresponding to the target download task based on the task details information.

[0036] The target download information acquisition module is used to query a preset configuration table based on the target task name to obtain the target download information corresponding to the target task name. The target download information includes the download path, data index, and output directory.

[0037] The target data report acquisition module is used to acquire the target data report corresponding to the data index from the big data set corresponding to the download path, and store the target data report in the storage space corresponding to the output directory;

[0038] The target data report visualization module is used to respond to a data query command containing the data index, read the target data report corresponding to the data index from the storage space corresponding to the output directory, and visualize the target data report corresponding to the data index.

[0039] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data report generation method.

[0040] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data report generation method.

[0041] The aforementioned data report generation method, apparatus, computer, and storage medium allow for the direct query of a preset configuration table based on the target task name to obtain target download information, including the download path, data index, and output directory. The target data report is then downloaded and stored based on this information. This process requires no human intervention, enabling automatic downloading and storage of target data reports, thus improving development efficiency. Furthermore, it eliminates the need for repetitive development of identical target data reports, reducing development costs. The method automatically responds to data query commands, retrieving target data reports without requiring separate development of generation rules for each report, further reducing human involvement. By automating the generation of target data reports, this method can automatically generate target data reports from large datasets according to a defined process, improving generation efficiency and saving development costs. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of an application environment for a data report generation method according to an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of a data report generation method according to an embodiment of the present invention;

[0045] Figure 3 This is another flowchart of a data report generation method in one embodiment of the present invention;

[0046] Figure 4 This is another flowchart of a data report generation method in one embodiment of the present invention;

[0047] Figure 5 This is another flowchart of a data report generation method in one embodiment of the present invention;

[0048] Figure 6 This is another flowchart of a data report generation method in one embodiment of the present invention;

[0049] Figure 7 This is another flowchart of a data report generation method in one embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of a data report generation device in one embodiment of the present invention;

[0051] Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The data report generation method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment shown. Specifically, this data report generation method is applied in a data report generation system, which includes, for example, […]. Figure 1 The diagram illustrates a client and server that communicate over a network to quickly generate data reports. The client, also known as the user terminal, is the program that provides local services to the client, corresponding to the server. The server can be a standalone server or a server cluster consisting of multiple servers.

[0054] In one embodiment, such as Figure 2 As shown, a data report generation method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0055] S201: Obtain the target download task, which includes task details;

[0056] S202: Based on the task details, obtain the target task name corresponding to the target download task;

[0057] S203: Query the preset configuration table based on the target task name to obtain the target download information corresponding to the target task name. The target download information includes the download path, data index and output directory.

[0058] S204: Obtain the target data report corresponding to the data index from the big data set corresponding to the download path, and store the target data report in the storage space corresponding to the output directory;

[0059] S205: In response to a data query instruction containing the data index, read the target data report corresponding to the data index from the storage space corresponding to the output directory, and visualize the target data report corresponding to the data index.

[0060] The target download task refers to the task that needs to be downloaded. Task details refer to the specific download information included in the target download task. For example, in the insurance business, the target download task might include downloading insurance sales details and downloading employee performance data. The task details information could include specific download information for downloading insurance sales details; for example, the task details information might include downloading insurance sales details for a specific department from January to June.

[0061] As an example, in step S201, the server obtains the target download task and the task details information contained in the target download task. For example, in the insurance business field, the server obtains the target download task for downloading insurance sales details, and identifies the task details information contained in the target download task for downloading insurance sales details as downloading insurance sales details of a certain department from January to June. In this example, obtaining the target download task and the task details information within the target download task facilitates the subsequent determination of the target task name corresponding to the target download task based on the task details information.

[0062] The target task name refers to the name of the target download task determined based on the task details information.

[0063] As an example, in step S202, after obtaining the task details information, the server determines the target task name corresponding to the target download task based on the task details information. In this example, determining the target task name facilitates subsequent calls to the target task name and queries of the preset configuration table. For example, in the insurance business field, when the target download task is to download insurance sales details, and the task details information contained in the target download task is to download insurance sales details of a certain department from January to June, the target task name is determined to be download_act_pol_ben, which facilitates subsequent calls to the target task name and queries of the preset configuration table.

[0064] The preset configuration table refers to a pre-defined configuration table used to determine the target download information. The download path refers to the path of the large dataset containing the data to be downloaded during the target download task. The data index is used to determine the location directory of the data corresponding to the target download task within the large dataset. The output directory refers to the storage location of the target data report after the target download task is completed and the corresponding target data report is obtained. The large dataset is the data source for the data to be downloaded in the target download task. The target data report is a data table generated by arranging the downloaded data in a certain order. In essence, each target download task corresponds to data that needs to be downloaded. The dataset containing this data is the large dataset, and the location where the large dataset is stored is the download path. The location of the data within the large dataset needs to be found using the data index. The target data report formed from the downloaded data needs to be stored in the corresponding location for easy subsequent querying without repeated generation. The location where the target data report needs to be stored is the output directory.

[0065] Among them, the target download information is the information that needs to be obtained to achieve the target download task, and is used to achieve the target download task.

[0066] As an example, in step S203, after determining the target task name corresponding to the target download task, the server queries a preset configuration table based on the target task name to obtain the target download information corresponding to the target task name, including the download path, data index, and output directory. Understandably, in the preset configuration table, each target task name has corresponding target download information; that is, for a given target task name, there exists target download information including a given download path, a given data index, and a given output directory. For example, in the insurance business field, when the target task name corresponding to the target download task is determined to be download_act_pol_ben, the server queries the preset configuration table based on the target task name to obtain the determined download path / hdfs / act / download_act_pol_ben / proc_date, the determined data index sx_hx_safe / Pol_ben, and the determined output directory / sftp / act / download_act_pol_ben / proc_date, among other target download information. In this example, the target download information can be obtained by directly querying the preset configuration table based on the target task name, reducing human intervention, improving development efficiency, saving development costs, and facilitating the automatic download of the data corresponding to the target download task based on the target download information to obtain the target data report.

[0067] As an example, in step S204, after obtaining the target download information, including the download path, data index, and output directory, the server automatically retrieves the target data report corresponding to the data index from the large data set corresponding to the download path, and stores the target data report in the storage space corresponding to the output directory for easy subsequent querying and retrieval. For example, in the insurance business field, after determining the target download information such as the download path / hdfs / act / download_act_pol_ben / proc_date, the data index sx_hx_safe / Pol_ben, and the output directory / sftp / act / download_act_pol_ben / proc_date, the server directly retrieves the target data report corresponding to the data index from the large data set corresponding to the download path based on the above target download information, and stores the target data report in the storage space corresponding to the output directory. In this example, the download and storage of the target data report based on the download path, data index, and output directory requires no human intervention, enabling automatic download and storage of the target data report, improving the development efficiency of the target data report, and reducing the development cost of the target data report.

[0068] The data query command refers to an instruction that queries the target data report stored in the output directory based on the data index. In other words, once the target data report corresponding to the target download task is downloaded, it is automatically stored in the output directory. When a data query command containing the data index is obtained, the target data report with the same data index is directly queried in the output directory based on the data index.

[0069] As an example, in step S205, after the server downloads and stores the target data report corresponding to the target download task, when it obtains a data query instruction containing the data index, it queries the data index of each target data report stored in the storage space in the output directory based on the data query instruction containing the data index, obtains the target data report with the same data index as in the data query instruction, and visualizes the target data report. For example, in the insurance business field, after downloading and storing the target data report corresponding to the target download task of downloading insurance sales details, when it obtains a data query instruction containing the data index corresponding to the target download task, it reads the target data report corresponding to the data index from the storage space corresponding to the output directory based on the data query instruction containing the data index, and visualizes the target data report obtained when the target download task is to download insurance sales details. In this example, in response to a data query command containing a data index, the target data report corresponding to the data index is directly read from the storage space corresponding to the output directory, and the target data report is visualized. This method automatically responds to data query commands and obtains the target data report without the need to develop separate rules for generating each target data report, reducing human involvement. This not only improves the efficiency of target data report generation but also saves on the development cost of target data reports.

[0070] The data report generation method provided in this embodiment directly queries a preset configuration table based on the target task name to obtain target download information, including download path, data index, and output directory. The method then downloads and stores the target data report based on the download path, data index, and output directory. This process requires no human intervention, enabling automatic downloading and storage of target data reports, improving development efficiency, and eliminating the need for repetitive development of identical target data reports, thus reducing development costs. The method automatically responds to data query commands and retrieves target data reports without requiring separate development of generation rules for each target data report, further reducing human involvement. By automating the generation of target data reports, this method can automatically generate target data reports in large datasets according to a specific process, improving generation efficiency and saving development costs.

[0071] In one embodiment, step S201, namely obtaining the target download task, includes: periodically querying the system's distributed tasks to obtain the target download task from the system's distributed tasks.

[0072] Among them, the system distributed task refers to several tasks of different dimensions that exist in the data report generation system.

[0073] As an example, the server periodically queries the system's distributed tasks to obtain the target download tasks, facilitating the subsequent generation of target data reports based on these tasks. Understandably, a data report generation system may contain tasks of different dimensions, including target download tasks and data retrieval tasks. It's necessary to periodically query the system's distributed tasks to determine the target download tasks, enabling the subsequent generation of target data reports. In this example, periodically querying the system's distributed tasks to obtain the target download tasks automates the acquisition of these tasks without manual intervention, making the acquisition of target download tasks more accurate.

[0074] In one embodiment, such as Figure 3 As shown, after step S201, that is, after obtaining the target download task, the following steps are also included:

[0075] S301: Display the task configuration interface and obtain the task type corresponding to the target download task determined based on the task configuration interface;

[0076] S302: If the task type is a report download task, then execute the process of obtaining the target task name corresponding to the target download task based on the task details information;

[0077] S303: If the task type is a real-time data download task, then retrieve the target data corresponding to the data index of the real-time data download task from the big data set corresponding to the download path of the real-time data download task, and visualize the target data corresponding to the data index.

[0078] The task configuration interface is used to configure and monitor the target download task, determining its corresponding task type. Task type refers to the type of the target download task, including two types: report download tasks and real-time data download tasks. Report download tasks generate target data reports. Real-time data download tasks acquire specific data in real time without generating target data reports. Understandably, when applying data report generation methods to different domains, it is often necessary not only to acquire target data reports for a specific time period but also to acquire real-time data information. Therefore, the target download task types include report download tasks and real-time data download tasks. After acquiring the target download task, it is also necessary to determine its task type to implement the target download task in the appropriate manner.

[0079] As an example, in step S301, the server displays a task configuration interface and identifies the interface to obtain the task type corresponding to the target download task. Understandably, different task types correspond to different task configuration interfaces, and the task type of the target download task can be determined by identifying the task configuration interface. In this example, obtaining the task type corresponding to the target download task facilitates accurate determination of the target download task's task type and enables the implementation of the corresponding target download task based on different task types.

[0080] As an example, in step S302, when the server determines that the task type is a report download task, it directly executes steps S202 to S205 to obtain the target task name corresponding to the target download task based on the task details, thus realizing the process of generating, downloading, storing, and visualizing the target data report. In this example, when the task type is determined to be a report download task, the server directly executes steps to obtain the target task name corresponding to the target download task based on the task details, which facilitates the subsequent generation, downloading, storage, and visualization of the target data report corresponding to the target download task.

[0081] As an example, in step S303, when the server determines that the task type is a real-time data download task, it obtains the download path and data index corresponding to the real-time data download task, and retrieves the target data corresponding to the data index of the real-time data download task from the large data set corresponding to the download path of the real-time data download task. The server then visualizes and displays the target data corresponding to the data index, thus realizing the real-time data download task. In this example, when the task type is determined to be a real-time data download task, the server retrieves the target data corresponding to the data index of the real-time data download task from the large data set corresponding to the download path of the real-time data download task and visualizes and displays the target data corresponding to the data index. This requires no manual intervention and no need to generate target data reports, making it more convenient, faster, and more real-time.

[0082] In this embodiment, the task type corresponding to the target download task is obtained, which facilitates accurate determination of the task type and enables the implementation of the corresponding target download task based on different task types. When the task type is determined to be a report download task, the target task name corresponding to the target download task is obtained directly based on the task details information, which facilitates the subsequent generation, downloading, storage, and visualization of the target data report corresponding to the target download task. When the task type is determined to be a real-time data download task, the target data corresponding to the data index of the real-time data download task is obtained without manual intervention or the generation of the target data report, which is more convenient, faster, and more real-time.

[0083] In one embodiment, such as Figure 4 As shown, step S204, which involves retrieving the target data report corresponding to the data index from the large dataset corresponding to the download path, includes:

[0084] S401: Determine the target file header based on the output file type corresponding to the target data report;

[0085] S402: Obtain the raw data corresponding to the data index from the large data set corresponding to the download path;

[0086] S403: Fill the original data into the data report corresponding to the header of the target file, and obtain the target data report corresponding to the data index.

[0087] The output file type refers to the file type of the target data report to be generated, including .rpt, .csv, and .txt types. The target file header refers to the header corresponding to the output file type.

[0088] As an example, in step S401, the server obtains the output file type corresponding to the target data report from the target download information. Based on the output file type of the target data report, it concatenates all fields in the target download information to form the target file header. Understandably, different output file types correspond to different target file headers for target data reports. Therefore, it is necessary to obtain the output file type of the target data report and determine the corresponding target file header based on it, facilitating subsequent data filling and generation of the target data report based on the target file header.

[0089] The raw data is downloaded from a large dataset based on a data index and is used to compose the target data report.

[0090] As an example, in step S402, the server obtains the large dataset corresponding to the target download task based on the download path in the target download information, and retrieves the original data to be downloaded in the target download task based on the data index in the target download information. In this example, the original data is retrieved directly based on the download path and data index without manual intervention, which can improve data download efficiency.

[0091] As an example, in step S403, after downloading the raw data used to compose the target data report, the server fills the raw data into the data report corresponding to the header of the target file and obtains the target data report corresponding to the data index. In this example, the server fills in the raw data according to the header of the target file to generate the target data report without manual intervention, which can improve the efficiency of target data report generation.

[0092] In this embodiment, the target file header is determined, and the original data corresponding to the data index is obtained from the big data set corresponding to the download path. The original data is then filled into the data report corresponding to the target file header to generate the target data report corresponding to the data index. This target data report generation method does not require manual intervention and realizes automated generation of target data reports, which can save labor costs and improve the efficiency of target data report generation.

[0093] In one embodiment, step S402 further includes target download conditions in the target download information. Target download conditions refer to conditions for fine-grained filtering of data within the big data. For example, in the insurance business field, the task details information corresponding to the target download task is to download the sales details of the sales department for the first half of the year. The target download conditions can be other additional conditions; for example, the target download conditions could be to download the sales details of female salespersons in even-numbered months of the first half of the year, thereby achieving fine-grained filtering of the big data.

[0094] In one embodiment, such as Figure 5 As shown, step S402, which involves retrieving the original data corresponding to the data index from the large data set corresponding to the download path, includes:

[0095] S501: If the target download information contains a target partition field, then based on the target partition field and the target download conditions, retrieve the original data corresponding to the data index from the big data set corresponding to the download path;

[0096] S502: If the target download information does not contain the target partition field, then according to the target download conditions, obtain the original data corresponding to the data index from the big data set corresponding to the download path.

[0097] The target partition field indicates whether multiple target data reports need to be generated according to certain download rules. For example, if the task details for the target download task are to download the sales details for the first half of the year from the sales department, the target partition field could be to download the sales details for each month of the first half of the year separately. If the target partition field exists, then six target data reports need to be downloaded simultaneously; if the target partition field does not exist, then only one target data report needs to be downloaded directly.

[0098] As an example, in step S501, when the server determines that the target download information contains a target partition field, it retrieves the original data corresponding to the data index from the big data set corresponding to the download path based on the target partition field and the target download conditions. All the original data is grouped according to the target partition field so as to generate a target data report that matches the number of fields in the target partition field. For example, in the insurance business field, the server determines that a target partition field exists and that the task details information corresponding to the target download task is to download the sales details of the sales department for the first half of the year. The target partition field can be to download the sales details of each month in the first half of the year for the sales department. The target download condition is to obtain the sales details of female salespersons in even-numbered months. The server retrieves the original data corresponding to the sales details of female salespersons in even-numbered months in the first half of the year from the big data set corresponding to the download path so as to generate three target data reports based on the original data.

[0099] As an example, in step S502, when the server determines that the target download information does not contain the target partition field, it retrieves the original data corresponding to the data index from the big data set corresponding to the download path, based on the target download conditions, so as to generate a target data report based on the original data. For example, in the insurance business field, if the server determines that the target partition field does not exist and that the task details information corresponding to the target download task is to download the sales details of the sales department for the first half of the year, and the target download condition is to obtain the sales details of female salespersons in even-numbered months, it retrieves the original data corresponding to the sales details of female salespersons in even-numbered months of the first half of the year from the big data set corresponding to the download path, so as to generate a target data report based on the original data.

[0100] In this embodiment, it is determined whether the target download information contains a target partition field, which facilitates the subsequent more accurate acquisition of target data reports that meet the purpose. This process obtains the required raw data from big data without human intervention, which can improve the efficiency of raw data acquisition and save development costs.

[0101] In one embodiment, in step S402, the target download information further includes a target table name, which refers to the name of the target table containing the target partition field. Understandably, the target partition field corresponding to the target download information is stored in the target table. The target table has the same table name as the standard table in the preset configuration table that stores the standard partition field, facilitating subsequent determination of the standard table and the corresponding standard partition field based on the target table name. The standard table refers to the table in the preset configuration table that contains the partition field. The standard partition field refers to the partition field contained in the standard table, used to verify the target partition field.

[0102] In one embodiment, such as Figure 6As shown, before step S501, that is, before obtaining the original data corresponding to the data index from the large data set corresponding to the download path based on the target partition field and the target download conditions, the data report generation method further includes:

[0103] S601: Query the standard table based on the target table name to obtain the standard partition field;

[0104] S602: Based on the standard partition field, validate the target partition field and determine the target validation result;

[0105] S603: If the target verification result is successful, then execute the process of retrieving the original data corresponding to the data index from the big data set corresponding to the download path based on the target partition field and the target download conditions.

[0106] As an example, in step S601, the server obtains the target table where the target partition field is located, determines the target table name corresponding to the target table, matches a standard table with the same name as the target table in a preset configuration file, and obtains the partition field from the standard table as the standard partition field. In this example, the standard partition field is obtained based on the target table name, which facilitates subsequent verification of the target partition field based on the standard partition field.

[0107] The target verification result is used to determine whether the target partition field has been successfully verified, including two verification results: successful verification and failed verification.

[0108] As an example, in step S602, the server validates the target partition field based on the standard partition field to determine the target validation result. In this example, the server compares and matches the target partition field with the standard partition field. If the target partition field matches the standard partition field, it indicates a successful match, and the target validation result is a successful validation. If the target partition field does not match the standard partition field, it indicates a failed match, and the target validation result is a failed validation. In this example, validating the target partition field based on the standard partition field and obtaining the target validation result helps determine if the target partition field is incorrect, facilitating more accurate acquisition of the target data report subsequently.

[0109] As an example, in step S603, when the server determines that the target verification result is successful, it retrieves the original data corresponding to the data index from the large data set corresponding to the download path, based on the target partition field and the target download conditions. In this example, when the server determines that the target verification result is successful, it confirms that the target partition field is correct and continues to execute step S501, facilitating the subsequent retrieval of a more accurate target data report. When the server determines that the target verification result is unsuccessful, it determines that the target partition field is incorrect, terminates the target download task, and displays a task failure message, avoiding the retrieval of incorrect reports and saving data resources.

[0110] In this embodiment, the target partition field is validated based on the standard partition field to determine the target validation result. A more accurate target data report is generated based on the target validation result, so that the target data report has better accuracy.

[0111] In one embodiment, step S204, which involves storing the target data report to the storage space corresponding to the output directory, further includes:

[0112] S2041: If the number of target data reports is at least two, then a thread pool is used to transfer the target data reports to the storage space corresponding to the output directory for storage.

[0113] S2042: If there is only one target data report, the target data report is directly transmitted to the storage space corresponding to the output directory for storage.

[0114] The thread pool is used to transfer at least two target data reports.

[0115] As an example, in step S2041, after obtaining the target data report, the server determines the number of target data reports. If it is determined that the number of target data reports is at least two, a thread pool is used to store at least two target data reports in the storage space corresponding to the output directory. In this example, when the number of target data reports is at least two, a thread pool is used to transfer the target data reports to the storage space corresponding to the output directory, eliminating the need for manual intervention, improving storage efficiency, and saving labor costs.

[0116] As an example, in step S2042, when the server determines that there is only one target data report, it directly transmits the target data report to the storage space corresponding to the output directory for storage. This process does not require manual intervention, which improves storage efficiency and saves labor costs.

[0117] In this example, when storing the target data report, the number of target data reports is determined, and different storage methods are used to store the target data reports based on the number of target data reports, which helps to improve data processing efficiency.

[0118] In one embodiment, such as Figure 7 As shown, after step S205, that is, after the visualization of the target data report corresponding to the data index, the data report generation method further includes:

[0119] S701: Count the number of target download tasks corresponding to each target data report stored in the output directory within a preset time period before the current time.

[0120] S702: If the number of tasks in the target download task is not greater than the preset threshold, then delete the target data report corresponding to the target download task in the output directory.

[0121] Here, "current time" refers to the moment after the target data report is visualized. "Preset time period" refers to a preset period of time prior to the current time. "Number of tasks" refers to the number of target download tasks with the same task details information corresponding to the target data report.

[0122] As an example, in step S701, after visualizing the target data report, the server counts the number of target download tasks corresponding to each target data report stored in the output directory within a preset time period prior to the current time. Understandably, due to the limited storage space in the output directory, it is necessary to obtain the number of tasks for the same target download task within the preset time period prior to the current time in order to determine whether the target data report generated by the target download task is a frequently used data report, thereby saving storage space.

[0123] The preset threshold refers to a preset value used to determine the number of tasks.

[0124] As an example, in step S702, after obtaining the number of tasks corresponding to each target data report, the server determines the relationship between the number of tasks and a preset threshold. If it is determined that the number of tasks for several target download tasks is not greater than the preset threshold, the target data reports corresponding to those tasks are deleted from the output directory to save storage space in the output directory. If it is determined that the number of tasks for several target download tasks is greater than the preset threshold, the target data reports corresponding to those tasks are retained in the output directory. Understandably, if the number of tasks for a target download task is not greater than the preset threshold, it indicates that the target download task is not a frequently used download task, and the corresponding target data report is also not a frequently used data report. Therefore, deleting the target data reports corresponding to infrequently used download tasks not only saves storage space in the output directory but also reduces data redundancy in the target data reports in the output directory, improving report development efficiency.

[0125] In this embodiment, when the number of target download tasks is no greater than a preset threshold, the target data report corresponding to the target download task is deleted from the output directory. This not only saves storage space in the output directory but also reduces data redundancy in the target data reports in the output directory, thereby improving report development efficiency.

[0126] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0127] In one embodiment, a data report generation apparatus is provided, which corresponds one-to-one with the data report generation method described in the above embodiments. For example... Figure 8 As shown, the data report generation device includes a target download task acquisition module 801, a target task name acquisition module 802, a target download information acquisition module 803, a target data report acquisition module 804, and a target data report visualization module 805. Detailed descriptions of each functional module are as follows:

[0128] The target download task acquisition module 801 is used to acquire target download tasks, which include task details.

[0129] The target task name acquisition module 802 is used to obtain the target task name corresponding to the target download task based on the task details information.

[0130] The target download information acquisition module 803 is used to query a preset configuration table based on the target task name to obtain the target download information corresponding to the target task name. The target download information includes the download path, data index and output directory.

[0131] The target data report acquisition module 804 is used to acquire the target data report corresponding to the data index from the big data set corresponding to the download path, and store the target data report to the storage space corresponding to the output directory.

[0132] The target data report visualization module 805 is used to respond to a data query command containing the data index, read the target data report corresponding to the data index from the storage space corresponding to the output directory, and visualize the target data report corresponding to the data index.

[0133] In one embodiment, the target download task acquisition module 801 includes:

[0134] The target download task acquisition submodule is used to periodically query the system's distributed tasks and retrieve the target download tasks from among them.

[0135] In another embodiment, the data report generation apparatus further includes:

[0136] The task type acquisition module is used to display the task configuration interface and obtain the task type corresponding to the target download task determined based on the task configuration interface.

[0137] The first processing module is used to obtain the target task name corresponding to the target download task based on the task details if the task type is a report download task.

[0138] The second processing module is used to, if the task type is a real-time data download task, retrieve the target data corresponding to the data index of the real-time data download task from the big data set corresponding to the download path of the real-time data download task, and visualize the target data corresponding to the data index.

[0139] In one embodiment, the target data report acquisition module 804 includes:

[0140] The target file header submodule is used to determine the target file header based on the output file type corresponding to the target data report;

[0141] The raw data acquisition submodule is used to obtain the raw data corresponding to the data index from the big data set corresponding to the download path;

[0142] The target data report acquisition submodule is used to fill the original data into the data report corresponding to the header of the target file and to acquire the target data report corresponding to the data index.

[0143] In one embodiment, the raw data acquisition submodule includes:

[0144] The first raw data acquisition unit is used to acquire the raw data corresponding to the data index from the big data set corresponding to the download path, based on the target partition field and the target download conditions, if the target download information contains a target partition field.

[0145] The second raw data acquisition unit is used to acquire the raw data corresponding to the data index from the big data set corresponding to the download path, based on the target download conditions, if the target download information does not contain the target partition field.

[0146] The target download information also includes the target download conditions.

[0147] In another embodiment, the data report generation apparatus further includes:

[0148] The standard partition field retrieval module is used to query a standard table based on the target table name and retrieve the standard partition field.

[0149] The target verification result determination module verifies the target partition field based on the standard partition field and determines the target verification result.

[0150] The execution module is used to retrieve the original data corresponding to the data index from the big data set corresponding to the download path, based on the target partition field and the target download conditions, if the target verification result is successful.

[0151] The target download information also includes the target table name.

[0152] In another embodiment, the data report generation apparatus further includes:

[0153] The task quantity output module is used to count the number of target download tasks corresponding to each target data report stored in the output directory within a preset time period before the current time.

[0154] The target data report output module is used to delete the target data report corresponding to the target download task from the output directory if the number of tasks in the target download task is not greater than a preset threshold.

[0155] Specific limitations regarding the data report generation device can be found in the limitations of the data report generation method described above, and will not be repeated here. Each module in the aforementioned data report generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data used or generated during the execution of a data report generation method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data report generation method.

[0157] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data report generation method described in the above embodiment, for example... Figure 2 As shown in S201-S205, or Figures 3 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the data report generation device, for example... Figure 8 The functions of the target download task acquisition module 801, target task name acquisition module 802, target download information acquisition module 803, target data report acquisition module 804, and target data report visualization module 805 shown are not described in detail here to avoid repetition.

[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the data report generation method described in the above embodiment, for example... Figure 2 As shown in S201-S205, or Figures 3 to 7 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the data report generation device, for example... Figure 8 The functions of the target download task acquisition module 801, target task name acquisition module 802, target download information acquisition module 803, target data report acquisition module 804, and target data report visualization module 805 shown are not described again here to avoid repetition. The computer-readable storage medium may be non-volatile or volatile.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0161] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating data reports, characterized in that, include: Obtain the target download task, which includes task details information; Based on the task details, obtain the target task name corresponding to the target download task; The target download information is obtained by querying a preset configuration table based on the target task name. The target download information includes the download path, data index, and output directory; the target download information also includes the target download conditions and the target table name. Determine the target file header based on the output file type corresponding to the target data report; Obtain the original data corresponding to the data index from the large data set corresponding to the download path; The original data is filled into the data report corresponding to the header of the target file, the target data report corresponding to the data index is obtained, and the target data report is stored in the storage space corresponding to the output directory. In response to a data query instruction containing the data index, the target data report corresponding to the data index is read from the storage space corresponding to the output directory, and the target data report corresponding to the data index is visualized and displayed. The step of obtaining the original data corresponding to the data index from the big data set corresponding to the download path includes: If the target download information contains a target partition field, then the standard table is queried according to the target table name to obtain the standard partition field; based on the standard partition field, the target partition field is validated to determine the target validation result; if the target validation result is successful, then the original data corresponding to the data index is obtained from the big data set corresponding to the download path according to the target partition field and the target download conditions.

2. The data report generation method as described in claim 1, characterized in that, The acquisition of the target download task includes: Periodically query the system's distributed tasks to obtain the target download task from the system's distributed tasks.

3. The data report generation method as described in claim 1, characterized in that, After obtaining the target download task, the data report generation method further includes: Display the task configuration interface and obtain the task type corresponding to the target download task determined based on the task configuration interface; If the task type is a report download task, then the step of obtaining the target task name corresponding to the target download task based on the task details information is executed; If the task type is a real-time data download task, then the target data corresponding to the data index of the real-time data download task is obtained from the big data set corresponding to the download path of the real-time data download task, and the target data corresponding to the data index is visualized.

4. The data report generation method as described in claim 1, characterized in that, The step of obtaining the original data corresponding to the data index from the big data set corresponding to the download path also includes: If the target download information does not contain a target partition field, then based on the target download conditions, the original data corresponding to the data index is obtained from the big data set corresponding to the download path.

5. The data report generation method as described in claim 1, characterized in that, After visually displaying the target data report corresponding to the data index, the data report generation method further includes: The number of target download tasks corresponding to each target data report stored in the output directory is counted within a preset time period prior to the current time. If the number of tasks in the target download task is not greater than a preset threshold, then the target data report corresponding to the target download task is deleted from the output directory.

6. A data report generation apparatus, used to implement the data report generation method according to any one of claims 1 to 5, characterized in that, The device includes: The target download task acquisition module is used to acquire target download tasks, which include task details information. The target task name acquisition module is used to obtain the target task name corresponding to the target download task based on the task details information. The target download information acquisition module is used to query a preset configuration table based on the target task name to obtain the target download information corresponding to the target task name. The target download information includes the download path, data index, and output directory. The target data report acquisition module is used to acquire the target data report corresponding to the data index from the big data set corresponding to the download path, and store the target data report in the storage space corresponding to the output directory; The target data report visualization module is used to respond to a data query command containing the data index, read the target data report corresponding to the data index from the storage space corresponding to the output directory, and visualize the target data report corresponding to the data index.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data report generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data report generation method as described in any one of claims 1 to 5.

Images