Report-based data processing methods, devices, and servers

Abstract

The present application provides a report-based data processing method, device and server, which relates to data analysis technology. The method includes: responding to received data processing instructions, wherein the data processing instructions are used to instruct the acquisition of data files and the processing of data files, to obtain multiple data files. According to the preset warehousing script, multiple data files are read and processed, and the read data are stored in the data warehouse table in the database corresponding to the server; wherein the warehousing script is used to instruct the reading of data in the data file. According to the preset processing script, the data in the data warehouse table is processed to obtain the processed target data, and the target data is stored in the target table to complete the data processing operation. The method of the present application adopts script processing throughout the process, and there is no need to introduce an application program. It only needs to schedule the preset script with a timing scheduling tool to run, which solves the technical problem of low data processing efficiency.

Description

Technical Field

[0001] This application relates to data analysis technology, and more particularly to a report-based data processing method, apparatus, and server. Background Technology

[0002] Currently, with the increase in data, the requirements for data processing operations are also increasing.

[0003] In existing technologies, data processing operations generally involve the server acquiring the data to be processed, and then performing calculations and other processing operations on the data in an Oracle database (Oracle Database, Oracle) deployed on the server.

[0004] However, in existing technologies, the processing time for calculations and other operations on the data to be processed in Oracle databases is relatively long, resulting in low processing efficiency. Summary of the Invention

[0005] This application provides a report-based data processing method, apparatus, and server to solve the technical problem of low data processing efficiency.

[0006] Firstly, this application provides a data processing method based on reports, including:

[0007] In response to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition of data files and the processing of the data files, thereby acquiring multiple data files;

[0008] According to a preset inbound script, multiple data files are read and processed, and the read data is stored in a data warehouse table in a database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files;

[0009] The data in the data warehouse table is processed according to a preset processing script to obtain the processed target data, and the target data is stored in the target table to complete the data processing operation; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to preset processing steps.

[0010] Furthermore, the processing step instructions include data sending instructions and data acquisition instructions; the step of processing the data in the data warehouse table according to a preset processing script to obtain the processed target data includes:

[0011] According to the data sending instructions in the preset processing script, the data in the data warehouse table and the data acquisition instructions in the processing script are sent to each of the multiple other servers, so that each of the other servers processes the received data according to the data acquisition instructions to obtain the preliminary processed data.

[0012] Receive initial data sent by each other server;

[0013] The initial data sent by each other server is aggregated and processed to obtain the processed target data.

[0014] Furthermore, the step of reading and processing multiple data files according to a preset inbound script, and storing the read data into a data warehouse table in the database corresponding to the server, includes:

[0015] According to the preset data entry script, multiple data files are read and processed to obtain the first delimiter between each record in the data files and the second delimiter between each field in the record;

[0016] Based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table, the data in each read record is stored in the data warehouse table of the database corresponding to the server according to the first delimiter between each record and the second delimiter between each field in the record.

[0017] Furthermore, the method also includes:

[0018] The target data in the target table is exported according to the preset export script to obtain the exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format;

[0019] The target file is stored in a preset path so that other servers can retrieve the target file from that path.

[0020] Secondly, this application provides a report-based data processing apparatus, comprising:

[0021] An acquisition unit is configured to respond to a received data processing instruction, wherein the data processing instruction is configured to instruct the acquisition of data files and the processing of the data files, thereby acquiring multiple data files;

[0022] The reading unit is used to read and process multiple data files according to a preset inbound script, and store the read data into a data warehouse table in a database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files;

[0023] The processing unit is used to process the data in the data warehouse table according to a preset processing script to obtain the processed target data;

[0024] A storage unit is used to store the target data into a target table to complete the data processing operation; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to preset processing step instructions.

[0025] Furthermore, the processing step instructions include data sending instructions and data acquisition instructions; the processing unit includes:

[0026] The sending module is used to send the data in the data warehouse table and the data acquisition instruction in the processing script to each of the multiple other servers according to the data sending instruction in the preset processing script, so that each of the other servers processes the received data according to the data acquisition instruction to obtain the preliminary processed data.

[0027] The receiving module is used to receive the initial data sent by each other server;

[0028] The aggregation module is used to aggregate the preliminary data sent by each other server to obtain the processed target data.

[0029] Furthermore, the reading unit includes:

[0030] The reading module is used to read and process multiple data files according to a preset loading script, and obtain the first delimiter between each record in the data file and the second delimiter between each field in the record;

[0031] The storage module is used to store the data of each read record into a data warehouse table in the database corresponding to the server, based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table.

[0032] Furthermore, the device also includes:

[0033] An export unit is used to export the target data in the target table according to a preset export script to obtain an exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format;

[0034] A storage unit is used to store the target file in a preset path so that other servers can obtain the target file in the path.

[0035] Thirdly, this application provides a server, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect.

[0036] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0037] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0038] This application provides a report-based data processing method, apparatus, and server. Responding to a received data processing instruction, the instruction instructs the acquisition and processing of multiple data files. Multiple data files are read and processed according to a preset data entry script, and the read data is stored in a data warehouse table within a database corresponding to the server. The data entry script instructs the reading of data from the data files. The data in the data warehouse table is then processed according to a preset processing script to obtain processed target data, which is then stored in a target table to complete the data processing operation. The processing script instructs the processing of the data in the data warehouse table according to preset processing steps. In this solution, multiple data files are read and processed according to a preset data entry script, and the read data is stored in a data warehouse table within a database corresponding to the server. The data in the data warehouse table is then processed according to a preset processing script to obtain processed target data, which is then stored in the target table. Therefore, the entire processing is script-based, eliminating the need for an application program; only a timer tool is required to schedule the preset script, thus solving the technical problem of low data processing efficiency. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0040] Figure 1 A flowchart illustrating a report-based data processing method provided in an embodiment of this application;

[0041] Figure 2 A flowchart illustrating another report-based data processing method provided in this application embodiment;

[0042] Figure 3 A flowchart illustrating another report-based data processing method provided in this application embodiment;

[0043] Figure 4 A schematic diagram of the structure of a report-based data processing device provided in this application embodiment;

[0044] Figure 5 A schematic diagram of another report-based data processing device provided in this application embodiment;

[0045] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of this application.

[0046] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0048] In one example, in existing technologies, data processing typically involves a server retrieving the data to be processed and performing calculations and other processing operations on the data within an Oracle database deployed on the server. However, in existing technologies, the time spent performing calculations and other processing operations on the data in the Oracle database is relatively long, resulting in low processing efficiency.

[0049] This application provides a data processing method, apparatus, and device based on reports, which aims to solve the above-mentioned technical problems in the prior art.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0052] 101. Responding to the received data processing instruction, wherein the data processing instruction is used to instruct the acquisition of data files and the processing of the data files, and to acquire multiple data files.

[0053] For example, the execution entity in this embodiment can be a server, which may include data warehouse tools (Hive), etc. Data files are generated based on data, which can be data of various business types. Each data file includes multiple records, each with a primary key identifier, multiple fields, and a delimiter between each record and between each field within a record. If the server receives a data processing instruction, it needs to acquire multiple data files. The server can acquire data files by sending data processing instructions to other devices, causing those devices to send corresponding data files to the server upon receiving the instructions, and the server receiving the data files from the other devices; alternatively, other devices may actively send data files to the server, etc., without limitation.

[0054] 102. According to the preset inbound script, read and process multiple data files, and store the read data into the data warehouse table in the database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files.

[0055] For example, the inbound script is used to instruct read commands to read data from data files. For instance, the read command can determine the data in each record and the corresponding data for each field in the data file by reading the delimiters between each record and between each field in the record. By processing multiple data files according to the preset inbound script, the server can identify the first delimiter between each record and the second delimiter between each field in the data file. Based on these delimiters, the server determines the data in each record and the corresponding data for each field in the data file, and stores the determined data in the data warehouse table (Hive warehouse table) in the database corresponding to the server.

[0056] 103. Process the data in the data warehouse table according to the preset processing script to obtain the processed target data, and store the target data in the target table to complete the data processing operation; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to the preset processing steps.

[0057] For example, a processing script is used to instruct data in a data warehouse table to be processed according to preset processing step instructions. The processing script is generated based on at least one processing step instruction based on preset processing step information. According to the preset processing script, the server can process the data in the data warehouse table to obtain processed target data, and store the target data in the target table, thereby completing the data migration.

[0058] For example, a pre-defined processing script is used to determine the total consumption of each record in the data warehouse table. The script includes data sending instructions and data retrieval instructions. During processing, according to the pre-defined processing instructions, the server can sum the data in each record of the data warehouse table to obtain the target sum, and then store the target data in the target table. Specifically, during the summation process, if the amount of data to be summed on the server is large, the server can first send the data from the data warehouse table to each of multiple other servers according to the data sending instructions in the pre-defined processing script, and send the data retrieval instructions from the processing script to each of the other servers. When each other server receives the data retrieval instructions and the data, it processes the received data according to the data retrieval instructions to obtain preliminary processed data, and then sends the preliminary data to the server. After receiving the preliminary data from each other server, the server summarizes the multiple preliminary data sets to obtain the target data, stores the target data in the target table, and thus completes the data processing operation.

[0059] In this embodiment, in response to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition and processing of data files, and multiple data files are acquired. Multiple data files are read and processed according to a preset data entry script, and the read data is stored in a data warehouse table in a database corresponding to the server; wherein the data entry script is used to instruct the reading of data from the data files. The data in the data warehouse table is processed according to a preset processing script to obtain processed target data, and the target data is stored in a target table to complete the data processing operation; wherein the processing script is used to instruct the processing of data in the data warehouse table according to preset processing steps. In this solution, multiple data files are read and processed according to a preset data entry script, and the read data is stored in a data warehouse table in a database corresponding to the server. The data in the data warehouse table is processed according to a preset processing script to obtain processed target data, and the target data is stored in the target table. Therefore, the entire processing is script-based, requiring no application program; only a timed scheduling tool needs to schedule the preset script to run, solving the technical problem of low data processing efficiency.

[0060] Figure 2 A flowchart illustrating another report-based data processing method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0061] 201. In response to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition of data files and the processing of the data files, and to acquire multiple data files.

[0062] For example, this step can be referred to Figure 1 Step 101 in the text will not be repeated here.

[0063] 202. According to the preset loading script, read and process multiple data files to obtain the first delimiter between each record in the data file and the second delimiter between each field in the record; wherein, the loading script is used to indicate the read instructions for reading data from the data file.

[0064] For example, the server can execute the BEGIN command before executing the import script. The BEGIN command means that the import script will execute the action specified in BEGIN immediately before reading any input lines. The delimiters in the specified actions correspond to the delimiters and other symbols in the data warehouse table. For example, RS specifies the input record delimiter, FS specifies the input field delimiter, OFS specifies the output field delimiter, ORS specifies the output record delimiter, the gsub command removes special delimiters and spaces, newlines, and tabs, and the msck repair table command is mainly used to solve the problem that data written to the Hive partition table via hdfs-put cannot be queried in Hive. After the symbols are set, the import script can recognize the first delimiter between each record in the data file and the second delimiter between each field in the record. The above code is not only applicable to scenarios with such field and record separation methods, but also applicable to file data with other distinguishable field and record separation methods.

[0065] 203. Based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table, the data in each read record is stored in the data warehouse table in the database corresponding to the server according to the first delimiter between each record and the second delimiter between each field in the record.

[0066] For example, since the delimiter set in the inbound script is consistent with the delimiter in the target table creation statement, the server can determine the data in each record in the data file and the data corresponding to each field in the record based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table. The first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table. After generating a txt file, the determined txt file is stored in the data warehouse table in the database corresponding to the server.

[0067] 204. The processing step instructions include data sending instructions and data retrieval instructions; according to the data sending instructions in the preset processing script, the data in the data warehouse table and the data retrieval instructions in the processing script are sent to each of the multiple other servers, so that each other server processes the received data according to the data retrieval instructions to obtain the preliminary processed data; wherein, the processing script is used to indicate the processing instructions for processing the data.

[0068] For example, a processing script is used to instruct the processing of data in a data warehouse table according to preset processing step instructions. The processing script is generated based on at least one preset processing step instruction. For instance, the processing script includes a data sending instruction and a data retrieval instruction. During processing, to prevent data skew, the Hive execution engine and execution queue are first specified, and then the specific processing logic is written. This processing logic refers to preset data processing rules, and preset processing instructions (processing instructions refer to HQL language specific to Hive, such as including Hive parameters) are generated based on the processing logic. According to the preset processing instructions, the server can first send the data in the data warehouse table to each of multiple other servers according to the data sending instruction in the preset processing script, and then send the data retrieval instruction in the processing script to each of the multiple other servers. When each other server receives the data retrieval instruction and the data, it processes the received data according to the data retrieval instruction to obtain preliminary processed data, and then sends the preliminary data to the server.

[0069] 205. Receive preliminary data sent by each other server.

[0070] For example, the server can receive preliminary data sent by each of the other servers. Specifically, the multiple other servers can be pre-divided into groups, and the server can receive preliminary data sent by each of the other servers in each group.

[0071] 206. Summarize and process the preliminary data sent by each other server to obtain the processed target data.

[0072] For example, the server can aggregate the preliminary data sent by each group of other servers to obtain the processed target data.

[0073] 207. Store the target data in the target table to complete the data processing operation.

[0074] For example, the server can store the target data in the target table, thereby completing the data processing operation.

[0075] 208. Export the data in the target table according to the preset export script to obtain the exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format.

[0076] For example, a pre-defined export script is used to export data from the target table according to a pre-defined format. This pre-defined format is identical to the data file format; that is, the delimiters and other symbols in the exported target file are the same as those in the data file, facilitating user identification and application across various systems. Specifically, the export script requires the parameter `Date_No`, which is the data's time base, used by the `Hive_Sql` variable to retrieve data under that time base. Then, the export script (i.e., the Hive command) exports the target data as a file, resulting in the exported target file. The `showHeader` parameter specifies that column names should not be displayed in the query results, the `outputformat` parameter specifies the result display mode, `delimiterForDSV` specifies the delimiter for the partition values ​​in the output format, the `set` command sets Hive's execution queue to default, and the `sed` command replaces special characters in the results and outputs them as a `.dat` file.

[0077] 209. Store the target file in a preset path so that other servers can obtain the target file from the path.

[0078] For example, the server can store the target file in a preset path, making it easy for other servers to retrieve the target file from that path. Specifically, the server parses the target file according to a preset Hive scheduling script and places it on another server (which could be an Oracle server). Then, the other server generates view tables. Finally, a preset reporting tool calls the data from the Oracle server and displays the data. The preset reporting tool is a tool independent of the Oracle server.

[0079] The technical solutions in this application involve the collection, storage, use, processing, transmission, provision, and disclosure of information such as financial data or user data, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0080] In this embodiment, in response to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition and processing of data files, multiple data files are acquired. The multiple data files are read and processed according to a preset data entry script to obtain a first delimiter between each record in the data files and a second delimiter between each field in the records; wherein the data entry script is used to instruct the reading of data from the data files. Based on the mapping relationship between the first delimiter in the data files and a first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data files and a second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table, the data from each read record is stored in a data warehouse table in the database corresponding to the server, according to the first delimiter between each record and the second delimiter between each field in the record. The processing steps include data sending instructions and data retrieval instructions. Based on the data sending instructions in the preset processing script, data from the data warehouse table and the data retrieval instructions in the processing script are sent to each of the multiple other servers. Each other server then processes the received data according to the data retrieval instructions to obtain preliminary processed data. The processing script indicates the processing instructions for data processing. The system receives the preliminary data from each other server. It then summarizes and processes the preliminary data from each other server to obtain the processed target data. The target data is stored in the target table to complete the data processing operation. The data in the target table is exported according to the preset export script to obtain the exported target file. The export script indicates that the target data is exported according to a preset format. The target file is stored in a preset path so that other servers can retrieve the target file from that path. Therefore, the entire process is script-based, requiring no application program; only a timer tool needs to schedule the preset script to run, solving the technical problem of low data processing efficiency and addressing the Hive data skew issue.

[0081] For example, Figure 3 A flowchart illustrating another report-based data processing method provided in this application embodiment is shown below. Figure 3 As shown, the server obtains the data file, processes it to obtain the target table, exports the target file corresponding to the target table, and schedules the target file to other servers. After obtaining the target file, the other servers parse it to obtain the target table and generate a view table. Finally, the preset report display tool calls and displays the data in the Oracle server.

[0082] Figure 4A schematic diagram of the structure of a report-based data processing device provided in this application embodiment is shown below. Figure 4 As shown, the device includes:

[0083] The acquisition unit 31 is used to respond to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition of data files and to process the data files, thereby acquiring multiple data files.

[0084] The reading unit 32 is used to read and process multiple data files according to a preset inbound script, and store the read data into a data warehouse table in the database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files.

[0085] Processing unit 33 is used to process the data in the data warehouse table according to the preset processing script to obtain the processed target data.

[0086] Storage unit 34 is used to store target data into a target table to complete data processing operations; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to preset processing step instructions.

[0087] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0088] Figure 5 This is a schematic diagram of another report-based data processing device provided in an embodiment of this application. Figure 4 Based on the illustrated embodiments, as Figure 5 As shown, the processing step instructions include data sending instructions and data acquisition instructions; the processing unit 33 includes:

[0089] The sending module 331 is used to send the data in the data warehouse table and the data acquisition instruction in the processing script to each of the multiple other servers according to the data sending instruction in the preset processing script, so that each other server processes the received data according to the data acquisition instruction to obtain the preliminary processed data.

[0090] The receiving module 332 is used to receive preliminary data sent by each other server.

[0091] The aggregation module 333 is used to aggregate the preliminary data sent by each other server to obtain the processed target data.

[0092] In one example, reading unit 32 includes:

[0093] The reading module 321 is used to read and process multiple data files according to a preset data entry script, and to obtain the first delimiter between each record in the data file and the second delimiter between each field in the record.

[0094] Storage module 322 is used to store the data of each read record into a data warehouse table in the database corresponding to the server, based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table.

[0095] In one example, the device also includes:

[0096] The export unit 41 is used to export the target data in the target table according to the preset export script to obtain the exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format.

[0097] Storage unit 42 is used to store the target file in a preset path so that other servers can obtain the target file from the path.

[0098] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0099] Figure 6 This application provides a schematic diagram of the structure of a server, as shown in the embodiment of the present application. Figure 6 As shown, the server includes: a memory 51 and a processor 52.

[0100] The memory 51 stores a computer program that can run on the processor 52.

[0101] Processor 52 is configured to perform the methods provided in the embodiments described above.

[0102] The server also includes a receiver 53 and a transmitter 54. The receiver 53 is used to receive instructions and data sent by external devices, and the transmitter 54 is used to send instructions and data to external devices.

[0103] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of a server, enables the server to perform the methods provided in the above embodiments.

[0104] This application also provides a computer program product, which includes: a computer program stored in a readable storage medium, at least one processor of the server can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the server to perform the solution provided in any of the above embodiments.

[0105] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0106] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method based on reports, characterized in that, include: In response to a received data processing instruction, wherein the data processing instruction is used to instruct the acquisition of data files and the processing of the data files, thereby acquiring multiple data files; According to a preset inbound script, multiple data files are read and processed, and the read data is stored in a data warehouse table in a database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files; The data in the data warehouse table is processed according to a preset processing script to obtain the processed target data, and the target data is stored in the target table to complete the data processing operation; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to preset processing step instructions; the processing script is at least one processing step instruction generated based on at least one preset processing step information; The processing step instructions include data sending instructions and data acquisition instructions; The step of processing the data in the data warehouse table according to a preset processing script to obtain the processed target data includes: According to the data sending instructions in the preset processing script, the data in the data warehouse table and the data acquisition instructions in the processing script are sent to each of the multiple other servers, so that each of the other servers processes the received data according to the data acquisition instructions to obtain the preliminary processed data. Receive preliminary data sent by each other server; summarize and process the preliminary data sent by each other server to obtain the processed target data.

2. The method according to claim 1, characterized in that, The step of reading and processing multiple data files according to a preset inbound script, and storing the read data into a data warehouse table in a database corresponding to the server, includes: According to the preset data entry script, multiple data files are read and processed to obtain the first delimiter between each record in the data files and the second delimiter between each field in the record; Based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table, the data in each read record is stored in the data warehouse table of the database corresponding to the server according to the first delimiter between each record and the second delimiter between each field in the record.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The target data in the target table is exported according to the preset export script to obtain the exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format; The target file is stored in a preset path so that other servers can retrieve the target file from that path.

4. A report-based data processing device, characterized in that, include: An acquisition unit is configured to respond to a received data processing instruction, wherein the data processing instruction is configured to instruct the acquisition of data files and the processing of the data files, thereby acquiring multiple data files; The reading unit is used to read and process multiple data files according to a preset inbound script, and store the read data into a data warehouse table in a database corresponding to the server; wherein, the inbound script is used to instruct the reading of data from the data files; The processing unit is used to process the data in the data warehouse table according to a preset processing script to obtain the processed target data; A storage unit is used to store the target data into a target table to complete the data processing operation; wherein, the processing script is used to instruct the data in the data warehouse table to be processed according to preset processing step instructions; the processing script is at least one processing step instruction generated based on at least one preset processing step information; The processing step instructions include data sending instructions and data acquisition instructions; The processing unit includes: The sending module is used to send the data in the data warehouse table and the data acquisition instruction in the processing script to each of the multiple other servers according to the data sending instruction in the preset processing script, so that each of the other servers processes the received data according to the data acquisition instruction to obtain the preliminary processed data. The receiving module is used to receive the initial data sent by each other server; The aggregation module is used to aggregate the preliminary data sent by each other server to obtain the processed target data.

5. The apparatus according to claim 4, characterized in that, The reading unit includes: The reading module is used to read and process multiple data files according to a preset loading script, and obtain the first delimiter between each record in the data file and the second delimiter between each field in the record; The storage module is used to store the data of each read record into a data warehouse table in the database corresponding to the server, based on the mapping relationship between the first delimiter in the data file and the first preset delimiter in the target table, and the mapping relationship between the second delimiter in the data file and the second preset delimiter in the target table, wherein the first preset delimiter is the delimiter between each record in the target table, and the second preset delimiter is the delimiter between each field in each record in the target table.

6. The apparatus according to claim 4 or 5, characterized in that, The device further includes: An export unit is used to export the target data in the target table according to a preset export script to obtain an exported target file; wherein, the export script is used to instruct the target data to be exported in a preset format; A storage unit is used to store the target file in a preset path so that other servers can obtain the target file in the path.

7. A server, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method of any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-3.

Images