A data processing method, device, storage medium and equipment

Abstract

The present application discloses a data processing method, device, storage medium and equipment, which obtain various data files, the data files include a first table file and a compressed file, the first table file has a business number, the compressed file is decompressed to obtain a second table file, the second table file has a business number, based on the preset correspondence between the business number and the subject folder, the first table file and the second table file are stored in the corresponding subject folder, the subject folder is imported into the kettle tool, and the kettle tool is triggered to perform a data extraction task. Among them, the data extraction task is used to extract the preset input field from the target table file according to the preset file path, and pre-process the input field, and import the pre-processed input field into the preset data table. It can be seen that compared with the prior art, the present application can avoid manual participation in the collection and verification of data, and the efficiency is significantly improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, apparatus, storage medium and device. Background Art

[0002] For large enterprises, normal business monitoring work mainly includes four links: data acquisition and issuance, data collection and verification, data processing and verification, and monitoring report preparation. In the data acquisition and issuance, as well as data processing and verification, each subsidiary under the name of a large enterprise will obtain business monitoring data from the business background, and integrate the business monitoring data with the system data issued by the headquarters of the large enterprise to generate a data file, and then send the data file to the preset operation monitoring workbench (specifically the server), and the operation monitoring workbench will upload the data file to the headquarters of the large enterprise. Finally, the headquarters of the large enterprise will be responsible for the data collection and verification, as well as the monitoring report preparation.

[0003] How to improve the processing efficiency of data collection and verification has become an urgent problem that large enterprises need to solve. Summary of the invention

[0004] The applicant found that: during the data collection process, since the formats of the data files uploaded by each subsidiary are mainly spreadsheet files (such as Excel files) and compressed files (such as compressed packages), a large number of manual intervention operations (such as decompressing compressed files, classifying spreadsheet files, etc.) are required, resulting in low efficiency. In addition, during the data verification process, some data files have problems such as non-standard data formats, inconsistent data dimensions, and data redundancy, which require a lot of time to solve these problems, thereby reducing efficiency.

[0005] The present application provides a data processing method, apparatus, storage medium and device, the purpose of which is to improve the efficiency of data collection and verification.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A data processing method, comprising:

[0008] Acquire each data file; the data file includes a first table file and a compressed file; the first table file has a business number;

[0009] Decompressing the compressed file to obtain a second table file; the second table file has a business number;

[0010] Based on the preset correspondence between the business number and the subject folder, storing the first table file and the second table file in the corresponding subject folder;

[0011] Import the subject folder into the kettle tool, and trigger the kettle tool to execute a data extraction task; wherein the data extraction task is used to extract preset input fields from a target table file according to a preset file path, and preprocess the input fields, and import the preprocessed input fields into a preset data table; the target table file includes the first table file and the second table file.

[0012] Optionally, also include:

[0013] The file path, the input field, and the data table are preconfigured in the kettle tool.

[0014] Optionally, the data extraction task includes a first preprocessing job, and the first preprocessing job is used to preprocess the input field according to a preset first rule;

[0015] The preprocessing of the input field according to the preset first rule comprises:

[0016] Taking a preset reference field as a benchmark requirement, performing string correction on the input field to obtain a field that meets the benchmark requirement; wherein the string correction means includes at least one of character replacement, space deletion, and character deletion;

[0017] The fields meeting the benchmark requirements are used as the pre-processed input fields.

[0018] Optionally, the data extraction task includes a second preprocessing job, and the second preprocessing job is used to preprocess the input field according to a preset second rule;

[0019] The preprocessing of the input field according to the preset second rule includes:

[0020] Identify the input field and determine the object described by the input field;

[0021] Based on the correspondence between the object and the preset expression method, the input field is converted using the corresponding expression method to obtain a new field;

[0022] The new field is used as the pre-processed input field.

[0023] Optionally, the data extraction task includes a third preprocessing job, and the third preprocessing job is used to preprocess the input field according to a preset third rule;

[0024] The preprocessing of the input field according to the preset third rule includes:

[0025] For the same multiple input fields, selecting the input field with the latest generation time from the multiple input fields as the target input field;

[0026] The step of importing the preprocessed input fields into a preset data table includes:

[0027] Import the target input field into a preset data table.

[0028] Optionally, storing each table file in the subject folder corresponding to the business number preset by itself includes:

[0029] The target table files under each of the subject folders are sorted in order of the acquisition time of the target table files from early to late.

[0030] A data processing device, comprising:

[0031] An acquisition unit, used for acquiring each data file; the data file includes a first table file and a compressed file; the first table file has a business number;

[0032] A decompression unit, used for decompressing the compressed file to obtain a second table file; the second table file has a business number;

[0033] A storage unit, configured to store the first table file and the second table file in the corresponding subject folder based on a preset correspondence between the business number and the subject folder;

[0034] An extraction unit is used to import the subject folder into a kettle tool and trigger the kettle tool to execute a data extraction task; wherein the data extraction task is used to extract preset input fields from a target table file according to a preset file path, and preprocess the input fields, and import the preprocessed input fields into a preset data table; the target table file includes the first table file and the second table file.

[0035] Optionally, also include:

[0036] A configuration unit is used to preconfigure the file path, the input field, and the data table in the kettle tool.

[0037] A computer-readable storage medium comprises a stored program, wherein the program executes the data processing method.

[0038] A data processing device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0039] The memory is used to store programs, and the processor is used to run programs, wherein the data processing method is executed when the program is run.

[0040] The technical solution provided by the present application obtains various data files, the data files include a first table file and a compressed file, the first table file has a business number, the compressed file is decompressed to obtain a second table file, the second table file has a business number, based on the preset correspondence between the business number and the subject folder, the first table file and the second table file are stored in the corresponding subject folder, the subject folder is imported into the kettle tool, and the kettle tool is triggered to perform the data extraction task. Among them, the data extraction task is used to extract the preset input field from the target table file according to the preset file path, and pre-process the input field, and import the pre-processed input field into the preset data table. The target table file includes the first table file and the second table file. It can be seen that compared with the prior art, the present application can avoid manual participation in the collection and verification of data, and the efficiency is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of another data processing method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The embodiment of the present application takes a power grid enterprise as an example and provides a data processing method which is applied to the backend system of the power grid enterprise. Specifically, it can be executed by a system host to improve the processing efficiency of data collection and verification.

[0047] like Figure 1 FIG. 1 is a schematic diagram of a data processing method provided in an embodiment of the present application, comprising the following steps:

[0048] S101: calling a web crawler process to obtain various data files from the operation monitoring workbench.

[0049] Among them, the so-called web crawler, also known as web spider, web robot, and web chaser, is a process that automatically captures network information according to preset rules.

[0050] It should be emphasized that the data file includes a first table file and a compressed file, and the first table file has a preset business number.

[0051] S102: Decompress the compressed file to obtain a second table file.

[0052] The specific implementation process of decompressing the compressed file is common knowledge familiar to those skilled in the art and will not be described in detail here.

[0053] It should be noted that the first table file and the second table file are only used to distinguish the source of the table file, and there is no specific association between the two. Of course, the second table file also has a preset business number.

[0054] S103: Based on a preset correspondence between the business number and the subject folder, the first table file and the second table file are stored in the corresponding subject folder.

[0055] Among them, business types include but are not limited to: power grid infrastructure project management, power grid infrastructure process, small infrastructure project management, daily substation operation and inspection, daily transmission operation and inspection, daily distribution operation and inspection, distribution network fault emergency repair process, business expansion and registration process, basic electricity fee management, power factor electricity fee management, electricity fee management, electricity quantity electricity fee refund, electricity user management, prepaid electricity fee management, fixed asset disposal management and fixed asset depreciation and fully depreciated assets, totaling 16 types.

[0056] It should be noted that by storing each form file in a subject folder corresponding to its own preset business number, form files of the same business type can be stored in the same subject folder, thereby realizing the classification of each form file.

[0057] In addition, in the embodiment of the present application, the various table files under each subject folder can be sorted in order from early to late according to the acquisition time of each table file, so as to realize the sorting of each table file.

[0058] S104: Pre-configure the file path, input field, and data table in the kettle tool.

[0059] Among them, the so-called kettle tool is an open source data migration (Extract Transform Load, ETL) tool. Kettle is commonly known as a kettle in Chinese. The kettle tool can be used to integrate data in various formats into a specified format.

[0060] It should be noted that S104 may be executed before S101, S102, or S103, or executed in parallel with S101, S102, or S103.

[0061] S105: Import the subject folder into the kettle tool, and trigger the kettle tool to execute the data extraction task.

[0062] The data extraction task is used to extract input fields from the target table file according to the file path, preprocess the input fields, and import the preprocessed input fields into the data table. The target table file includes a first table file and a second table file.

[0063] It should be noted that in the process of executing the data extraction task, the kettle tool will pre-create and generate three kinds of information (which can be understood as script files) based on the pre-configured file path and input field, namely job information, task information, and operation requirement information. In the process of obtaining the data file, the background system will also create two kinds of information at the same time, namely table file information and download file information (script files used to identify data files). In the process of storing the first table file and the second table file in the corresponding subject folder, the background system will also create two kinds of information at the same time, namely subject information and subject unit information (script files used to locate the first table file and the second table file). Specifically, in the process of the kettle tool executing the data extraction task, the above 7 script files will be called to implement the data extraction task. Of course, the specific implementation method of the kettle tool calling the above 7 script files is common knowledge familiar to those skilled in the art, and will not be repeated here.

[0064] In the embodiment of the present application, the above-mentioned 7 types of information all follow a pre-configured rule table (ie, data structure).

[0065] Taking a two-dimensional table (a method of expressing an existing data structure) as an example, the specific rule table can be seen in Table 1 below.

[0066] Table 1

[0067]

[0068] In the above Table 1, the job information (i.e., the script file of the preprocessing job, i.e., the job file) includes the job code (i.e., the code of the job file), the job name (i.e., the specific name of the job file), the job file path (i.e., the disk path of the job file), the job description (i.e., the specific operation mode of the preprocessing, such as string replacement, string space removal, character deletion, etc.), and whether to allow running (i.e., whether the job file is being executed);

[0069] The subject information (i.e., the script file of the subject folder) includes the subject number (i.e., the number of the business type), the subject name (i.e., the specific name of the business type), and the file type (i.e., the type of the subject folder);

[0070] The table file information (i.e. the script file of the table file) includes the file path (i.e. the disk path of the table file) and the last update time (i.e. the time point when the table file was last changed);

[0071] Download file information (i.e., the script file of the communication protocol configured when obtaining the data file, i.e., the ftp file) includes the file path (i.e., the disk path of the ftp file), the last update time (i.e., the time point when the ftp file was last changed), and whether it has been processed (i.e., whether the ftp file is being called);

[0072] The task information (i.e., the script file of the data extraction task, i.e., the task file) includes the subject number (i.e., the number of the business type), the task type (i.e., the type of data operation performed by the task file, such as deleting data, writing data, etc.), the task file path (i.e., the disk path of the task file), whether it is being allowed (i.e., whether the task file is being run), the task name (i.e., the specific name of the task file), the task code (i.e., the code of the task file), and the code (i.e., the identity of the task file);

[0073] The operation requirement information (i.e. the script file used to run the task file, i.e. the run file) includes the subject number (i.e. the number of the business type), the province (i.e. the extension file name of the run file), whether it needs to be run (i.e. whether the run file is run), and the required time (i.e. the time point for running the run file);

[0074] The subject unit information (i.e., the script file used to record the sender of the data file) includes the subject number (i.e., the number of the business type), the province name (the specific name of the sender), and the sequence number (the preset sequence number of the sender).

[0075] It should be emphasized that the data extraction task includes a first preprocessing job, a second preprocessing job, and a third preprocessing job. The first preprocessing job is used to preprocess the input field according to the preset first rule, the second preprocessing job is used to preprocess the input field according to the preset second rule, and the third preprocessing job is used to preprocess the input field according to the preset third rule.

[0076] Specifically,

[0077] The specific implementation process of preprocessing the input field according to the preset first rule is: taking the preset reference field as the benchmark requirement, performing string correction on the input field to obtain a field that meets the benchmark requirements, and using the field that meets the benchmark requirements as the preprocessed input field. Among them, the means of string correction include but are not limited to character replacement, space deletion, and character deletion. In an embodiment of the present application, a field in the power grid infrastructure project management table can be used as a reference field. Taking the input field "11 / 10 / 2020" as an example, the meaning represented by the input field is the date. Taking the reference field "YYYY / MM / DD" as the benchmark requirement for the date, "11 / 10 / 2020" is replaced with characters to obtain "2020 / 10 / 11", and "2020 / 10 / 11" is used as the preprocessed input field.

[0078] In practical applications, preprocessing the input field according to the preset first rule can solve the problem of non-standard data format in the data verification process.

[0079] The specific implementation process of preprocessing the input field according to the preset second rule is: identify the input field, determine the object described by the input field, based on the correspondence between the object and the preset expression method, use the corresponding expression method to convert the input field to obtain a new field, and use the new field as the preprocessed input field. Assuming that the objects include "provincial company abbreviation" and "other communication methods", "provincial company abbreviation" corresponds to the place name (that is, the place name is used as the expression method), and "other communication methods" corresponds to "other" (that is, "other" is used as the expression method). The existing first input field is "State Grid Anhui Electric Power" and the second input field is "AC A0606". The first input field is identified and it is determined that the object described by the first input field is "provincial company abbreviation", then the first input field is converted using the place name to obtain the "Anhui" field, the second input field is identified and it is determined that the object described by the second input field is "other communication methods", then the second input field is converted using "other" to obtain the "other" field, and the "Anhui" field and the "other" field are used as the preprocessed input fields.

[0080] In practical applications, preprocessing the input field according to the preset second rule can solve the problem of inconsistent data dimensions in the data verification process.

[0081] The specific implementation process of preprocessing the input field according to the preset third rule is: for the same multiple input fields, the input field with the latest generation time is selected from the multiple input fields as the target input field. In addition, any input field can also be selected from the multiple input fields as the target input field. In this case, the specific implementation method of importing the preprocessed input field into the preset data table is: importing the target input field into the preset data table.

[0082] In practical applications, preprocessing the input field according to the preset third rule can solve the problem of data redundancy in the data verification process.

[0083] In an embodiment of the present application, the specific implementation logic of the above S101-S105 can be written in Java language. Specifically, the existing JNA (Java Native Access) tool class can be called to perceive the change details of the target table file in the subject folder in real time, so as to trigger the kettle tool to perform the data extraction task.

[0084] In summary, compared with the prior art, this embodiment can avoid manual participation in data collection and verification, and the efficiency is significantly improved.

[0085] It should be noted that S104 mentioned in the above embodiment, pre-configuring the file path, input field, and data table in the kettle tool, is an optional specific implementation method in the data processing method provided by this application. In addition, the first preprocessing job, the second preprocessing job, and the third preprocessing job mentioned in the above embodiment are also optional preprocessing jobs in the data extraction task described in this application. Of course, the data extraction task also includes other preprocessing jobs. To this end, the steps shown in the above embodiment can be summarized as follows: Figure 2 The process shown.

[0086] like Figure 2 FIG. 1 is a schematic diagram of another data processing method provided in an embodiment of the present application, comprising the following steps:

[0087] S201: Acquire various data files.

[0088] The data file includes a first table file and a compressed file, and the first table file has a business number.

[0089] S202: Decompress the compressed file to obtain a second table file.

[0090] Among them, the second table file has a business number.

[0091] S203: Based on a preset correspondence between the business number and the subject folder, the first table file and the second table file are stored in the corresponding subject folder.

[0092] S204: Import the subject folder into the kettle tool, and trigger the kettle tool to execute the data extraction task.

[0093] Among them, the data extraction task is used to extract preset input fields from the target table file according to the preset file path, and preprocess the input fields, and import the preprocessed input fields into the preset data table. The target table file includes a first table file and a second table file.

[0094] In summary, compared with the prior art, this embodiment can avoid manual participation in data collection and verification, and the efficiency is significantly improved.

[0095] Corresponding to the data processing method provided in the above embodiment, the embodiment of the present application also provides a data processing device.

[0096] like Figure 3 FIG. 1 is a schematic diagram of a data processing device provided in an embodiment of the present application, comprising:

[0097] The acquisition unit 100 is used to acquire various data files. The data files include a first table file and a compressed file, and the first table file has a business number.

[0098] The decompression unit 200 is used to decompress the compressed file to obtain a second table file. The second table file has a business number.

[0099] The storage unit 300 is used to store the first table file and the second table file in the corresponding subject folder based on the preset corresponding relationship between the business number and the subject folder.

[0100] The storage unit 300 is further used to sort the target table files under each subject folder in the order of the acquisition time of the target table files from early to late.

[0101] The extraction unit 400 is used to import the subject folder into the kettle tool and trigger the kettle tool to perform a data extraction task. The data extraction task is used to extract a preset input field from a target table file according to a preset file path, preprocess the input field, and import the preprocessed input field into a preset data table. The target table file includes a first table file and a second table file.

[0102] The data extraction task includes a first preprocessing operation, and the first preprocessing operation is used to preprocess the input field according to a preset first rule. The specific process of the extraction unit 400 for preprocessing the input field according to the preset first rule includes: taking a preset reference field as a benchmark requirement, performing string correction on the input field to obtain a field that meets the benchmark requirement, and using the field that meets the benchmark requirement as the preprocessed input field, and the means of string correction includes at least one of character replacement, space deletion, and character deletion.

[0103] The data extraction task includes a second preprocessing operation, and the second preprocessing operation is used to preprocess the input field according to the preset second rule. The specific process of the extraction unit 400 for preprocessing the input field according to the preset second rule includes: identifying the input field, determining the object described by the input field, based on the correspondence between the object and the preset expression method, using the corresponding expression method, converting the input field to obtain a new field, and using the new field as the preprocessed input field.

[0104] The data extraction task includes a third preprocessing operation, and the third preprocessing operation is used to preprocess the input field according to the preset third rule. The specific process of the extraction unit 400 for preprocessing the input field according to the preset third rule includes: for the same multiple input fields, selecting the input field with the latest generation time from the multiple input fields as the target input field. In this case, importing the preprocessed input field into the preset data table includes: importing the target input field into the preset data table.

[0105] The configuration unit 500 is used to pre-configure the file path, input field, and data table in the kettle tool.

[0106] In summary, obtain each data file, the data file includes a first table file and a compressed file, the first table file has a business number, decompress the compressed file to obtain a second table file, the second table file has a business number, based on the preset correspondence between the business number and the subject folder, store the first table file and the second table file in the corresponding subject folder, import the subject folder into the kettle tool, and trigger the kettle tool to perform the data extraction task. Among them, the data extraction task is used to extract the preset input field from the target table file according to the preset file path, and pre-process the input field, and import the pre-processed input field into the preset data table. The target table file includes the first table file and the second table file. It can be seen that compared with the prior art, the present application can avoid manual participation in the collection and verification of data, and the efficiency is significantly improved.

[0107] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the data processing method provided by the present application.

[0108] The present application also provides a data processing device, including: a processor, a memory and a bus. The processor and the memory are connected via a bus, the memory is used to store programs, and the processor is used to run programs, wherein when the program is run, the data processing method provided by the present application is executed, including the following steps:

[0109] Acquire each data file; the data file includes a first table file and a compressed file; the first table file has a business number;

[0110] Decompressing the compressed file to obtain a second table file; the second table file has a business number;

[0111] Based on the preset correspondence between the business number and the subject folder, storing the first table file and the second table file in the corresponding subject folder;

[0112] Import the subject folder into the kettle tool, and trigger the kettle tool to execute a data extraction task; wherein the data extraction task is used to extract preset input fields from a target table file according to a preset file path, and preprocess the input fields, and import the preprocessed input fields into a preset data table; the target table file includes the first table file and the second table file.

[0113] Optionally, also include:

[0114] The file path, the input field, and the data table are preconfigured in the kettle tool.

[0115] Optionally, the data extraction task includes a first preprocessing job, and the first preprocessing job is used to preprocess the input field according to a preset first rule;

[0116] The preprocessing of the input field according to the preset first rule comprises:

[0117] Taking a preset reference field as a benchmark requirement, performing string correction on the input field to obtain a field that meets the benchmark requirement; wherein the string correction means includes at least one of character replacement, space deletion, and character deletion;

[0118] The fields meeting the benchmark requirements are used as the pre-processed input fields.

[0119] Optionally, the data extraction task includes a second preprocessing job, and the second preprocessing job is used to preprocess the input field according to a preset second rule;

[0120] The preprocessing of the input field according to the preset second rule includes:

[0121] Identify the input field and determine the object described by the input field;

[0122] Based on the correspondence between the object and the preset expression method, the input field is converted using the corresponding expression method to obtain a new field;

[0123] The new field is used as the pre-processed input field.

[0124] Optionally, the data extraction task includes a third preprocessing job, and the third preprocessing job is used to preprocess the input field according to a preset third rule;

[0125] The preprocessing of the input field according to the preset third rule includes:

[0126] For the same multiple input fields, the input field with the latest generation time is selected from the multiple input fields as the target input field.

[0127] The step of importing the preprocessed input fields into a preset data table includes:

[0128] Import the target input field into a preset data table.

[0129] Optionally, storing each table file in the subject folder corresponding to the business number preset by itself includes:

[0130] The target table files under each of the subject folders are sorted in order of the acquisition time of the target table files from early to late.

[0131] If the functions described in the method of the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: include: Acquire each data file; the data file includes a first table file and a compressed file; The first form file has a business number; Decompressing the compressed file to obtain a second table file; the second table file has a business number; Based on the preset correspondence between the business number and the subject folder, storing the first table file and the second table file in the corresponding subject folder; Import the subject folder into the kettle tool, and trigger the kettle tool to execute a data extraction task; wherein the data extraction task is used to extract a preset input field from a target table file according to a preset file path, pre-process the input field, and import the pre-processed input field into a preset data table; the target table file includes the first table file and the second table file; Wherein, the data extraction task includes a second preprocessing operation, and the second preprocessing operation is used to preprocess the input field according to a preset second rule; The preprocessing of the input field according to the preset second rule includes: Identify the input field and determine the object described by the input field; Based on the correspondence between the object and the preset expression method, the input field is converted using the corresponding expression method to obtain a new field; The new field is used as the pre-processed input field.

2. The method according to claim 1, characterized in that: Also includes: The file path, the input field, and the data table are preconfigured in the kettle tool.

3. The method according to claim 1, characterized in that The data extraction task also includes a first preprocessing operation, wherein the first preprocessing operation is used to preprocess the input field according to a preset first rule; The preprocessing of the input field according to the preset first rule comprises: Taking a preset reference field as a benchmark requirement, performing string correction on the input field to obtain a field that meets the benchmark requirement; wherein the string correction means includes at least one of character replacement, space deletion, and character deletion; The fields meeting the benchmark requirements are used as the pre-processed input fields.

4. The method according to claim 1, characterized in that: The data extraction task also includes a third preprocessing operation, and the third preprocessing operation is used to preprocess the input field according to a preset third rule; The preprocessing of the input field according to the preset third rule includes: For the same multiple input fields, selecting the input field with the latest generation time from the multiple input fields as the target input field; The step of importing the preprocessed input fields into a preset data table includes: Import the target input field into a preset data table.

5. The method according to claim 1, characterized in that The storing of each table file in the subject folder corresponding to the business number preset by itself includes: The target table files under each of the subject folders are sorted in order of the acquisition time of the target table files from early to late.

6. A data processing device, characterized in that: include: An acquisition unit, used for acquiring each data file; the data file includes a first table file and a compressed file; The first form file has a business number; A decompression unit, used for decompressing the compressed file to obtain a second table file; the second table file has a business number; A storage unit, configured to store the first table file and the second table file in the corresponding subject folder based on a preset correspondence between the business number and the subject folder; An extraction unit, used for importing the subject folder into a kettle tool, and triggering the kettle tool to execute a data extraction task; wherein the data extraction task is used for extracting a preset input field from a target table file according to a preset file path, and preprocessing the input field, and importing the preprocessed input field into a preset data table; the target table file includes the first table file and the second table file; Wherein, the data extraction task includes a second preprocessing operation, and the second preprocessing operation is used to preprocess the input field according to a preset second rule; The preprocessing of the input field according to the preset second rule includes: Identify the input field and determine the object described by the input field; Based on the correspondence between the object and the preset expression method, the input field is converted using the corresponding expression method to obtain a new field; The new field is used as the pre-processed input field.

7. The device according to claim 6, characterized in that Also includes: A configuration unit is used to preconfigure the file path, the input field, and the data table in the kettle tool.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the data processing method according to any one of claims 1 to 5.

9. A data processing device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program executes the data processing method according to any one of claims 1 to 5 when it is run.

Images