A method for analyzing configuration fields of big data cleaning

By setting data type IDs in the Excel template and automatically calling the parsing configuration, the problem of high workload and error rate caused by developers manually writing parsing programs is solved, and an efficient data entry process is achieved.

CN112183047BActive Publication Date: 2024-11-05SICHUAN SHANGTONG PAYMENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011050140.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-29
Publication Date
2024-11-05
Estimated Expiration
2040-09-29

AI Technical Summary

Technical Problem

In existing technologies, developers need to manually write and test parsing programs based on new data templates each time, which results in a large workload and is prone to errors.

Method used

By setting the data type ID of the Excel template and uploading it to the program configuration file directory, the cleaning program automatically calls the corresponding parsing configuration to parse the fields, including automatic import, matching, and saving of the parsing process, supporting single, multiple, and batch uploads.

Benefits of technology

It reduced the workload of developers, lowered the error rate, and improved the efficiency of data entry.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112183047B_ABST
    Figure CN112183047B_ABST
Patent Text Reader

Abstract

The present invention discloses a big data cleaning configuration field parsing method, including the following specific steps: S1, setting the type ID of the file, the user sets the type ID of the file through the data of the Excel template, and the data type corresponding to each field can be a string, an integer, or a floating point number; S2, establishing a program configuration file directory, and uploading the type ID of the setting file to the program configuration file directory; S3, template data cleaning, when the user performs template data cleaning, the Excel template is uploaded and the ID of the template type is selected; S4, field parsing, the cleaning program automatically calls the corresponding field of the corresponding parsing configuration for parsing according to the data source, and the template ID. The type ID of the file is set through the data of the Excel template, and the ID is uploaded to the program configuration file directory. The cleaning program automatically calls the corresponding field of the corresponding parsing configuration for parsing according to the data source, and the template ID is uploaded to the program configuration file directory. The cleaning program automatically calls the corresponding field of the corresponding parsing configuration for parsing according to the data source, thereby reducing the workload of the staff.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of big data cleaning field analysis, and specifically is a big data cleaning configuration field analysis method. Background Art

[0002] With the advent of the big data era, the massive amount of data is increasing rapidly. Various industries can integrate and readjust existing resources with the support of big data technology, improve industry operation efficiency, tap the huge potential of the industry, and need to continuously collect data.

[0003] The biggest difference at present is that when people need to collect new data, developers need to write the corresponding parsing program according to the new template every time, and then test and update it to the production server before they can enter the new data into the database, which results in a huge workload, more business processes and easy errors. Summary of the invention

[0004] The purpose of the present invention is to provide a big data cleaning configuration field parsing method to solve the problem that when personnel need to collect new data, developers need to write corresponding parsing programs according to new templates every time, and then test and update them to the production server before new data can be entered into the database, which results in a large workload.

[0005] To achieve the above object, the present invention provides the following technical solution: a big data cleaning configuration field parsing method, comprising the following specific steps:

[0006] S1. Set the file type ID. The user sets the file type ID through the data in the Excel template. The data type corresponding to each field can be string, integer, or floating point number.

[0007] S2. Create a program configuration file directory and upload the type ID of the setting file to the program configuration file directory;

[0008] S3, template data cleaning. When the user cleans the template data, upload the Excel template and select the ID of the template type;

[0009] S4, field parsing, the cleaning program will automatically call the corresponding fields of the corresponding parsing configuration for parsing based on the data source and template ID.

[0010] As a further solution of the present invention: the establishment of the program configuration file directory includes a plurality of program configuration file directory modules, each of the program configuration file directory modules includes a program configuration file directory detection template and a program configuration file directory storage module;

[0011] The plurality of program configuration file directory modules operate independently of each other and correspond to the data type corresponding to each field;

[0012] The program configuration file directory detection template is used to detect whether data loss occurs when uploading the type ID of the file, and an intermediate data service module is provided in the program configuration file directory detection template, wherein the intermediate data service module can divide the data into multiple intermediate point segments;

[0013] The program configuration file directory storage module stores the IDs uploaded to the multiple program configuration file directory modules in a database.

[0014] As a further solution of the present invention: the two end points of the intermediate point segment of the intermediate data service module are used as input point X and output point Y, and the input point X and output point Y are detected and information is collected through the program configuration file directory detection template;

[0015] When the uploaded data is not lost, the data information A1 and A2 of the input point X and the data information B1 and B2 of the output point Y are used to generate a quadratic equation according to the first formula, as shown in Formula 1:

[0016] Y=KX (1)

[0018] When the uploaded data is lost, Formula 1 can be transformed into Formula 2:

[0019] Y=KX+B (2)

[0021] Where B is the data intercept, generating formula (3):

[0022] Y=X+B (3)

[0024] The coordinates of the input point X are (A1, A2), and the coordinates of the output point Y are (B1, B2).

[0025] As a further solution of the present invention: the template data cleaning includes a single upload module, a multiple upload module, and a batch upload module;

[0026] The single upload module selects the ID of the Excel template individually and uploads them one by one. When selecting, the ID of the template is numbered as 1, 2, 3, 4...N;

[0027] The multiple upload module performs multiple combined uploads, and when selecting, the template ID is numbered as one of 12, 13...1N or 123, 134...13N or 2678,...2345N;

[0028] The batch upload module performs batch combination upload, and the template ID number is 123456...N when selecting.

[0029] As a further solution of the present invention: when uploading template data for cleaning, the single-item uploading module, the multiple-item uploading module and the batch uploading module are uploaded independently, and two or three of them are simultaneously uploaded in combination and then uploaded independently.

[0030] As a further solution of the present invention: the field parsing includes an automatic import parsing module, an automatic matching parsing module, and a saving parsing module;

[0031] The automatic import parsing module is used to automatically import the cleaning program data template ID according to the uploaded format;

[0032] The automatic matching and parsing module is used to automatically match the program data template ID in the format uploaded by the automatic import and parsing module and perform parsing;

[0033] The saving and parsing module is used to save the program data template ID parsed by the automatic matching and parsing module.

[0034] As a further solution of the present invention: the automatic matching and parsing module includes a data recovery module and a data clearing module;

[0035] The data recovery module is used to recover the ID of the data when the automatic matching and parsing module finds that the data is wrong or has a gap;

[0036] The data clearing module clears the ID of the data when the automatic matching and parsing module parses data that is repeated or wrong.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The file type ID is set through the data in the Excel template, and the ID is uploaded to the program configuration file directory. The cleaning program will automatically call the corresponding field of the corresponding parsing configuration for parsing based on the data source and template ID, thereby reducing the workload of the staff and avoiding errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A diagram showing the steps of field analysis of a big data cleaning configuration according to the present invention;

[0040] Figure 2 A flow chart for establishing a program configuration file directory for the present invention;

[0041] Figure 3 A flow chart of template data cleaning of the present invention;

[0042] Figure 4 This is a flow chart of field parsing of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0044] See also Figure 1-Figure 4 As shown, in an embodiment of the present invention, a method for parsing a big data cleaning configuration field includes the following specific steps:

[0045] S1. Set the file type ID. The user sets the file type ID through the data in the Excel template. The data type corresponding to each field can be string, integer, or floating point number.

[0046] S2. Create a program configuration file directory and upload the type ID of the setting file to the program configuration file directory;

[0047] S3, template data cleaning. When the user cleans the template data, upload the Excel template and select the ID of the template type;

[0048] S4, field parsing, the cleaning program will automatically call the corresponding fields of the corresponding parsing configuration for parsing based on the data source and template ID.

[0049] In this embodiment, a server based on a big database is used. The big data server is a combination of data files. The data file combination creates objects, views, texts, functions, pictures, and various combination files. A file type ID is set on the data file, where the ID can correspond to a string, an integer, a floating point number, etc. Example 2

[0050] In conjunction with Example 1, the difference is that, Figure 2 As shown, establishing a program configuration file directory includes multiple program configuration file directory modules, each of which includes a program configuration file directory detection template and a program configuration file directory storage module; multiple program configuration file directory modules run independently of each other and correspond to the data type corresponding to each field;

[0051] Among them, the ID can correspond to a string, an integer, or a floating point number and be uploaded to different program configuration file directory modules, and each corresponding ID can be uploaded to the template data cleaning separately or in combination.

[0052] The program configuration file directory detection template is used to detect whether data loss occurs when uploading the type ID of the file. The program configuration file directory detection template is provided with an intermediate data service module, wherein the intermediate data service module can divide the data into multiple intermediate point segments; the program configuration file directory saving module saves the IDs uploaded to multiple program configuration file directory modules in the database.

[0053] The two endpoints of the intermediate point segment of the intermediate data service module are used as the input point X and the output point Y, and the input point X and the output point Y are detected and information is collected through the program configuration file directory detection template;

[0054] When the uploaded data is not lost, the data information A1 and A2 of the input point X and the data information B1 and B2 of the output point Y are used to generate a quadratic equation according to the first formula, as shown in Formula 1:

[0055] Y=KX (1)

[0057] When the uploaded data is lost, Formula 1 can be transformed into Formula 2:

[0058] Y=KX+B (2)

[0060] Where B is the data intercept, generating formula (3):

[0061] Y=X+B (3)

[0063] The coordinates of the input point X are (A1, A2), and the coordinates of the output point Y are (B1, B2).

[0064] Among them, through the program configuration file directory detection template detection analysis, it can be seen from formula 1 that when no intercept is generated, it can be known that the data is not discarded. When the program configuration file directory detection template detection analysis is performed, when formula 1 can be transformed into formula 2 and formula 3, an intercept appears, and the intercept indicates that data is missing, and the missing part is the part displayed by the intercept. Example 3

[0065] Combining Example 1 with Example 2, the difference is that, Figure 3 As shown, template data cleaning includes single-item upload module, multiple-item upload module, and batch upload module;

[0066] Single upload module, select the ID of the Excel template individually and upload them one by one. When selecting, the template ID is numbered as 1, 2, 3, 4...N;

[0067] Multiple upload module, for multiple combined uploads, when selecting, the template ID number is one of 12, 13...1N or 123, 134....13N or 2678,...2345N;

[0068] Batch upload module, for batch combination upload, when selecting, the template ID number is 123456...N.

[0069] When uploading template data for cleaning, upload the single upload module, multiple upload module, and batch upload module independently, or upload two or all three simultaneously and then upload them independently. Example 4

[0070] Combining Example 1 with Example 2 and Example 3, the difference is that, Figure 4 As shown, field parsing includes automatic import parsing module, automatic matching parsing module and saving parsing module;

[0071] Automatic import parsing module, used to clean program data template ID and automatically import it according to the uploaded format;

[0072] The automatic matching and parsing module is used to automatically match the program data template ID in the format uploaded by the automatic import and parsing module and perform parsing;

[0073] The save analysis module is used to save the program data template ID analyzed by the automatic matching analysis module.

[0074] The automatic matching and parsing module includes a data recovery module and a data clearing module;

[0075] Data recovery module: when the automatic matching and parsing module finds that the data is wrong or has a gap, the data recovery module is used to recover the ID of the data;

[0076] Clear data module: when the automatic matching and parsing module finds that the data is duplicated or wrong, the ID of the data will be cleared through the clear data module.

[0077] To summarize, the file type ID is set through the data in the Excel template, and the ID is uploaded to the program configuration file directory. The cleaning program will automatically call the corresponding field of the corresponding parsing configuration for parsing based on the data source and template ID, thereby reducing the workload of the staff.

[0078] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0079] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode includes only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for analyzing fields in a big data cleaning configuration format, characterized in that: The specific steps include: S1. Set the file type ID. The user sets the file type ID through the data in the Excel template. The data type corresponding to each field includes string, integer, and floating point number. S2. Create a program configuration file directory and upload the type ID of the setting file to the program configuration file directory; S3, template data cleaning. When the user cleans the template data, upload the Excel template and select the ID of the template type; S4, field parsing, the cleaning program will automatically call the corresponding fields of the corresponding parsing configuration for parsing according to the data source and template ID; Wherein, the field parsing includes an automatic import parsing module, an automatic matching parsing module, and a saving parsing module; The automatic import parsing module is used to automatically import the cleaning program data template ID according to the uploaded format; The automatic matching and parsing module is used to automatically match the program data template ID in the format uploaded by the automatic import and parsing module and perform parsing; The saving and parsing module is used to save the program data template ID parsed by the automatic matching and parsing module; The automatic matching and parsing module includes a data recovery module and a data clearing module; The data recovery module is used to recover the ID of the data when the automatic matching and parsing module finds that the data is wrong or has a gap; The data clearing module clears the ID of the data when the automatic matching and parsing module parses data that is repeated or wrong.

2. A method for analyzing fields of a big data cleaning configuration according to claim 1, characterized in that: The establishment of the program configuration file directory includes a plurality of program configuration file directory modules, each of which includes a program configuration file directory detection template and a program configuration file directory storage module; The plurality of program configuration file directory modules operate independently of each other and correspond to the data type corresponding to each field; The program configuration file directory detection template is used to detect whether data loss occurs when uploading the type ID of the file, and an intermediate data service module is provided in the program configuration file directory detection template, wherein the intermediate data service module can divide the data into multiple intermediate point segments; The program configuration file directory storage module stores the IDs uploaded to the multiple program configuration file directory modules in a database.

3. A method for analyzing fields of a big data cleaning configuration according to claim 2, characterized in that: The two endpoints of the intermediate point segment of the intermediate data service module are used as input point X and output point Y, and the input point X and output point Y are detected and information is collected through the program configuration file directory detection template; When the uploaded data is not lost, the data information A1 and A2 of the input point X and the data information B1 and B2 of the output point Y are used to generate a quadratic equation according to the first formula, as shown in Formula 1: Y=KX (1) When the uploaded data is lost, Formula 1 can be transformed into Formula 2: Y=KX+B (2) Where B is the data intercept, generating formula (3): Y=X+B (3) The coordinates of the input point X are (A1, A2), and the coordinates of the output point Y are (B1, B2).

4. A method for analyzing fields of a big data cleaning configuration according to claim 1, characterized in that: The template data cleaning includes a single upload module, a multiple upload module, and a batch upload module; The single upload module selects the ID of the Excel template individually and uploads them one by one. When selecting, the ID of the template is numbered as 1, 2, 3, 4...N; The multiple upload module performs multiple combined uploads, and when selecting, the template ID is numbered as one of 12, 13...1N or 123, 134...13N or 2678,...2345N; The batch upload module performs batch combination upload, and the template ID number is 123456...N when selecting.

5. A method for analyzing fields of a big data cleaning configuration according to claim 4, characterized in that: When uploading template data for cleaning, upload the single upload module, multiple upload module, and batch upload module independently, or upload two or all three simultaneously and then upload them independently.