An operation method for data analysis and processing based on data sources with indefinite formats

Abstract

The present application relates to the technical field of data processing, in particular to a data analysis and processing operation method based on data source format uncertainty, which can realize data acquisition, analysis, storage and use for the data source with uncertain data field format, content and sequence. The present application designs data storage table and data mapping table according to the number, format and type of data field of data source, and completes data collection by analyzing and storing the content and relationship of data field in data source. When using data, the data field format item and unique identifier corresponding to the identification information of input data are searched in the data mapping table according to the identification information, and the target data is obtained by comparing the data field format item and unique identifier in the data storage table to complete data use. The present application solves the technical problem of low data access efficiency, improves the data access efficiency, and can be applied to data analysis and processing, storage and use of data source format variability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data parsing and processing operation method based on indefinite data source format. BACKGROUND

[0002] Traditional data source processing methods are mostly fixed formats, and the mapping relationship is obtained through data source field name, number and order and stored in a database. However, many projects require variable formats of data sources, because the test formats and test categories are not unique, and storing all test format and test category mapping relationships into a database often consumes a lot of time and effort, resulting in low data access efficiency, and further resulting in the project being difficult to complete within the expected time. Therefore, the traditional data source processing method is often not suitable for variable data source formats.

[0003] Therefore, there is currently a need for a data parsing and processing operation method based on indefinite data source format. SUMMARY

[0004] The summary section is intended to introduce the concepts of the application in a simplified form, which will be described in more detail in the detailed description section. The summary section is not intended to identify key or essential features of the claimed technology, nor is it intended to limit the scope of the claimed technology.

[0005] In order to solve the technical problem of low data access efficiency, the present application provides a data parsing and processing operation method based on indefinite data source format.

[0006] The present application provides a data parsing and processing operation method based on indefinite data source format, which comprises:

[0007] Obtaining a test template file, determining the data field name and data field format of the test data included in the test template file according to the test template file, parsing the data field name and data field format to obtain the data format and data mapping relationship, and storing the data format and data mapping relationship into a data mapping table;

[0008] Obtaining a test file, obtaining a test file category identifier and a data field format according to the test file, comparing the data mapping table through the test file category identifier and the data field format, and storing the test file content into a data storage table according to the data mapping relationship of the data mapping table;

[0009] Obtaining a test identifier and a test file category identifier, comparing the data storage table through the test identifier and the test file category identifier, and obtaining target data from the data storage table.

[0010] Further, the test template file is acquired, the data field name and the data field format of the test data included in the test template file are determined according to the test template file, the data field name and the data field format are parsed, the data format and the data mapping relationship are obtained, the data format and the data mapping relationship are stored into the data mapping table, and the method comprises the following steps:

[0011] The test template file is acquired from the template upload interface;

[0012] The data field name and the data field format of the test data are acquired by traversing the test template file;

[0013] The data field name and the data field format of the test data are parsed, the data format and the data mapping relationship are obtained;

[0014] The database field format is acquired, the data format and the data mapping relationship obtained are matched, and the matching result is saved;

[0015] The data format and the data mapping relationship are stored into the data mapping table through the matching result.

[0016] Further, the test file is acquired, the test file category identifier and the data field format are acquired according to the test file, the data mapping table is compared through the test file category identifier and the data field format, the test file content is stored into the data storage table according to the data mapping relationship of the data mapping table, and the method comprises the following steps:

[0017] The test file and the test file category identifier are acquired from the test upload interface;

[0018] If the test file is a single test file, the test file is traversed, the data field format corresponding to the test file is acquired, the data mapping table is compared, the mapping relationship corresponding to the test file is acquired, and the test data included in the test file is stored into the data storage table through the mapping relationship corresponding to the test file;

[0019] If the test file is at least two test files, the at least two test files are traversed, the data field format corresponding to the test file is acquired, the data mapping table is compared, the mapping relationship corresponding to the test file is acquired, and the test data included in the test file is stored into the data storage table through the mapping relationship corresponding to the test file;

[0020] If the test file is a test data file to be uploaded, the test unique identifier and the test data file category name are acquired, the data mapping table is compared, the mapping relationship corresponding to the test file is acquired, and the test data included in the test file is stored into the data storage table through the mapping relationship corresponding to the test file;

[0021] If the test file is a compressed package file, the compressed package file is decompressed by a decompression function, all files obtained by decompression are traversed, the test file is obtained, the field mapping relationship of the test file is obtained, the data mapping table is compared, the mapping relationship corresponding to the test file is obtained, and the test data included in the test file is stored in the data storage table through the mapping relationship corresponding to the test file.

[0022] Further, the test identifier and the test file category identifier are obtained, the target data is obtained from the data storage table by comparing the test identifier and the test file category identifier with the data storage table, and the target data is obtained from the data storage table by comparing the test identifier and the test file category identifier with the data storage table.

[0023] The test file category identifier and the test identifier are obtained from the data acquisition interface;

[0024] The mapping relationship is obtained by comparing the test file category identifier and the test identifier obtained from the data acquisition interface with the data storage table.

[0025] The target data is obtained through the obtained mapping relationship.

[0026] Further, the test template file is used to determine the test type, the test content and the test format.

[0027] Further, the data field format includes the number of data fields, the name of each data field format and the name of the data field to be uploaded file category; the test file includes a structured data test file, an unstructured data test file and a test file compression package; and the test file category identifier is used to determine the test template category.

[0028] Further, the data mapping relationship includes:

[0029] The data mapping table is matched according to the test template file data field format and the data field content;

[0030] The data storage table is matched according to the test file category identifier and the test file field format;

[0031] The data storage table is matched according to the test data file name to be uploaded and the test identifier.

[0032] Further, the data mapping table is used to store the test template, the test field name and the test field unit.

[0033] Further, the test data file to be uploaded includes a video file, a picture file, an indefinite format file, a text file and a table file; and the data storage table is used to store matrix data, two-dimensional data and test data.

[0034] Further, the test identifier is used to determine a string of the test that does not contain Chinese characters.

[0035] The present application has the following advantages:

[0036] The present application provides a data source format indefinite data analysis and processing operation method, which can realize efficient data processing, storage, uploading and use for data source with variable data field name, variable field number and variable field order, greatly improving data access efficiency. The data storage table and data mapping table are designed according to the format of data source field content. By comparing the database field format and the indefinite data source format, the data format is stored in the data mapping table to realize data mapping relationship storage. When using data, the data in the data storage table is found through the mapping relationship in the data mapping table to complete data use. This method can be used for data processing, storage and use of data source format variable, greatly improving the data access mode and the data access efficiency, and can be widely applied to various test scenes. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art and the advantages thereof, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0038] Figure 1 A flow chart of a data source format indefinite data analysis and processing operation method according to the present application;

[0039] Figure 2 A template storage flow chart according to the present application;

[0040] Figure 3 A test storage flow chart according to the present application;

[0041] Figure 4 A test use flow chart according to the present application;

[0042] Figure 5 Another flow chart of a data source format indefinite data analysis and processing operation method according to the present application. DETAILED DESCRIPTION

[0043] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined object of the application, the specific implementation, structure, features and effects of the technical solutions proposed according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0045] The present application provides a data source format indefinite data parsing and processing operation method, which comprises the following steps:

[0046] Obtain the test template file, determine the data field name and data field format of the test data included in the test template file according to the test template file, parse the data field name and data field format to obtain the data format and data mapping relationship, and store the data format and data mapping relationship to the data mapping table.

[0047] Obtain the test file, obtain the test file category identifier and data field format according to the test file, compare the data mapping table through the test file category identifier and data field format, and store the test file content to the data storage table according to the data mapping relationship of the data mapping table.

[0048] Obtain the test identifier and test file category identifier, compare the data storage table through the test identifier and test file category identifier, and obtain the target data from the data storage table.

[0049] The above steps are described in detail as follows:

[0050] Reference Figure 1 , shows the flow of some embodiments of a data source format indefinite data parsing and processing operation method according to the present application. The data source format indefinite data parsing and processing operation method comprises the following steps:

[0051] Step S1, obtain the test template file, determine the data field name and data field format of the test data included in the test template file according to the test template file, parse the data field name and data field format to obtain the data format and data mapping relationship, and store the data format and data mapping relationship to the data mapping table.

[0052] In some embodiments, the test template file can be acquired, the data field name and data field format of the test data included in the test template file are determined according to the test template file, the data field name and data field format are parsed to obtain the data format and data mapping relationship, and the data format and data mapping relationship are stored in the data mapping table.

[0053] The test template file is used to determine the test type, test content and test format. Before uploading the test file, the test template file containing the test file format needs to be parsed to obtain the format of the test file and upload it to the database because the test file format is unknown. The test file format can represent the content of the test file, such as the number of rows and columns, whether there is a space, whether it is a DATE type or an INT type, the number of DATEs, and the number of INTs. The data mapping table is used to store the test template, test field name and test field unit. The test field unit can be a common physical unit that may be contained in the test data and involves mathematical operations. The test template table, test name table and test unit table in the data mapping table are used to store the test format. The test field unit is used for unit conversion. The test template file records the test type, test content and test format of the test.

[0054] As shown in Figure 2 , the present step can include the following steps:

[0055] Firstly, the test template file is acquired from the template upload interface.

[0056] Secondly, the data field name and data field format of the test data are acquired by traversing the test template file.

[0057] Thirdly, the data field name and data field format of the test data are parsed to obtain the data format and data mapping relationship.

[0058] Fourthly, the database field format is acquired, matched with the obtained data format and data mapping relationship, and the matching result is saved.

[0059] Fifthly, the data format and data mapping relationship are stored in the data mapping table through the matching result.

[0060] Step S2, the test file is acquired, the test file category identifier and data field format are acquired according to the test file, the data mapping table is compared through the test file category identifier and data field format, the test file content is stored in the data storage table according to the data mapping relationship of the data mapping table.

[0061] In some embodiments, the test file can be acquired, the test file category identifier and the data field format can be acquired according to the test file, the data mapping table can be compared through the test file category identifier and the data field format, and the test file content can be stored to the data storage table according to the data mapping relationship of the data mapping table.

[0062] The data field format includes the number of data fields, the name of each format of the data field, and the name of the category of the file to be uploaded, which are used to compare the format of the test in the test template table and the field name of the file to be uploaded in the test data table. The test file includes a structured data test file, an unstructured data test file, and a test file compression package. The test file is used to store test structured data and test unstructured data. The test file category identifier is used to determine the test template category. The data mapping relationship includes matching the data mapping table according to the test template file data field format and the data field content, matching the data storage table according to the test file category identifier and the test file field format, matching the data storage table according to the test data file name to be uploaded and the test identifier, forming a test format according to the test template file data field format and the data field content, comparing the test format and the database field format, and storing the test format to the test template table, the test name table, and the test unit table; matching the test data table, the matrix data table, and the two-dimensional data table according to the test file category identifier and the test file field format; and matching the matrix data table and the two-dimensional data table according to the test data file name to be uploaded and the test identifier. The data storage table is used to store matrix data, two-dimensional data, and test data. The test data table, the matrix data table, and the two-dimensional data table in the data storage table are used to store test data, test matrix data, and test two-dimensional data. The test file category identifier is used to quickly lock the format to which the test belongs.

[0063] As shown in Figure 3 , the present step can include the following steps:

[0064] Firstly, the test file and the test file category identifier are acquired from the test upload interface.

[0065] Secondly, if the test file is a single test file, the test file is traversed, the data field format corresponding to the test file is acquired, the data mapping table is compared, the mapping relationship corresponding to the test file is acquired, and the test data included in the test file is stored to the data storage table through the mapping relationship corresponding to the test file.

[0066] Thirdly, if the test file is at least two test files, the at least two test files are traversed, the data field format corresponding to the test file is acquired, the data mapping table is compared, the mapping relationship corresponding to the test file is acquired, and the test data included in the test file is stored to the data storage table through the mapping relationship corresponding to the test file.

[0067] In the fourth step, if the test file is a test data file to be uploaded, a test unique identifier and a test data file category name are obtained, a data mapping table is compared, a mapping relationship corresponding to the test file is obtained, and test data included in the test file is stored in a data storage table through the mapping relationship corresponding to the test file.

[0068] The test data to be uploaded includes a video file, a picture file, an indefinite format file, a text file and a table file, and test data can be quickly accessed by comparing the file category name and the test identifier.

[0069] In the fifth step, if the test file is a compressed package file, the compressed package file is decompressed through a decompression function, all files obtained by decompression are traversed, a test file is obtained, a test file field mapping relationship is obtained, a data mapping table is compared, a mapping relationship corresponding to the test file is obtained, and test data included in the test file is stored in a data storage table through the mapping relationship corresponding to the test file.

[0070] The compressed package file contains five folders, which are a picture folder, a video folder, an indefinite format folder, a table folder and a text folder. The table folder contains a test file and a test data file, and the text folder contains a test matrix file and a test two-dimensional data file.

[0071] In step S3, a test identifier and a test file category identifier are obtained, a data storage table is compared through the test identifier and the test file category identifier, and target data is obtained from the data storage table.

[0072] In some embodiments, a test identifier and a test file category identifier can be obtained, a data storage table can be compared through the test identifier and the test file category identifier, and target data can be obtained from the data storage table.

[0073] The test identifier is used to determine a string of the test that does not contain Chinese characters. The target data can be data that needs to be obtained when using data.

[0074] As shown in FIG. 1, the present step can include the following steps: Figure 4

[0075] In the first step, a test file category identifier and a test identifier are obtained from a data acquisition interface.

[0076] In the second step, a mapping relationship is obtained by comparing a data storage table through the test file category identifier and the test identifier obtained from the data acquisition interface.

[0077] In the third step, target data is obtained through the obtained mapping relationship.

[0078] ​The data mapping table is formed by storing the test template file with the test format into the database, data source format indefinite data analysis and processing operation are realized, and test access mode is effectively improved.

[0079] In conclusion, the application can include: database design, data storage table, data mapping table. Data storage, store data content and data mapping relationship. Data use, obtain data mapping relationship according to input data, and obtain target data in the data storage table through the mapping relationship.

[0080] Another flowchart of the application can be as shown in Figure 5 The specific steps can include the following steps:

[0081] Firstly, the test template file is obtained from the template upload interface.

[0082] Secondly, the test format is stored into the data mapping table.

[0083] Thirdly, the test file and test file category identifier are obtained from the test upload interface.

[0084] Fourthly, the test file format is obtained and compared with the data mapping table.

[0085] Fifthly, the test data is stored into the data storage table.

[0086] Sixthly, the data storage table is compared through the test file category identifier and test identifier.

[0087] Seventhly, the target data is obtained through the mapping relationship.

[0088] The application provides a good solution to the current situation of data source format indefiniteness, greatly facilitates user test data access, and can be applied to various test scenes.

[0089] The above embodiments are only used to illustrate the technical solutions of the application, rather than limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application, and should be included in the protection scope of the application.

Claims

1. A method for data parsing and processing operation based on data source format uncertainty, characterized in that, The method comprises the following steps: obtaining a test template file, determining data field names and data field formats of test data included in the test template file according to the test template file, parsing the data field names and the data field formats to obtain data formats and data mapping relationships, and storing the data formats and the data mapping relationships into a data mapping table; obtaining a test file, obtaining a test file category identifier and a data field format according to the test file, comparing the test file category identifier and the data field format with the data mapping table, storing test file content into a data storage table according to data mapping relationships of the data mapping table; obtaining a test identifier and a test file category identifier, comparing the test identifier and the test file category identifier with the data storage table, and obtaining target data from the data storage table; The method of obtaining a test template file, determining data field names and data field formats of test data included in the test template file according to the test template file, parsing the data field names and the data field formats to obtain data formats and data mapping relationships, and storing the data formats and the data mapping relationships into a data mapping table comprises: obtaining the test template file from a template upload interface; traversing the test template file to obtain data field names and data field formats of test data; parsing the data field names and the data field formats of the test data to obtain data formats and data mapping relationships; obtaining database field formats, matching the obtained data formats and data mapping relationships, and saving a matching result; storing the data formats and the data mapping relationships into the data mapping table through the matching result; The method of obtaining a test file, obtaining a test file category identifier and a data field format according to the test file, comparing the test file category identifier and the data field format with the data mapping table, and storing test file content into a data storage table according to data mapping relationships of the data mapping table comprises: obtaining the test file and the test file category identifier from a test upload interface; if the test file is a single test file, traversing the test file to obtain a data field format corresponding to the test file, comparing the data field format with the data mapping table, obtaining a mapping relationship corresponding to the test file, and storing test data included in the test file into the data storage table through the mapping relationship corresponding to the test file; if the test file is at least two test files, traversing the at least two test files to obtain data field formats corresponding to the test files, comparing the data field formats with the data mapping table, obtaining mapping relationships corresponding to the test files, and storing test data included in the test files into the data storage table through the mapping relationships corresponding to the test files; if the test file is a test data file to be uploaded, obtaining a test unique identifier and a test data file category name, comparing the test unique identifier and the test data file category name with the data mapping table, obtaining a mapping relationship corresponding to the test file, and storing test data included in the test file into the data storage table through the mapping relationship corresponding to the test file. If the test file is a compressed package file, the compressed package file is decompressed by a decompression function, all files obtained by decompression are traversed, the test file is obtained, a field mapping relationship of the test file is obtained, a data mapping table is compared, a mapping relationship corresponding to the test file is obtained, and test data included in the test file are stored in a data storage table through the mapping relationship corresponding to the test file.

2. The method according to claim 1, wherein, The test identifier and the test file category identifier are obtained, target data is obtained from the data storage table by comparing the test identifier and the test file category identifier with the data storage table, and the target data is obtained from the data storage table. The test file category identifier and the test identifier are obtained from the data acquisition interface. The mapping relationship is obtained by comparing the test file category identifier and the test identifier obtained from the data acquisition interface with the data storage table. The target data is obtained through the obtained mapping relationship.

3. The method of claim 1, wherein the data source format is not fixed. The test template file is used to determine a test type, test content and test format.

4. The method of claim 1, wherein the data source format is not fixed. The data field format includes a data field number, a data field each format name and a data field to-be-uploaded file category name; the test file includes a structured data test file, an unstructured data test file and a test file compressed package; and the test file category identifier is used to determine a test template category.

5. The method of claim 1, wherein the data source format is not fixed. The data mapping relationship includes: The data mapping table is matched according to the test template file data field format and the data field content; The data storage table is matched according to the test file category identifier and the test file field format; The data storage table is matched according to the to-be-uploaded test data file name and the test identifier.

6. The method of claim 1, wherein the data source format is not fixed. The data mapping table is used to store a test template, a test field name and a test field unit.

7. The method of claim 1, wherein the data source format is not fixed. The to-be-uploaded test data file includes a video file, a picture file, an indefinite format file, a text file and a table file; and the data storage table is used to store matrix data, two-dimensional data and test data.

8. The method of claim 2, wherein the data source format is not fixed. The test identifier is used to determine a string of the test that does not contain Chinese characters.

Images