Method and system for processing multi-source heterogeneous data report of complex product

Abstract

The invention provides a processing method and system for a multi-source heterogeneous data report of a complex product. The method comprises the steps that a data analysis template of a configuration file is set according to the content of a test file of the product; reading the content of the test file line by line according to the data analysis template, and storing the content into a memory; using an SHA-256 algorithm to calculate and read hash values of file contents in the memory, comparing the hash values with all hash values in a storage file table in a database, if the same hash values do not exist, reading all tables of the test file based on the data analysis template, and storing the tables in a plurality of lists; and storing the content of the first table of the different types of tables into a new list by taking the column number and the column header content of the tables as the division basis of the table types, and storing the list into a database table according to the data file storage information of the data analysis template. According to the method, data format conversion is realized, unified structured storage of the data is facilitated, and support is provided for unified analysis and processing of the data.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method and system for processing multi-source heterogeneous data reports of complex products. Background Art

[0002] In the context of the rapid integration of modern industry and information technology, data management for complex products has become a critical component of enterprise operational decision-making. Complex products, such as high-end manufacturing equipment, large-scale system engineering, and intelligent connected products, often involve massive amounts of heterogeneous data from multiple sources. This data originates from diverse business systems, sensor networks, user feedback, third-party data sources, and other sources, and features diverse formats and complex structures. Effectively integrating, processing, and analyzing this multi-source, heterogeneous data is crucial for improving product design, optimizing production processes, predicting maintenance failures, and enhancing user experience.

[0003] Traditional data processing methods often struggle to cope with the multi-source, heterogeneous data of complex products. On the one hand, inconsistent data formats and a lack of data standards complicate data integration, making it difficult to form a comprehensive and accurate data view. On the other hand, data processing efficiency issues limit the immediate mining and utilization of data value, hindering the ability to respond quickly to decisions. Furthermore, data quality and consistency are pain points that traditional methods struggle to effectively address. Erroneous or inconsistent data can lead to misleading analytical results, negatively impacting business operations.

[0004] To address these challenges, the industry has begun exploring and adopting a series of advanced report processing methods and technical frameworks, aiming to achieve efficient integration and intelligent processing of multi-source, heterogeneous data. These methods typically include automated table creation and standardized processing, ensuring efficient processing of massive amounts of data.

[0005] To sum up, the multi-source heterogeneous data report processing method for complex products was developed in the context of responding to the challenges of data complexity, improving data value mining capabilities, and supporting intelligent decision-making of enterprises, providing enterprises with powerful data management and analysis support. Summary of the Invention

[0006] The present invention proposes a method and system for processing multi-source heterogeneous data reports of complex products, aiming to achieve efficient integration and intelligent processing of multi-source heterogeneous test data files.

[0007] The technical solutions for achieving the purpose of the present invention are:

[0008] A method for processing multi-source heterogeneous data reports for complex products, including:

[0009] Step 1: Set the data parsing template of the configuration file according to the content of the product test file; the data parsing template includes a table start marker, a table end marker, a table content separator, a key information identification formula, and data file storage information. The data file storage information is set according to the table content in the test data file;

[0010] Step 2: Read the test file content line by line according to the data parsing template and store it in memory;

[0011] Step 3: Use the SHA-256 algorithm to calculate the hash value of the file content read into the memory, and compare the hash value with all the hash values in the file table stored in the database. If there is no identical hash value, proceed to step 4.

[0012] Step 4: Based on the data parsing template, read all the tables in the test file and store them in lists A1, A2, A3, ..., An, where n is the number of tables in the test file;

[0013] Step 5: For lists A1, A2, A3, ..., An, the table types are classified based on the number of columns and the column header content. The contents of the first table of different types are stored in lists B1, B2, B3, ..., Bm. Based on the data file storage information of the data parsing template, the lists are stored in the database table, where m is the number of table types, m≤n.

[0014] Furthermore, the key information identification formula consists of keywords and regular expressions. The keywords are used to locate the position of the key information, and the regular expressions are used to match the key information. The key information of different products corresponds to different regular expressions. The line number where the keyword is located is located by searching the file line by line, and the content in the line where the keyword is located is matched using regular expressions to extract the key information.

[0015] Furthermore, step 2 specifically includes: extracting key information from the test file content according to the key information identification formula in the data parsing template, searching the entire file for content identical to the keyword, recording the line number containing the keyword, and storing the relevant content after each line of the keyword in the Key list, using the regular expression in the key information identification formula to match the information in each item in the Key list. If multiple information is matched, only the first information is used as the key information of the test file. If no match is found, the set default number is used instead.

[0016] Furthermore, the hash value in step 3 is:

[0017] file_hash=SHA-256(STR_File)

[0018] Where STR_File is the test file content, which is a byte array of arbitrary length. SHA-256 is the hash calculation function. file_hash is a positive integer in the range [0, 256), which is the hash value of STR_File.

[0019] Furthermore, step 4 specifically includes: identifying all tables in the test file based on the table start marker and the table end marker, determining the start position and end position of the table, identifying all tables in the test file, storing the start position and end position of the i-th table in the list Ai, dividing the table content by row according to the table content separator, and storing it in the list Ai. For the N test tables in the test file, there are corresponding lists A1, A2, A3, ..., An.

[0020] Furthermore, the step 5 specifically includes:

[0021] Step 5.1: Visualize tables of types B1, B2, B3, ..., and Bm in sequence, store specific columns in the database as needed, and set the names and types of the corresponding fields in the database.

[0022] Step 5.2: Convert the data file entry information DX of the test data file and the entry information D1, D2, D3, ..., Dp of different types of test files in the database into strings and sort them in ASCII order. The sorted strings are DX_new and D1_new, D2_new, D3_new, ..., Dp_new, where p is the number of all different file types contained in the database table.

[0023] In step 5.3, the string DX_new is compared with D1_new, D2_new, D3_new, ..., Dp_new one by one. If there is no string DY_new that is exactly the same as DX_new, and Y is 1, 2, ..., p, a new database table is created according to the set database field information, and all table information of the file is stored in the newly created data table according to the table type. Otherwise, a new data table is not created, and all table information of the file is stored in the matching DY type database table according to the table type.

[0024] A system for processing multi-source heterogeneous data reports for complex products, including:

[0025] The data parsing template design unit sets the data parsing template of the configuration file according to the content of the product test file; the data parsing template includes a table start marker, a table end marker, a table content separator, a key information identification formula, and data file storage information. The data file storage information is set according to the table content in the test data file;

[0026] The reading unit reads the test file content line by line according to the data parsing template and stores it into the memory;

[0027] The parsing unit uses the SHA-256 algorithm to calculate the hash value of the file content read into the memory, compares the hash value with all the hash values in the file table stored in the database, and if there is no identical hash value, reads all the tables of the test file based on the data parsing template and stores them in lists A1, A2, A3, ..., An, where n is the number of tables in the test file;

[0028] For the warehousing unit, for lists A1, A2, A3, ..., An, the table types are divided based on the number of columns and the column header content. The contents of the first table of different types are stored in lists B1, B2, B3, ..., Bm. Based on the data file warehousing information of the data parsing template, the lists are stored in the database table. m is the number of table types, m≤n.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes to establish a data conversion model by summarizing and analyzing multi-source heterogeneous test data of complex products. The model provides a conversion template for establishing a data source (test data to be analyzed) and a target data (database storage format). The template can perform unified data conversion, data filtering and cleaning on various types of engineering data, extract data from the data source, and finally save the results as target data, realize data format conversion, facilitate unified structured data storage, and provide support for unified data analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart for multi-source heterogeneous data report processing.

[0031] Figure 2 Configure an example graph for parsing templates. DETAILED DESCRIPTION

[0032] The testing process of complex products involves multiple steps, and each step will generate a corresponding test file. The test file often contains two parts: product key information and product test information. Product key information generally includes information such as product number and product test type, while product test information generally includes test information of each unit of the product, such as the input voltage and output voltage of the unit. Since complex products are often composed of hundreds or even thousands of units, and each unit will generate multiple test information during the testing process, each test process of a complex product will generate tens of thousands or even hundreds of thousands of test data. At the same time, different types of complex products often have large differences in the unit composition, so the test information generated each time is also different. The present invention proposes a multi-source heterogeneous test data processing method to realize the collection and management of test data of different complex products.

[0033] Combine Figure 1 A method for processing a complex product multi-source heterogeneous data report includes the following steps:

[0034] Step 1: Data parsing template configuration. The data parsing template includes table identification markers, table content separators, key information identification formulas, and data file storage information settings. According to the table definition in the comprehensive test data file content, set the table start marker (table_start), table end marker (table_end), and table content separator (table_split); according to the definition of the table key information, set the corresponding key information identification formula. The key information identification formula consists of keywords (key_word) and regular expressions. The keyword is responsible for locating the key information position, and the regular expression is responsible for matching the key information. According to the key information needs of different products, set the corresponding regular expression. By searching the entire file line by line to locate the line number where the keyword is located, use regular expressions to match the content in the line where the keyword is located to extract the key information; set the data file storage information according to the table content in the test data file. The data file storage information is responsible for storing the corresponding column information of different tables in the test data file into the data. A complex product TXT type parsing template such as Figure 2 shown.

[0035] Step 2: Read the file line by line, read the original content in the data file into the memory, extract the key information of the test file content according to the key information identification formula in the configuration file, search the entire file for content identical to the keyword, and record the line number containing the keyword, and store the relevant content after each line of the keyword in the Key list, use the regular expression in the key information identification formula to match the information in each item of the Key list, if multiple information are matched, only the first information is used as the key information of the test file, if no match is found, then the default number is used instead.

[0036] Step 3: Use the SHA-256 algorithm to calculate the hash value file_hash of the file content read into the memory. Compare the hash value with all hash values in the file table stored in the database. If the hash values are the same, the subsequent file import is not performed. If the hash values are not the same, subsequent data processing is performed according to the configuration.

[0037] file_hash=SHA-256(STR_File)

[0038] STR_File is the test data file content, which is a byte array of arbitrary length. SHA-256 is the hash calculation function. file_hash is a positive integer in the range [0,256), which is the hash value of STR_File.

[0039] Step 4: Read the relevant content information from the configuration file. Identify all tables in the test file based on the table start and end markers, and determine the table's start and end positions. After identifying all tables in the test data file, store the start and end positions of each individual table in list A1. The table contents are divided into rows based on the table delimiter and stored in list A1. For N test tables in the test data file, this corresponds to lists A1, A2, A3, ..., and An (n is the number of tables in the test file).

[0040] Step 5: For lists A1, A2, A3...An, the number of columns and column header contents of the table are used as the basis for dividing the table types, and the contents of the first table of different types are stored in lists B1, B2, B3...Bm (m is the number of table types, m≤n). Tables of types B1, B2, B3...Bm are presented on the page in a visual manner in turn. The user chooses to set the database to store specific columns as needed, and sets the name and type of the corresponding field in the database. After the setting is completed, the data file storage information DX of the test data file is compared with the storage configuration information of each test file D1, D2, D3...Dp (p is the number of all different file types contained in the file type database table) in the file type data table.

[0041] Step 5.1: Convert the data file entry information DX and the data file entry information D1, D2, D3...Dp in the database into strings and sort them in ASCII order. The sorted strings are DX_new and D1_new, D2_new, D3_new...Dp_new (p is the number of all different file types contained in the file type database table).

[0042] Step 5.2: Compare the string DX_new with D1_new, D2_new, D3_new...Dp_new one by one. If there is no string DY_new (Y is one of 1, 2...p) that is exactly the same as DX_new, then create a new database table based on the database field information set by the user, and store all the table information of the file in the newly created data table according to the table type. Otherwise, do not create a new data table, and store all the table information of the file in the DY (Y is one of 1, 2...p) type database table according to the table type.

[0043] Through step 5, the data file content is written into the corresponding data table according to the data file storage information setting, and the storage information of the test data file is updated.

[0044] This embodiment also provides a system for processing complex product multi-source heterogeneous data reports, including:

[0045] The data parsing template design unit sets the data parsing template of the configuration file according to the content of the product test file; the data parsing template includes a table start marker, a table end marker, a table content separator, a key information identification formula, and data file storage information. The data file storage information is set according to the table content in the test data file;

[0046] The reading unit reads the test file content line by line according to the data parsing template and stores it into the memory;

[0047] The parsing unit uses the SHA-256 algorithm to calculate the hash value of the file content read into the memory, compares the hash value with all the hash values in the file table stored in the database, and if there is no identical hash value, reads all the tables of the test file based on the data parsing template and stores them in lists A1, A2, A3, ..., An, where n is the number of tables in the test file;

[0048] For the warehousing unit, for lists A1, A2, A3, ..., An, the table types are divided based on the number of columns and the column header content. The contents of the first table of different types are stored in lists B1, B2, B3, ..., Bm. Based on the data file warehousing information of the data parsing template, the lists are stored in the database table. m is the number of table types, m≤n.

[0049] This embodiment also provides a computer storage medium, which stores an executable program. The executable program is executed by a processor to implement the steps of the method for processing complex product multi-source heterogeneous data reports.

[0050] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0051] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for processing multi-source heterogeneous data reports of complex products, characterized by: include: Step 1: Set the data parsing template of the configuration file according to the test file content of the product; The data parsing template includes a table start marker, a table end marker, a table content separator, a key information identification formula, and data file storage information. The data file storage information is set according to the table content in the test data file; Step 2: Read the test file content line by line according to the data parsing template and store it in memory; Step 3: Use the SHA-256 algorithm to calculate the hash value of the file content read into the memory, and compare the hash value with all the hash values in the file table stored in the database. If there is no identical hash value, proceed to step 4. Step 4: Based on the data parsing template, read all the tables in the test file and store them in lists A1, A2, A3, ..., An, where n is the number of tables in the test file; Step 5: For lists A1, A2, A3, ..., An, the table types are classified based on the number of columns and the column header content. The contents of the first table of different types are stored in lists B1, B2, B3, ..., Bm. Based on the data file storage information of the data parsing template, the lists are stored in the database table, where m is the number of table types, m≤n.

2. A method for processing complex product multi-source heterogeneous data reports according to claim 1, characterized in that: The key information identification formula consists of keywords and regular expressions. Keywords are used to locate the position of key information, and regular expressions are used to match key information. The key information of different products corresponds to different regular expressions. The line number where the keyword is located is located by searching the file line by line, and the content in the line where the keyword is located is matched using regular expressions to extract key information.

3. The method for processing complex product multi-source heterogeneous data reports according to claim 2, characterized in that: The step 2 specifically includes: extracting key information from the test file content according to the key information identification formula in the data parsing template, searching the entire file for content identical to the keyword, recording the line number containing the keyword, and storing the relevant content after each line of the keyword in the Key list, using the regular expression in the key information identification formula to match the information in each item of the Key list. If multiple information are matched, only the first information is used as the key information of the test file. If no match is found, the set default number is used instead.

4. The method for processing complex product multi-source heterogeneous data reports according to claim 2, characterized in that: The hash value in step 3 is: file_hash=SHA-256(STR_File) Where STR_File is the test file content, which is a byte array of arbitrary length. SHA-256 is the hash calculation function. file_hash is a positive integer in the range [0, 256), which is the hash value of STR_File.

5. The method for processing complex product multi-source heterogeneous data reports according to claim 2, characterized in that: Step 4 specifically includes: identifying all tables in the test file based on the table start marker and table end marker, determining the start and end positions of the table, identifying all tables in the test file, storing the start and end positions of the i-th table in list Ai, dividing the table content by row according to the table content delimiter, and storing them in list Ai. For the N test tables in the test file, there are corresponding lists A1, A2, A3, ..., An.

6. The method for processing complex product multi-source heterogeneous data reports according to claim 2, characterized in that: The step 5 specifically includes: Step 5.1: Visualize tables of types B1, B2, B3, ..., and Bm in sequence, store specific columns in the database as needed, and set the names and types of the corresponding fields in the database. Step 5.2: Convert the data file entry information DX of the test data file and the entry information D1, D2, D3, ..., Dp of different types of test files in the database into strings and sort them in ASCII order. The sorted strings are DX_new and D1_new, D2_new, D3_new, ..., Dp_new, where p is the number of all different file types contained in the database table. In step 5.3, the string DX_new is compared with D1_new, D2_new, D3_new, ..., Dp_new one by one. If there is no string DY_new that is exactly the same as DX_new, and Y is 1, 2, ..., p, a new database table is created according to the set database field information, and all table information of the file is stored in the newly created data table according to the table type. Otherwise, a new data table is not created, and all table information of the file is stored in the matching DY type database table according to the table type.

7. A system for processing complex product multi-source heterogeneous data reports that implements the method of any one of claims 1 to 6, characterized in that: include: Data parsing template design unit, which sets the data parsing template of the configuration file according to the content of the product's test file; The data parsing template includes a table start marker, a table end marker, a table content separator, a key information identification formula, and data file storage information. The data file storage information is set according to the table content in the test data file; The reading unit reads the test file content line by line according to the data parsing template and stores it into the memory; The parsing unit uses the SHA-256 algorithm to calculate the hash value of the file content read into the memory, compares the hash value with all the hash values in the file table stored in the database, and if there is no identical hash value, reads all the tables of the test file based on the data parsing template and stores them in lists A1, A2, A3, ..., An, where n is the number of tables in the test file; For the warehousing unit, for lists A1, A2, A3, ..., An, the table types are divided based on the number of columns and the column header content. The contents of the first table of different types are stored in lists B1, B2, B3, ..., Bm. Based on the data file warehousing information of the data parsing template, the lists are stored in the database table. m is the number of table types, m≤n.

8. A computer storage medium, characterized in that The computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the method for processing complex product multi-source heterogeneous data reports as described in any one of claims 1 to 6.

Images