A method and apparatus for parsing multi-source heterogeneous test data of a server.

Abstract

This application provides a method and apparatus for parsing multi-source heterogeneous test data of a server. The method includes: receiving a byte stream of multi-source heterogeneous test data from different components of the server under test; identifying explicit encoding declaration bits of the byte stream, matching the corresponding first encoding method, and if the matching fails, predicting the first encoding method based on the byte features of the byte stream; parsing the byte stream based on the first encoding method and outputting a string in a first format; cleaning the string based on the character features of the string in the first format to obtain clean text; constructing a structured data storage template; calling multiple parsing tool modules to sequentially parse the clean text, identifying the field values ​​corresponding to the fields in the data storage template, filling the data storage template, and obtaining the parsing result of the multi-source heterogeneous test data. The method and apparatus can realize automated and standardized parsing of multi-source heterogeneous server test data across vendors, versions, and formats.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a method and apparatus for parsing multi-source heterogeneous test data of servers. Background Technology

[0002] As core equipment in data center and cloud computing infrastructure, the performance and stability of servers largely depend on the configuration and operational status of the BIOS (Basic Input / Output System) and BMC (Baseboard Management Controller) firmware. During server research, development, testing, and maintenance, engineers frequently need to collect and analyze test data from the BIOS and BMC, such as BIOS hardware configuration parameters, sensor data monitored by the BMC (including temperature, voltage, fan speed, power consumption, etc.), and hardware health status information, to verify the server's reliability and stability under different environmental and version conditions.

[0003] Currently, the industry typically relies on vendor-provided tools or self-written parsing scripts to process server test data. For well-structured JSON data, standard JSON parsing libraries (such as json.loads) are commonly used; for key-value text, string splitting methods based on delimiters (such as colons and equal signs) are frequently employed; and for tabular text, regular expressions or simple row and column splitting techniques are often used for information extraction. These methods can achieve basic data parsing functions when dealing with data of a single, known format.

[0004] However, due to the existence of numerous devices from different manufacturers, models, and firmware versions in the server industry, the BIOS and BMC test data output by these devices exhibit high heterogeneity in format, structure, and semantics. First, the data format is heterogeneous; test data may exist in various forms such as JSON, key-value pairs, and tabular text, lacking a unified, adaptive parsing framework. This necessitates writing and maintaining different parsing logic for different formats, resulting in low automation and fragile processes. Second, the data semantics are heterogeneous; different manufacturers use inconsistent naming conventions for fields representing the same physical quantity or state, employ mixed units, and represent abnormal states differently. This makes direct comparison of cross-vendor and cross-version data impossible, heavily relying on manual identification and normalization, which is inefficient and prone to errors. Furthermore, existing data processing workflows lack robustness to encoding differences and noise from control characters, and the data comparison process is usually based on line-by-line comparison of the original text, failing to understand the semantics and structure of the data. This leads to numerous irrelevant format differences being falsely reported, while critical changes in business values ​​may be missed. Summary of the Invention

[0005] In view of this, this application provides a method and apparatus for parsing multi-source heterogeneous server test data, which can solve the problem of automated and standardized parsing of multi-source heterogeneous server test data across vendors, versions, and formats.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for parsing multi-source heterogeneous test data of a server, the method comprising:

[0008] Receive byte streams of heterogeneous test data from different components of the server under test;

[0009] Identify the explicit encoding declaration bits of the byte stream, match the corresponding first encoding method, and if the match fails, predict the first encoding method based on the byte features of the byte stream;

[0010] The byte stream is parsed based on the first encoding method, and the output is a string in the first format;

[0011] The string is cleaned based on the character features of the first format string to obtain clean text;

[0012] Construct a structured data storage template;

[0013] Multiple parsing tool modules are invoked to sequentially parse the clean text, identify the field values ​​corresponding to the fields in the data storage template, fill the data storage template, and obtain the parsing results of the multi-source heterogeneous test data.

[0014] A second aspect of this application provides a device for parsing multi-source heterogeneous test data of a server, the device comprising a receiving module, an identification module, a parsing module, a cleaning module, a construction module, and a processing module;

[0015] The receiving module is used to receive byte streams of multi-source heterogeneous test data from different components of the server under test;

[0016] The identification module is used to identify the explicit encoding declaration bits of the byte stream, match the corresponding first encoding method, and if the matching fails, predict the first encoding method based on the byte features of the byte stream.

[0017] The parsing module is used to parse the byte stream based on the first encoding method and output a string in the first format;

[0018] The cleaning module is used to clean the string based on the character features of the string in the first format to obtain clean text;

[0019] The building module is used to build structured data storage templates;

[0020] The processing module is used to call multiple parsing tool modules to sequentially parse the clean text, identify the field values ​​corresponding to the fields in the data storage template, fill the data storage template, and obtain the parsing results of the multi-source heterogeneous test data.

[0021] The method and apparatus for parsing heterogeneous test data from multiple sources provided in this application identify the key data types and match the most accurate parsing method for data from different sources. This automatically and accurately extracts key information from the heterogeneous data. In particular, by constructing a structured data storage template and calling multiple parsing tool modules to sequentially parse the clean text, the field values ​​corresponding to the fields in the template are identified and filled into the template. This directly achieves semantic and structural unification of heterogeneous data from multiple sources, solving the problem that heterogeneous server test data from multiple sources cannot be uniformly and automatically parsed due to differences in encoding, format, and structure. By clearly dividing the processing into three stages—encoding recognition and decoding, data cleaning, and format parsing and template filling—unified access and standardized output of heterogeneous data are achieved. In this way, regardless of the vendor or version of the server component from which the data comes, as long as it goes through this process, it can ultimately be transformed into a structurally unified data storage template, laying the foundation for subsequent data comparison, analysis, and storage, and greatly improving the automation level and system adaptability of data processing. Specifically, by identifying the explicit encoding declaration bits of the byte stream to predict the first encoding method and uniformly decoding it into a first format string, the problem of garbled characters and parsing failures caused by inconsistent character encoding is directly solved. This ensures that raw byte streams from different operating systems, firmware, or tools can be correctly and losslessly converted into a unified character format that can be processed by subsequent processes, guaranteeing the reliability and consistency of data input from the source. Furthermore, by cleaning the strings based on character features to obtain clean text, non-business noise such as ASCII control characters and ANSI escape sequences mixed in the raw data is effectively removed, avoiding parser misjudgments or crashes that these hidden characters may cause, allowing core business data to be presented clearly and accurately. Finally, by constructing a structured data storage template and calling multiple parsing tool modules to sequentially parse and fill the template, heterogeneous raw text is mapped to a unified structured model. This achieves semantic normalization of multi-source heterogeneous data, enabling data of different formats and structures to be understood and processed by the same process, thus solving the problem of automated analysis caused by format heterogeneity. Attached Figure Description

[0022] Figure 1 A flowchart of an embodiment of the method for parsing multi-source heterogeneous test data of servers provided in this application;

[0023] Figure 2 This is a schematic diagram of the second embodiment of the device for parsing multi-source heterogeneous test data of the server provided in this application. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0025] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0027] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0028] Example 1

[0029] Figure 1 This is a flowchart of an embodiment of the method for parsing multi-source heterogeneous test data of a server provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0030] S101. Receive byte streams of multi-source heterogeneous test data from different components of the server under test.

[0031] It should be noted that the multi-source heterogeneous test data from different components of the server under test mainly originates from two firmware components in the server hardware system: the BIOS and the BMC. During server R&D, testing, and maintenance, specific query commands are sent to these components to obtain their status, configuration, and sensor information. Specifically, this data includes BIOS configuration data and BMC sensor and status data. BIOS configuration data records key parameters such as server hardware initialization, boot sequence, CPU and memory operating status, and PCIe device configuration. The BMC, as the server's remote management controller, is responsible for monitoring the real-time health status of the hardware. Therefore, BMC sensor and status data can include temperature data, such as CPU core temperature, motherboard air intake and exhaust temperatures, and hard drive temperature; voltage data, such as the input / output voltage of each power supply; fan data, such as the speed of each fan; power consumption data, such as the real-time power consumption of the entire system or individual components; hardware status data, such as the insertion and removal status of memory modules, the health status of hard drives, and the status of power supply units; in addition, the BMC may also include hardware topology and event logs, such as server node configuration and hardware logs.

[0032] It should also be noted that "multi-source" refers to data coming from servers of different manufacturers (such as Huawei, Dell, and HP), different models, different firmware versions, and even commands from different functions within the same server. Heterogeneity manifests in format heterogeneity, structural heterogeneity, and semantic heterogeneity. Specifically, format heterogeneity can include semi-structured JSON, key-value pair text, and tabular text; structural heterogeneity refers to data organization logic that is not fixed, such as being segmented by logical modules, hardware topology, or time series, or even mixed together; semantic heterogeneity refers to inconsistent naming, units, and outlier representations of fields representing the same physical quantity or state.

[0033] Furthermore, test data is received in real-time from the BIOS and BMC of the server under test via command-line tool calls and data interface acquisition. Specifically, BIOS test data can be obtained by reading BIOS configuration parameters through the firmware interaction interface during the server boot phase or through dedicated configuration tools. For example, during the server POST (Power-On Self-Test) phase, a query command is sent through the BIOS configuration tool provided by the manufacturer, or after the operating system boots, commands such as dmidecode are called through the SMBIOS (System Management BIOS) interface to obtain data such as the BIOS firmware version and hardware initialization status. The data is returned to the test terminal in byte stream format. BMC test data can be collected through command-line tools or API interfaces based on the IPMI (Intelligent Platform Management Interface) protocol or the manufacturer's custom remote management protocol. Data collection via command-line tools includes executing commands such as ipmitool sdr (read sensor data) and ipmitool fru (read hardware asset information) in the test terminal, with the BMC directly returning byte stream data containing sensor values ​​and status. Data collection via API interfaces includes, for BMCs supporting RESTful APIs, receiving byte stream data encapsulated in JSON format by sending HTTP requests.

[0034] Furthermore, it should be noted that the byte stream here refers to the raw binary data sequence returned by the aforementioned command, which has not been interpreted using any character encoding. It is the lowest-level representation of data during transmission and storage and has not yet been decoded into a human-readable string. Specifically, the byte stream here may include byte order markers, valid text content bytes, control characters and escape sequence bytes, as well as potentially encoded errors or irrelevant binary data (which may contain some meaningless binary data due to transmission errors or the characteristics of tool output).

[0035] Furthermore, it should be noted that the multi-source heterogeneous test data mainly includes semi-structured JSON data, typically originating from BMCs that support RESTful APIs. Its byte stream content is a string conforming to JSON syntax and may contain complex nested structures. Key-value pair text data, mostly output by command-line tools, is presented in the form of "key[delimiter]value," with delimiters often being colons, equal signs, or consecutive spaces. Tabular text data is commonly found in the results of system information query commands. Its byte stream content has a row-column structure, with columns separated by tabs, vertical bars, or consecutive spaces, and may include table headers. In addition to the encoded bytes of valid text characters, the raw byte streams of this data may also contain noise from non-business data such as BOM, ASCII control characters, and ANSI escape sequences.

[0036] S102. Identify the explicit encoding declaration bit of the byte stream, match the corresponding first encoding method, and if the match fails, predict the first encoding method based on the byte characteristics of the byte stream.

[0037] It should be noted that due to differences in the manufacturer, firmware version, and data transmission interface of the tested servers, the received byte stream may use different character encoding formats such as UTF-8, UTF-16LE, and GBK. If a fixed encoding is used for parsing directly, and an incorrect encoding is used for decoding, subsequent parsing will result in garbled strings, causing the entire process to fail. Therefore, before attempting any structured parsing, the byte stream must first be correctly decoded into a unified string format (preferably UTF-8).

[0038] Specifically, identifying the explicit encoding declaration bits of the byte stream and matching them with the corresponding first encoding method includes:

[0039] (1) Read a preset number of bytes from the beginning of the byte stream.

[0040] It should be noted that the preset number is determined based on the length range of common Byte Order Marks (BOMs), which are typically between 2 and 4 bytes long. Specifically, by calling the file or stream read operation, the first 4 bytes are read from the absolute beginning of the byte stream into a temporary byte array or buffer. If the total length of the byte stream is less than 4 bytes, all bytes are read.

[0041] (2) Compare the read bytes with the predefined encoding tag mapping table.

[0042] It should be noted that the predefined encoding tag mapping table contains explicit declaration bit-to-encoding type correspondences for common encodings in server test data scenarios. For example, Table 1 shows an encoding tag mapping table as illustrated in this application:

[0043] Table 1 Encoding Tag Mapping Table

[0044] Explicit encoding declaration bits (hexadecimal) Explicitly encode the declared bits (byte representation). Corresponding encoding type EF BB BF b'\xef\xbb\xbf' UTF-8 FF FE b'\xff\xfe' UTF-16LE FE FF b'\xfe\xff' UTF-16BE FF FE 00 00 b'\xff\xfe\x00\x00' UTF-16BE 00 00 FE FF b'\x00\x00\xfe\xff' UTF-32LE

[0045] Please refer to Table 1 and compare the data read in step (1) byte by byte with each explicit encoded declaration bit sequence in the above mapping table. During the comparison, if the read data is greater than or equal to the declaration bit length of a certain row in the mapping table, and the first N bytes of the read data (N is the declaration bit length) are completely consistent with the declaration bits of that row, then the comparison is considered successful; if the read data does not match the declaration bits of all rows, then the comparison is considered unsuccessful.

[0046] (3) If the comparison is consistent, it is determined that there is an explicit encoding declaration bit, and the matched encoding type is determined as the first encoding method.

[0047] It should be noted that the first encoding method refers to the character encoding scheme used to decode the current byte stream, determined by a successful BOM match. Specifically, upon a match, the matched encoding type (e.g., UTF-8) is recorded, and in practice, the identified BOM byte sequence is removed from the byte stream (because the BOM is metadata, not the text content itself). Then, the remaining byte stream after removing the BOM is decoded using the determined first encoding method, outputting a uniform string (e.g., UTF-8 format).

[0048] The above describes the case where there is an explicit encoding declaration bit and the corresponding encoding type is matched. However, there may also be cases where there is no explicit encoding declaration bit or it is not recognized, i.e., the match fails. In this case, the statistical encoding guessing process is entered.

[0049] Specifically, the first encoding method for predicting byte features based on the byte stream includes:

[0050] (1) Extract the byte features of the byte stream.

[0051] It should be noted that byte characteristics refer to quantitative statistical indicators extracted from byte stream data that reflect its encoding patterns. Specifically, the chardet or cchardet libraries in Python can be used to analyze the statistical characteristics of byte streams. Among these, the frequency of occurrence of each byte value (0x00-0xFF) in the byte stream is analyzed, and the distribution of byte values ​​differs significantly across different encodings. For example, UTF-8 encoded English characters are 1 byte (0x00-0x7F), while Chinese characters are 3 bytes (first byte 0xE0-0xEF, second byte 0x80-0xBF, last byte 0x80-0xBF), with a large number of 0x80-0xBF bytes. ASCII encoded byte values ​​range from 0x00-0x7F (with the highest bit being 0). GBK / GB2312 (Chinese) encoded Chinese characters consist of 2 bytes, with the first byte value being 0x81-0xFE and the second byte value being 0x40-0x7E or 0x80-0xFE; therefore, bytes above 0x80 account for a high proportion.

[0052] Furthermore, verify whether the byte stream conforms to the syntax rules of the specific encoding. For example, for UTF-8 compliance, check whether the multi-byte sequence conforms to the rule that the high-order byte of the first byte is 11xxxxxx and the high-order byte of subsequent bytes is 10xxxxxx; for GBK / GB2312 (Chinese) compliance, check whether the double-byte sequence conforms to the rule that the first byte is 0x81-0xFE and the second byte is 0x40-0x7E or 0x80-0xFE. Finally, statistically analyze the occurrence of characteristic characters of specific encodings. For example, for UTF-16 encoding, since 2 bytes are used to represent a character, English characters are followed by a 0x00 byte, so the 0x00 byte appears frequently; for ASCII encoding, there is no 0x00 byte (except for special control characters), and the byte values ​​of common characters (0x3A, 0x3D, 0x6F, 0x6B, 0x66, 0x61, 0x69, 0x6C) appear frequently. This operation transforms the raw, unstructured byte stream into a set of structured, quantifiable feature indicators, providing a data foundation for the next step of confidence calculation.

[0053] (2) Based on the predefined candidate encoding set, calculate the confidence score for each candidate encoding according to the byte characteristics.

[0054] The candidate encoding set is a predefined set of all possible encoding types in the application scenario of this application, such as {UTF-8, GBK, GB2312, UTF-16, ASCII, ...}. The confidence score is a value between 0 and 1, used to quantify the probability that a byte stream conforms to a specific encoding; the higher the score, the greater the probability. Furthermore, an analysis function can be defined for each encoding in the candidate encoding set. This analysis function is a detection logic designed for a specific encoding rule; its input is byte features, and its output is the confidence score. The analysis functions corresponding to different encodings differ in their specific implementations, but all follow a unified feature extraction-rule verification-score calculation paradigm. These analysis functions each rely on the core features defined by their encoding standard, and ultimately output a comparable normalized confidence value. This function receives the byte features extracted in step (1) and outputs a confidence score according to the encoding rules. For example, for UTF-8, the confidence score is mainly based on UTF-8 compliance features. If the vast majority (e.g., more than 99%) of the multi-byte sequences in the byte stream conform to UTF-8 rules and there are no invalid sequences, a high score (e.g., 0.99) is given. For ASCII, the confidence score is based on the byte value distribution. If all byte values ​​are in the range of 0x00-0x7F, it is determined to be ASCII with a confidence score of 1.0. For GBK / GB2312, the analysis function checks the frequency of sequences in the byte stream that conform to the GBK double-byte encoding range (first byte 0x81-0xFE, second byte 0x40-0xFE) and calculates the score by combining the matching of common Chinese character sequences. In specific implementations, these analysis functions can be implemented by directly integrating or referring to the algorithm logic of mature libraries (such as Python's chardet or cchardet). In addition, for each candidate encoding, the confidence score can be calculated by multi-feature weighted summation based on the byte features extracted in step (1), with the weights set according to the probability of encoding occurrence in the server scenario. For example, taking UTF-8 as an example, the matched features are: UTF-8 compliant sequence proportion ≥ 90%, 0x80-0xBF byte proportion ≥ 30% (including Chinese characters); no large number of 0x00 bytes (proportion < 5%). In this case, the confidence score calculation formula is 0.4 × compliant sequence proportion + 0.3 × 0x80-0xBF byte proportion + 0.3 × (1 - 0x00 byte proportion) (it should be noted that the total weight = 1). Specifically, if a byte stream has a UTF-8 compliant sequence proportion of 95%, a 0x80-0xBF byte proportion of 35%, and a 0x00 byte proportion of 2%, then the confidence score = 0.4 × 0.95 + 0.3 × 0.35 + 0.3 × (1 - 0.02) = 0.891.

[0055] (3) According to the predefined coding priority order, select the highest priority coding from the candidate coding with a confidence score greater than the preset threshold and determine it as the first coding method.

[0056] It's important to note that to avoid misjudgments due to similar confidence levels, priority should be set based on the frequency of encoding occurrence in the server scenario, while a confidence threshold should be set to filter low-confidence encodings. In practice, this can be based on server industry data statistics (e.g., UTF-8 accounts for approximately 60%, ASCII approximately 25%, GBK approximately 14%, and UTF-16LE approximately 1%). In this case, the priority order can be set as: UTF-8 > ASCII > GBK > UTF-16LE. Furthermore, a minimum confidence threshold (e.g., 0.6) can be set based on practical experience with the chardet library.

[0057] Based on the above operations, the system first filters out all codes with confidence scores below the preset threshold. Among the remaining candidate codes, they are sorted according to coding priority, and the code with the highest priority is selected as the final first coding method. If the confidence scores of all candidate codes do not reach the threshold, the process is judged as a failure, and an alarm mechanism can be triggered to notify manual intervention or log recording.

[0058] Thus, through the two-tiered encoding matching mechanism described above, a progressive identification process from explicit declaration to statistical inference is achieved, effectively improving the coverage and accuracy of encoding detection. The first-tier BOM matching offers the highest reliability, enabling fast and unambiguous encoding determination for byte streams with explicit declarations. The second-tier statistical prediction serves as an important supplement and fallback mechanism, addressing encoding determination issues in scenarios with no BOM or unidentified BOM by analyzing the inherent statistical characteristics of the byte stream. The combination of these two tiers ensures that this method maintains high robustness and adaptability even when facing complex and varied encoding situations in server testing environments.

[0059] S103. Parse the byte stream based on the first encoding method and output a string in the first format.

[0060] It should be noted that binary byte sequences are meaningless, therefore the byte stream needs to be parsed using a first encoding method to transform it into a semantic character sequence. This process, based on the character mapping rules of the selected encoding scheme, translates the original byte sequence, which has no specific meaning, bit by bit into a character sequence containing clear business meaning.

[0061] Based on the preceding description, the first encoding method is the character encoding scheme ultimately determined in step S102 for decoding the current byte stream. If a match is successfully made in S102 using the explicit encoding declaration bit (BOM), then the first encoding method is the matched encoding type. If the BOM match fails, the first encoding method is the candidate encoding with the highest confidence level, exceeding a preset threshold, predicted through statistical features. Regardless of the path used to determine it, the first encoding method will serve as the sole basis for decoding the byte stream. During the parsing process, the system calls the decoding interface corresponding to the first encoding method in the programming language's standard library (such as the bytes.decode method in Python), passing the first encoding method as a key parameter. This interface strictly adheres to the specifications of the selected encoding, mapping the byte sequence to the corresponding Unicode code points, and then combining them into a complete string object. This process is a standard system-level operation, ensuring that the conversion from bytes to characters is accurate and as expected.

[0062] Furthermore, if the first encoding method is determined by matching explicit encoding declaration bits, the explicit encoding declaration bits in the header must be removed from the original byte stream first, because these bytes are only metadata identifying the encoding type and do not belong to the business content of the test data. If they are retained, invalid characters will appear after parsing, interfering with subsequent processing. If the first encoding method is determined by guessing through statistical encoding, the byte stream has no redundant encoding identifiers and can directly enter the parsing stage.

[0063] To address the differentiated parsing for different primary encoding methods, the encoding parsing interface of a programming language (such as Python) needs to be invoked to accurately match and parse according to the primary encoding method, ensuring no garbled characters and no semantic loss. For details, please refer to the descriptions of relevant technologies; they will not be elaborated here. Furthermore, to address illegal bytes or encoding errors encountered during decoding, a fault-tolerance mechanism must be adopted. For example, using the `errors='replace'` mode to replace illegal bytes with uniform placeholders, or using the `errors='ignore'` mode to directly ignore illegal bytes. These two strategies ensure that the decoding process is not interrupted by local byte errors, thereby guaranteeing the robustness and continuity of the entire data processing flow. Finally, the decoded output string is uniformly represented in the system as a primary format; in this application, the primary format is a UTF-8 encoded string object.

[0064] It's important to note that, firstly, the primary format is UTF-8 encoded string objects because UTF-8 supports all characters globally (English, Chinese, and special symbols in server test data), fully covering the character type requirements of BIOS / BMC test data. Secondly, core Python libraries for text processing (such as the re regular expression library for cleaning control characters, the json library for parsing JSON format, and the pandas library for processing tabular data) all default to UTF-8 encoding for string processing, eliminating the need for additional encoding conversion and reducing process complexity. Furthermore, UTF-8 is byte-order indistinguishable (no need to distinguish between big-endian and little-endian), avoiding secondary parsing errors caused by byte order differences in subsequent steps (unlike UTF-16, which always needs to distinguish between LE and BE, easily leading to errors due to missing identifiers).

[0065] S104. Clean the string based on the character features of the string in the first format to obtain clean text.

[0066] It should be noted that after S103 decoding, a unified string is obtained. However, the information generated by BIOS / BMC may contain noisy characters such as debugging information / memory dumps, terminal interactive output, and binary data remnants. This noise can disrupt structured parsing, cause logical errors, and reduce readability and analytical value. Therefore, it is necessary to filter out all control characters and escape sequences that have no business significance and extract the clean text containing only core test data.

[0067] Specifically, the step of cleaning the string based on the character features of the first format string to obtain clean text includes:

[0068] Character features refer to the distribution patterns and existence patterns of specific categories of characters in a string. Extraction is independent of the specific business source of the data, but rather based on the general encoding attributes and sequence patterns of characters. Specifically, the system scans the entire string using a predefined set of general rules to identify and locate two common types of noise features:

[0069] (1) Identify and remove ASCII control characters from the string.

[0070] It's important to note that ASCII control characters refer to characters in the ASCII encoding table located in the ranges of 0x00 to 0x1F (decimal 0-31) and 0x7F (DEL). These characters are not printable characters but are used to control the behavior of devices such as terminals and printers. Specifically, you can determine whether a string is a control character by iterating through it and checking the Unicode code point of each character. Using regular expressions is one of the most efficient ways to achieve this. For example, in Python, you can use regular expressions to match all ASCII control characters and replace them with an empty string.

[0071] (2) Identify and remove ANSI escape sequences from the string.

[0072] It's important to note that ANSI escape sequences are a set of standard code sequences used to control the display format of text terminals (such as color, font, and cursor position). They typically begin with an escape character (ESC, ASCII code 0x1B or octal \033), followed by a left square bracket "[", several parameters (numbers and semicolons), and a closing command letter (usually 'm'). Similarly, regular expressions can be used to precisely match and remove these sequences.

[0073] Through the two distinct cleaning operations described above, the original string can be purified into a well-organized, clean text containing only the core test data.

[0074] S105. Construct a structured data storage template.

[0075] It should be noted that although the clean text output by S104 has removed noise, it still retains the heterogeneous form of the original data. For example, one manufacturer's BMC sensor data is in tabular format, while another manufacturer's is in JSON format. Even within the same manufacturer, the key-value pairs in the text fields output by different commands may have different names. If parsing is directly based on the original format, separate logic needs to be developed for each format, making automation and scalability difficult. Therefore, a common structure is extracted from the heterogeneous text to construct a unified storage template. By sampling the clean text, the boundary between fixed delimiters and variable data regions is found. Then, candidate field names are identified based on the delimiters. Finally, field types are defined according to the data characteristics of the variable regions, ultimately forming a unified, clearly defined, and multi-format-adaptable storage template.

[0076] The construction of the structured data storage template includes:

[0077] (1) Sample the clean text and identify fixed delimiters and variable data regions.

[0078] It should be noted that fixed delimiters refer to characters or combinations of characters that appear repeatedly in clean text and are used to divide data structures. They do not carry business data meaning and only serve as structural identifiers, such as colons (:) and equal signs (=) in key-value pairs, tabs (\t), vertical bars (|), and consecutive spaces in tables, or quotation marks ("), colons (:), and curly braces ({}) in JSON. Variable data areas refer to text segments between fixed delimiters that carry core BIOS / BMC business data. It should also be noted that because clean text may contain a large number of repetitive structures (such as tables with dozens of rows of data or key-value pairs with dozens of records), it is not necessary to analyze the entire text. Instead, a representative portion of the text lines should be extracted for analysis to improve efficiency. For example, only the first 10-20 lines need to be sampled (covering all structure types), which ensures recognition accuracy while reducing computational overhead.

[0079] In practice, the system scans the sampled text, looking for recurring, regular delimiters. For example, if the scan finds that multiple lines of text contain the pattern "key: value", then a colon followed by a space is identified as a fixed delimiter. The part before the delimiter (key) and the part after the delimiter (value) are identified as variable data areas.

[0080] (2) Identify the content between the fixed delimiters as candidate field names, and map the candidate field names to the fields in the data storage template.

[0081] It should be noted that candidate field names are unstandardized field identifiers extracted directly from the raw data; the fields in the data storage template are system-defined, unified, and standardized field names. In practice, based on the delimiters identified in the previous step, the system can parse out the original candidate field names. Then, the system initiates a field mapping process, comparing the candidate field names with a standard field name library through semantic similarity matching. An algorithm (such as calculating string similarity) is used to determine which standard field name it most likely corresponds to. Further, a secondary determination can be made by combining data type, location, and context analysis. For details on this process, please refer to the descriptions of relevant technologies; they will not be elaborated here.

[0082] This step translates vendor-defined and diverse field names into standardized field names with clear semantics that are consistent within the system, thus resolving the issue of semantic heterogeneity.

[0083] (3) Based on the data pattern of the variable data area, predict and define the data type of the corresponding field in the data storage template.

[0084] It should be noted that data pattern refers to the statistical characteristics and format regularity of the data, such as the proportion of numerical values, the proportion of numerical values ​​plus units, the proportion of enumerations, and the proportion of unique values; data type refers to the type of value defined for each field in the data storage template, such as integer, floating-point, string, boolean, enumeration, etc. Specifically, the system analyzes the content of the variable data area (i.e., the data value part) and predicts its data type through a series of rules or statistical models. For example, if most values ​​can be matched by the regular expression ^[+-]?\d+$, it is predicted to be an integer; if the value matches ^[+-]?(\d+(\.\d+)?)$, it is predicted to be a floating-point (these are all determined through corresponding features, which are not shown one by one; please refer to the relevant technology introduction).

[0085] This step predefines the type of each field's value in the template, guiding subsequent parsing tools to perform correct type conversions and enabling more accurate comparisons during data comparison. It also helps optimize storage and computation.

[0086] S106. Call multiple parsing tool modules to sequentially parse the clean text, identify the field values ​​corresponding to the fields in the data storage template, fill the data storage template, and obtain the parsing results of the multi-source heterogeneous test data.

[0087] It's important to note that multiple parsing tool modules refer to a set of dedicated parsers designed using a plug-in or strategy pattern, each designed for different data formats (such as JSON, key-value pairs, and tables). Each parser is an independent module responsible for handling a specific data format. Sequential parsing refers to the parsing process attempting different parsers sequentially according to a priority strategy. The typical order is to first try the most structurally sound and information-rich format (JSON), then try the next highest priority format (key-value pairs), and finally try tables. This parsing strategy ensures efficiency and success rate. Field values ​​refer to the actual data content extracted from the clean text, corresponding to the standardized field names defined in the data storage template. Finally, the extracted field values ​​are filled into the corresponding positions according to the structure and data type defined in the template. This is a mapping process from heterogeneous source data to a unified data model. The final parsing result is a standard, structured data object (such as a JSON object or a specific class instance), the specific structure of which is defined by the template built in S105, containing all standardized test data.

[0088] In practice, the parsing is performed in the order of the JSON module, key-value pair module, and table module. Specifically, multiple parsing tool modules are called to parse the clean text sequentially, including:

[0089] (1) Use a JSON parsing function to parse the clean text.

[0090] The fastest and most standard JSON parsing path is to directly call the JSON parsing functions in the programming language's standard library (such as Python's json.loads() function) to parse the clean text. If the text is well-formatted JSON, this step will succeed immediately, parsing it into a dictionary or list object in memory and proceeding to the subsequent template filling stage.

[0091] (2) If parsing fails, perform at least one syntax compliance repair operation on the clean text and then try parsing again.

[0092] In practical applications, the JSON output by BIOS / BMC may not be entirely standard, containing minor syntactic flaws (not structural errors). Directly discarding it and switching modules would reduce the parsing success rate. Therefore, targeted fixes are made to ensure compatibility with such scenarios, improving the parsing success rate of JSON data through fault tolerance. At least one syntactic compliance fix includes: removing redundant commas at the end of the clean text; adding double quotes to unquoted keys in the clean text; and removing or escaping illegal control characters in the clean text. After one or more rounds of fixes, many parsing failures caused by minor formatting issues are resolved, significantly improving the parsing coverage of JSON formats.

[0093] When JSON parsing fails, key-value pair formats are tried first, specifically including:

[0094] (1) Filter out blank lines and comment lines in the clean text.

[0095] Based on the inherent characteristics of the key-value pair format, all lines containing only whitespace characters (spaces, tabs) and comment lines starting with specific characters (such as #, / / ) are removed to eliminate interference from non-data lines.

[0096] (2) Calculate the proportion of the remaining text lines that contain key-value delimiters.

[0097] Iterate through each remaining line and check if it contains common key-value separators, such as colons and equal signs. The ratio is calculated as: number of lines containing separators / total number of valid lines.

[0098] (3) When the ratio is greater than or equal to the preset threshold, the text format is determined to be a key-value pair format.

[0099] Based on server scenario experience, an empirical threshold is set. If the proportion of lines containing delimiters exceeds this threshold, the entire text is determined to be in key-value pair format. Otherwise, key-value pair parsing is abandoned, and table parsing is switched to table parsing.

[0100] (4) For text that is determined to be in key-value pair format, the key name and key value are parsed out based on the delimiter.

[0101] For each line, use a defined delimiter to split the line into two parts. The first part, after removing leading and trailing spaces, is used as the key name, and the second part, after removing leading and trailing spaces, is used as the key value.

[0102] It should also be noted that key-value texts often contain proprietary and special terminology, so key name validity checks are added to ensure that the key is not a pure number and is not confused with invalid identifiers (such as ok, fail); field uniqueness checks are also added, as key-value keys generally will not be duplicated.

[0103] For text that does not conform to either JSON or key-value pair format, the final step is to attempt to parse it as a table, specifically including:

[0104] (1) Split the clean text into text lines and remove the blank lines in the text lines.

[0105] Specifically, with The clean text is split by the delimiter to obtain a single-line list; then blank lines (lines containing only whitespace characters) and pure noise lines (such as lines containing only delimiters) are removed, thus retaining the real table data rows and avoiding blank lines interfering with column count and delimiter recognition.

[0106] (2) Use the delimiters in the candidate delimiter set to split the split text lines in turn, and count the frequency of the number of columns obtained after each delimiter.

[0107] It should be noted that the candidate delimiter set usually includes tabs, vertical lines, commas, and two or more consecutive spaces. For each candidate delimiter, iterate through all valid rows, split each row according to the delimiter, and obtain the number of columns in each row. For each delimiter, calculate the percentage of occurrence of each column number after splitting. In this way, the delimiter that makes the table row and column structure most uniform can be found.

[0108] (3) The number of columns with the highest frequency of occurrence is determined as the standard number of columns in the table, and the corresponding delimiter is determined as the valid delimiter.

[0109] Specifically, for each candidate delimiter, the column number with the highest frequency is taken. The highest frequencies of all candidate delimiters are compared, and the delimiter with the highest frequency is selected as the valid delimiter. The column number with the highest frequency corresponding to this valid delimiter is the standard column number of the table (the number of columns that all data rows should follow). Here, the valid delimiter is the boundary of the table columns, and the standard column number ensures the regularity of the table, which can lay the foundation for subsequent header identification and data extraction.

[0110] (4) Analyze the character composition of each cell in the first row. If each cell in the first row is composed of non-pure numeric characters and the number of columns is consistent with the standard number of columns, then the first row is determined to be the header; otherwise, the first row is determined to be the data.

[0111] Specifically, extract the first row of valid rows, split the first row according to the valid delimiter, obtain the content of each cell in the first row, and check the content of each cell in the first row. If all cells are not purely numbers (i.e., contain letters, Chinese characters, underscores, etc.), and the number of columns in the row is equal to the standard number of columns, then this row is determined to be the header. Otherwise, it means that the table has no header, and the first row is the data.

[0112] By identifying whether or not a table header exists, it becomes easier to understand the semantics of the table data. When there is a table header, the header content will be used for field mapping; when there is no table header, it is necessary to rely on techniques such as column profiling for inference.

[0113] It should be noted that, in order to achieve flexibility and scalability in multi-format parsing and to solve the core problem of semantic heterogeneity of tabular data, this application adopts a plug-in parser architecture and an intelligent field mapping method.

[0114] Specifically, the parsing tool module adopts a plug-in design. The system defines a general parser interface, which is implemented by all parsers targeting specific formats (such as JSON, key-value pairs, and tables). Based on the data format recognition result, the system can dynamically load and call the corresponding parser plugin, thereby parsing the clean text and filling it into the structured data storage template.

[0115] Each parser plugin outputs a unified, standardized data object, which preferably contains two parts: metadata (recording contextual information such as data source and timestamp) and data (carrying the core data list). Each item in the data list represents a standardized data unit, and its structure is designed to preserve the hierarchical relationship of the original data (through the path field), the standardized semantics (through the key field), and the original information (through the raw field).

[0116] Furthermore, to address the heterogeneity of data from different manufacturers in terms of field naming and numerical unit representation, an intelligent field mapping and data standardization mechanism is integrated into the parsing process. This includes numerical and unit splitting: for values ​​that may contain a mixture of numerical and physical units, the system uses a physical quantity processing library (such as Python's Pint library) for automatic identification and splitting, filling the numerical value and standardized unit into the `value` and `unit` fields respectively; and multi-dimensional field mapping: for tabular data, especially in scenarios with inconsistent header naming, a dynamic mapping method based on column profiling and multi-dimensional signal fusion is proposed.

[0117] Specifically, column profiling describes the attributes of each column of data by analyzing its statistical characteristics. The core characteristics include numeric_ratio (numerical ratio), unit_ratio (unit ratio), enum_ratio (enumeration ratio), and unique_ratio (unique value ratio).

[0118] In addition, a fusion scoring model is provided. Based on column profiles, the system integrates multiple signals such as word literal / semantic similarity, data type matching, location prior, and neighbor prior to calculate a comprehensive confidence score for the matching degree between each column and the predefined standard field.

[0119] Preferably, the data type matching degree can be obtained by weighted calculation of the column profile features. For example, the numerical matching degree can be calculated using the formula Numeric=a*numeric_ratio+b*unit_ratio, where a and b are preset weight coefficients. The comprehensive confidence score is a weighted sum of signals from each dimension; for example, when there is a table header, semantic similarity can be given a higher weight. Finally, the system maps the column with the highest confidence score exceeding the dynamic threshold to the corresponding standard field.

[0120] It should also be noted that the intelligent field mapping method possesses self-learning capabilities to continuously improve its accuracy in ongoing use. Specifically, this includes lexicon expansion and threshold adaptation. Lexicon expansion refers to the system adding the original header text to the thesaurus of the standard field after each successful field mapping. Simultaneously, for state enumeration fields, the system dynamically expands its recognition lexicon, employing a strategy combining global and local lexicons to balance general applicability with the terminology habits of specific vendors. Threshold adaptation means that the dynamic threshold can be updated based on the average confidence level of historical mapping results, enabling the system to adapt to the data quality of different vendors.

[0121] After obtaining the parsing results of the multi-source heterogeneous test data, this application further provides an automated data comparison method to accurately identify changes in the server state before and after the test. This method is based on the premise that all heterogeneous data has been successfully parsed and normalized into a unified structured object.

[0122] Specifically, the method includes:

[0123] (1) Obtain the corresponding analysis results before and after the test.

[0124] It should be noted that "before testing" and "after testing" here are broad concepts, referring to the periods before and after firmware version upgrades, hardware configuration changes, different stress test phases, or any two scenarios requiring state comparison. Based on predefined comparison task configurations, the system retrieves two standardized parsing result files generated at specified times or under specified conditions from storage, which have already undergone the aforementioned steps.

[0125] (2) Construct data mapping tables for the parsing results before and after the test, respectively, where the combination of the path and field name of the data entry is used as a unique identifier.

[0126] It's important to note that the purpose of constructing the data mapping table is to quickly locate and access each piece of data. Since the `data` in the parsed result is a list, directly traversing it is inefficient. Furthermore, because server configuration and status data are hierarchical, a single field name may appear repeatedly in different modules. Therefore, a combination of the data entry's path and field name can be used to construct a globally unique identifier. Further, the system traverses the `data` list of the parsed result, using the unique identifier as the key and the corresponding entire data entry as the value, to construct two efficient data dictionaries (i.e., mapping tables) in memory: one before the test and one after.

[0127] (3) Traverse the union of the unique identifiers of all data entries before and after the test.

[0128] It's important to note that the core of the comparison is identifying all possible changes, including additions, missing items, and modifications. Therefore, the system needs to examine all data entries that existed before and after the test. Specifically, the system first obtains the set of all keys in the mapping table before the test and the set of all keys in the mapping table after the test, and then calculates the union of these two sets. This union contains all the data points that need to be examined.

[0129] (4) Identify the changed fields based on the status of the unique identifier in the data mapping table and the comparison results of the corresponding field values.

[0130] It should be noted that for each unique identifier in the union set, the comparison of its state and value determines the difference type. If a unique identifier exists only in the post-test mapping table, the field is determined to be newly added; if a unique identifier exists only in the pre-test mapping table, the field is determined to be missing; if a unique identifier exists in both mapping tables, the field value comparison logic is entered.

[0131] Specifically, the system extracts and compares the values ​​corresponding to the unique identifier before and after the test. To improve the intelligence and accuracy of the comparison, the comparison process considers the data type and unit. For numerical data with physical units (such as temperature, voltage, and rotational speed), the system identifies the unit and judges based on a preset error threshold. Only when the absolute value of the difference between two values ​​is greater than the threshold is it judged as a change, thus avoiding a large number of meaningless differences caused by minor fluctuations in the sensor. If the units of the data before and after are inconsistent but convertible (such as 1MB and 1024KB), the system will convert them to standard units before comparison. In addition, for timestamp data marked with unit="time", the system can choose to ignore the comparison or perform exact matching according to the configuration. If the two values ​​are different (and do not meet the above ignore or tolerance conditions), the field is judged as a change, and the value before and after the change is recorded.

[0132] Using this method, the system can automatically output a structured difference report, accurately pointing out all the new, missing, and changed fields before and after the test, along with detailed contextual information (path, field name, new and old values), which greatly improves the efficiency and accuracy of the server testing and verification process.

[0133] The method provided in this embodiment firstly clarifies the specific types and acquisition methods of BIOS configuration data and BMC sensor data at the data input source, and defines in detail the complex components that the byte stream may contain, such as BOM and control characters. This allows the method to closely fit the real-world scenario of server testing, greatly improving the relevance and feasibility of the solution in practical applications, and ensuring that the method can handle complex and imperfect raw data. Secondly, in the decoding and cleaning stages, a confidence calculation and priority selection mechanism based on statistical features is introduced, and it is specifically explained that regular expressions are used to remove control characters and ANSI sequences. This not only enhances the accuracy of encoding detection and the fault tolerance capability in the absence of BOM, but also ensures the purity of the text content input to the subsequent parsing process through efficient cleaning methods. This fundamentally improves the stability and anti-interference capability of the entire system process and avoids parsing failures caused by data pollution. Furthermore, in the parsing and structuring phase, the specific process of constructing a structured data storage template through sampling, delimiter identification, field mapping, and data type prediction is detailed. Subsequently, by defining sequential parsing strategies for JSON, key-value pairs, and tables, and providing specific parsing algorithms for each format, including fault tolerance repair, validity verification, header recognition, and column delimiter detection, adaptation to multi-source heterogeneous formats and high-success-rate parsing are achieved. Finally, by introducing a unified data object and designing a data unit structure that preserves the original hierarchy, standardized semantics, and original information, the goal of constructing and populating a structured data storage template is achieved. Finally, the function of transforming highly heterogeneous data into highly normalized, structured results that can be directly used for automated comparison is implemented.

[0134] Example 2

[0135] Corresponding to the aforementioned embodiment of a method for parsing multi-source heterogeneous test data of a server, this application also provides an embodiment of a device for parsing multi-source heterogeneous test data of a server.

[0136] Figure 2 This is a schematic diagram of the second embodiment of the server multi-source heterogeneous test data parsing device provided in this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes a receiving module 210, an identification module 220, a parsing module 230, a cleaning module 240, a construction module 250, and a processing module 260.

[0137] The receiving module 210 is used to receive byte streams of multi-source heterogeneous test data from different components of the server under test.

[0138] The identification module 220 is used to identify the explicit encoding declaration bit of the byte stream, match the corresponding first encoding method, and if the matching fails, predict the first encoding method based on the byte features of the byte stream.

[0139] The parsing module 230 is used to parse the byte stream based on the first encoding method and output a string in the first format;

[0140] The cleaning module 240 is used to clean the string based on the character features of the string in the first format to obtain clean text;

[0141] The construction module 250 is used to construct a structured data storage template;

[0142] The processing module 260 is used to call multiple parsing tool modules to sequentially parse the clean text, identify the field values ​​corresponding to the fields in the data storage template, fill the data storage template, and obtain the parsing results of the multi-source heterogeneous test data.

[0143] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0144] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0145] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0146] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for parsing multi-source heterogeneous test data of a server, characterized in that, The method comprises: receiving a byte stream of multi-source heterogeneous test data from different components of a server under test; identifying an explicit encoding declaration bit of the byte stream, matching a corresponding first encoding mode, and if the matching fails, predicting a first encoding mode based on byte characteristics of the byte stream; parsing the byte stream based on the first encoding mode and outputting a string in a first format; cleaning the string based on character characteristics of the string in the first format to obtain a pure text; constructing a structured data storage template; sequentially parsing the pure text by calling multiple parsing tool modules to identify field values corresponding to fields in the data storage template, fill the data storage template, and obtain a parsing result of the multi-source heterogeneous test data; wherein the multiple parsing tool modules refer to a set of plug-in or strategy mode designed special parsers for different data formats, each parser is an independent module, responsible for processing a specific data format; the sequential parsing refers to sequentially calling the parsing tool modules corresponding to the data formats according to a preset priority, if the current parsing tool module is successfully parsed, the subsequent parsing is terminated; if the parsing fails, the next priority parsing tool module is called, until the parsing is successful or all parsing tool modules are tried, and the preset priority is in turn JSON format parsing, key-value pair format parsing and table format parsing.

2. The method of claim 1, wherein, The identification of the explicit encoding declaration bit of the byte stream, the matching of the corresponding first encoding mode comprises: reading a preset number of bytes at the beginning of the byte stream; comparing the read bytes with a predefined encoding marker mapping table; if the comparison is consistent, it is determined that there is an explicit encoding declaration bit, and the matched encoding type is determined as the first encoding mode.

3. The method of claim 1, wherein, The prediction of the first encoding mode based on the byte characteristics of the byte stream comprises: extracting byte characteristics of the byte stream; based on a predefined candidate encoding set, calculating a confidence score for each candidate encoding according to the byte characteristics; according to a predefined encoding priority order, selecting the highest priority encoding from the candidate encodings with a confidence score greater than a preset threshold to determine the first encoding mode.

4. The method of claim 1, wherein, The cleaning of the string based on the character characteristics of the string in the first format to obtain a pure text comprises: identifying and removing ASCII control characters in the string; identifying and removing ANSI escape sequences in the string.

5. The method of claim 1, wherein, The construction of the structured data storage template comprises: sampling the pure text to identify fixed delimiters and variable data regions; identifying the content between the fixed delimiters as candidate field names, and mapping the candidate field names to the fields in the data storage template; predicting and defining the data type of the corresponding field in the data storage template according to the data mode of the variable data region.

6. The method of claim 1, wherein, The sequential parsing of the pure text by calling multiple parsing tool modules comprises: filtering empty lines and comment lines in the pure text; statistically analyzing the proportion of lines containing key-value separators in the remaining text lines; When the ratio is greater than or equal to a preset threshold, it is determined that the text format is a key-value pair format; For the text determined as the key-value pair format, the key name and the key value are parsed based on the separator.

7. The method of claim 1, wherein, The calling of the multiple parsing tool modules for sequentially parsing the pure text further includes: The pure text is parsed using a JSON parsing function; If the parsing fails, at least one syntax compliance repair operation is performed on the pure text, and the parsing is attempted again.

8. The method of claim 1, wherein, The calling of the multiple parsing tool modules for sequentially parsing the pure text further includes: The pure text is split into text lines, and the empty lines in the text lines are removed; The split text lines are cut using separators in a candidate separator set one by one, and the occurrence frequency of the column number obtained after cutting by each separator is counted; The column number with the highest occurrence frequency is determined as the standard column number of the table, and the corresponding separator is determined as the effective separator; The character composition of each cell in the first row is analyzed, if each cell in the first row is composed of non-numeric characters and the column number is consistent with the standard column number, the first row is determined as the table header, otherwise the first row is determined as the data.

9. The method of claim 1, wherein, After the parsing result of the multi-source heterogeneous test data is obtained, the following steps are included: The parsing results before and after the test are obtained; Data mapping tables are constructed for the parsing results before and after the test respectively, wherein the combination of the path and the field name of a data entry is used as a unique identifier; The union set of the unique identifiers of all data entries before and after the test is traversed; According to the comparison result of the state of the unique identifier in the data mapping table and the corresponding field value, the field that has changed is identified.

10. A server multi-source heterogeneous test data parsing device, characterized in that, The device includes a receiving module, an identifying module, a parsing module, a cleaning module, a constructing module, and a processing module; The receiving module is configured to receive a byte stream of multi-source heterogeneous test data from different components of a server under test; The identifying module is configured to identify an explicit encoding declaration bit of the byte stream, match a corresponding first encoding mode, and if the matching fails, predict a first encoding mode based on the byte characteristics of the byte stream; The parsing module is configured to parse the byte stream based on the first encoding mode and output a string in a first format; The cleaning module is configured to clean the string based on the character characteristics of the string in the first format and obtain a pure text; The constructing module is configured to construct a structured data storage template; The processing module is configured to sequentially parse the pure text by calling multiple parsing tool modules, identify the field values corresponding to the fields in the data storage template, fill the data storage template, and obtain the parsing result of the multi-source heterogeneous test data. The multiple analysis tool modules are a set of special analyzers for different data formats designed by using a plug-in or strategy mode, each analyzer being an independent module responsible for processing a specific data format; the sequential analysis refers to sequentially calling the analysis tool modules corresponding to the data formats according to preset priorities, terminating subsequent analysis if the current analysis tool module is successfully analyzed, and continuing to call the analysis tool module of the next priority if the analysis fails, until the analysis is successful or all analysis tool modules are tried.

Images