Data acquisition system and method applied to engineering algorithm logs

Abstract

The invention relates to a data acquisition system and method applied to an engineering algorithm log, and the method comprises the steps: enabling a data format in the log to accord with a preset structure through presetting a format of an initial log file, and analyzing the preset initial log file into a log file of a hierarchical dictionary structure through a hierarchical division rule, therefore, when a data calling instruction including a multi-dimensional data acquisition rule is received, recursive matching and traversal can be performed on the log file of the hierarchical dictionary structure based on a multi-dimensional instruction with a hierarchical structure, and target data parameters are extracted. Therefore, it can be guaranteed that the process of collecting the target data parameters is faster, the obtained target data parameters better meet expected requirements, homonymous result data irrelevant to rules in the data calling instruction can be effectively eliminated, the accuracy and rapidness of screening data from the log file are effectively improved, and the data collection efficiency is greatly improved. The method has the advantages of being easy to operate, stable in operation, high in accuracy and good in adaptability.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and specifically to a data acquisition system and method for engineering algorithm logs. Background Technology

[0002] To better monitor equipment operation and analyze key parameters, logs are typically used to record the equipment's operating status. However, existing logs are usually unstructured, and data may only be associated with timestamps. This leads to difficulties in data extraction during subsequent analysis, hindering the collection and analysis of log data. This is especially true for data in specialized fields, where users are concerned not only with the data results but also with the reasoning process leading to those results. Existing log formats and data collection schemes cannot enable operators to quickly and effectively obtain the target data related to the reasoning process.

[0003] Therefore, how to provide users with a more efficient way to retrieve target object data from logs is a problem that urgently needs to be solved. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the present invention proposes a data acquisition system and method for engineering algorithm logs that is easy to operate, has high data retrieval accuracy, stable performance, and is easy to use.

[0005] To achieve the above objectives, the data acquisition system and method of the present invention for engineering algorithm logs are as follows: In a first aspect, embodiments of the present invention propose a data acquisition system for engineering algorithm logs, characterized in that the system comprises: The data source generation module is used to generate an initial log file corresponding to the device's operating data. The log file includes data in a preset structured format that corresponds to the operating data and is generated based on a preset structure. The data parsing module is used to parse the initial log file into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures based on the data in the preset structured format. The data acquisition rule configuration module is used to obtain data call instructions including multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure. The data structuring output module responds to the data call instruction by recursively matching and traversing the log file with the hierarchical dictionary structure to extract the target data parameters.

[0006] In some embodiments, the data in the preset structured format in the initial log file is constructed using a nested structure, and the core fields of the data in the preset structured format include operator name, operator parameter category, operator parameter name, and parameter value.

[0007] In some embodiments, the data parsing module includes: The text reading unit is used to recognize the text content of the initial log file and generate the corresponding string; The parsing rule matching unit is used to match the text content of the initial log file using regular expressions based on the preset core fields to obtain the corresponding matching results; The data extraction unit extracts core fields from the initial log file based on the matching results; The hierarchical dictionary construction unit extracts core fields from the initial log file, extracts strings associated with the core fields from the initial log file, and assembles a log file with a hierarchical dictionary structure including corresponding multi-level key-value pairs.

[0008] In some embodiments, the data parsing module further includes: An error handling unit is used to provide an error message when the data parsing module malfunctions.

[0009] In some embodiments, the data collection rule configuration module includes: The rule configuration interface unit provides an interface that allows users to input data retrieval commands for multi-dimensional data collection rules; The rule serialization and processing unit is used to convert the data call instruction obtained from the rule configuration interface unit into a data call instruction in a preset format.

[0010] In some embodiments, the rule configuration interface unit is further configured to generate a corresponding prompt instruction when the received data call instruction does not conform to the preset configuration rules.

[0011] In some embodiments, the data structuring output module is further configured to generate a corresponding target parameter list based on the extracted target data parameters, and then output and display it.

[0012] In some embodiments, the initial log file is dynamically generated using JSON Lines format.

[0013] In some embodiments, the log file may further include timestamp data, log level data, and runtime status data.

[0014] Secondly, embodiments of the present invention propose a data acquisition method for engineering algorithm logs, characterized in that the method includes: Obtain an initial log file corresponding to the device's operating data, wherein the log file includes data in a preset structured format generated based on a preset structure and corresponding to the operating data; Based on the data in the preset structured format, the initial log file is parsed into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures; Obtain data retrieval instructions that include multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure; In response to the data retrieval instruction, the log file of the hierarchical dictionary structure is recursively matched and traversed to extract the target data parameters.

[0015] The beneficial effects of the data acquisition system and method for engineering algorithm logs of the present invention are as follows: This data acquisition system and method for engineering algorithm logs pre-formats the initial log file, ensuring the data format conforms to a pre-defined structure. Through hierarchical partitioning rules, the initial log file is parsed into a hierarchical dictionary-structured log file. When a data retrieval command containing multi-dimensional data acquisition rules is received, the system recursively matches and traverses the hierarchical dictionary-structured log file based on these multi-dimensional commands to extract the target data parameters. This ensures a faster process for acquiring target data parameters, and the obtained parameters better meet the expected requirements. It effectively excludes data with the same name that is unrelated to the rules in the data retrieval command, significantly improving the accuracy and speed of data filtering from the log file and greatly enhancing data acquisition efficiency. The data acquisition system and method for engineering algorithm logs of this invention are characterized by ease of operation, stable operation, high accuracy, and good adaptability. Attached Figure Description

[0016] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention.

[0017] Figure 1 This is a schematic diagram of a data acquisition system for engineering algorithm logs according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the initial log file code with identification in one embodiment.

[0019] Figure 3 This is a schematic diagram of the data acquisition rule configuration module in one embodiment.

[0020] Figure 4This is a schematic diagram of the code for a preset format data retrieval instruction in one embodiment.

[0021] Figure 5 This is a schematic diagram of the target data parameters in one embodiment.

[0022] Figure 6 This is a flowchart of a data acquisition method applied to engineering algorithm logs in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that in this article, relational terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0025] The inventors of this solution discovered that in the existing technology, during the extraction of data from some engineering logs, the same parameter name in the engineering log may represent multiple different situations. However, operators need the value corresponding to the parameter name that conforms to specific rules. When the log data volume is particularly large, the existing traversal method is not a quick and effective way to find the required parameter. At the same time, for the operation of some complex equipment, multiple different methods may be used to detect the same object. As a result, there may be multiple parameter values ​​in the log corresponding to a certain detection parameter of the test object, but they all use the same parameter name. This makes it impossible for users to distinguish which operating method the obtained parameter was obtained from, thus making it impossible to achieve effective data screening.

[0026] Current technologies for parsing collected log data into structured information commonly include: 1. Extracting key fields from unstructured logs by defining regular expression matching rules. However, complex regular expressions are difficult to maintain and have high performance overhead. 2. Parsing log data based on predefined log templates (such as JSON templates, XML templates), but this requires predefined templates and has limited support for unstructured logs. 3. Performing word segmentation and part-of-speech tagging on log text to extract key information, but this is suitable for processing logs in natural language format, requires natural language processing technology, and has high computational overhead. 4. Parsing data based on predefined rules, such as Drools and easy rules, but rule writing is complex and maintenance costs are high. 5. Unstructured log data is irregular, making data extraction difficult. The data configuration and parsing process is cumbersome and lacks flexibility.

[0027] To address these issues, embodiments of the present invention provide the following solutions: In a first aspect, the present invention provides a data acquisition system for engineering algorithm logs, comprising: The data source generation module is used to generate an initial log file corresponding to the device's operating data. The log file includes data in a preset structured format corresponding to the operating data, generated based on a preset structure. The initial log file is dynamically generated using JSON Lines format. The log file also includes timestamp data, log level data, and operating status data. The data parsing module is used to parse the initial log file into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures based on the data in the preset structured format. The data acquisition rule configuration module is used to obtain data call instructions including multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure. The data structuring output module responds to the data call instruction by recursively matching and traversing the log file with the hierarchical dictionary structure to extract the target data parameters.

[0028] In specific implementation, the data in the preset structured format in the initial log file is composed of a nested structure, and the core fields of the data in the preset structured format include operator name, operator parameter category, operator parameter name and parameter value.

[0029] In this embodiment, the data parsing module includes: The text reading unit is used to recognize the text content of the initial log file and generate the corresponding string; The parsing rule matching unit is used to match the text content of the initial log file using regular expressions based on the preset core fields to obtain the corresponding matching results. The matching results can be marked with specified identifiers. The data extraction unit extracts core fields from the initial log file based on the matching results; The hierarchical dictionary construction unit extracts core fields from the initial log file, extracts strings associated with the core fields from the initial log file, and assembles a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures. An error handling unit is used to provide an error message when the data parsing module malfunctions.

[0030] In this embodiment, the hierarchical dictionary building unit is a stable abstraction layer. When the log data structure changes, only the parsing rules or collection rules need to be adapted, and the core engine and user interface can remain stable, which solves the problem of high maintenance costs.

[0031] In this embodiment, the data collection rule configuration module includes: The rule configuration interface unit provides an interface for inputting data retrieval commands for multi-dimensional data collection rules. Based on the design of the rule configuration interface unit, domain experts can define data collection tasks without coding by simply configuring the system, which greatly reduces the barrier to entry and achieves the separation of business and technology. The rule serialization and processing unit is used to convert the data call instruction obtained from the rule configuration interface unit into a data call instruction in a preset format. This unit is mainly used to convert manually input instructions into computer-readable instructions.

[0032] In this embodiment, the rule configuration interface unit is also used to generate a corresponding prompt instruction when the received data call instruction does not conform to the preset configuration rules.

[0033] In this embodiment, the data structuring output module is also used to generate a corresponding target parameter list based on the extracted target data parameters and to output and display it. This structural design allows for the output of a structured data list, enabling data analysis to be refined to the granularity of each specific field and value, thus achieving a "data usability" effect.

[0034] This data acquisition system for engineering algorithm logs pre-formats initial log files, ensuring the data conforms to a pre-defined structure. Through hierarchical partitioning rules, it parses the initial log file into a hierarchical dictionary-structured log file. When a data retrieval command containing multi-dimensional data acquisition rules is received, the system recursively matches and traverses the hierarchical dictionary-structured log file based on these multi-dimensional commands to extract the target data parameters. This ensures faster target data parameter acquisition and that the obtained parameters better meet expectations. It effectively excludes duplicate data with the same name unrelated to the rules in the data retrieval command, significantly improving the accuracy and speed of data filtering from the log file and greatly enhancing data acquisition efficiency. It is characterized by ease of operation, stable operation, high accuracy, and good adaptability.

[0035] Furthermore, this solution completely avoids NLP (Natural Language Processing) and knowledge graphs, adopting a technical path of "source structured generation" and "deterministic rule parsing". Through the data source generation module, it fundamentally avoids the generation of "chaos" and has overwhelming performance and reliability advantages.

[0036] The aforementioned equipment can be used in civil aircraft. This data acquisition system applied to engineering algorithm logs can be applied to the field of intensity data acquisition and analysis for flight engineering algorithms, and can constitute a data acquisition system based on flight engineering algorithm logs. The following description, using the above embodiment applied to log reading in civil aircraft, further illustrates the solution in this embodiment: Strength data acquisition and analysis using flight engineering algorithms enables real-time monitoring of key parameters such as stress, strain, and vibration of aircraft structures. This data is crucial for assessing the health of aircraft structures and predicting potential structural problems. Simultaneously, by comparing actual flight data with design expectations, engineers can verify the accuracy of their designs and make necessary optimizations and improvements. However, with the increasing complexity of aircraft systems, flight engineering algorithm structures are becoming more complex and larger, resulting in an exponential increase in the number and size of logs. Data acquisition and analysis based on log files need to meet higher requirements for real-time performance, security, and compliance. In a field like flight engineering, which demands 100% accuracy, the mis-extraction of a single key parameter can lead to catastrophic analytical errors, compromising system reliability. Therefore, efficiently extracting strength data (such as structural strength and vibration intensity) from logs is a critical issue.

[0037] By employing the aforementioned system and its technical framework of data acquisition and parsing, the visualization of flight engineering algorithm intensity data can be achieved. The technical problems that can be solved include generating structured log files that support efficient writing and parsing; constructing a flexible and scalable data parsing engine; and dynamically configuring data acquisition rules to improve performance and reduce subsequent maintenance costs.

[0038] like Figure 1 As shown, when the data acquisition system described above, applied to engineering algorithm logs, is applied to data acquisition based on flight engineering algorithm logs, the structure of the system can be found in [reference needed]. Figure 1 As shown, the system includes: Data source generation module: This module allows for the creation of custom log file structures (i.e., the log file structure can be manually defined during system design). It dynamically generates .log format flight engineering algorithm log files (i.e., initial logs) programmatically. The module supports efficient writing, storage, and subsequent parsing of intensity data. Other log file formats can also be used in other embodiments. See Flight Engineering Algorithm Log File Structure Figure 2 (It should be noted that,) Figure 2 The document only provides a segment of daily master data code including a hierarchical structure; the actual logs contain much more data. Figure 2 To facilitate understanding, corresponding locations in the code are highlighted with boxes, and the categories of the core fields corresponding to the code are indicated. The file content adopts a structured format (JSON Lines), and each log record adopts a nested structure, containing the following core fields: operator name, used to identify the operator module called by the algorithm; operator parameter category, determining whether the current parameter is a calculation input parameter or a result output parameter; operator parameter name, describing the configuration or parameter name of the operator at runtime; parameter value, recording the specific value of the parameter. It should be noted that this hierarchical structure is a better hierarchical structure designed by the inventor, but in practical applications, other hierarchical structures can also be used to limit the log structure. In addition, the log file also includes auxiliary fields such as timestamp, log level, and running status to ensure the integrity, traceability, and analyzability of the logs.

[0039] Data Parsing Engine: This parsing engine includes a text reading unit, a parsing rule matching unit, a data extraction unit, a hierarchical dictionary construction unit, and an error handling unit. Through regular expression matching, data extraction, and dictionary construction, it parses the log text file generated by the data source generation module into a hierarchical dictionary structure containing multi-level key-value pairs. Its modular design and error handling mechanism give the parsing engine good scalability and robustness, making it suitable for text parsing tasks with similar formats.

[0040] Text reading unit: Reads the contents of a text file and inputs it as a string.

[0041] Parsing rule matching unit: In this embodiment, four key parts of the text are matched using four regular expressions to obtain the matching results.

[0042] In this embodiment, matching can be performed in the following manner: This includes the start, end, and termination positions of the calculation, the parameter type (input or output parameter), and returns the matching index and the matching result. When the returned matching index is 1 or 2, it indicates a match at the start or end position of the calculation, and the matching result is the operator name or function name. When the returned matching index is 3 or 4, it indicates a match at the parameter type. When index=3, the matching result is the function's input parameter and its value; when index=4, the matching result is the function's output parameter and its value.

[0043] Data extraction module: Extracts information such as operator name, function name, input parameters, output parameters, and parameter values ​​from the matching results.

[0044] Hierarchical dictionary construction module: Organizes the extracted data into a hierarchical dictionary structure. In this embodiment, the hierarchical dictionary structure is: {operator name:{function name:{parameter category:{parameter name:parameter value}}}.

[0045] Error handling module: Handles possible exceptions during parsing. If the log format does not meet expectations, a ValueError instruction is thrown; if the input or output parameter value is not in a valid JSON format, a json.JSONDecodeError instruction is thrown. This implements timely error reporting functionality.

[0046] like Figure 3 As shown, the dynamic data acquisition rule configuration module (i.e., the data acquisition rule configuration module) includes a rule configuration interface unit and a rule serialization and processing unit.

[0047] The rule configuration interface unit provides a configurable front-end interface, supporting users to dynamically define multi-dimensional data collection rules and verify the validity of user-inputted rules in real time. For example, rules may include the flight engineering algorithm name to which the data to be collected belongs, the parameter display name, and the parameter source. Simultaneously, the interface implements user interaction logic, including adding, editing, deleting, and combining rules; it supports various input components (such as text input, drop-down selection, etc.) to adapt to the configuration requirements of different rule types. In other words, this rule configuration interface unit can dynamically render the data call command specifications that need to be filled in, presenting them to the user in a visual form, while simultaneously executing user interaction logic and validating and providing error prompts for user input in the corresponding positions.

[0048] Rule Serialization and Processing Unit: This unit serializes user-configured data collection rules into structured JSON data. Specifically, it converts user input into the graphical interface into a specified format, enabling the system to read and process related instructions. This includes serializing received data call instructions, parsing and querying rules, and generating data such as... Figure 4 The computer-readable instructions in the format shown can be stored and transmitted to the server to enable the system to run.

[0049] The data structures of the relevant computer-readable instructions are as follows: Figure 4 As shown, it also includes information such as operator parameter names, function names, parameter categories, and parameter names. It is important to note that... Figure 4 The Chinese content is for explanation and interpretation purposes only, and may not be included in practical applications.

[0050] This solution provides an interface to apply rules to data filtering and querying, supporting flexible and scalable data processing needs.

[0051] The data structuring output module recursively matches and traverses the hierarchical dictionary data according to the data acquisition rules (i.e., data retrieval instructions), extracts the target data parameter values ​​that conform to the rules, and organizes them into a structured data list for output. This list can be directly used for data visualization.

[0052] Figure 5 This is a list structure in one embodiment. "titles" stores the display names, "columns" stores the parameter names filtered according to data collection rules, and "data" stores the extracted parameter values.

[0053] The above embodiments, through the technical means of a data source generation module and a data parsing engine, solve the technical problems of efficient generation, storage, and parsing of flight engineering algorithm log files. The data source generation module supports custom log file structures and dynamically generates structured log files using JSON Lines format, ensuring the integrity, traceability, and analyzability of the logs. The data parsing engine, through modular design and utilizing regular expressions and hierarchical dictionary construction techniques, achieves efficient parsing and structured output of log files. This allows for better matching of data that conforms to the collection rules, obtaining matching parameter values ​​derived from specified algorithms and other conditions, avoiding increased workload due to misidentification caused by identical parameter names. Furthermore, the dynamic data collection rule configuration module allows users to flexibly define and verify data collection rules, and applies these rules to data filtering and querying through a rule serialization and processing unit, further enhancing the system's flexibility and scalability. It can flexibly match corresponding target data objects based on user input, without being limited by fixed conditions. Ultimately, this application achieves efficient generation, storage, parsing, and structured output of flight engineering algorithm logs, providing a reliable foundation for intensity data visualization.

[0054] Secondly, the present invention also provides a data acquisition method for engineering algorithm logs, wherein the method includes: Obtain an initial log file corresponding to the device's operating data, wherein the log file includes data in a preset structured format generated based on a preset structure and corresponding to the operating data; Based on the data in the preset structured format, the initial log file is parsed into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures; specifically including: The text content of the initial log file is recognized, and the corresponding string is generated; Based on the preset core fields, regular expressions are used to match the text content of the initial log file to obtain the corresponding matching results; Based on the matching results, extract the core fields from the initial log file; Based on the core fields extracted from the initial log file, the strings associated with the core fields in the initial log file are extracted and a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures is formed. In practice, when the data parsing module malfunctions, an error message is given. Obtain data retrieval instructions that include multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure; specifically including: An interface for providing data retrieval commands that allow input of multi-dimensional data collection rules; The data retrieval instruction obtained from the rule configuration interface unit is converted into a data retrieval instruction in a preset format; When the received data call command does not conform to the preset configuration rules, a corresponding prompt command is generated; In response to the data call instruction, the log file of the hierarchical dictionary structure is recursively matched and traversed to extract the target data parameters; Based on the extracted target data parameters, a corresponding target parameter list is generated and displayed.

[0055] In practice, the initial log file is dynamically generated using JSON Lines format, and the log file also includes timestamp data, log level data, and running status data.

[0056] In this embodiment, the data in the preset structured format in the initial log file is composed of a nested structure, and the core fields of the preset structured format data include operator name, operator parameter category, operator parameter name, and parameter value.

[0057] The data collection method described above for engineering algorithm logs can also be applied to the field of civil aircraft engineering technology. When this method is applied to log data collection in the field of civil aircraft engineering, such as... Figure 6 As shown, the specific operating steps are as follows: Step 1: In the data source generation module, design and implement the structure of the log file according to the flight engineering algorithm, including the algorithm function name, input and output parameters, parameter values, etc. (i.e., define the algorithm log file structure).

[0058] Step 2: Generate a compliant .log log file.

[0059] Step 3: Use the data parsing engine to parse the unstructured data in the log file into hierarchical dictionary data.

[0060] Step 3.1: Use the text reading module to iterate through the log file content line by line and output it as a string.

[0061] Step 3.2: Use the tools in the parsing rule matching module to perform regular expression matching on the string. If the match is successful, output the matching identifier index and the matching result; if the match fails, read the next line of the log file. Since this embodiment uses JSON Lines format to generate logs, it is possible to read the log file line by line.

[0062] Step 3.3: Use the tools in the data extraction module to extract information such as function name, parameter type, and parameter value from the matching results. If any exceptions occur during processing, use the error handling module to throw exception information.

[0063] Step 3.4: Using the matching identifier `index`, organize the function name, parameter type, parameter value, and other information into a hierarchical dictionary structure: `{function name: {parameter type: {parameter name: parameter value}}`. Step 4: Compress the parsed dictionary data into a .gzip file and save it.

[0064] Step 5: Dynamically generate multi-dimensional data collection rules through a configurable front-end page and serialize them into JSON data.

[0065] Step 5.1: Enter the dynamic interface for front-end rule configuration.

[0066] Step 5.2: Click the "Add Rule" button on the page. Three fields will pop up: Flight Engineering Algorithm Name, Parameter Display Name, and Parameter Source. Enter the field information as needed.

[0067] Step 5.3: If the input field information does not conform to the specifications, such as the input flight engineering algorithm name not matching the algorithm library, or the parameter source not being found in the logs, the system will validate the input information and provide a prompt for non-compliant input. Click "OK". If the input field information conforms to the specifications, one rule is successfully configured.

[0068] Step 5.4: Continue clicking the "Add Rule" button until all rules have been entered. The system also supports deleting and modifying each rule.

[0069] Step 6: Unzip the file generated in Step 4, and recursively match and traverse the hierarchical dictionary structure according to the data collection rules to extract the target data parameters that meet the rules.

[0070] Step 7: Use the tools in the data structuring output module to organize the extracted target data parameters into a list, which is used for data visualization.

[0071] The advantages of this embodiment compared to existing technologies are as follows: 1. It addresses the parsing challenges at the source, achieving deterministic processing: By defining logs in JSON Lines format through the "Data Source Generation Module," the complex "Natural Language Understanding" problem is reduced to a simple "Syntax Parsing" problem, overcoming the uncertainties and low reliability caused by relying on NLP technology, and ensuring 100% accuracy in data processing. 2. It improves performance and reduces overhead: By adopting a lightweight combination of "regular expression matching" and "hierarchical dictionary construction," it avoids the huge performance bottlenecks brought by computationally intensive technologies such as NLP and knowledge graphs, resulting in faster parsing speed and extremely low resource consumption, making it suitable for high real-time data acquisition scenarios such as flight engineering. 3. Configurable business rules significantly lower the barrier to entry: The "Dynamic Data Acquisition Rule Configuration Module" provides a user-friendly front-end interface, allowing domain experts to define data acquisition rules without writing code, solving the problems of high barriers to entry and rigidity, achieving decoupling of business and technology, and significantly improving system flexibility and maintainability. 4. End-to-end optimized collaboration: This solution is a highly cohesive vertical domain solution. From log generation and storage to parsing and querying, each step is designed collaboratively, solving the problems of complex integration and high operation and maintenance costs caused by scattered general-purpose tools.

[0072] Thirdly, embodiments of the present invention also provide a chip for performing the method in the first aspect described above. Specifically, the chip includes a processor for calling and running a computer program from a memory, such that a device having the chip installed performs the method in the second aspect described above.

[0073] Furthermore, in a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method of the second aspect of the present invention.

[0074] For example, machine-readable storage media can include, but are not limited to, various known and unknown types of non-volatile memory.

[0075] Fifthly, embodiments of the present invention also provide a computer program product, including computer program instructions that cause a computer to perform the method described in the second aspect above.

[0076] Those skilled in the art will understand that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions in different ways for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] In the embodiments of this application, the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another system. Furthermore, the coupling between the various units can be direct coupling or indirect coupling. Additionally, the functional units in the embodiments of this application can be integrated into a processing unit, or they can exist as separate physical entities, etc.

[0078] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0079] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a machine-readable storage medium. Therefore, the technical solution of this application can be embodied in the form of a software product, which can be stored in a machine-readable storage medium. This software product may include several instructions to cause an electronic device to execute all or part of the processes of the technical solution described in the embodiments of this application. The aforementioned storage medium may include various media capable of storing program code, such as ROM, RAM, removable disk, hard disk, magnetic disk, or optical disk.

[0080] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A data acquisition system for engineering algorithm logs, characterized in that, The system includes: The data source generation module is used to generate an initial log file corresponding to the device's operating data. The log file includes data in a preset structured format that corresponds to the operating data and is generated based on a preset structure. The data parsing module is used to parse the initial log file into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures based on the data in the preset structured format. The data acquisition rule configuration module is used to obtain data call instructions including multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure. The data structuring output module responds to the data call instruction by recursively matching and traversing the log file with the hierarchical dictionary structure to extract the target data parameters.

2. The data acquisition system for engineering algorithm logs according to claim 1, characterized in that, The data in the preset structured format in the initial log file is composed of a nested structure, and the core fields of the preset structured format data include operator name, operator parameter category, operator parameter name, and parameter value.

3. The data acquisition system for engineering algorithm logs according to claim 2, characterized in that, The data parsing module includes: The text reading unit is used to recognize the text content of the initial log file and generate the corresponding string; The parsing rule matching unit is used to match the text content of the initial log file using regular expressions based on the preset core fields to obtain the corresponding matching results; The data extraction unit extracts core fields from the initial log file based on the matching results; The hierarchical dictionary construction unit extracts core fields from the initial log file, extracts strings associated with the core fields from the initial log file, and assembles a log file with a hierarchical dictionary structure including corresponding multi-level key-value pairs.

4. The data acquisition system for engineering algorithm logs according to claim 3, characterized in that, The data parsing module also includes: An error handling unit is used to provide an error message when the data parsing module malfunctions.

5. The data acquisition system for engineering algorithm logs according to claim 1, characterized in that, The data acquisition rule configuration module includes: The rule configuration interface unit provides an interface that allows users to input data retrieval commands for multi-dimensional data collection rules; The rule serialization and processing unit is used to convert the data call instruction obtained from the rule configuration interface unit into a data call instruction in a preset format.

6. The data acquisition system for engineering algorithm logs according to claim 5, characterized in that, The rule configuration interface unit is also used to generate a corresponding prompt instruction when the received data call instruction does not conform to the preset configuration rules.

7. The data acquisition system for engineering algorithm logs according to claim 1, characterized in that, The data structuring output module is also used to generate a corresponding target parameter list based on the extracted target data parameters, and to output and display it.

8. The data acquisition system for engineering algorithm logs according to claim 1, characterized in that, The initial log file is dynamically generated using JSON Lines format.

9. The data acquisition system for engineering algorithm logs according to claim 1, characterized in that, The log file also includes timestamp data, log level data, and runtime status data.

10. A data acquisition method for engineering algorithm logs, characterized in that, The method includes: Obtain an initial log file corresponding to the device's operating data, wherein the log file includes data in a preset structured format generated based on a preset structure and corresponding to the operating data; Based on the data in the preset structured format, the initial log file is parsed into a log file with a hierarchical dictionary structure including corresponding multi-level key-value pair structures; Obtain data retrieval instructions that include multi-dimensional data acquisition rules, wherein the data acquisition rules correspond to the preset structure; In response to the data retrieval instruction, the log file of the hierarchical dictionary structure is recursively matched and traversed to extract the target data parameters.

Images