A method and device for collecting formatted data containing a large number of known repeated fields

Through the combination of log generation and processing modules, the problem of irregular log collection format, repeated fields occupying bandwidth and confidential information leakage is solved, and the method of efficient collection and processing of structured logs is realized, ensuring data accuracy and confidentiality sex.

CN118689857BActive Publication Date: 2024-11-26PLA DALIAN NAVAL ACADEMY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411162033.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

In the fields of surface formation confrontation and big data data collection, there are irregular log formats, repeated fields occupancy bandwidth, and confidential information leakage during log collection, resulting in low data processing efficiency and security risks.

Method used

The log generation module processes non-structural data in a preset format, combines the key-value storage module to perform desensitization processing and repetitive field replacement, generates formatted and standardized logs, and divides, compresses and restores them through the log collection and data processing modules, and imports Data warehouse.

Benefits of technology

It realizes efficient collection of structured logs from multiple servers, reduces the risk of abnormal data processing, saves bandwidth, and ensures data confidentiality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118689857B_ABST
    Figure CN118689857B_ABST
Patent Text Reader

Abstract

The present application provides a formatted data collection method and device containing a large number of known repeated fields, which relates to the field of data collection technology. The log generation module processes and desensitizes the unstructured data to be processed according to the preset case log format, generates a formatted and standardized case log, and stores it in the log file; the log collection module performs segmentation and compression processing and sends it to the data processing module; the data processing module performs data restoration on the case log after data processing according to the key value, and imports the restored data into the data warehouse. The above method can quickly obtain accurate structured logs of surface formation data based on the log generation module, the log collection module, the log collection module and the key value storage module, and at the same time realizes the desensitization processing and rapid transmission of the surface formation data, and is not easy to leak the confidential information of the surface formation data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of data collection, and in particular to a method and device for collecting formatted data containing a large number of known repeated fields. Background Art

[0002] In the field of surface formation confrontation and big data data collection, an important step for the confrontation decision system before training the decision algorithm is to collect formation case data. The case data is usually printed into logs. At present, in the processing of large quantities of case logs, case logs often do not have a unified and standardized format, and the log collection method is to directly send all text information from the collection end to the preprocessing end.

[0003] Therefore, there are several problems: the collected log format is not standardized, and the logs often have no reasonable structure, which requires a lot of work to format the logs during preprocessing, and the chaotic log structure will also cause preprocessing exceptions; there are also a large number of repeated fields in the case logs, which occupy a lot of bandwidth during data transmission, thus affecting the normal business processing; there is also confidential information in the logs, and there is a problem of leakage.

[0004] Based on this, there is an urgent need for a method to efficiently collect a large amount of structured case logs from multiple servers. Summary of the invention

[0005] The present application provides a method and device for collecting formatted data including a large number of known repeated fields to solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for collecting formatted data containing a large number of known repeated fields is provided, the method comprising:

[0007] Through the log generation module, the unstructured data to be processed is processed according to the preset case log format to obtain a formatted case log, and according to the fields set by _system_encrypt_fields_ in the key-value storage module, the formatted case log is desensitized according to the field order to generate a formatted and standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields;

[0008] The log collection module obtains the case log from the log file, and after segmentation and compression, sends the processed case log to the data processing module;

[0009] The processed case log is received through the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module to obtain the data in JSON format, and imported into the data warehouse.

[0010] In an optional embodiment of the present application, in the key-value storage module, a hash table is used to store key-value pair data, and the key-value pair data includes fields that are not subject to collection processing, replacement processing, and need to be desensitized in the formatted log; wherein, _system_ignore_fields_ is used to set fields in the formatted log that are not subject to collection processing; _system_encrypt_fields_ is used to set fields in the formatted log that need to be desensitized, and the replacement processing is to use a pre-set simple symbol to replace repeated long fields in the formatted log.

[0011] In an optional embodiment of the present application, the log generation module includes a case log format design submodule, a log desensitization submodule and a log printing submodule;

[0012] The log generation module processes the unstructured data to be processed according to the preset case log format to obtain a formatted case log, and desensitizes the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module according to the field order to generate a formatted and standardized case log and store it in the log file, including:

[0013] Through the case log format design submodule, the unstructured data to be processed is formatted according to the preset case log format to obtain a formatted case log;

[0014] The formatted case log is desensitized according to the field order by the log desensitization submodule to obtain a formatted and standardized case log, wherein the desensitization refers to encrypting or not displaying the confidential fields in the formatted case log according to the fields set by _system_encrypt_fields_ in the key-value storage module;

[0015] The formatted and standardized case log is printed into a log file for storage through the log printing submodule.

[0016] In an optional embodiment of the present application, the preset case log format is as follows:

[0017] yyyy-MM-dd HH:mm:ss fffffffff zzz|First application name|Service name|Subservice name|Cloud service address|Log level|Second application name|Log category|Method name|Data name{"Attribute 1": "Value 1", "Attribute 2": "Value 2",...}|Description;

[0018] Among them, yyyy-MM-dd HH:mm:ss fffffffff zzz represents time information, yyyy-MM-dd represents year, month, and day respectively, HH:mm:ss represents hour, minute, and second respectively, fffffffff represents nanosecond, and zzz represents time zone; log levels include severe error, error, warning, general log, and debug log; the data name {"attribute 1": "value 1", "attribute 2": "value 2",...} is followed by a JSON string used to store data in the case log; log categories include at least the red force, blue force, naval environment, combat mission, and action plan of the surface formation; method names include at least create, update, read, and delete.

[0019] In an optional embodiment of the present application, the log generation module processes the unstructured data to be processed according to a preset case log format to obtain a formatted case log, including:

[0020] Obtaining log data of a surface formation in a designated sea area as unstructured data to be processed, wherein the unstructured data to be processed includes at least basic information of the surface formation, navigation track and position data, environmental parameter data, equipment status data, personnel operation data and task execution data;

[0021] A custom extractor is used to parse and extract key fields in the unstructured data to be processed, wherein the custom extractor is a neural network model trained by log data of a surface formation in a specified sea area;

[0022] The key fields are matched according to the preset case log format to obtain the formatted case log.

[0023] In an optional embodiment of the present application, the log collection module includes: a log extraction submodule, a log segmentation submodule, a log compression submodule and a log forwarding submodule;

[0024] The log collection module obtains the case log from the log file, and after segmentation and compression, sends the processed case log to the data processing module, including:

[0025] Collect case logs from the log file line by line through the log extraction submodule;

[0026] Through the log segmentation submodule, the line starting with the date and the subsequent lines not starting with the date are merged into one log file;

[0027] Through the log compression submodule, the key value is read from the key-value storage module, and the duplicate long fields of the merged log file are replaced. The fields set in _system_ignore_fields_ in the key-value pair data are not collected to obtain the processed case log;

[0028] The processed case log is sent to the data processing module through the log forwarding submodule.

[0029] In an optional embodiment of the present application, the data processing module includes a data receiving submodule, a data restoring submodule and a data forwarding submodule;

[0030] The processed case log is received through the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module, and the restored data in JSON format is imported into the data warehouse, including:

[0031] Receive the processed case logs sent by the log collection module through the data receiving submodule;

[0032] Through the data restoration submodule, based on reading the key-value pair data from the key-value storage module, the repeated long fields of the processed case logs are restored and converted into JSON format data;

[0033] The data in JSON format is imported into the data warehouse through the data forwarding submodule.

[0034] In a second aspect of an embodiment of the present application, a formatted data collection device containing a large number of known repeated fields is provided, the device comprising: a log generation module, a log collection module, a data processing module and a key-value storage module;

[0035] A log generation module, used to process the unstructured data to be processed according to a preset case log format to obtain a formatted case log, and desensitize the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module in the field order, generate a formatted and standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields;

[0036] The log collection module is used to obtain the case log from the log file, and after segmentation and compression, send the processed case log to the data processing module;

[0037] The data processing module is used to receive the processed case log, and restore the processed case log according to the key-value pair data stored in the key-value storage module, restore the data to obtain the JSON format, and import it into the data warehouse.

[0038] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the formatted data collection methods containing a large number of known repeated fields described in the first aspect.

[0039] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the formatted data collection method containing a large number of known repeated fields described in any one of the first aspects above is implemented.

[0040] The present application includes the following advantages: the present application provides a method and device for collecting formatted data containing a large number of known repeated fields, wherein the non-structured data to be processed is processed according to a preset case log format by a log generation module to obtain a formatted case log, and the formatted case log is desensitized according to the field set by _system_encrypt_fields_ in the key-value storage module in the order of the fields to generate a formatted and standardized case log, and stored in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields; the case log is obtained from the log file by a log collection module, and the processed case log is sent to a data processing module after segmentation and compression; the processed case log is received by the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module to obtain data in JSON format, and the data is imported into a data warehouse. The above method can quickly obtain accurate structured logs of surface formation data based on the log generation module, log collection module, log collection module and key-value storage module, while realizing desensitization processing and rapid transmission of surface formation data, and is not easy to leak confidential information of surface formation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1It is a flowchart of the steps of a method for collecting formatted data containing a large number of known repeated fields provided by an embodiment of the present application;

[0043] Figure 2 It is an architectural diagram of a formatted data acquisition device including a large number of known repeated fields provided in an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In the field of surface formation confrontation and big data data collection, an important step for the confrontation decision system before training the decision algorithm is to collect formation case data. The case data is usually printed into logs. At present, in the processing of large quantities of case logs, case logs often do not have a unified and standardized format, and the log collection method is to directly send all text information from the collection end to the preprocessing end.

[0047] Therefore, there are several problems: the collected log format is not standardized, and the logs often have no reasonable structure, which requires a lot of work to format the logs during preprocessing, and the chaotic log structure will also cause preprocessing exceptions; there are also a large number of repeated fields in the case logs, which occupy a lot of bandwidth during data transmission, thus affecting the normal business processing; there is also confidential information in the logs, and there is a problem of leakage.

[0048] In order to efficiently collect a large number of structured case logs from multiple servers, the present application discloses, in a first aspect, a method for collecting formatted data containing a large number of known repeated fields, see Figure 1 , Figure 1 : is a flowchart of a method for collecting formatted data containing a large number of known repeated fields provided in an embodiment of the present application, the method comprising the following steps:

[0049] Step 101: Process the unstructured data to be processed according to the preset case log format through the log generation module to obtain a formatted case log, and perform desensitization processing on the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module according to the field order, generate a formatted standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields;

[0050] Step 102: Obtain case logs from the log file through the log collection module, segment and compress the case logs, and send the processed case logs to the data processing module;

[0051] Step 103: Receive the processed case log through the data processing module, and restore the processed case log according to the key-value pair data stored in the key-value storage module to obtain the data in JSON format, and import it into the data warehouse.

[0052] When implementing step 101, when efficiently collecting a large number of structured case logs from multiple servers, the non-standard format of the collected case logs will lead to low efficiency and many failures in subsequent preprocessing, and each log requires a data processing method. Based on this, in order to unify the log format, the log generation module is used to process the unstructured data to be processed according to the preset case log format to obtain a formatted case log, and the formatted case log is desensitized according to the field order according to the field set by _system_encrypt_fields_ in the key-value storage module, and a formatted standardized case log is generated and stored in a log file. The log generation module is used to generate a formatted standardized case log, which is convenient for the subsequent extraction and conversion of the log, and is also convenient for the operation and maintenance personnel to view and retrieve.

[0053] In an optional embodiment of the present application, a large number of repeated fields in the case log data, although convenient for viewing, occupy a large amount of bandwidth during data transmission, and affect the processing of normal business. After formatting the log, a hash table that can support concurrent reading is used to replace the repeated fields with simple symbols, and then replace them with the original values ​​during data processing. Therefore, the above-mentioned key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields; refer to Figure 1, Table 1 is a hash table supporting concurrent reading provided by an embodiment of the present application. Specifically, in the above-mentioned key-value storage module, a hash table is used to store key-value pair data, and the above-mentioned key-value pair data includes fields that are not collected, replaced, and need to be desensitized for the formatted log; wherein, _system_ignore_fields_ is used to set the fields that are not collected in the formatted log; _system_encrypt_fields_ is used to set the fields that need to be desensitized in the formatted log, and the replacement process is to replace the repeated long fields in the formatted log with pre-set simple symbols, such as #*1, #*2. The key values ​​of the remaining fields can be edited. The formatted log here refers to the case log after formatting, including formatted case log and formatted standardized case log.

[0054] Table 1 Hash tables that support concurrent reading

[0055]

[0056] In an optional embodiment of the present application, the above-mentioned log generation module includes a case log format design submodule, a log desensitization submodule and a log printing submodule; through the log generation module, the non-structured data to be processed is processed according to the preset case log format to obtain a formatted case log, and according to the fields set by _system_encrypt_fields_ in the key-value storage module, the formatted case log is desensitized in the order of fields to generate a formatted standardized case log, and stored in the log file, which specifically includes the following processes: first, the non-structured data to be processed is formatted according to the preset case log format by the case log format design submodule to obtain a formatted case log; then, the formatted case log is desensitized according to the field order by the log desensitization submodule to obtain a formatted standardized case log, and the above-mentioned desensitization refers to encrypting or not displaying the confidential fields in the formatted case log according to the fields set by _system_encrypt_fields_ in the key-value storage module; finally, the above-mentioned formatted standardized case log is printed into the log file for storage by the log printing submodule.

[0057] In an optional embodiment of the present application, the preset case log format used in the above-mentioned log generation module is as follows: yyyy-MM-dd HH:mm:ss fffffffff zzz|first application name|service name|subservice name|cloud service address|log level|second application name|log category|method name|data name{"attribute 1": "value 1", "attribute 2": "value 2",…}|description; wherein, yyyy-MM-dd HH:mm:ss fffffffff zzz represents time information, yyyy-MM-dd represents year, month and day respectively, HH:mm:ss represents hour, minute and second respectively, fffffffff represents nanosecond, and zzz represents time zone; the first application name is the name of the entire platform or system, which is used to identify in the data warehouse for easy retrieval; the service name is the name of the platform module; the subservice name is the name of the microservice in the module; the cloud service address is the IP address and host name; the log levels include Fatal (serious error), Error (error), Warn (warning), Info (general log) and Debug (debug log); the second application name is the name of the application started on the machine used; the log category is the category to which the case log belongs, including at least the red force, blue force, naval combat environment, combat mission and action plan of the surface formation; the method name is what processing the log is used for, including at least creation, update, read and deletion; the data name {"attribute 1": "value 1", "attribute 2": "value 2",...} is followed by a JSON string, which is used to store the data in the case log. Combine the predefined case log format and rule engine to generate formatted logs through matching and replacement. Use machine learning or user feedback to continuously optimize and adjust the preset case log format to adapt to changes in the log format.

[0058] In an optional embodiment of the present application, the unstructured data to be processed is processed according to a preset case log format through a log generation module to obtain a formatted case log, which specifically includes the following processes: first, the log data of the surface formation in the specified sea area is obtained as the unstructured data to be processed, and the above-mentioned unstructured data to be processed includes at least basic information of the surface formation, navigation trajectory and position data, environmental parameter data, equipment status data, personnel operation data and task execution data; then, a custom extractor is used to parse and extract key fields in the unstructured data to be processed, and the above-mentioned custom extractor is a neural network model trained by the log data of the surface formation in the specified sea area; finally, the key fields are matched according to the preset case log format to obtain the above-mentioned formatted case log.

[0059] Optionally, the basic information of the surface formation may include the formation composition and mission instructions. The formation composition records the model, number, tonnage, weapon configuration and other information of each ship in the formation. The mission instructions specify the mission objectives, start and end times, navigation routes and special requirements of the formation. The navigation track and position data may include GPS positioning data and navigation track diagrams. The GPS positioning data records the longitude and latitude, speed, heading and other position information of each ship in the formation in real time. The navigation track diagram is a formation navigation track diagram drawn based on GPS data for subsequent analysis and evaluation. Environmental parameter data include meteorological data, hydrological data and sea area data. Meteorological data include meteorological conditions such as wind direction, wind speed, temperature, humidity, visibility, etc. Hydrological data include hydrological parameters such as water depth, water flow speed, tidal conditions, wave height, etc. The sea area environment records the topography, channel conditions, obstacle locations and other information of the sea area where the formation is sailing. Equipment status data includes navigation equipment, weapon systems and power systems. Navigation equipment records the operating status and performance parameters of navigation auxiliary equipment such as radar, sonar, and communication equipment. Weapon systems include the readiness status, launch status, and hit rate of weapons such as missiles and artillery. Power systems record the operating status, fuel consumption, and fault information of power equipment such as engines and propellers. Personnel operation data includes command instructions, operation records, and emergency handling. Command instructions record the various instructions issued by the formation commander and their execution status. Operation records include specific operation records of crew members in various links such as navigation, combat, and maintenance. Emergency handling records the emergency handling process and results of the formation when encountering emergencies. Mission execution data includes mission progress, results evaluation, and lessons learned. Mission progress records the progress and completion of the formation's mission execution in real time. Results evaluation evaluates and summarizes the results achieved by the formation in the process of executing the mission. Lessons learned summarizes the lessons learned and improvement measures of the formation in the process of mission execution. The above data are of great significance for analyzing formation performance, optimizing mission execution strategies, evaluating equipment reliability, and improving personnel skills.

[0060] Optionally, the pre-trained model used by the custom extractor used in the log generation module can be a basic model such as a recurrent neural network (RNN), a long short-term memory network (LSTM) or a Transformer, and the extraction of key fields in the unstructured data to be processed can be achieved through technologies such as word embedding and attention mechanism. When unstructured data needs to be processed in real time, stream processing technology can be used to parse, format and index log data in real time. For example, stream processing frameworks such as ApacheKafka and Apache Flink can be used to collect and process log data in real time.

[0061] When step 102 is specifically implemented, after the unstructured data is formatted, the case log is obtained from the above log file through the log collection module, and after segmentation and compression, the processed case log is sent to the data processing module.

[0062] In an optional embodiment of the present application, the above-mentioned log collection module includes: a log extraction submodule, a log segmentation submodule, a log compression submodule and a log forwarding submodule; through the log collection module, the case log is obtained from the log file, and after segmentation and compression processing, the processed case log is sent to the data processing module, which specifically includes the following processes: first, through the log extraction submodule, the case log is collected from the log file by line; then, through the log segmentation submodule, the line starting with the date and the subsequent lines not starting with the date are merged into a log file; further, through the log compression submodule, the key value is read from the key-value storage module, the duplicate long fields of the merged log file are replaced, and the fields set in _system_ignore_fields_ in the key-value pair data are not collected to obtain the processed case log; finally, through the log forwarding submodule, the processed case log is sent to the data processing module.

[0063] The processing of the log extraction submodule and the log segmentation submodule is illustrated. A log often has many lines and starts with a date. The following are two logs, but each log has multiple lines. The following is a list of abbreviations. An example of the merged log: The first log: 2024-07-10 15:23:24 21291 CST | Application 1 | Service 1 | Subservice 1 | 192.168.0.2 | Info | Application Name 1 | Blue Force | Update | Data Name {"Attribute 1": "Value 1", "Attribute 2": "Value 2", ...} | Description; System information: No abnormality was found. The second log: 2024-07-10 15:23:26 12252 CST | Application 1 | Service 1 | Subservice 1 | 192.168.0.2 | Error | Application Name 1 | Blue Force | Delete | Data Name {"Attribute 1": "Value 1", "Attribute 2": "Value 2", ...} | Description; System Information: System Exception: Memory overflow.

[0064] The processing of the log compression submodule is illustrated as an example. In the log data obtained by the surface fleet, a long field such as "maritime fleet platform data analysis system server node A" often appears. If the following key value is stored in the key-value storage module in advance: key (#*1), value (artificial intelligence log analysis system server a), then the third log: 2024-07-1015:23:24 21291 CST | Application 1 | Service 1 | Subservice 1 | 192.168.0.2 | Info | Application Name 1 | Blue Force | Update | Data Name {"Attribute 1": "Artificial Intelligence Log Analysis System Server a", "Attribute 2": "Value 2", ...} | Description; when printed, it will be changed to: 2024-07-10 15:23:24 21291 CST | Application 1 | Service 1 | Subservice 1 | 192.168.0.2 | Info | Application Name 1 | Blue Force | Update | Data Name {"Attribute 1": “#*1”, “attribute 2”: “value 2”,…}|description; Through the above processing, storage space can be saved.

[0065] When step 103 is specifically implemented, the processed case log is restored and converted into data in JSON format, and the data in JSON format is stored in the data warehouse for subsequent retrieval and use. Specifically, the processed case log is received through the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module, and the restored data in JSON format is imported into the data warehouse.

[0066] In an optional embodiment of the present application, the above-mentioned data processing module includes a data receiving sub-module, a data restoration sub-module and a data forwarding sub-module; through the data processing module, the processed case log is received, and the processed case log is restored according to the key-value pair data stored in the key-value storage module, and the restored data is obtained in JSON format and imported into the data warehouse, which specifically includes the following processes: first, through the data receiving sub-module, the processed case log sent by the log collection module is received; then, through the data restoration sub-module, based on reading the key-value pair data from the key-value storage module, the processed case log is restored with repeated long fields, and converted into JSON format data; finally, through the data forwarding sub-module, the JSON format data is imported into the data warehouse.

[0067] For example, if the received case log is: 2024-07-10 15:23:24 21291 CST | Application 1 | Service 1 | Sub-service 1 | 192.168.0.2 | Info | Application name 1 | Blue force | Update | Data name {"Attribute 1": "#*1", "Attribute 2": "Value 2", ...} | Description; restore it according to the key (#*1) and value (artificial intelligence log analysis system server a), the restored case log is as follows: 2024-07-10 15:23:24 21291 CST | Application 1 | Service 1 | Sub-service 1 | 192.168.0.2 | Info | Application name 1 | Blue force | Update | Data name {"Attribute 1": "Artificial intelligence log analysis system server a", "Attribute 2": "Value 2", ...} | Description; then convert it into JSON format data and send it to the data warehouse.

[0068] The embodiment of the present application provides a formatted data collection method containing a large number of known repeated fields, wherein the unstructured data to be processed is processed according to the preset case log format by the log generation module to obtain a formatted case log, and the formatted case log is desensitized according to the field set by _system_encrypt_fields_ in the key-value storage module in the order of the fields, a formatted standardized case log is generated, and stored in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields; the case log is obtained from the log file by the log collection module, and the processed case log is sent to the data processing module after segmentation and compression; the processed case log is received by the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module, and the data in JSON format is restored and imported into the data warehouse. The above method can quickly obtain accurate structured logs of surface formation data based on the log generation module, the log collection module, the log collection module and the key-value storage module, and at the same time realize the desensitization and rapid transmission of surface formation data, and it is not easy to leak confidential information of surface formation data.

[0069] In a second aspect of the embodiment of the present application, a formatted data collection device including a large number of known repeated fields is provided, referring to Figure 2 , Figure 2 It is an architecture diagram of a formatted data collection device including a large number of known repeated fields provided by an embodiment of the present application, wherein the device includes: a log generation module, a log collection module, a data processing module and a key-value storage module;

[0070] The log generation module 201 is used to process the unstructured data to be processed according to the preset case log format to obtain a formatted case log, and desensitize the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module according to the field order, generate a formatted and standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields;

[0071] The log collection module 202 is used to obtain the case log from the log file, and after segmentation and compression, send the processed case log to the data processing module;

[0072] The data processing module 203 is used to receive the processed case log, and restore the processed case log according to the key-value pair data stored in the key-value storage module to obtain the data in JSON format, and import it into the data warehouse.

[0073] Among them, the log generation module, the log collection module and the key-value storage module used in the log collection module are used to store key-value pair data using a hash table, and the key-value pair data includes fields that are not subject to collection processing, replacement processing, and need to be desensitized in the formatted log; wherein, _system_ignore_fields_ is used to set the fields that are not subject to collection processing in the formatted log; _system_encrypt_fields_ is used to set the fields that need to be desensitized in the formatted log, and the replacement processing is to replace the repeated long fields in the formatted log with a pre-set simple symbol.

[0074] Wherein, the log generation module includes a case log format design submodule, a log desensitization submodule and a log printing submodule;

[0075] The log generation module is used to process the unstructured data to be processed according to the preset case log format to obtain a formatted case log, and desensitize the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module in the field order to generate a formatted and standardized case log and store it in a log file, including:

[0076] The case log format design submodule is used to format the unstructured data to be processed according to the preset case log format to obtain a formatted case log;

[0077] The log desensitization submodule is used to perform desensitization processing on the formatted case log in field order to obtain a formatted and standardized case log. The desensitization processing refers to encrypting or not displaying the confidential fields in the formatted case log according to the fields set by _system_encrypt_fields_ in the key-value storage module;

[0078] The log printing submodule is used to print the formatted and standardized case log into a log file for storage.

[0079] The preset case log format used in the log generation module is as follows:

[0080] yyyy-MM-dd HH:mm:ss fffffffff zzz|First application name|Service name|Subservice name|Cloud service address|Log level|Second application name|Log category|Method name|Data name{"Attribute 1": "Value 1", "Attribute 2": "Value 2",...}|Description; yyyy-MM-dd HH:mm:ss fffffffff zzz represents time information, yyyy-MM-dd represents year, month, and day respectively, HH:mm:ss represents hour, minute, and second respectively, fffffffff represents nanosecond, and zzz represents time zone; Log levels include severe error, error, warning, general log, and debug log; Data name{"Attribute 1": "Value 1", "Attribute 2": "Value 2",...} is followed by a JSON string used to store data in the case log; Log categories include at least the red force, blue force, naval combat environment, combat mission, and action plan of the surface formation, and method names include at least create, update, read, and delete.

[0081] Wherein, the log generation module includes:

[0082] An acquisition submodule is used to acquire log data of a surface formation in a designated sea area as unstructured data to be processed, wherein the unstructured data to be processed includes at least basic information of the surface formation, navigation track and position data, environmental parameter data, equipment status data, personnel operation data and task execution data;

[0083] An extraction submodule, used for parsing and extracting key fields in the unstructured data to be processed by using a custom extractor, wherein the custom extractor is a neural network model trained by log data of a surface formation in a specified sea area;

[0084] The matching submodule is used to match the key fields according to the preset case log format to obtain the formatted case log.

[0085] The log collection module includes:

[0086] The log collection module is used to obtain the case log from the log file, and after segmentation and compression, send the processed case log to the data processing module, including:

[0087] A log extraction submodule, used for collecting case logs from the log file line by line;

[0088] The log segmentation submodule is used to merge the lines starting with a date and the subsequent lines that do not start with a date into one log file;

[0089] The log compression submodule is used to read the key value from the key-value storage module, replace the repeated long fields in the merged log file, and not collect the fields set in _system_ignore_fields_ in the key-value pair data to obtain the processed case log;

[0090] The processed case log is sent to the data processing module through the log forwarding submodule.

[0091] Wherein, the data processing module includes:

[0092] The data processing module is used to receive the processed case logs, restore the processed case logs according to the key-value pair data stored in the key-value storage module, and import the restored data in JSON format into the data warehouse, including:

[0093] The data receiving submodule is used to receive the processed case logs sent by the log collection module;

[0094] The data restoration submodule is used to restore the repeated long fields of the processed case logs based on reading the key-value pair data from the key-value storage module and convert them into JSON format data;

[0095] The data forwarding submodule is used to import the data in the JSON format into the data warehouse.

[0096] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 3 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 3 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a formatted data collection method containing a large number of known repeated fields disclosed in an embodiment of the present application.

[0097] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, a formatted data collection method containing a large number of known repeated fields disclosed in the embodiment of the present application is implemented.

[0098] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0099] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0103] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0104] The above is a detailed introduction to a method and device for collecting formatted data containing a large number of known repeated fields provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A formatted data collection method containing a large number of known repeating fields, It is characterized in that The method comprises: Through the log generation module, the unstructured data to be processed is processed according to the preset case log format to obtain a formatted case log, and according to the fields set by _system_encrypt_fields_ in the key-value storage module, the formatted case log is desensitized according to the field order to generate a formatted and standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields; The log collection module obtains the case log from the log file, and after segmentation and compression, sends the processed case log to the data processing module; The processed case log is received through the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module to obtain the data in JSON format, and is imported into the data warehouse; Through the log generation module, the unstructured data to be processed is processed according to the preset case log format to obtain a formatted case log, including: Obtaining log data of a surface formation in a designated sea area as unstructured data to be processed, wherein the unstructured data to be processed includes at least basic information of the surface formation, navigation track and position data, environmental parameter data, equipment status data, personnel operation data and task execution data; A custom extractor is used to parse and extract key fields in the unstructured data to be processed, wherein the custom extractor is a neural network model trained by log data of a surface formation in a specified sea area; The key fields are matched according to the preset case log format to obtain the formatted case log.

2. The method for collecting formatted data containing a large number of known repeated fields according to claim 1, It is characterized in that In the key-value storage module, a hash table is used to store key-value pair data, and the key-value pair data includes fields that are not subject to collection processing, replacement processing, and need to be desensitized in the formatted log; wherein, _system_ignore_fields_ is used to set the fields that are not subject to collection processing in the formatted log; _system_encrypt_fields_ is used to set the fields that need to be desensitized in the formatted log, and the replacement processing is to replace the repeated long fields in the formatted log with a pre-set simple symbol.

3. The method for collecting formatted data containing a large number of known repeated fields according to claim 1, It is characterized in that The log generation module includes a case log format design submodule, a log desensitization submodule and a log printing submodule; The log generation module processes the unstructured data to be processed according to the preset case log format to obtain a formatted case log, and desensitizes the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module according to the field order to generate a formatted and standardized case log and store it in the log file, including: Through the case log format design submodule, the unstructured data to be processed is formatted according to the preset case log format to obtain a formatted case log; The formatted case log is desensitized according to the field order by the log desensitization submodule to obtain a formatted and standardized case log, wherein the desensitization refers to encrypting or not displaying the confidential fields in the formatted case log according to the fields set by _system_encrypt_fields_ in the key-value storage module; The formatted and standardized case log is printed into a log file for storage through the log printing submodule.

4. The method for collecting formatted data containing a large number of known repeated fields according to claim 1, It is characterized in that The preset case log format is as follows: yyyy-MM-dd HH:mm:ss fffffffff zzz|First application name|Service name|Subservice name|Cloud service address|Log level|Second application name|Log category|Method name|Data name{"Attribute 1": "Value 1", "Attribute 2": "Value 2",...}|Description; Among them, yyyy-MM-dd HH:mm:ss fffffffff zzz represents time information, yyyy-MM-dd represents year, month, and day respectively, HH:mm:ss represents hour, minute, and second respectively, fffffffff represents nanosecond, and zzz represents time zone; log levels include severe error, error, warning, general log, and debug log; the data name {"attribute 1": "value 1", "attribute 2": "value 2", ...} is followed by a JSON string used to store data in the case log; log categories include at least the red force, blue force, naval environment, combat mission, and action plan of the surface formation; method names include at least create, update, read, and delete.

5. The method for collecting formatted data containing a large number of known repeated fields according to claim 2, It is characterized in that The log collection module includes: a log extraction submodule, a log segmentation submodule, a log compression submodule and a log forwarding submodule; The log collection module obtains the case log from the log file, and after segmentation and compression, sends the processed case log to the data processing module, including: Collect case logs from the log file line by line through the log extraction submodule; Through the log segmentation submodule, the line starting with the date and the subsequent lines not starting with the date are merged into one log file; Through the log compression submodule, the key value is read from the key-value storage module, and the duplicate long fields of the merged log file are replaced. The fields set in _system_ignore_fields_ in the key-value pair data are not collected to obtain the processed case log; The processed case log is sent to the data processing module through the log forwarding submodule.

6. The method for collecting formatted data containing a large number of known repeated fields according to claim 1, It is characterized in that The data processing module includes a data receiving submodule, a data restoring submodule and a data forwarding submodule; The processed case log is received through the data processing module, and the processed case log is restored according to the key-value pair data stored in the key-value storage module to obtain the data in JSON format, and is imported into the data warehouse, including: Receive the processed case logs sent by the log collection module through the data receiving submodule; Through the data restoration submodule, based on reading the key-value pair data from the key-value storage module, the repeated long fields of the processed case logs are restored and converted into JSON format data; The data in JSON format is imported into the data warehouse through the data forwarding submodule.

7. A formatted data collection device containing a large number of known repeating fields, It is characterized in that The device comprises: a log generation module, a log collection module, a data processing module and a key-value storage module; A log generation module, used to process the unstructured data to be processed according to a preset case log format to obtain a formatted case log, and desensitize the formatted case log according to the field set by _system_encrypt_fields_ in the key-value storage module in the field order, generate a formatted and standardized case log, and store it in a log file, wherein the key-value storage module stores key-value pair data for replacing or desensitizing repeated long fields; The log collection module is used to obtain the case log from the log file, and after segmentation and compression, send the processed case log to the data processing module; A data processing module is used to receive the processed case logs, and restore the processed case logs according to the key-value pair data stored in the key-value storage module to obtain data in JSON format, and import the data into the data warehouse; The log generation module includes: An acquisition submodule is used to acquire log data of a surface formation in a designated sea area as unstructured data to be processed, wherein the unstructured data to be processed includes at least basic information of the surface formation, navigation track and position data, environmental parameter data, equipment status data, personnel operation data and task execution data; An extraction submodule, used for parsing and extracting key fields in the unstructured data to be processed by using a custom extractor, wherein the custom extractor is a neural network model trained by log data of a surface formation in a specified sea area; The matching submodule is used to match the key fields according to the preset case log format to obtain the formatted case log.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the formatted data collection method containing a large number of known repeated fields according to any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the formatted data collection method containing a large number of known repeated fields described in any one of claims 1 to 6 is implemented.