Server log file processing method and device

Abstract

The present application provides a method and device for processing server log documents, in which the data storage structure of the row storage in the search engine is disabled, and the method includes: receiving a log write request; extracting non-text type and text type data according to the attribute information of the target log document; in the case of extracting first data of non-text type, writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field; in the case of extracting second data of text type, establishing a target data index for the second data; writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index. Through the present application, the problem of low storage efficiency of server log documents is solved, and the effect of improving the storage efficiency of server log documents is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a method and apparatus for processing server log documents. Background Art

[0002] With the continuous development of server technology and industry, the use of search engines to store and analyze server log data is gaining increasing attention. Currently, search engines have designed three data storage structures to meet diverse application scenarios. These structures store server log data differently for different search engine functions. However, this storage design can lead to data redundancy during log storage, resulting in low storage efficiency for server log documents. Summary of the Invention

[0003] The embodiments of the present application provide a method and apparatus for processing server log documents, so as to at least solve the problem of low storage efficiency of server log documents in the related art.

[0004] According to one embodiment of the present application, a method for processing server log documents is provided, which is applied to a search engine deployed on a server, wherein the search engine is used to store log documents of server logs for the server, and the data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled, and the row storage data storage structure is disabled. The method comprises: receiving a log write request initiated by the server, wherein the log write request is used to request that a target log document be written to the search engine, and the target log document is a log document corresponding to a target server log generated on the server; in response to the log write request, extracting non-text type data and text type data from the target log document according to attribute information of the target log document; data, wherein the attribute information is used to indicate the data type of the document data included in the target log document; in the case of extracting the first data of the non-text type, the first data field included in the first data is written into the data storage structure of the column storage and the first mapping relationship between the target log document and the first data field is stored; in the case of extracting the second data of the text type, a target data index is established for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after the second data is segmented and the target log document; the second data field included in the second data is written into the data storage structure of the column storage, the second mapping relationship between the target log document and the second data field is stored, and the target data index is stored.

[0005] In an exemplary embodiment, the extracting non-text type data and text type data from the target log document according to the attribute information of the target log document includes: extracting a mapping configuration corresponding to the target log document as the attribute information from the search engine, wherein the mapping configuration is used to indicate the field attributes of each field in the document to be stored by the search engine; using the mapping configuration to divide the target field included in the target log document into a first field belonging to the non-text type and a second field belonging to the text type; extracting the first data from the position of the first field in the target log document, and extracting the second data from the position of the second field in the target log document.

[0006] In an exemplary embodiment, the search engine includes a first coordination node and one or more sharding nodes, and the receiving of the log write request initiated by the server includes: the first coordination node receiving the log write request from the transmission port of the server; the first coordination node generating a document name of the target log document; the first coordination node searching for the primary sharding node of the target log document from the multiple sharding nodes based on the document name and the storage settings of the search engine, wherein the storage settings are used to indicate the storage status of the server log stored in the search engine, and the primary sharding node is used to store the target log document; the first coordination node sending the log write request to the primary sharding node; and the primary sharding node receiving the log write request.

[0007] In an exemplary embodiment, after writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, the method also includes: receiving a log query request initiated by the server, wherein the log query request is used to request to find document data that meets the query conditions from the target log document; extracting target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested by the log query request; searching and returning target document data that meets the query conditions based on the target keywords, the result requirement type, the first mapping relationship, the second mapping relationship, and the target data index.

[0008] In an exemplary embodiment, the method of searching and returning the target document data according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship and the target data index includes: searching for a reference log document matching the target keyword from the target log document according to the target data index; extracting the first data field corresponding to the reference log document from the data storage structure of the column storage according to the first mapping relationship corresponding to the reference log document and extracting the second data field corresponding to the reference log document from the data storage structure of the column storage according to the second mapping relationship corresponding to the reference log document; in the case where the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, integrating the extracted data into the data storage structure of the column storage. The first data field corresponding to the reference log document and the second data field corresponding to the extracted reference log document are used to obtain third data that meets the query condition, and the third data is sent to the server, wherein the target document data includes the third data; in the case where the result requirement type is used to indicate that the log query request requests to obtain part of the fields of the document data, the target field attribute in the query condition is detected, and the data field whose field attribute matches the target field attribute is filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, to obtain fourth data that meets the query condition, and the fourth data is sent to the server, wherein the target document data includes the fourth data.

[0009] In an exemplary embodiment, the method further includes: in the event that the word segmentation rules of the search engine change, creating a reference data index having the same index attributes as the target data index, wherein the word segmentation rules are used to indicate how the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rules, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; and deleting the target data index when the reference data index update is completed.

[0010] In an exemplary embodiment, updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule includes: creating an index alias for the target data index and setting the read-write status of the target data index to a read-only state, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field with reference to the changed word segmentation rule and recording the third mapping relationship between each reference keyword and the second data field; updating the reference data index according to the second mapping relationship and the third mapping relationship.

[0011] According to another embodiment of the present application, a device for processing server log documents is provided, which is applied to a search engine deployed on a server, wherein the search engine is used to store log documents of server logs for the server, and the data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled, and the row storage data storage structure is disabled. The device includes: a first receiving module, which is used to receive a log write request initiated by the server, wherein the log write request is used to request that a target log document be written to the search engine, and the target log document is a log document corresponding to a target server log generated on the server; a first extraction module, which is used to respond to the log write request and extract non-text type data and text type data from the target log document according to attribute information of the target log document. , wherein the attribute information is used to indicate the data type of the document data included in the target log document; a first storage module is used to write the first data field included in the first data into the data storage structure of the column storage and store the first mapping relationship between the target log document and the first data field when the first data of the non-text type is extracted; a second storage module is used to establish a target data index for the second data when the second data of the text type is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and to write the second data field included in the second data into the data storage structure of the column storage, store the second mapping relationship between the target log document and the second data field, and store the target data index.

[0012] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0013] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0014] Through the present application, since a column-stored data storage structure is enabled in a search engine for storing log documents of server logs for a server while a row-stored data storage structure is disabled, the search engine receives a log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case where non-text type first data is extracted, the first data field included in the first data is written into the column-stored data storage structure and the first mapping relationship between the target log document and the first data field is stored, and in the case where text type second data is extracted, a mapping relationship is established for the second data for indicating that the second data has been segmented. The target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are all stored, which fully records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a hardware structure block diagram of a server device for a method for processing server log documents according to an embodiment of the present application;

[0016] Figure 2 is a flowchart of a method for processing a server log document according to an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of a data storage structure according to an embodiment of the present application;

[0018] Figure 4 This is a flow chart of a log file writing method according to an embodiment of the present application;

[0019] Figure 5 This is a flow chart of a log query method according to an embodiment of the present application;

[0020] Figure 6 This is a structural block diagram of a server log document processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure diagram of a server device for a method of processing server log documents according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 The server device includes a processor 102 (only one of which is shown) (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. The server device may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing server log documents in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communication provider of a server device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] In this embodiment, a method for processing server log documents is provided, which is applied to a search engine deployed on a server. The search engine is used to store log documents of the server log for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled, and the row storage data storage structure is disabled. Figure 2 Flowchart of a method for processing a server log document according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0027] Step S202: receiving a log writing request initiated by the server, wherein the log writing request is used to request writing a target log document into the search engine, the target log document being a log document corresponding to a target server log generated on the server;

[0028] Step S204, in response to the log write request, extracting non-text type data and text type data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document;

[0029] Step S206 , when first data of a non-text type is extracted, writing a first data field included in the first data into a data storage structure of a column store and storing a first mapping relationship between the target log document and the first data field;

[0030] Step S208: When second data of text type is extracted, a target data index is established for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; the second data field included in the second data is written into a data storage structure stored in a column, the second mapping relationship between the target log document and the second data field is stored, and the target data index is stored.

[0031] Through the above steps, since the data storage structure of column storage is enabled in the search engine for storing the log document of the server log for the server and the data storage structure of row storage is disabled, the search engine receives the log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case of extracting the first data of non-text type, writes the first data field included in the first data into the data storage structure of column storage and stores the first mapping relationship between the target log document and the first data field, and in the case of extracting the second data of text type, establishes a method for indicating that the second data has been segmented for the second data. Then, the target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage, and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are all stored, which fully records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved.

[0032] Optionally, in an embodiment of the present application, the search engine deployed on the server is used to store log documents of the server log for the server, and at the same time facilitate query and screening of log documents. The search engine deployed on the server can be ES (Elasticsearch, a distributed, RESTful-style search and analysis engine). In order to meet a variety of application scenarios, ES has designed three data storage structures (Store Filed, Doc Values, _Source) in terms of data storage. _Source is mainly used to store the original JSON documents submitted to ES, Doc Value is mainly used to provide ES with grouping, aggregation, and sorting capabilities. Store Filed is mainly used to store a single field of the original JSON document submitted to ES. Among them, Source and Store Filed both belong to row-stored data storage structures, and DocValue belongs to column-stored data storage structures.

[0033] Optionally, in an embodiment of the present application, the server log of the server includes but is not limited to various logs such as normal operation logs, fault logs, and test logs.

[0034] Optionally, in an embodiment of the present application, the log document of the server log may be, but is not limited to, set to have a fixed file format.

[0035] Optionally, in the embodiment of the present application, Figure 3 Schematic diagram of a data storage structure according to an embodiment of the present application. Figure 3 As shown, the left side is the row storage data structure, and the right side is the column storage data structure. The row storage data structure shown on the left can be used to store the original JSON document and a single field of the original JSON document. The column storage data structure shown on the right is very suitable for data warehouse and big data analysis applications because it only reads and processes the column data actually required in the query instead of the entire row data when processing aggregation, filtering and connection operations of large amounts of data, thereby reducing I / O operations and memory usage, and achieving better performance.

[0036] Optionally, in an embodiment of the present application, enabling the data storage structure of column storage in the search engine and disabling the data storage structure of row storage includes but is not limited to modifying the original data storage method in ES. After the index is created, removing the Source field and Store Filed field writing logic, retaining the original data storage method of Doc Values, modifying the original data storage logic of DocValues, and adding the Doc Values ​​storage method of the text attribute field according to the field type (whether it is a word segmentation field).

[0037] In the embodiment provided in step S202, the log write request is used to request that the target log document be written to the search engine. The target log document is the log document corresponding to the target server log generated on the server, or it may be the corresponding log document generated by the server based on the faulty server log received on other faulty servers. This application does not make specific limitations on this.

[0038] In the embodiment provided in step S204, the attribute information of the log document is used to indicate the data type of the document data included in the target log document, and may be, but is not limited to, a mapping configuration for indicating the field attributes of each field in the document to be stored by the search engine.

[0039] Optionally, in an embodiment of the present application, the extracting of non-text type data and text type data from the target log document according to the attribute information of the target log document includes but is not limited to querying the log type of the target log document according to the attribute information; querying the log format corresponding to the log type from the server, wherein the log format is used to indicate the meaning of the fields stored in different storage locations in the target log document and the field data types corresponding to the fields with different meanings; searching for the first storage location corresponding to the field whose field data type is the non-text type according to the log format, and searching for the second storage location corresponding to the field whose field data type is the text type according to the log format; extracting the non-text type data from the first storage location, and extracting the text type data from the second storage location. Through the above steps, text type data and non-text type data are accurately extracted from the log document, laying a good foundation for using different data storage methods for different types of data in the future, avoiding the search engine from being unable to be used normally due to data being stored in the search engine in an inappropriate manner, and improving the storage efficiency of the server log document to a certain extent.

[0040] In the embodiment provided in step S206, writing the first data field included in the first data into the data storage structure of the column storage includes dividing the storage space in the data storage structure of the column storage into a first storage space and a second storage space, writing the first data field into the first storage space, and the second storage space is used to store text type data.

[0041] Optionally, in an embodiment of the present application, storing the first mapping relationship between the target log document and the first data field includes but is not limited to generating a feature code of the target log document according to a feature code generation algorithm provided by the server, and storing the mapping relationship between the feature code of the target log document and the first data field as the first mapping relationship, wherein the feature codes of the log documents stored in the search engine are different.

[0042] In the embodiment provided in step S208, establishing a target data index for the second data includes but is not limited to segmenting the extracted second data with reference to the segmentation rules to obtain one or more keywords, and establishing a mapping relationship between the keywords and the target log document as the target data index.

[0043] Optionally, in an embodiment of the present application, writing the second data field included in the second data into the data storage structure of the column storage includes but is not limited to calculating the data ratio of the first data and the second data after completing the extraction operation of all the first data and the second data in the target log document, allocating a first data space for the first data and a second data space for the second data from the storage space corresponding to the data storage structure of the column storage with reference to the data ratio and the data volume of the target log document, writing the second data field included in the second data into the second data space, and the first data space is used to store the first data field included in the first data.

[0044] As an optional implementation, non-text type data and text type data are extracted from the target log document respectively according to the attribute information of the target log document, including: extracting the mapping configuration corresponding to the target log document as attribute information from the search engine, wherein the mapping configuration is used to indicate the field attributes of each field in the document to be stored by the search engine; using the mapping configuration to divide the target field included in the target log document into a first field belonging to the non-text type and a second field belonging to the text type; extracting the first data from the position of the first field in the target log document, and extracting the second data from the position of the second field in the target log document.

[0045] Optionally, in an embodiment of the present application, the mapping configuration corresponding to each log document is recorded in the search engine, that is, the field attributes of each field in each log document to be stored by the search engine are recorded. When the field attributes of the field are determined, it can be determined whether the corresponding field is a non-text type or a text type, and then text type data can be extracted at the position of the text type field, and non-text type data can be extracted at the position of the non-text type field.

[0046] As an optional implementation, the search engine includes a first coordination node and one or more sharding nodes, and receives a log write request initiated by a server, including: the first coordination node receives the log write request from the transmission port of the server; the first coordination node generates a document name of the target log document; the first coordination node searches for a primary sharding node of the target log document from multiple sharding nodes based on the document name and the storage settings of the search engine, wherein the storage settings are used to indicate the storage status of the server log stored in the search engine, and the primary sharding node is used to store the target log document; the first coordination node sends the log write request to the primary sharding node; and the primary sharding node receives the log write request.

[0047] Optionally, in an embodiment of the present application, a search engine may be configured with, but not limited to, multiple engine nodes, including a coordination node and one or more sharding nodes, wherein the coordination node is configured to receive log write requests from the server, and determine which sharding node to distribute the log write request to for further execution based on setting information such as the number of sharding nodes and the log storage conditions on each sharding node, and send the log write request to the primary sharding node after determining the corresponding primary sharding node.

[0048] Optionally, in an embodiment of the present application, after the primary sharding node receives the log write request, the primary sharding node responds to the log write request, and the primary sharding node extracts non-text type data and text type data from the target log document according to the attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document; in the case of extracting first data of non-text type, the primary sharding node writes the first data field included in the first data into the data storage structure of the column storage and stores the first mapping relationship between the target log document and the first data field; in the case of extracting second data of text type, the primary sharding node establishes a target data index for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; the primary sharding node writes the second data field included in the second data into the data storage structure of the column storage, stores the second mapping relationship between the target log document and the second data field, and stores the target data index.

[0049] Through the above steps, multiple shard nodes are arranged in the search engine, which can enable processing of multiple log write requests at the same time, thereby improving the processing efficiency of log write requests, that is, improving the storage efficiency of server log documents.

[0050] Optionally, in the embodiment of the present application, Figure 4 This is a flow chart of a log file writing method according to an embodiment of the present application. Figure 4 As shown, the method includes the following steps:

[0051] Step S401: Modify the original data storage method. After the index is created, remove the Source field and StoreField field writing logic, retain the original Doc Values ​​data storage method, modify the original Doc Values ​​data storage logic, and add the Doc Values ​​storage method for the text attribute field based on the field type (whether it is a word segmentation field).

[0052] Step S402: Receive a data write request (i.e., a log write request) from the client (i.e., the server), generate a document ID, determine the primary shard node to which the document should be written based on the document ID and index settings (e.g., the number of shards), and forward the request to the primary shard node.

[0053] In step S403, the primary sharding node receives the request, parses the document JSON data according to the received document JSON data and the defined mapping relationship, performs word segmentation on the fields that need to be segmented using Lucence (an open source full-text search engine toolkit), creates an inverted index (i.e., the target data index), and records TransLog (a transaction log that records persistent information about index operations).

[0054] Step S404: Determine based on the field data type (whether it is a word segmentation field). If it is a non-text attribute field, directly write the data to Doc Values ​​(i.e., the data storage structure of column storage). If it is a text field attribute, write the original field data to Doc Values ​​storage after the inverted index is created.

[0055] Step S405: Create a mapping relationship between the document ID and the fields in the Doc Values ​​storage (i.e., a first mapping relationship and a second mapping relationship), and record Translog information;

[0056] Step S406: synchronize the shard data to the shard replica data node, write the shard data, and return the response result to the client.

[0057] As an optional implementation, after writing the second data field included in the second data into a data storage structure stored in a column, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, the method further includes: receiving a log query request initiated by the server, wherein the log query request is used to request to find document data that meets the query conditions from the target log document; extracting target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested by the log query request; searching and returning the target document data that meets the query conditions based on the target keywords, result requirement type, first mapping relationship, second mapping relationship, and target data index.

[0058] Optionally, in an embodiment of the present application, generally speaking, the time for completing the operation of writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field will be earlier than the time for completing the operation of writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index. Therefore, it can be considered that after writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, it is also after writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field.

[0059] Through the above steps, document data query is achieved in a storage mode in which all document data is stored in a column-based data storage structure, thereby improving the feasibility of storing all document data in a column-based data storage structure.

[0060] As an optional implementation, the target document data is searched and returned according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship and the target data index, including: searching for a reference log document matching the target keyword from the target log document according to the target data index; extracting the first data field corresponding to the reference log document from the data storage structure stored in the column according to the first mapping relationship corresponding to the reference log document and extracting the second data field corresponding to the reference log document from the data storage structure stored in the column according to the second mapping relationship corresponding to the reference log document; in the case where the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, integrating the extracted reference data into the data storage structure stored in the column according to the first mapping relationship corresponding to the reference log document. The first data field corresponding to the log document and the second data field corresponding to the extracted reference log document are examined to obtain third data that meets the query condition, and the third data is sent to the server, wherein the target document data includes the third data; in the case where the result requirement type is used to indicate that the log query request requests to obtain partial fields of the document data, the target field attribute in the query condition is detected, and data fields whose field attributes match the target field attributes are filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document to obtain fourth data that meets the query condition, and the fourth data is sent to the server, wherein the target document data includes the fourth data.

[0061] Through the above steps, full query and custom field query are realized, the query angle of log document data query is expanded, and the query efficiency of log document data is improved.

[0062] Optionally, in the embodiment of the present application, Figure 5 This is a flow chart of a log query method according to an embodiment of the present application. Figure 5 As shown, the method includes the following steps:

[0063] Step S501: Modify the ES (i.e., search engine) source code and transform the query method. The query interface adds the type type (i.e., result requirement type) and the fileds attribute (i.e., target field attribute). The type attribute has two values: all and part. The default value is all. The fileds attribute is in String array format. When the type type is all (the fileds attribute can be empty), all fields of the document are queried from Doc Values ​​(i.e., the data storage structure of column storage) for response. When the type type is part (the fileds attribute cannot be empty), the specified field value is queried from DocValues ​​based on the fileds array field value in the request for response.

[0064] For example, to request all fields in a response, you can use the following method:

[0065]

[0066] Step S502: Receive a client (i.e., server) data query request (i.e., log query request), and the coordination node parses the query statement and determines which shards to query based on the index mapping and settings information.

[0067] Step S503: All document IDs (docId) that meet the query conditions are found through full-text search. The coordination node determines the shard location information of the data based on the document ID and forwards the query request to the data node containing the target shard based on the location information.

[0068] Step S504: After receiving the query request, the data node first parses the request parameters. If the type is all (i.e., obtaining all fields), it queries all the document fields from Doc Values ​​according to DocId and responds. If the type is part (i.e., obtaining some fields), it queries for some fields and then obtains the field attributes to be responded from the fileds array. Then, it queries the specified field values ​​from Doc Values ​​according to DocId and responds.

[0069] In step S505, the data node returns the retrieved data to the coordination node. The coordination node needs to aggregate the results from different shards, sort and paginate the results as needed, and return the results to the client (ie, the server).

[0070] As an optional implementation, the method also includes: when the search engine's word segmentation rules change, creating a reference data index with the same index attributes as the target data index, wherein the word segmentation rules are used to indicate how the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rules, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; and deleting the target data index when the reference data index update is completed.

[0071] Through the above steps, index reconstruction is achieved in a storage mode in which all document data is stored in a column-based data storage structure, thereby improving the feasibility of storing all document data in a column-based data storage structure.

[0072] As an optional implementation, updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule includes: creating an index alias for the target data index and setting the read-write status of the target data index to a read-only status, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field with reference to the changed word segmentation rule and recording the third mapping relationship between each reference keyword and the second data field; updating the reference data index according to the second mapping relationship and the third mapping relationship.

[0073] Through the above steps, during the index reconstruction process, an index alias is created for the initial index, i.e., the target data index, and the read-write status of the target data index is set to read-only, ensuring that no new index items are written to the initial index during the index reconstruction process, avoiding repeated operations. At the same time, it also ensures that the index can be used normally during the index reconstruction process, avoiding the index reconstruction process affecting the normal use of the search engine.

[0074] As an optional implementation, the present application also provides a method for index reconstruction, which includes the following steps:

[0075] Step S1: Modify the index and rebuild the source code, and change the data reading method from the Source field to reading data from the DocValues ​​storage;

[0076] Step S2, create a new index and copy the mapping and settings from the old index;

[0077] Step S3: Set an alias for the original index and set the original index to read-only mode;

[0078] Step S4: Call the ReIndex operation to query the original data from Doc Values ​​and regenerate the inverted index and sorting fields based on the mapping relationship and word segmentation field settings;

[0079] Step S5: After the new index is updated, the alias of the old index name is added to the new index, and the old index and the copy are deleted.

[0080] Through the embodiments given in this application, by transforming the data storage method of ES, the Source field and Store Filed field writing logic are removed after the index creation is completed, the original data storage method of Doc Values ​​is retained, the original data storage logic of DocValues ​​is modified, and the DocValues ​​storage method of the text attribute field is added according to the field type (whether it is a word segmentation field). This can reduce the problem of redundant data storage, and the columnar storage method used by Doc Values ​​can improve the data compression ratio. At the same time, the characteristic of Doc Values ​​using system memory for data writing and query cache is utilized, which reduces the use of JVM allocated memory, improves the program running efficiency, and increases the utilization rate of system memory. The query method has been modified, adding the "type" and "fileds" attributes to the query interface. The "type" attribute has two values: "all" and "part," with the default value being "all." The "fileds" attribute is a String array. When "type" is "all" (the "fileds" attribute can be empty), the response is based on a query of all document fields from "Doc Values." When "type" is "part" (the "fileds" attribute cannot be empty), the response is based on a query of the specified field values ​​from "Doc Values" based on the values ​​of the "fileds" array fields in the request. This modification to the query method allows all retrieved data to be output uniformly from "Doc Values." A query no longer requires supplementing text field data from "Source" or "Doc Values," reducing data interaction and improving request response efficiency. The index rebuild source code has been modified. When performing index migration, the original data is retrieved from "Doc Values," and the inverted index and sort fields are regenerated based on the mapping relationship and word segmentation field settings. This resolves the issue of index rebuild functionality being unavailable due to data storage changes. These measures can significantly improve cluster device memory and hard disk utilization, reduce hardware costs, and improve data retrieval efficiency, thereby enhancing enterprise profitability.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0082] In this embodiment, a device for processing server log documents is also provided, which is applied to a search engine deployed on a server. The search engine is used to store log documents of server logs for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the data storage structure of column storage is enabled, and the data storage structure of row storage is disabled. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and those that have been explained will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.

[0083] Figure 6 is a structural block diagram of a server log document processing device according to an embodiment of the present application, such as Figure 6 As shown, the device includes:

[0084] A first receiving module 602 is configured to receive a log writing request initiated by a server, wherein the log writing request is configured to request writing a target log document into a search engine, the target log document being a log document corresponding to a target server log generated on the server;

[0085] A first extraction module 604 is configured to, in response to the log write request, extract non-text data and text data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document;

[0086] A first storage module 606 is configured to, when first data of a non-text type is extracted, write a first data field included in the first data into a column-stored data storage structure and store a first mapping relationship between the target log document and the first data field;

[0087] The second storage module 608 is used to establish a target data index for the second data when text-type second data is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and is used to write the second data field included in the second data into a column-stored data storage structure, store the second mapping relationship between the target log document and the second data field, and store the target data index.

[0088] Through the above steps, since the data storage structure of column storage is enabled in the search engine for storing the log document of the server log for the server and the data storage structure of row storage is disabled, the search engine receives the log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case of extracting the first data of non-text type, writes the first data field included in the first data into the data storage structure of column storage and stores the first mapping relationship between the target log document and the first data field, and in the case of extracting the second data of text type, establishes a method for indicating that the second data has been segmented for the second data. Then, the target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage, and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are all stored, which fully records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved.

[0089] As an optional implementation, the first extraction module includes: a first extraction unit, used to extract the mapping configuration corresponding to the target log document from the search engine as attribute information, wherein the mapping configuration is used to indicate the field attributes of each field in the document to be stored by the search engine; a division unit, used to use the mapping configuration to divide the target field included in the target log document into a first field belonging to a non-text type and a second field belonging to a text type; a second extraction unit, used to extract first data from the position of the first field in the target log document, and to extract second data from the position of the second field in the target log document.

[0090] As an optional embodiment, the search engine includes a first coordination node and one or more sharding nodes, and the first receiving module includes: a first receiving unit, which is used by the first coordination node to receive a log write request from the transmission port of the server; a generating unit, which is used by the first coordination node to generate the document name of the target log document; a first searching unit, which is used by the first coordination node to search for the primary sharding node of the target log document from multiple sharding nodes according to the document name and the storage settings of the search engine, wherein the storage settings are used to indicate the storage status of the server log stored in the search engine, and the primary sharding node is used to store the target log document; a sending unit, which is used by the first coordination node to send the log write request to the primary sharding node; and a second receiving unit, which is used by the primary sharding node to receive the log write request.

[0091] As an optional embodiment, the device also includes: a second receiving module, used to receive a log query request initiated by the server, wherein the log query request is used to request to find document data that meets the query conditions from the target log document; a second extraction module, used to extract target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested by the log query request; a search module, used to search and return target document data that meets the query conditions based on the target keywords, result requirement type, first mapping relationship, second mapping relationship and target data index.

[0092] As an optional embodiment, the search module includes: a second search unit, configured to search for a reference log document matching a target keyword from a target log document according to a target data index; a third extraction unit, configured to extract a first data field corresponding to the reference log document from a column-stored data storage structure according to a first mapping relationship corresponding to the reference log document, and to extract a second data field corresponding to the reference log document from a column-stored data storage structure according to a second mapping relationship corresponding to the reference log document; an integration unit, configured to, when the result requirement type indicates that the log query request requests to obtain all fields of the document data, integrate the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document to obtain third data that meets the query condition, and send the third data to the server, wherein the target document data includes the third data; a detection unit, configured to, when the result requirement type indicates that the log query request requests to obtain some fields of the document data, detect the target field attribute in the query condition, filter out data fields whose field attributes match the target field attributes from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, obtain fourth data that meets the query condition, and send the fourth data to the server, wherein the target document data includes the fourth data.

[0093] As an optional embodiment, the device also includes: a creation module, which is used to create a reference data index with the same index attributes as the target data index when the search engine's word segmentation rules change, wherein the word segmentation rules are used to indicate how the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; an update module, which is used to update the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rules, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; and a deletion module, which is used to delete the target data index when the reference data index update is completed.

[0094] Optionally, the update module includes: a creation unit, used to create an index alias for the target data index and set the read-write status of the target data index to a read-only status, wherein the index alias is used to obtain the target data index after the reference data index is created; a fourth extraction unit, used to extract each reference keyword from the second data field with reference to the changed word segmentation rules and record a third mapping relationship between each reference keyword and the second data field; an update unit, used to update the reference data index according to the second mapping relationship and the third mapping relationship.

[0095] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0096] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0097] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0098] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0101] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0102] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0103] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0104] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0105] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for processing server log files, characterized in that: A method is applied to a search engine deployed on a server, the search engine being used to store log documents of server logs for the server, the data storage structures supported by the search engine including row storage and column storage, the column storage data storage structure being enabled in the search engine and the row storage data storage structure being disabled, the method comprising: receiving a log writing request initiated by the server, wherein the log writing request is used to request writing a target log document into the search engine, and the target log document is a log document corresponding to a target server log generated on the server; In response to the log write request, extracting non-text type data and text type data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document; When the non-text type first data is extracted, writing a first data field included in the first data into the data storage structure of the column store and storing a first mapping relationship between the target log document and the first data field; In the case where second data of the text type is extracted, a target data index is established for the second data, wherein the target data index is used to indicate a mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; a second data field included in the second data is written into the data storage structure of the column storage, a second mapping relationship between the target log document and the second data field is stored, and the target data index is stored.

2. The method according to claim 1, characterized in that The extracting non-text type data and text type data from the target log document according to the attribute information of the target log document includes: Extracting a mapping configuration corresponding to the target log document from the search engine as the attribute information, wherein the mapping configuration is used to indicate field attributes of each field in the document to be stored by the search engine; Using the mapping configuration, divide the target field included in the target log document into a first field belonging to the non-text type and a second field belonging to the text type; The first data is extracted from the position of the first field in the target log document, and the second data is extracted from the position of the second field in the target log document.

3. The method according to claim 1, characterized in that The search engine includes a first coordination node and one or more shard nodes. The receiving the log writing request initiated by the server includes: The first coordinating node receives the log write request from the transmission port of the server; The first coordinating node generates a file name for the target log file; The first coordinating node searches for a primary sharding node for the target log document from the plurality of sharding nodes based on the document name and the storage setting of the search engine, wherein the storage setting is used to indicate a storage status of a storage server log in the search engine, and the primary sharding node is used to store the target log document; The first coordinating node sends the log write request to the primary sharding node; The primary shard node receives the log write request.

4. The method according to claim 1, wherein After writing the second data field included in the second data into the data storage structure of the column store, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, the method further includes: receiving a log query request initiated by the server, wherein the log query request is used to request to search for document data that meets a query condition from the target log document; Extracting target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate a field condition of the document data requested to be obtained by the log query request; Target document data that meets the query condition is searched and returned according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship, and the target data index.

5. The method according to claim 4, characterized in that The searching and returning the target document data according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship, and the target data index includes: Searching the target log document for a reference log document that matches the target keyword according to the target data index; extracting a first data field corresponding to the reference log document from the column-stored data storage structure according to the first mapping relationship corresponding to the reference log document, and extracting a second data field corresponding to the reference log document from the column-stored data storage structure according to the second mapping relationship corresponding to the reference log document; In a case where the result requirement type indicates that the log query request requests to obtain all fields of document data, integrating the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document to obtain third data that meets the query condition, and sending the third data to the server, wherein the target document data includes the third data; In a case where the result requirement type is used to indicate that the log query request requests to obtain partial fields of document data, the target field attributes in the query condition are detected, and data fields whose field attributes match the target field attributes are filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, to obtain fourth data that meets the query condition, and send the fourth data to the server, wherein the target document data includes the fourth data.

6. The method according to claim 1, characterized in that The method further comprises: In the event that the word segmentation rules of the search engine are changed, creating a reference data index having the same index attributes as the target data index, wherein the word segmentation rules are used to indicate how the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship, and the changed word segmentation rule, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; When the update of the reference data index is completed, the target data index is deleted.

7. The method according to claim 6, characterized in that The updating of the reference data index according to the second data field, the second mapping relationship, and the changed word segmentation rule includes: Creating an index alias for the target data index and setting the read / write state of the target data index to a read-only state, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field with reference to the changed word segmentation rule and recording a third mapping relationship between each reference keyword and the second data field; The reference data index is updated according to the second mapping relationship and the third mapping relationship.

8. A server log file processing device, characterized in that: A search engine deployed on a server is used to store log documents of server logs for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled, and the row storage data storage structure is disabled. The apparatus comprises: A first receiving module is configured to receive a log writing request initiated by the server, wherein the log writing request is used to request writing a target log document into the search engine, and the target log document is a log document corresponding to a target server log generated on the server; a first extraction module, configured to, in response to the log write request, extract non-text data and text data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate a data type of the document data included in the target log document; a first storage module configured to, when extracting the first data of the non-text type, write a first data field included in the first data into the data storage structure of the column store and store a first mapping relationship between the target log document and the first data field; A second storage module is used to establish a target data index for the second data when the second data of the text type is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and to write the second data field included in the second data into the data storage structure of the column storage, store the second mapping relationship between the target log document and the second data field, and store the target data index.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images