Server log document processing method and device

Abstract

The invention provides a server log document processing method and device, a data storage structure of row storage in a search engine is forbidden, and the method comprises the following steps: receiving a log write-in request; extracting data of a non-text type and a text type according to the attribute information of the target log document; under the condition that first data of a non-text type is extracted, writing a first data field included in the first data into a data storage structure of column storage, and storing a first mapping relationship between the target log document and the first data field; under the condition that the second data of the text type is extracted, establishing a target data index for the second data; and writing a second data field included in the second data into a data storage structure of the column storage, storing a second mapping relationship between the target log document and the second data field, and storing the target data index. By means of the method and device, the problem that the storage efficiency of the server log document is low is solved, and the effect of improving the storage efficiency of the server log document is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the computer field, and in particular, to a method and device for processing server log documents. Background Art

[0002] With the continuous development of server technology and industry, people are paying more and more attention to the method of using search engines to store and analyze server log data. At present, in order to meet various application scenarios, search engines have designed three data storage structures in terms of data storage. According to the different functions of search engines, different data storage structures store server log data in different ways. However, this storage design will lead to certain data redundancy in the process of log data storage, and the storage efficiency of server log documents is low. Summary of the invention

[0003] The embodiments of the present application provide a method and device for processing server log documents, so as to at least solve the problem of low storage efficiency of server log documents in the related art.

[0004] According to an embodiment of the present application, a method for processing server log documents is provided, which is applied to a search engine deployed on a server, wherein the search engine is used to store log documents of server logs for the server, and the data storage structures supported by the search engine include row storage and column storage. In the search engine, the data storage structure of the column storage is enabled, and the data storage structure of the row storage is disabled. The method comprises: receiving a log write request initiated by the server, wherein the log write request is used to request that a target log document be written to the search engine, and the target log document is a log document corresponding to a target server log generated on the server; in response to the log write request, extracting non-text type data and text type data from the target log document according to attribute information of the target log document; data, wherein the attribute information is used to indicate the data type of the document data included in the target log document; in the case of extracting the first data of the non-text type, writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field; in the case of extracting the second data of the text type, establishing a target data index for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index.

[0005] In an exemplary embodiment, the extracting non-text type data and text type data from the target log document respectively according to the attribute information of the target log document includes: extracting a mapping configuration corresponding to the target log document from the search engine as the attribute information, wherein the mapping configuration is used to indicate the field attributes of each field in the document to be stored by the search engine; using the mapping configuration to divide the target field included in the target log document into a first field belonging to the non-text type and a second field belonging to the text type; extracting the first data from the position of the first field in the target log document, and extracting the second data from the position of the second field in the target log document.

[0006] In an exemplary embodiment, the search engine includes a first coordination node and one or more shard nodes, and the receiving of the log write request initiated by the server includes: the first coordination node receives the log write request from the transmission port of the server; the first coordination node generates a document name of the target log document; the first coordination node searches for the primary shard node of the target log document from the multiple shard nodes according to the document name and the storage setting of the search engine, wherein the storage setting is used to indicate the storage status of the server log stored in the search engine, and the primary shard node is used to store the target log document; the first coordination node sends the log write request to the primary shard node; and the primary shard node receives the log write request.

[0007] In an exemplary embodiment, after writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, the method also includes: receiving a log query request initiated by the server, wherein the log query request is used to request to find document data that meets the query conditions from the target log document; extracting target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested to be obtained by the log query request; searching and returning the target document data that meets the query conditions according to the target keywords, the result requirement type, the first mapping relationship, the second mapping relationship, and the target data index.

[0008] In an exemplary embodiment, the method of searching and returning the target document data according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship and the target data index includes: searching for a reference log document matching the target keyword from the target log document according to the target data index; extracting a first data field corresponding to the reference log document from the data storage structure stored in the column according to the first mapping relationship corresponding to the reference log document and extracting a second data field corresponding to the reference log document from the data storage structure stored in the column according to the second mapping relationship corresponding to the reference log document; and, in the case where the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, integrating the extracted data into the data storage structure; The first data field corresponding to the reference log document and the second data field corresponding to the extracted reference log document are used to obtain third data that meets the query condition, and the third data is sent to the server, wherein the target document data includes the third data; in the case where the result requirement type is used to indicate that the log query request requests to obtain part of the fields of the document data, the target field attribute in the query condition is detected, and the data field whose field attribute matches the target field attribute is filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, to obtain fourth data that meets the query condition, and the fourth data is sent to the server, wherein the target document data includes the fourth data.

[0009] In an exemplary embodiment, the method further includes: in the event that the word segmentation rule of the search engine changes, creating a reference data index having the same index attributes as the target data index, wherein the word segmentation rule is used to indicate the manner in which the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; and deleting the target data index when the update of the reference data index is completed.

[0010] In an exemplary embodiment, updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule includes: creating an index alias for the target data index and setting the read-write status of the target data index to a read-only status, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field with reference to the changed word segmentation rule and recording the third mapping relationship between each reference keyword and the second data field; and updating the reference data index according to the second mapping relationship and the third mapping relationship.

[0011] According to another embodiment of the present application, a device for processing server log documents is provided, which is applied to a search engine deployed on a server, wherein the search engine is used to store log documents of server logs for the server, and the data storage structures supported by the search engine include row storage and column storage. In the search engine, the data storage structure of the column storage is enabled, and the data storage structure of the row storage is disabled. The device includes: a first receiving module, used to receive a log write request initiated by the server, wherein the log write request is used to request that a target log document be written to the search engine, and the target log document is a log document corresponding to a target server log generated on the server; and a first extraction module, used to respond to the log write request and extract non-text type data and text type data from the target log document according to attribute information of the target log document. , wherein the attribute information is used to indicate the data type of the document data included in the target log document; a first storage module is used to write the first data field included in the first data into the data storage structure of the column storage and store the first mapping relationship between the target log document and the first data field when the first data of the non-text type is extracted; a second storage module is used to establish a target data index for the second data when the second data of the text type is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and to write the second data field included in the second data into the data storage structure of the column storage, store the second mapping relationship between the target log document and the second data field, and store the target data index.

[0012] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0013] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0014] Through the present application, since a data storage structure of column storage is enabled and a data storage structure of row storage is disabled in a search engine for storing log documents of server logs for a server, the search engine receives a log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case where non-text type first data is extracted, a first data field included in the first data is written into the data storage structure of column storage and a first mapping relationship between the target log document and the first data field is stored, and in the case where text type second data is extracted, a mapping relationship is established for the second data for indicating that the second data has been segmented. The target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage, and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are stored, which completely records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a hardware structure block diagram of a server device of a method for processing server log documents according to an embodiment of the present application;

[0016] Figure 2 is a flowchart of a method for processing a server log document according to an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of a data storage structure according to an embodiment of the present application;

[0018] Figure 4 is a flow chart of a log document writing method according to an embodiment of the present application;

[0019] Figure 5 is a flow chart of a log query method according to an embodiment of the present application;

[0020] Figure 6 It is a structural block diagram of a server log document processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a method for processing a server log document according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 The server device includes a processor 102 (only one is shown) (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. A person skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing server log documents in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] In this embodiment, a method for processing a server log document is provided, which is applied to a search engine deployed on a server. The search engine is used to store log documents of the server log for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled, and the row storage data storage structure is disabled. Figure 2 is a flow chart of a method for processing a server log document according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0027] Step S202, receiving a log writing request initiated by the server, wherein the log writing request is used to request to write a target log document into the search engine, and the target log document is a log document corresponding to a target server log generated on the server;

[0028] Step S204, in response to the log writing request, extracting non-text type data and text type data from the target log document according to the attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document;

[0029] Step S206, when first data of a non-text type is extracted, a first data field included in the first data is written into a data storage structure of column storage and a first mapping relationship between the target log document and the first data field is stored;

[0030] Step S208, when extracting second data of text type, establish a target data index for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; write the second data field included in the second data into a data storage structure stored in a column, store the second mapping relationship between the target log document and the second data field, and store the target data index.

[0031] Through the above steps, since the data storage structure of column storage is enabled and the data storage structure of row storage is disabled in the search engine for storing the log document of the server log, the search engine receives the log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case of extracting the first data of non-text type, writes the first data field included in the first data into the data storage structure of column storage and stores the first mapping relationship between the target log document and the first data field, and in the case of extracting the second data of text type, establishes a mapping relationship for indicating that the second data has been segmented for the second data. The target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage, and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are stored, which completely records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved.

[0032] Optionally, in an embodiment of the present application, the search engine deployed on the server is used to store log documents of the server log for the server, and at the same time facilitate query and screening of log documents. The search engine deployed on the server can be ES (Elasticsearch, a distributed, RESTful-style search and analysis engine). In order to meet a variety of application scenarios, ES has designed three data storage structures (Store Filed, Doc Values, _Source) in terms of data storage. _Source is mainly used to store the original JSON documents submitted to ES, Doc Value is mainly used to provide ES with grouping, aggregation, and sorting capabilities. Store Filed is mainly used to store a single field of the original JSON document submitted to ES. Among them, Source and Store Filed both belong to row storage data storage structures, and DocValue belongs to column storage data storage structures.

[0033] Optionally, in an embodiment of the present application, the server log of the server includes various logs such as, but not limited to, normal operation logs, fault logs, and test logs.

[0034] Optionally, in an embodiment of the present application, the log document of the server log may be, but is not limited to, set to have a fixed file format.

[0035] Optionally, in the embodiments of the present application, Figure 3 Schematic diagram of a data storage structure according to an embodiment of the present application. Figure 3 As shown, the left side is the row storage data structure, and the right side is the column storage data structure. The row storage data structure shown on the left side can be used to store the original JSON document and a single field of the original JSON document, and the column storage data structure shown on the right side is very suitable for data warehouse and big data analysis applications, because when it processes the aggregation, filtering and connection operations of large amounts of data, it only reads and processes the column data actually required in the query, rather than the entire row of data, thereby reducing I / O operations and memory usage, and achieving better performance.

[0036] Optionally, in an embodiment of the present application, the data storage structure of column storage in the search engine is enabled, and the data storage structure of row storage is disabled, including but not limited to modifying the original data storage method in ES. After the index is created, the Source field and the Store Filed field writing logic are removed, the original data storage method of Doc Values ​​is retained, the original data storage logic of DocValues ​​is modified, and according to the field type (whether it is a word segmentation field), the Doc Values ​​storage method of the text attribute field is added.

[0037] In the embodiment provided in step S202, the log write request is used to request that the target log document be written to the search engine. The target log document is a log document corresponding to the target server log generated on the server, and may also be a corresponding log document generated by the server based on the faulty server logs received on other faulty servers. This application does not make specific limitations on this.

[0038] In the embodiment provided in step S204, the attribute information of the log document is used to indicate the data type of the document data included in the target log document, and may be but is not limited to a mapping configuration for indicating the field attributes of each field in the document to be stored by the search engine.

[0039] Optionally, in an embodiment of the present application, the extracting of non-text type data and text type data from the target log document according to the attribute information of the target log document includes but is not limited to querying the log type of the target log document according to the attribute information; querying the log format corresponding to the log type from the server, wherein the log format is used to indicate the meaning of the fields stored in different storage locations in the target log document and the field data types corresponding to the fields with different meanings; searching the first storage location corresponding to the field whose field data type is the non-text type according to the log format, and searching the second storage location corresponding to the field whose field data type is the text type according to the log format; extracting the non-text type data from the first storage location, and extracting the text type data from the second storage location. Through the above steps, text type data and non-text type data are accurately extracted from the log document, laying a good foundation for adopting different data storage methods for different types of data in the future, avoiding the search engine from being unable to be used normally due to the data being stored in the search engine in an inappropriate manner, and improving the storage efficiency of the server log document to a certain extent.

[0040] In the embodiment provided in step S206, writing the first data field included in the first data into the data storage structure of column storage includes dividing the storage space in the data storage structure of column storage into a first storage space and a second storage space, writing the first data field into the first storage space, and the second storage space is used to store text type data.

[0041] Optionally, in an embodiment of the present application, storing the first mapping relationship between the target log document and the first data field includes but is not limited to generating a feature code of the target log document according to a feature code generation algorithm provided by the server, and storing the mapping relationship between the feature code of the target log document and the first data field as the first mapping relationship, wherein the feature codes of the log documents stored in the search engine are different.

[0042] In the embodiment provided in step S208, establishing a target data index for the second data includes but is not limited to segmenting the extracted second data with reference to the segmentation rules to obtain one or more keywords, and establishing a mapping relationship between the keywords and the target log document as the target data index.

[0043] Optionally, in an embodiment of the present application, writing the second data field included in the second data into the data storage structure of the column storage includes but is not limited to calculating the data ratio of the first data and the second data after completing the extraction operation of all the first data and the second data in the target log document, allocating a first data space for the first data and a second data space for the second data from the storage space corresponding to the data storage structure of the column storage with reference to the data ratio and the data volume of the target log document, writing the second data field included in the second data into the second data space, and the first data space is used to store the first data field included in the first data.

[0044] As an optional implementation, non-text type data and text type data are extracted from the target log document respectively according to the attribute information of the target log document, including: extracting a mapping configuration corresponding to the target log document as attribute information from a search engine, wherein the mapping configuration is used to indicate the field attributes of each field in the document to be stored by the search engine; using the mapping configuration to divide the target field included in the target log document into a first field belonging to a non-text type and a second field belonging to a text type; extracting first data from the position of the first field in the target log document, and extracting second data from the position of the second field in the target log document.

[0045] Optionally, in an embodiment of the present application, a mapping configuration corresponding to each log document is recorded in the search engine, that is, the field attributes of each field in each log document to be stored by the search engine are recorded. When the field attributes of the field are determined, it can be determined whether the corresponding field is a non-text type or a text type, and then text type data can be extracted at the position of the text type field, and non-text type data can be extracted at the position of the non-text type field.

[0046] As an optional implementation, the search engine includes a first coordination node and one or more shard nodes, and receives a log write request initiated by a server, including: the first coordination node receives the log write request from the transmission port of the server; the first coordination node generates a document name of a target log document; the first coordination node searches for a primary shard node of the target log document from multiple shard nodes according to the document name and the storage settings of the search engine, wherein the storage settings are used to indicate the storage status of the server log stored in the search engine, and the primary shard node is used to store the target log document; the first coordination node sends the log write request to the primary shard node; and the primary shard node receives the log write request.

[0047] Optionally, in an embodiment of the present application, a search engine may be configured with, but not limited to, multiple engine nodes, including a coordination node and one or more shard nodes, wherein the coordination node is used to receive a log write request from the server, and determine which shard node to distribute the log write request to for further execution based on setting information such as the number of shard nodes and the conditions for storing logs on each shard node, and after determining the corresponding primary shard node, send the log write request to the primary shard node.

[0048] Optionally, in an embodiment of the present application, after the primary sharding node receives the log write request, the primary sharding node responds to the log write request, and the primary sharding node extracts non-text type data and text type data from the target log document according to the attribute information of the target log document, respectively, wherein the attribute information is used to indicate the data type of the document data included in the target log document; in the case of extracting non-text type first data, the primary sharding node writes the first data field included in the first data into a data storage structure stored in a column and stores a first mapping relationship between the target log document and the first data field; in the case of extracting text type second data, the primary sharding node establishes a target data index for the second data, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; the primary sharding node writes the second data field included in the second data into a data storage structure stored in a column, stores the second mapping relationship between the target log document and the second data field, and stores the target data index.

[0049] Through the above steps, multiple shard nodes are arranged in the search engine, which can make it possible to process multiple log writing requests at the same time, thereby improving the processing efficiency of log writing requests, that is, improving the storage efficiency of server log documents.

[0050] Optionally, in the embodiments of the present application, Figure 4 1 is a flow chart of a log file writing method according to an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:

[0051] Step S401, modify the original data storage method. When the index is created, remove the Source field and StoreField field writing logic, retain the original data storage method of Doc Values, modify the original data storage logic of Doc Values, and add the Doc Values ​​storage method of the text attribute field according to the field type (whether it is a word segmentation field);

[0052] Step S402, receiving a data write request (i.e., a log write request) from a Client (i.e., a server), generating a document ID, determining the primary shard node to which the document should be written according to the document ID and index settings (e.g., the number of shards), and forwarding the request to the primary shard node;

[0053] Step S403: The primary sharding node receives the request, parses the document JSON data according to the received document JSON data and the defined Mapping relationship, performs word segmentation on the fields that need to be segmented using Lucence (an open source full-text search engine toolkit), creates an inverted index (i.e., the target data index), and records TransLog (Transaction Log, a transaction log used to record persistent information of index operations) information;

[0054] Step S404: determine whether it is a word segmentation field according to the field data type. If it is a non-text attribute field, directly write the data into Doc Values ​​(i.e., the data storage structure of column storage). If it is a text field attribute, write the original field data into Doc Values ​​storage after the inverted index is created.

[0055] Step S405, creating a mapping relationship between the document id and the fields in the Doc Values ​​storage (i.e., a first mapping relationship and a second mapping relationship), and recording Translog information;

[0056] Step S406, synchronize the shard data to the shard replica data node, write the shard data, and return the response result to the Client.

[0057] As an optional implementation, after writing the second data field included in the second data into a data storage structure stored in a column, storing a second mapping relationship between the target log document and the second data field, and storing a target data index, the method further includes: receiving a log query request initiated by a server, wherein the log query request is used to request to search for document data that meets the query conditions from the target log document; extracting target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested to be obtained by the log query request; searching for and returning target document data that meets the query conditions based on the target keywords, result requirement type, the first mapping relationship, the second mapping relationship, and the target data index.

[0058] Optionally, in an embodiment of the present application, generally speaking, the time for completing the operation of writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field will be earlier than the time for completing the operation of writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index. Therefore, it can be considered that after writing the second data field included in the second data into the data storage structure of the column storage, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, it is also after writing the first data field included in the first data into the data storage structure of the column storage and storing the first mapping relationship between the target log document and the first data field.

[0059] Through the above steps, document data query is implemented in a storage mode in which all document data is stored in a column-based data storage structure, thereby improving the feasibility of storing all document data in a column-based data storage structure.

[0060] As an optional implementation, searching and returning target document data according to target keywords, result requirement type, first mapping relationship, second mapping relationship and target data index includes: searching for reference log documents matching the target keywords from target log documents according to the target data index; extracting the first data field corresponding to the reference log document from the data storage structure stored in columns according to the first mapping relationship corresponding to the reference log document and extracting the second data field corresponding to the reference log document from the data storage structure stored in columns according to the second mapping relationship corresponding to the reference log document; in the case where the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, integrating the extracted references The first data field corresponding to the reference log document and the second data field corresponding to the extracted reference log document are used to obtain third data that meets the query condition, and the third data is sent to the server, wherein the target document data includes the third data; in the case where the result requirement type is used to indicate that the log query request requests to obtain partial fields of the document data, the target field attribute in the query condition is detected, and data fields whose field attributes match the target field attributes are filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, so as to obtain fourth data that meets the query condition, and the fourth data is sent to the server, wherein the target document data includes the fourth data.

[0061] Through the above steps, full query and custom field query are realized, the query angle of log document data query is expanded, and the query efficiency of log document data is improved.

[0062] Optionally, in the embodiments of the present application, Figure 5 1 is a flow chart of a log query method according to an embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0063] Step S501, modify the ES (i.e., search engine) source code, transform the query method, add the type type (i.e., result requirement type) and fileds attribute (i.e., target field attribute) to the query interface, where the type attribute values ​​are all and part, the default value is all, and the fileds attribute is in String array format. When the type type is all (the fileds attribute can be empty), query all the fields of the document from Doc Values ​​(i.e., the data storage structure of column storage) for response. When the type type is part (the fileds attribute cannot be empty), query the specified field value from DocValues ​​according to the fileds array field value in the request for response.

[0064] For example, when requesting all fields of a response, you can use the following method:

[0065]

[0066] Step S502, receiving a data query request (i.e., a log query request) from a client (i.e., a server), the coordination node parses the query statement, and determines which shards need to be queried based on the mapping and settings information of the index;

[0067] Step S503: all document IDs (docld) that meet the query conditions are queried according to the full-text search. The coordination node determines the shard location information of the data according to the document ID, and forwards the query request to the data node containing the target shard according to the location information.

[0068] Step S 504: After receiving the query request, the data node first parses the request parameters. When the type is all (i.e., obtaining all fields), it queries all the fields of the document from Doc Values ​​according to Docld for response. When the type is part (i.e., obtaining some fields), it queries some fields, then obtains the field attributes that need to be responded from the fileds array, and then queries the specified field value from Doc Values ​​according to Docld for response.

[0069] Step S505, the data node returns the retrieved data to the coordination node, and the coordination node needs to aggregate the results from different shards, sort and paginate the results as needed, and return the results to the client (ie, the server).

[0070] As an optional implementation, the method also includes: when the word segmentation rules of the search engine change, creating a reference data index with the same index attributes as the target data index, wherein the word segmentation rules are used to indicate the way in which the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rules, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; and deleting the target data index after the reference data index update is completed.

[0071] Through the above steps, index reconstruction is achieved in a storage mode in which all document data is stored in a column-based data storage structure, thereby improving the feasibility of storing all document data in a column-based data storage structure.

[0072] As an optional implementation, updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule includes: creating an index alias for the target data index and setting the read-write status of the target data index to a read-only status, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field with reference to the changed word segmentation rule and recording the third mapping relationship between each reference keyword and the second data field; and updating the reference data index according to the second mapping relationship and the third mapping relationship.

[0073] Through the above steps, during the index reconstruction process, an index alias is created for the initial index, i.e., the target data index, and the read-write status of the target data index is set to read-only, ensuring that no new index items are written to the initial index during the index reconstruction process to avoid repeated operations. At the same time, it also ensures that the index can be used normally during the index reconstruction process to avoid the index reconstruction process affecting the normal use of the search engine.

[0074] As an optional implementation, the present application also provides a method for index reconstruction, which includes the following steps:

[0075] Step S1, modify the index and rebuild the source code, and change the data reading method from the Source field to reading data from the DocValues ​​storage;

[0076] Step S2, create a new index and copy the mapping and settings from the old index;

[0077] Step S3, setting an alias for the original index and setting the original index to read-only mode;

[0078] Step S4, calling the ReIndex operation, querying the original data from Doc Values, and regenerating the inverted index and sorting field according to the mapping relationship and the word segmentation field settings;

[0079] Step S5: After the new index is updated, the alias of the old index name is added to the new index, and the old index and the copy are deleted.

[0080] Through the embodiments given in this application, by transforming the data storage method of ES, the Source field and Store Field writing logic are removed after the index is created, the original data storage method of Doc Values ​​is retained, the original data storage logic of DocValues ​​is modified, and the DocValues ​​storage method of the text attribute field is added according to the field type (whether it is a word segmentation field). This can reduce the problem of redundant data storage, and improve the data compression ratio with the help of the columnar storage method used by Doc Values. At the same time, the characteristic of Doc Values ​​using system memory for data writing and query cache is utilized, which reduces the use of JVM allocated memory, can improve the program running efficiency, and increase the utilization rate of system memory. The query method is modified. The query interface adds type and fileds attributes. The type attribute values ​​are all and part. The default value is all. The fileds attribute is in String array format. When the type is all (the fileds attribute can be empty), all the fields of the document are queried from Doc Values ​​for response. When the type is part (the fileds attribute cannot be empty), the specified field value is queried from Doc Values ​​according to the fileds array field value in the request for response. By modifying the query method, all retrieved data can be uniformly output from Doc Values. A query no longer needs to supplement text field data through Source or Doc Values, which reduces data interaction and improves request response efficiency. Modify the index reconstruction source code. When performing index migration, query the original data from Doc Values, regenerate the inverted index and sorting field according to the mapping relationship and word segmentation field settings, and solve the problem of unavailable index reconstruction function caused by data storage changes. The above measures can greatly improve the utilization rate of cluster device memory and hard disk, reduce hardware costs, and improve data retrieval efficiency and enterprise benefits.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0082] In this embodiment, a server log document processing device is also provided, which is applied to a search engine deployed on a server. The search engine is used to store log documents of server logs for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the data storage structure of column storage is enabled, and the data storage structure of row storage is disabled. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0083] Figure 6 is a structural block diagram of a server log document processing device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0084] A first receiving module 602 is used to receive a log writing request initiated by a server, wherein the log writing request is used to request to write a target log document into a search engine, and the target log document is a log document corresponding to a target server log generated on the server;

[0085] A first extraction module 604, configured to respond to the log writing request and extract non-text type data and text type data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document;

[0086] A first storage module 606 is used to write a first data field included in the first data into a data storage structure stored in a column and store a first mapping relationship between the target log document and the first data field when first data of a non-text type is extracted;

[0087] The second storage module 608 is used to establish a target data index for the second data when text-type second data is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and is used to write the second data field included in the second data into a column-stored data storage structure, store the second mapping relationship between the target log document and the second data field, and store the target data index.

[0088] Through the above steps, since the data storage structure of column storage is enabled and the data storage structure of row storage is disabled in the search engine for the log document storing the server log for the server, the search engine receives the log write request initiated by the server, and extracts non-text type data and text type data from the target log document according to the attribute information of the target log document requested to be written to the search engine by the log write request, and in the case of extracting the first data of non-text type, writes the first data field included in the first data into the data storage structure of column storage and stores the first mapping relationship between the target log document and the first data field, and in the case of extracting the second data of text type, establishes a mapping relationship for indicating that the second data has been segmented for the second data. The target data index of the mapping relationship between each keyword and the target log document is obtained, the second data field included in the second data is written into the data storage structure of the column storage, and the second mapping relationship between the target log document and the second data field and the target data index are stored. Through the above method, all data fields of the server log document are stored in the storage structure of the column storage, and the first mapping relationship, the second mapping relationship and the target data index are stored, which completely records the information required for the normal use of the search engine, and reduces the storage space occupied while ensuring the normal use of the search engine. Therefore, the problem of low storage efficiency of the server log document can be solved, and the storage efficiency of the server log document can be improved.

[0089] As an optional implementation, the first extraction module includes: a first extraction unit, used to extract a mapping configuration corresponding to a target log document from a search engine as attribute information, wherein the mapping configuration is used to indicate field attributes of each field in a document to be stored by the search engine; a division unit, used to use the mapping configuration to divide a target field included in the target log document into a first field of a non-text type and a second field of a text type; a second extraction unit, used to extract first data from the position of the first field in the target log document, and to extract second data from the position of the second field in the target log document.

[0090] As an optional implementation, the search engine includes a first coordination node and one or more shard nodes, and a first receiving module includes: a first receiving unit, used by the first coordination node to receive a log write request from a transmission port of a server; a generating unit, used by the first coordination node to generate a document name of a target log document; a first searching unit, used by the first coordination node to search for a primary shard node of a target log document from multiple shard nodes according to the document name and a storage setting of the search engine, wherein the storage setting is used to indicate a storage status of a server log stored in the search engine, and the primary shard node is used to store the target log document; a sending unit, used by the first coordination node to send the log write request to the primary shard node; and a second receiving unit, used by the primary shard node to receive the log write request.

[0091] As an optional implementation, the device also includes: a second receiving module, used to receive a log query request initiated by a server, wherein the log query request is used to request to search for document data that meets the query conditions from a target log document; a second extraction module, used to extract target keywords and result requirement types from the query conditions, wherein the result requirement type is used to indicate the field status of the document data requested to be obtained by the log query request; a search module, used to search and return target document data that meets the query conditions based on the target keywords, result requirement type, first mapping relationship, second mapping relationship and target data index.

[0092] As an optional implementation, the search module includes: a second search unit, which is used to search for a reference log document matching a target keyword from a target log document according to a target data index; a third extraction unit, which is used to extract a first data field corresponding to the reference log document from a column-stored data storage structure according to a first mapping relationship corresponding to the reference log document, and extract a second data field corresponding to the reference log document from a column-stored data storage structure according to a second mapping relationship corresponding to the reference log document; an integration unit, which is used to integrate the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document when the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, obtain third data that meets the query condition, and send the third data to a server, wherein the target document data includes the third data; a detection unit, which is used to detect a target field attribute in the query condition when the result requirement type is used to indicate that the log query request requests to obtain some fields of the document data, filter out data fields whose field attributes match the target field attributes from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, obtain fourth data that meets the query condition, and send the fourth data to a server, wherein the target document data includes the fourth data.

[0093] As an optional implementation, the device also includes: a creation module, which is used to create a reference data index with the same index attributes as the target data index when the word segmentation rules of the search engine are changed, wherein the word segmentation rules are used to indicate the way in which the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; an update module, which is used to update the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rules, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; a deletion module, which is used to delete the target data index when the reference data index update is completed.

[0094] Optionally, the update module includes: a creation unit, used to create an index alias for the target data index and set the read-write status of the target data index to a read-only status, wherein the index alias is used to obtain the target data index after the reference data index is created; a fourth extraction unit, used to extract each reference keyword from the second data field with reference to the changed word segmentation rule and record a third mapping relationship between each reference keyword and the second data field; and an update unit, used to update the reference data index according to the second mapping relationship and the third mapping relationship.

[0095] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0096] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0097] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0098] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0101] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0102] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0103] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0104] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0105] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing server log files, characterized in that: A search engine deployed on a server is used to store log documents of server logs for the server. The data storage structures supported by the search engine include row storage and column storage. In the search engine, the column storage data storage structure is enabled and the row storage data storage structure is disabled. The method includes: Receiving a log writing request initiated by the server, wherein the log writing request is used to request to write a target log document into the search engine, and the target log document is a log document corresponding to a target server log generated on the server; In response to the log write request, extracting non-text type data and text type data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document; In the case where the non-text type first data is extracted, a first data field included in the first data is written into the data storage structure of the column storage and a first mapping relationship between the target log document and the first data field is stored; In the case where the second data of the text type is extracted, a target data index is established for the second data, wherein the target data index is used to indicate a mapping relationship between each keyword obtained after word segmentation of the second data and the target log document; a second data field included in the second data is written into the data storage structure of the column storage, a second mapping relationship between the target log document and the second data field is stored, and the target data index is stored.

2. The method according to claim 1, characterized in that The extracting non-text type data and text type data from the target log document according to the attribute information of the target log document respectively includes: Extracting a mapping configuration corresponding to the target log document from the search engine as the attribute information, wherein the mapping configuration is used to indicate field attributes of each field in the document to be stored by the search engine; Using the mapping configuration, dividing the target field included in the target log document into a first field belonging to the non-text type and a second field belonging to the text type; The first data is extracted from the position of the first field in the target log document, and the second data is extracted from the position of the second field in the target log document.

3. The method according to claim 1, characterized in that The search engine includes a first coordination node and one or more shard nodes. The receiving the log writing request initiated by the server includes: The first coordination node receives the log write request from the transmission port of the server; The first coordination node generates a file name of the target log file; The first coordination node searches for a primary shard node of the target log document from the multiple shard nodes according to the document name and the storage setting of the search engine, wherein the storage setting is used to indicate the storage status of the storage server log in the search engine, and the primary shard node is used to store the target log document; The first coordinating node sends the log write request to the primary sharding node; The log write request is received by the primary shard node.

4. The method according to claim 1, characterized in that: After writing the second data field included in the second data into the data storage structure of the column store, storing the second mapping relationship between the target log document and the second data field, and storing the target data index, the method further includes: Receiving a log query request initiated by the server, wherein the log query request is used to request to search for document data that meets a query condition from the target log document; Extracting a target keyword and a result requirement type from the query condition, wherein the result requirement type is used to indicate a field condition of the document data requested to be obtained by the log query request; The target document data satisfying the query condition is searched and returned according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship and the target data index.

5. The method according to claim 4, characterized in that The searching and returning the target document data according to the target keyword, the result requirement type, the first mapping relationship, the second mapping relationship and the target data index includes: Searching for a reference log document matching the target keyword from the target log document according to the target data index; Extracting a first data field corresponding to the reference log document from the data storage structure of the column storage according to the first mapping relationship corresponding to the reference log document, and extracting a second data field corresponding to the reference log document from the data storage structure of the column storage according to the second mapping relationship corresponding to the reference log document; In the case where the result requirement type is used to indicate that the log query request requests to obtain all fields of the document data, integrating the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document to obtain third data that meets the query condition, and sending the third data to the server, wherein the target document data includes the third data; In a case where the result requirement type is used to indicate that the log query request requests to obtain partial fields of document data, the target field attribute in the query condition is detected, and data fields whose field attributes match the target field attributes are filtered out from the first data field corresponding to the extracted reference log document and the second data field corresponding to the extracted reference log document, so as to obtain fourth data that meets the query condition, and send the fourth data to the server, wherein the target document data includes the fourth data.

6. The method according to claim 1, characterized in that The method further comprises: In the case where the word segmentation rule of the search engine is changed, creating a reference data index having the same index attributes as the target data index, wherein the word segmentation rule is used to indicate the way in which the search engine converts document data into one or more keywords, and the index attributes are used to indicate the index settings of the target data index; updating the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule, wherein the reference data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data field and the target log document; When the reference data index is updated, the target data index is deleted.

7. The method according to claim 6, characterized in that The updating of the reference data index according to the second data field, the second mapping relationship and the changed word segmentation rule includes: Creating an index alias of the target data index and setting the read-write state of the target data index to a read-only state, wherein the index alias is used to obtain the target data index after the reference data index is created; extracting each reference keyword from the second data field by referring to the changed word segmentation rule and recording a third mapping relationship between each reference keyword and the second data field; The reference data index is updated according to the second mapping relationship and the third mapping relationship.

8. A server log document processing device, characterized in that: A search engine deployed on a server, the search engine being used to store log documents of server logs for the server, the data storage structures supported by the search engine including row storage and column storage, the data storage structure of the column storage being enabled in the search engine, and the data storage structure of the row storage being disabled, the device comprising: A first receiving module, configured to receive a log writing request initiated by the server, wherein the log writing request is used to request to write a target log document into the search engine, and the target log document is a log document corresponding to a target server log generated on the server; A first extraction module, configured to respond to the log write request and extract non-text type data and text type data from the target log document according to attribute information of the target log document, wherein the attribute information is used to indicate the data type of the document data included in the target log document; A first storage module is used for, when the non-text type first data is extracted, writing a first data field included in the first data into the data storage structure of the column storage and storing a first mapping relationship between the target log document and the first data field; A second storage module is used to establish a target data index for the second data when the second data of the text type is extracted, wherein the target data index is used to indicate the mapping relationship between each keyword obtained after word segmentation of the second data and the target log document, and is used to write the second data field included in the second data into the data storage structure of the column storage, store the second mapping relationship between the target log document and the second data field, and store the target data index.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images