Log storage method and computing device

Abstract

The invention discloses a log storage method and computing equipment, relates to the technical field of computers, and can save human resources and time resources, flexibly cope with different log data and realize efficient storage of the log data. Specifically, the computing device can receive log data, determine constant information in the log data and variable information corresponding to the constant information, determine whether a log database contains a storage template corresponding to the constant information or not, generate the storage template based on the constant information under the condition that the log database does not contain the storage template, and send the generated storage template to the computing device. And storing the variable information in the storage template.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a log storage method and a computing device. BACKGROUND

[0002] The workflow of a log management system can include three stages of data collection, data storage and data query. Referring to FIG. 1, in the data collection stage, the log management system can receive log data sent by different data sources (such as source A and source B shown in FIG. 1). In the data storage stage, the log management system can store the log data sent by each data source into a database. In the data query stage (not shown in FIG. 1), the log management system can obtain corresponding log data from the database after receiving a query request for the log data from an operation and maintenance personnel. Figure 1 Figure 1 Figure 1

[0003] Continuing to refer to FIG. 1, the existing log management system adopts a centralized storage architecture, and requires that the log data from different sources must be manually adapted before being sent to the log management system to ensure that the format of the log data conforms to the unified requirements of the log management system. In the case that the formats of the log data from different sources are quite different, the manual adaptation method not only needs to consume a large amount of human resources and time resources, but also is difficult to quickly and accurately identify the complex and variable formats between different log data due to the subjectivity and limitations of manual operation, resulting in a lack of flexibility in the adaptation process. Figure 1 SUMMARY

[0004] Embodiments of the present application provide a log storage method and a computing device, which can not only save human resources and time resources, but also enable the computing device to flexibly cope with different log data and achieve efficient storage of the log data.

[0005] To achieve the above object, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a log storage method is provided, applied to a computing device, and the method includes: receiving log data, determining constant information in the log data and variable information corresponding to the constant information, determining whether a storage template corresponding to the constant information is contained in a log database, in a case where the storage template is not contained in the log database, generating the storage template based on the constant information, and storing the variable information in the storage template.

[0007] ​​​​Through the technical solution, the computing device can store the variable information of the log data into the log database based on the storage template corresponding to the constant information of the log data. In this way, without manual intervention, the computing device can flexibly cope with various different log data, automatically identify and match the storage templates corresponding to different log data, and store the log data. Efficient storage of log data is achieved, effectively saving human resources and time resources.

[0008] In addition, in the case where the computing device does not include the storage template corresponding to the constant information, the computing device can generate the storage template based on the constant information and store the variable information in the storage template. In this way, the level of automation of the computing device in processing log data can be effectively improved, reducing the involvement of manual labor, and the efficiency of managing log data can be further improved.

[0009] In an optional implementation, the storage template can be a table, and the generation of the storage template based on the constant information can specifically include: generating a column name corresponding to the constant information, and generating the table based on the column name.

[0010] The above technical solution describes a specific implementation of generating a table when the storage template is a table, which can effectively improve the realizability of the present solution.

[0011] In an optional implementation, the generation of the table based on the column name can specifically include: generating a table creation statement corresponding to the log database based on the column name, and creating the table in the log database based on the table creation statement.

[0012] Through the above technical solution, in the case where the storage template is a table created in the log database, the computing device can automatically generate a table creation statement corresponding to the log database, and then execute the table creation statement to establish the table in the log database. In this way, the efficiency of the computing device in generating the table creation statement can be effectively improved, and the efficiency of the computing device in storing log data can be effectively improved.

[0013] In an optional implementation, the generation of the table creation statement corresponding to the log database based on the column name can specifically include: generating the table creation statement corresponding to the log database based on the column name through a table creation statement model.

[0014] Through the above technical solution, the computing device can generate the table creation statement through the table creation statement model, which can effectively improve the efficiency of the computing device in generating the table creation statement, and further effectively improve the efficiency of the computing device in storing log data.

[0015] In an optional implementation, the generating the storage template based on the constant information can specifically include: generating the storage template based on the constant information and the preset field in a case where the constant information does not contain the preset field.

[0016] The preset field includes at least one of a data source field or a time field, the data source field is used to represent a data source of the generated log data, and the time field is used to represent a time of the generated log data.

[0017] Through the above technical solution, more field information can be contained in the obtained storage template, the information integrity of the log data is ensured, so that the user can query the log data more quickly and accurately based on the field information in the storage template in subsequent query of the log information, and the user experience is enhanced.

[0018] In an optional implementation, the constant information can include multiple, and before the storing the variable information in the storage template, the computing device can also verify whether each constant information is contained in the storage template. Correspondingly, the storing the variable information in the storage template can specifically include: in a case where each constant information is contained in the storage template, storing the variable information in the storage template.

[0019] Through the above technical solution, in a case where multiple constant information is contained in the log data, the computing device can verify whether each constant information is contained in the storage template, so that the phenomenon that any constant information is missing in the generated storage template and the variable information corresponding to the constant information cannot be stored in the storage template can be avoided.

[0020] In an optional implementation, the method can further include: receiving a query request for the log data input by a user, converting the query request into a query statement corresponding to the storage template, and querying the log data in the storage template based on the query statement.

[0021] Through the above technical solution, the user can use natural language to describe their log data query requirements, and the computing device can automatically generate and execute a corresponding query statement to query corresponding log data from the log database and output to the user. In this way, the convenience and user experience of the user querying the log data can be effectively improved, the difficulty and time of the user writing the query statement are reduced, and non-technical personnel can also easily realize the retrieval of the log data.

[0022] In an optional implementation, the converting the query request into the query statement corresponding to the storage template can specifically include: converting the query request into the query statement corresponding to the storage template through a query statement model.

[0023] By the technical solution, the computing device can generate a query statement through the query statement model, so that the efficiency of the computing device in generating the query statement is improved, and the efficiency of the computing device in querying the log data is improved.

[0024] In an optional implementation, the query statement model can be deployed in the computing device.

[0025] In an optional implementation, the storage template can be a table, and converting the query request into the query statement corresponding to the storage template can specifically include: determining key information from the query request, determining an identifier and a column name of a table corresponding to the key information, and generating the query statement based on the identifier and the column name of the table.

[0026] The key information includes at least one of time, a user identifier, or an entity to be queried.

[0027] The technical solution provides a specific implementation of the computing device in converting the query request into the query statement corresponding to the storage template when the storage template is a table, so that the realizability of the solution is improved.

[0028] In a second aspect, a log storage apparatus is provided, which includes functional units for performing any of the methods provided in the first aspect, and each functional unit performs an action through hardware or corresponding software executed by hardware. For example, the log storage apparatus can include a receiving unit, a processing unit, and a storage unit.

[0029] The receiving unit is configured to receive log data. The processing unit is configured to determine constant information in the log data and variable information corresponding to the constant information. The processing unit is further configured to determine whether a storage template corresponding to the constant information is included in a log database, and the log database is configured to store the log data. The storage unit is configured to generate the storage template based on the constant information and store the variable information in the storage template when the log database does not include the storage template.

[0030] In a third aspect, a computing device is provided, which includes a processor and a memory, and the processor is connected to the memory. The memory is configured to store computer execution instructions, and the processor executes the computer execution instructions stored in the memory, thereby implementing any of the log storage methods provided in the first aspect.

[0031] In a fourth aspect, a chip is provided, which includes a processor and an interface circuit. The interface circuit is configured to receive code instructions and transmit the code instructions to the processor. The processor is configured to run the code instructions to execute any of the log storage methods provided in the first aspect.

[0032] In a fifth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions, when the computer-executable instructions are run on a computer, cause the computer to perform any of the log storage methods provided in the first aspect.

[0033] In a sixth aspect, a computer program product is provided, which includes computer-executable instructions, when the computer-executable instructions are run on a computer, cause the computer to perform any of the log storage methods provided in the first aspect.

[0034] The technical effects brought by the implementation manners of the second aspect to the sixth aspect can refer to the technical effects brought by the implementation manners of the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 A workflow diagram of a log management system provided in the related art;

[0036] Figure 2 A structural schematic diagram of a computing device provided in the embodiments of the present application;

[0037] Figure 3 A flowchart of a log storage method provided in the embodiments of the present application;

[0038] Figure 4 A module interaction schematic diagram of a computing device provided in the embodiments of the present application;

[0039] Figure 5 A structural schematic diagram of a log storage apparatus provided in the embodiments of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0041] In the description of the present application, unless otherwise specified, “ / ” represents a “or” relationship between the objects before and after the “ / ”, for example, A / B can represent A or B; “and / or” in the present application is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural.

[0042] Also, in the description of the present application, unless otherwise specified, “plurality” means two or more than two. “At least one of the following” or the like means any combination of the items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be a single item or a plurality of items.

[0043] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using “first”, “second”, etc. The skilled in the art can understand that “first”, “second”, etc. do not limit the quantity and execution order, and “first”, “second”, etc. also do not necessarily mean different. At the same time, in the embodiments of the present application, “exemplary” or “for example” is used to represent as an example, illustration or description. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of “exemplary” or “for example” is intended to present the relevant concept in a specific manner, for ease of understanding.

[0044] The following introduces the terms related to the embodiments of the present application.

[0045] Log: A log is a file that records events that occur during the operation of a system, application, or network device. Logs can be used for performance monitoring, security auditing, or user behavior analysis, etc.

[0046] Large language model (LLM): LLM is an artificial intelligence model that can be trained with a large amount of data using a transformer architecture to learn the patterns and structures of language, so as to understand and generate natural language text. LLM can be applied to text summarization, document translation, document analysis, etc.

[0047] Prompt: A prompt is an input text used to guide the model to perform a specific task, with the purpose of helping the model understand the type of task the user wants to perform or the required output format.

[0048] The following exemplarily introduces the application scenarios of the embodiments of the present application.

[0049] In the field of information technology (IT) and network security, log data, as a key carrier for recording system running status, security events and operation behaviors, plays an irreplaceable role in system monitoring, security auditing and fault diagnosis, etc. The content of log data usually can include event timestamp, event category (such as error, warning or information), event content, user identification and context information of related system or application, etc.

[0050] The embodiment of the present application provides a log storage method applied to a computing device. After receiving log data, the computing device can determine constant information in the log data and variable information corresponding to the constant information, and store the variable information in the log data into a log database in a storage template corresponding to the constant information.

[0051] Through the above technical solution, the computing device can store the variable information of the log data into the log database based on the storage template corresponding to the constant information of the log data. In this way, without manual intervention, the computing device can flexibly cope with various different log data, automatically identify and match the storage templates corresponding to different log data, and store the log data. Efficient storage of log data is realized, and human resources and time resources are effectively saved.

[0052] The computing device in the embodiment of the present application can be a network device. The network device can include a server, etc. The server can be one physical server, two or more physical servers sharing different responsibilities, or a virtual server (also known as a virtual machine) running in a physical server, etc. For example, the server can be a blade server, a high-density server, a rack server or a tower server, etc.

[0053] It should be noted that the embodiment of the present application does not limit the device form of the computing device, and the system architecture of the computing device provided by the embodiment of the present application is described below taking the server as an example.

[0054] Figure 2 A schematic diagram of a computing device provided by the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the computing device can include a processor 101, a memory 102, a storage device 103, an input device 104 and an output device 105. Figure 2As shown, the computing device includes a processor 202, a memory 204. The processor 202 is connected with the memory 204 through a double data rate (DDR) bus 203. Here, the DDR bus 203 can also be replaced by other types of buses, and the embodiments of the present application do not limit the type of bus. In addition, the computing device also includes various input / output (I / O) devices 207, and the processor 202 can access these I / O devices 207 through a peripheral component interconnect express (PCIe) bus 205.

[0055] The processor 202 is the operation core and control core of the computing device. The processor 202 can include one or more processor cores 201. The processor 202 can be a super large scale integrated circuit. The operating system and other software programs are installed in the processor 202, so that the processor 202 can realize access to the memory 204 and various PCIe devices.

[0056] It can be understood that in the embodiments of the present application, the core 201 in the processor 202 can be a central processing unit (CPU), and can be an application specific integrated circuit (ASIC). The processor 202 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. In practical applications, the computing device can also include multiple processors.

[0057] The memory controller is a bus circuit controller inside the computing device for controlling the memory 204 and managing and planning data transmission between the memory 204 and the core 201. Through the memory controller, the memory 204 and the core 201 can exchange data. The memory controller can be a separate chip and is connected with the core 201 through a system bus.

[0058] Those skilled in the art can know that the memory controller can also be integrated into the processor 202, can also be built into the north bridge, and can also be a separate memory controller chip, and the embodiments of the present application do not limit the specific location and form of the memory controller. In actual application, the memory controller can control the necessary logic to write data into the memory 204 or read data from the memory 204. The memory controller can be a memory controller in a general-purpose processor, a special-purpose accelerator, a GPU, an FPGA, an embedded processor, etc.

[0059] The memory 204 is the main memory of the computing device. The memory 204 is usually used to store various running software in the operating system, input and output data, and information exchanged with the external storage, etc. In order to improve the access speed of the processor 202, the memory 204 needs to have the advantage of fast access speed. In the traditional computer system architecture, dynamic random access memory (DRAM) is usually used as the memory 204. The processor 202 can access the memory 204 at high speed through the memory controller, and perform read and write operations on any storage unit in the memory 204. In addition to DRAM, the memory 204 can also be other random access memories, such as static random access memory (SRAM), etc. In addition, the memory 204 can also be a read only memory (ROM). For read only memories, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc. The embodiments of the present application do not limit the number and type of the memory 204. In addition, the memory 204 can be configured to have a power retention function. The power retention function refers to that when the system is powered off and then powered on again, the data stored in the memory will not be lost. The memory 204 with the power retention function is called a non-volatile memory.

[0060] The I / O device 207 refers to a hardware that can perform data transmission, and can also be understood as a device connected to the I / O interface. Common I / O devices include network cards, printers, keyboards, mice, etc. All external storages can also be I / O devices, such as hard disks, floppy disks, optical disks, etc. The processor 202 can access each I / O device 207 through the PCIe bus 205. It should be noted that the PCIe bus 205 is only an example, and can be replaced by other buses, such as a unified (UB) bus, etc.

[0061] The baseboard management controller (BMC) 206 can perform firmware upgrade, management of running state, and troubleshooting of the computing device in a state where the computing device is not powered on. The processor 202 can access the baseboard management controller 206 through the PCIe bus 205. The baseboard management controller 206 can also be connected to at least one sensor. The sensor can obtain state data of the computing device, where the state data includes temperature data, current data, voltage data, and the like. The type of the state data is not limited in the present application.

[0062] The baseboard management controller 206 communicates with the processor 202 through the PCIe bus or other types of buses, for example, to transmit the obtained state data to the processor 202 for processing. The baseboard management controller 206 can also maintain the program code in the memory, including upgrade or recovery, and the like. The baseboard management controller 206 can also control the power supply circuit or the clock circuit in the computing device, and the like. In summary, the baseboard management controller 206 can manage the computing device through the above manners. However, the baseboard management controller 206 is an optional device. In some embodiments, the processor 202 can directly communicate with the sensor to directly manage and maintain the computing device.

[0063] The log storage method provided in the embodiments of the present application is described in detail below with the computing device as an example. Figure 3 A flowchart of a log storage method provided in the embodiments of the present application is shown in FIG. 3. As shown in FIG. 3, the method includes the following S301-S303. Figure 3

[0064] S301, receiving log data.

[0065] The log data can include: (1) operating system log, which is used to record various events and error information occurring in the running process of the operating system, and the like; (2) application program log, which is used to record various events and error information occurring in the running process of the application program, and the like; (3) network device (such as router, switch) log, which is used to record various events, network traffic, and configuration changes occurring in the running process of the network device, and the like; (4) security device (such as firewall, intrusion detection system) log, which is used to record various events, security events, and warning information occurring in the running process of the security device, and the like.

[0066] ​The format of the log data is not specifically limited in the embodiments of this application. For example, the log data can be in JavaScript object notation (JSON) format, extensible markup language (XML) format, or the like.

[0067] It can be understood that the computing device receives the log data in batches. That is, the computing device can receive multiple log data at the same time.

[0068] S302, constant information in the log data and variable information corresponding to the constant information are determined.

[0069] The constant information refers to static labels that are fixed and unchanged in the log data. For example, application identifier, server host name, data center location, log level, log type, or timestamp format, and the like.

[0070] The variable information refers to real-time data that dynamically changes in the log data. For example, the name of a user, user identifier, request parameter, response time, or error code recorded in the log data, and the like.

[0071] Taking log data as the following log A as an example, the constant information includes “UserName”, “Uid”, “UserState”, “Type”, “Description”, and “UserBehavior”, and the variable information includes “User1”, “123465”, “Normal”, “Ordinary User”, “Daily use of system users, limited permissions”, and “View file”.

[0072] Log A:

[0073] {

[0074] UserName: User1,

[0075] Uid: 123465,

[0076] UserState: Normal,

[0077] Type: Ordinary User,

[0078] Description: Daily use of system users, limited permissions,

[0079] UserBehavior: View file

[0080] }

[0081] The constant information and the variable information can correspond to each other one by one.

[0082] Specifically, after receiving the plurality of log data, the computing device can compare the plurality of log data, and determine fixed content and dynamic content in the plurality of log data. Then, the computing device can take the fixed content in the plurality of log data as constant information of the log data, and take the dynamic content in the plurality of log data as variable information of the log data.

[0083] In order to effectively improve the efficiency of the computing device in determining the constant information and the variable information in the log data, in an optional implementation, the computing device can be deployed with a log processing model, which can be used to determine the constant information and the variable information in the log data. Correspondingly, S302 can be specifically implemented as: inputting the log data into the log processing model to obtain the constant information in the log data and the variable information corresponding to the constant information.

[0084] S303, determining whether the log database contains a storage template corresponding to the constant information.

[0085] S304, in the case where the log database does not contain the storage template, generating the storage template based on the constant information, and storing the variable information in the storage template.

[0086] The log database is used to store the log data. The storage template is used to store the variable information in the log data.

[0087] The log database is not specifically limited in the embodiments of the present application. For example, the log database can be a relational database (Relational Databases) such as a structured query language (structured query language, SQL) database or a MySQL database.

[0088] The storage template is not specifically limited in the embodiments of the present application. For example, the storage template can be a table, a linked list, a graph structure, etc.

[0089] The storage template corresponding to the constant information refers to a storage template containing the constant information. For example, taking a table as the storage template, the table corresponding to the constant information refers to a table containing the constant information in the column name.

[0090] Specifically, the log database of the computing device can store storage templates corresponding to different constant information. After determining the constant information in the log data by the method described in S302, the computing device can determine the storage template corresponding to the constant information in the log data (hereinafter referred to as storage template A) from the storage templates stored in the log database of the computing device. In the case that the log database of the computing device contains storage template A, the computing device can store the variable information in the log data in storage template A. In the case that the log database of the computing device does not contain storage template A, the computing device can generate storage template A based on the constant information in the log data, and store the variable information in the log data in storage template A.

[0091] Through the above technical solution, the computing device can store the variable information of the log data into the log database based on the storage template corresponding to the constant information of the log data. In this way, without manual intervention, the computing device can flexibly cope with a variety of different log data, automatically identify and match the storage templates corresponding to different log data, and store the log data. Efficient storage of log data is achieved, effectively saving human and time resources.

[0092] The following describes the method of generating a storage template based on the constant information in the log data by taking a table as an example.

[0093] Specifically, the computing device can generate column names corresponding to the constant information in the log data, and generate a table based on the column names.

[0094] The present application does not limit the way in which the computing device generates column names corresponding to constant information. The following describes the way A and way B in detail.

[0095] Way A, the computing device can take the constant information as the column name.

[0096] For example, taking the constant information including "UserName", "Uid", "UserState", "Type", "Description" and "UserBehavior" as an example, the computing device can take "UserName", "Uid", "UserState", "Type", "Description" and "UserBehavior" as the column names of the table.

[0097] Way B, the computing device can adjust the constant information by using a preset modification rule to obtain the column name.

[0098] The preset modification rule is not limited in the embodiments of the present application. For example, the preset modification rule can be modifying the capital letters in the constant information into lower case letters. For another example, the preset modification rule can be modifying the capital letters in the constant information into lower case letters and adding a preset character between two words.

[0099] The preset character is not limited in the embodiments of the present application. For example, the preset character can be "+" or "_" and the like.

[0100] Exemplarily, in one embodiment, taking the log data as the log A, the constant information includes "UserName", "Uid", "UserState", "Type", "Description" and "UserBehavior", the preset modification rule can be modifying the capital letters in the constant information into lower case letters and adding a preset character between two words, taking the preset character as "_" for example, the computing device can take "user_name", "uid", "user_state", "type", "description" and "user_behavior" as column names.

[0101] In another embodiment, taking the log data as the log B, the constant information in the log B includes "ActivityId", "Uid", "AppName", "ActivityType", "ActivityDetail", "ActivityTime", "LogSource" and "RecordTime", the preset modification rule can be modifying the capital letters in the constant information into lower case letters and adding a preset character between two words, taking the preset character as "_" for example, the computing device can take "activity_id", "uid", "app_name", "activity_type", "activity_detail", "activity_time", "log_source" and "record_time" as column names.

[0102] Log B

[0103] {

[0104] ActivityId: 1,

[0105] Uid: 123465,

[0106] AppName: Document Viewer,

[0107] ActivityType: File Operation,

[0108] ActivityDetail: User viewed a file named 'annual report.pdf',

[0109] ActivityTime: 2024-01-01 12:00:00,

[0110] LogSource: User monitored system,

[0111] RecordTime: 2024-01-01 12:00:05,

[0112] }

[0113] After generating the column name by the above manner, the computing device can generate a table containing the column name.

[0114] Exemplarily, continuing to take the column name including "user_name", "uid", "user_state", "type", "description" and "user_behavior" as an example, the computing device can generate a table containing the column name, which can refer to Table 1 below.

[0115] Table 1

[0116] user_name uid user_state type description user_behavior

[0117] After generating the table by the above manner, the computing device can store the variable information in the log data under the column name corresponding to the constant information corresponding to the variable information, so that the variable information in the log data can be stored in the table.

[0118] Exemplarily, continuing to take the table generated by the computing device as Table 1 and the log data as the log A above as an example. The computing device can store the variable information "user 1", "123465", "normal", "ordinary user", "a user who uses the system daily, with limited permissions" and "viewing files" in the table 1 above, to obtain Table 2 below.

[0119] Table 2

[0120]

[0121] It can be understood that the table in the embodiments of the present application is a table created in a log database. In this context, the computing device generates a table based on the column name, which can be specifically implemented as: generating a table creation statement corresponding to the log database based on the column name, and creating a table in the log database based on the table creation statement.

[0122] Exemplarily, in an embodiment, taking the log database as an SQL database, the log data as the log A, and the column names including "user_name", "uid", "user_state", "type", "description", and "user_behavior" as an example, the table creation statement generated by the computing device can refer to the following table creation statement A. Subsequently, the computing device can run the table creation statement A to create the table 1 in the log database.

[0123] Table creation statement A

[0124]

[0125]

[0126] In another embodiment, taking the log database as an SQL database, the log data as the log B, and the column names including "activity_id", "uid", "app_name", "activity_type", "activity_detail", "activity_time", "log_source", and "record_time" as an example, the table creation statement generated by the computing device can refer to the following table creation statement B.

[0127]

[0128] Through the above technical solutions, the computing device can automatically generate a table creation statement corresponding to the log database, and then execute the table creation statement to establish a table in the log database. In this way, the efficiency of the computing device in generating a table creation statement in the log database can be effectively improved, and the efficiency of the computing device in storing log data can be effectively improved.

[0129] In order to make the storage template contain more fields, so that subsequent users can quickly query log data based on the fields in the storage template. In an optional implementation, the computing device can generate a storage template based on the constant information and the preset fields without containing the preset fields in the constant information of the log data.

[0130] The preset fields are not specifically limited in the embodiments of the present application. For example, the preset fields can include at least one of a data source field (also referred to as log_source) or a time field (also referred to as record_time).

[0131] The data source field is used to represent the data source of the generated log data, and the time field is used to represent the time of the generated log data.

[0132] Specifically, taking the storage template as a table as an example, the computing device can generate column names corresponding to constant information and preset fields in the log data, and generate the table based on the column names.

[0133] The specific way of generating the column names can refer to the above-mentioned way A and way B, which will not be described here.

[0134] Exemplarily, taking the table as a table created in the SQL database, the column names include "user_name", "uid", "user_state", "type", "description", "user_behavior", "log_source" and "record_time" as examples, the computing device can first generate the following table creation statement C based on the column names "user_name", "uid", "user_state", "type", "description", "user_behavior", "log_source" and "record_time". Then, the computing device can run the table creation statement C to create the following table 3 in the SQL database.

[0135] Table creation statement C

[0136] Output table creation statement:

[0137]

[0138]

[0139] Table 3

[0140] user_name uid user_state type description user_behavior log_source record_time

[0141] Through the above technical solutions, more field information can be included in the obtained storage template, ensuring the information integrity of the log data, so that the user can query the log data more quickly and accurately based on the field information in the storage template when querying the log information, thereby enhancing the user experience.

[0142] In order to effectively improve the efficiency of the computing device in generating the table creation statement, in an optional implementation, the computing device can be deployed with a table creation statement model, which is used to generate a table creation statement based on log data. Correspondingly, the above-mentioned generating a table creation statement corresponding to a log database based on column names can be replaced by: generating the table creation statement corresponding to the log database based on the column names through the table creation statement model.

[0143] The embodiment of the present application does not make specific limitation on the table building statement model. For example, the table building statement model can be an LLM, a multi-modal model, or an embedding model constructed based on a transformer network, a recursive neural network (RNN), or a convolutional neural network (CNN).

[0144] Specifically, after receiving the log data, the computing device can input the log data and a preset prompt word (hereinafter referred to as a first prompt word) into the table building statement model. The table building statement model can process the log data based on the first prompt word to obtain a corresponding table building statement.

[0145] The embodiment of the present application does not make specific limitation on the first prompt word. For example, taking the log database as a SQL database, the first prompt word can be: You are an expert in system log analysis, responsible for analyzing log data and generating a corresponding table building statement. To complete this task, please perform the following subtasks: 1) take the constant information of the log data as the column name, 2) take the variable information of the log data as the value in the column, 3) the table building statement should include the log_source column and the record_time column, where log_source represents the data source of the generated log data, and record_time represents the time of generating the log data, and 4) output the table building statement: {“Table build”:<SQL statement>}.

[0146] Through the above technical solution, the computing device can generate a table building statement through the table building statement model. In this way, the efficiency of the computing device in generating a table building statement can be effectively improved, and the efficiency of the computing device in storing log data can be effectively improved.

[0147] As can be known from the above description, the log data can include multiple constant information. In order to avoid the phenomenon that any constant information is missing in the generated storage template, resulting in the variable information corresponding to the constant information cannot be stored in the storage template, in an optional implementation manner, the computing device can further verify whether each constant information in the log data is included in the storage template before storing the variable information in the storage template, and store the variable information in the storage template in a case where each constant information in the log data is included in the storage template.

[0148] The process in which the computing device verifies whether each constant information in the log data is included in the storage template will be described below.

[0149] Specifically, after the computing device generates the storage template, the computing device can establish a corresponding relationship between each constant information in the log data and a field included in the storage template. In a case where the storage template includes a field corresponding to each constant information in the log data, the computing device can determine that the storage template includes each constant information in the log data. In a case where the storage template does not include a field corresponding to any constant information in the log data, the computing device can determine that the storage template does not include each constant information in the log data.

[0150] The embodiments of the present application do not make specific limitations on the corresponding relationship between the constant information generated by the computing device and the field included in the storage template. For example, the corresponding relationship between the constant information and the field included in the storage template can be represented in the form of a key-value pair, or can be represented in the form of a key-value pair in JSON format, as shown in the following corresponding relationship A.

[0151] Corresponding relationship A:

[0152]

[0153] Exemplarily, taking the log data as the log A, the storage template as the table 3, and the corresponding relationship represented in the form of the corresponding relationship A as an example, the corresponding relationship between each constant information in the log data and the column name of the table established by the computing device can refer to the following corresponding relationship B.

[0154] Corresponding relationship B:

[0155]

[0156]

[0157] As can be known from the above corresponding relationship A, the table 3 includes a field corresponding to each constant information in the log A, that is, the table 3 includes each constant information in the log A. Therefore, the computing device can store the variable information in the log A in the table 3.

[0158] In order to effectively improve the verification efficiency of the computing device on whether the storage template includes each constant information in the log data, in an optional implementation, the computing device can be deployed with a verification model, and the verification model is used to verify the storage template. Accordingly, the above verification of whether the storage template includes each constant information in the log data can be replaced by: inputting the storage template into the verification model to obtain a verification result.

[0159] The embodiments of the present application do not make specific limitations on the verification model. For example, the verification model can be an LLM, a multi-modal model, or an embedding model constructed based on a transformer network, a recursive neural network (RNN), or a convolutional neural network (CNN).

[0160] Specifically, after the computing device generates the storage template, the computing device can input the log data, the storage template, and a preset prompt word (referred to as a second prompt word below) into the verification model. The verification model can verify the storage template based on the second prompt word to determine whether each constant information in the log data is included in the storage template.

[0161] The embodiments of the present application do not make specific limitations on the second prompt word. For example, the second prompt word can be: You are an expert in system log analysis, responsible for verifying whether each constant information in the log data is included in the storage template. To complete this task, please perform the following sub-tasks: 1) establish a correspondence between each constant information in the log data and a field included in the storage template, and 2) output the correspondence.

[0162] Through the above technical solution, the computing device can verify whether each constant information in the log data is included in the storage template through the verification model. In this way, the verification efficiency of the computing device can be effectively improved.

[0163] In an optional implementation, the computing device can generate the meaning of the field included in the storage template based on the log data.

[0164] The embodiments of the present application do not make specific limitations on the timing at which the computing device generates the meaning of the field included in the storage template. For example, the computing device can generate the meaning of the field included in the storage template in the process of generating the correspondence between each constant information in the log data and the field included in the storage template, or can separately generate the meaning of the field included in the storage template.

[0165] Continuing with the log data as the log A and the storage template as the table 3, and the correspondence being represented in the manner shown by the correspondence A, the computing device generates the meaning of the field included in the storage template in the process of generating the correspondence between each constant information in the log data and the field included in the storage template. For example, the correspondence between each constant information in the log data and the column name of the table and the meaning of the field included in the storage template established by the computing device can refer to the following correspondence C.

[0166] Correspondence C:

[0167]

[0168]

[0169] In an optional implementation, the computing device can generate information of the storage template, such as a source of data in the storage template, a name of the storage template, a role of the storage template, and the like.

[0170] The embodiments of the present application do not make specific limitation on the information of the storage template generated by the computing device. For example, the information of the storage template can be represented by text, or represented by JSON format in a key-value pair manner, such as the following information A.

[0171] Information A:

[0172] {

[0173] “LogSource”: a source of data in the storage template,

[0174] “Table”: a name of the storage template,

[0175] “TableKnowledge”: a role of the storage template,

[0176] }

[0177] Exemplarily, continuing with the log data as the above log A and the storage template as the above table 3, and taking the name of the table 3 as users as an example, the information of the storage template generated by the computing device can be as follows:

[0178] {

[0179] “LogSource”: log A,

[0180] “Table”: users,

[0181] “TableKnowledge”: a table describing user information, containing basic information and behaviors of users,

[0182] }

[0183] Through the above technical solution, after the computing device generates the meaning of the field contained in the storage template and the information of the storage template, the computing device can better understand the role of the storage template and the meaning of each field contained in the storage template, so that the computing device can more accurately find the log data based on the role of the storage template and the meaning of each field contained in the storage template after receiving a query request for the log data.

[0184] Optionally, after the computing device generates the meaning of the field contained in the storage template and the information of the storage template, the computing device can store the meaning of the field contained in the storage template and the information of the storage template.

[0185] After the computing device stores the log data in the manner described above, the computing device can further receive a query request for the log data, convert the query request into a query statement corresponding to the storage template, and query the log data in the storage template based on the query statement. Figure 3

[0186] Specifically, taking a storage template as a table as an example, after the computing device receives a query request for the log data, the computing device can determine the key information from the query request, determine the identifier and column name of the table corresponding to the key information, and finally generate a query statement based on the identifier and column name of the table. Subsequently, the computing device can query the corresponding log data in the log database based on the query statement, and output the queried log data.

[0187] Embodiments of the present application do not specifically limit the key information. For example, the key information can include at least one of time, user identifier, or entity to be queried.

[0188] The entity can be a specific object, such as a person, a place, or an article, or an abstract concept, such as an event or an idea, etc.

[0189] By way of example, assuming that the query request for the log data is "which applications did user a use on December 12, 2024", where user a is a user identifier, December 12, 2024 is a time, and application is an entity to be queried. Correspondingly, the computing device can take user a, December 12, 2024, and application as key information, and determine the identifier and column name of the table corresponding to user a, December 12, 2024, and application. Assuming that the identifier of the table corresponding to user a, December 12, 2024, and application includes "users" and "user_app_activities", and the column name of the table corresponding to user a, December 12, 2024, and application includes the "user_name" column in the "users" table and the "time" column in the "user_app_activities" table. The computing device generates a query statement based on the identifier "users" and "user_app_activities" of the table and the column name "user_name" and "time", which can refer to the following query statement A.

[0190] Query statement A

[0191]

[0192] ​

[0193] Through the above technical solution, the user can use natural language to describe their log data query requirements, and the computing device can automatically generate and execute the corresponding query statement to query the corresponding log data from the log database and output to the user. In this way, the convenience and user experience of the user querying the log data can be effectively improved, and the difficulty and time of the user writing the query statement are reduced, so that non-technical personnel can also easily realize the retrieval of log data.

[0194] In order to effectively improve the efficiency of the computing device generating the query statement, in an optional implementation, the computing device can be deployed with a query statement model, which is used to convert the query request into the corresponding query statement. Accordingly, the above converting the query request into the query statement corresponding to the storage template can be replaced by: converting the query request into the query statement corresponding to the storage template through the query statement model.

[0195] The query statement model is not specifically limited in the embodiments of the present application. For example, the query statement model can be an LLM, a multi-modal model, or an embedding model constructed based on a transformer network, a recursive neural network (RNN), or a convolutional neural network (CNN).

[0196] Specifically, after the computing device receives the query request for the log data, the query request and a preset prompt word (referred to as a third prompt word below) can be input into the query statement model. The query statement model can process the query request based on the third prompt word to obtain the corresponding query statement.

[0197] The third prompt word is not specifically limited in the embodiments of the present application. For example, taking the log database as a SQL database, the third prompt word can be: You are an expert in data retrieval, responsible for generating accurate SQL query statements according to the natural language description of the user, to complete this task, please perform the following sub-tasks: 1), determine the key information from the query request input by the user, 2), determine the identifier and column name of the table corresponding to the key information, 3), generate a query statement based on the identifier and column name of the table, and perform syntax and logic checks on the query statement, 4), output the query statement.

[0198] Through the above technical solution, the computing device can generate the query statement through the query statement model, so that the efficiency of the computing device generating the query statement can be effectively improved, and the efficiency of the computing device querying the log data can be effectively improved.

[0199] In an optional implementation, the computing device can include a log analysis module, a query conversion module, and a determination module. The determination module is configured to determine whether the computing device includes a storage template corresponding to the constant information in the log data. The log analysis module is configured to generate the storage template based on the constant information, and store the variable information in the storage template. The query conversion module is configured to convert the query request into a query statement corresponding to the storage template.

[0200] The following describes the log storage method provided by the embodiments of the present application by taking the log analysis module, the query conversion module, and the determination module, and a table created in the SQL database as an example.

[0201] As shown in FIG. 6, the determination module can determine whether the computing device includes a table corresponding to the constant information in the log data after receiving the log data. In the case where the computing device includes the table corresponding to the constant information in the log data, the determination module can store the variable information in the log data in the table. In the case where the computing device does not include the table corresponding to the constant information in the log data, it indicates that the log data is sent by a new data source, and the determination module can send the log data to the log analysis module. Figure 4

[0202] After receiving the log data, the log analysis module can determine the constant information in the log data and the variable information corresponding to the constant information, and create a table including a column name corresponding to the constant information in the SQL database based on the constant information. Then, the log analysis module can store the variable information in the log data in the table.

[0203] After receiving the query request for the log data input by the user, the query conversion module can convert the query request into a query statement corresponding to the table in the SQL database, and query the corresponding log data from the table in the SQL database. After querying the log data, the query conversion module can output the log data.

[0204] The above mainly describes the scheme provided by the embodiments of the present application from the perspective of the method. In order to implement the above functions, the log storage apparatus includes the hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0205] ​The embodiments of the present application can divide the log storage device into functional modules according to the above method. For example, the log storage device can include various functional modules corresponding to various functional divisions, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware or in the form of a software functional module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0206] For example, Figure 5 A possible structure of the log storage device involved in the above embodiments is shown. The device includes a receiving unit 501, a processing unit 502 and a storage unit 503. The receiving unit 501 is configured to receive log data. The processing unit 502 is configured to determine constant information in the log data and variable information corresponding to the constant information. The processing unit 502 is further configured to determine whether the log database contains a storage template corresponding to the constant information. The storage unit 503 is configured to generate the storage template based on the constant information in the case that the log database does not contain the storage template, and store the variable information in the storage template.

[0207] Optionally, the storage template can be a table, and correspondingly, the storage unit 503 can be specifically configured to generate a column name corresponding to the constant information, and generate the table based on the column name.

[0208] Optionally, the storage unit 503 can be specifically configured to generate a table creation statement corresponding to the log database based on the column name, and create the table in the log database based on the table creation statement.

[0209] Optionally, the storage unit 503 can be specifically configured to generate a table creation statement corresponding to the log database based on the column name through a table creation statement model.

[0210] Optionally, the storage unit 503 can be specifically configured to generate the storage template based on the constant information and a preset field in the case that the constant information does not contain the preset field. The preset field includes at least one of a data source field or a time field. The data source field is used to represent a data source of the generated log data, and the time field is used to represent a time of the generated log data.

[0211] Optionally, the constant information can include multiple constant information, and the log storage device can further include a verification unit. The verification unit is specifically configured to verify whether each constant information is contained in the storage template. Correspondingly, the storage unit 503 can be specifically configured to store the variable information in the storage template in the case that each constant information is contained in the storage template.

[0212] Optionally, the log storage apparatus further comprises a query unit, which is configured to receive a query request for the log data input by a user, convert the query request into a query statement corresponding to the storage template, and query the log data in the storage template based on the query statement.

[0213] Optionally, the query unit is configured to convert the query request into the query statement corresponding to the storage template by using a query statement model.

[0214] Optionally, the storage template is a table, and the query unit is configured to determine key information from the query request, determine an identifier and a column name of the table corresponding to the key information, and generate the query statement based on the identifier and the column name of the table.

[0215] The key information includes at least one of a time, a user identifier, or an entity to be queried.

[0216] The specific description of the optional mode can be referred to the foregoing method embodiments, and will not be described here. In addition, the explanation and beneficial effect of any one of the log storage apparatuses provided above can be referred to the corresponding method embodiments, and will not be described here.

[0217] The embodiment of the present application further provides a computing device, which comprises a processor and a memory. The processor is connected with the memory. The memory stores computer execution instructions. The processor implements the data processing method in the foregoing embodiment when executing the computer execution instructions. The embodiment of the present application does not make any limitation on the specific form of the computing device. For example, the computing device can be a terminal apparatus or a network device. The terminal apparatus can be referred to as a terminal, a user equipment (UE), a terminal device, an access terminal, a user unit, a user station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a wireless communication device, a user agent or a user apparatus, etc. The terminal apparatus can be a mobile phone, an augmented reality (AR) device, a virtual reality (VR) device, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The network device can be a server, etc. The server can be one physical or logical server, or two or more physical or logical servers sharing different responsibilities and cooperating with each other to implement the functions of the server.

[0218] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program runs on a computer, the computer is caused to execute the method executed by any one of the computing devices provided above.

[0219] The explanation and beneficial effects of the related content in any one of the computer readable storage media provided above can refer to the corresponding embodiments described above, and will not be repeated here.

[0220] The embodiment of the present application further provides a chip. The chip integrates a control circuit and one or more ports for realizing the functions of the computing device. Optionally, the functions supported by the chip can refer to the above, and will not be repeated here. Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by a program instructing the related hardware. The program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or the processor mentioned above can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0221] The embodiments of the present application further provide a computer program product containing instructions, which, when executed on a computer, cause the computer to perform any of the above-mentioned embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc.

[0222] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided by the embodiments of the present application, such as but not limited to the above-mentioned memory, computer-readable storage medium and communication chip, etc., are all non-volatile (non-transitory).

[0223] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more servers, data centers, etc. integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0224] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to an advantage.

[0225] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to an advantage.

[0225] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to an advantage.

Claims

1. A log storage method characterized by, Applied to a computing device, the method comprises: receiving log data; determining constant information in the log data and variable information corresponding to the constant information; determining whether a storage template corresponding to the constant information is contained in a log database used for storing log data; in the case that the storage template is not contained in the log database, generating the storage template based on the constant information, and storing the variable information in the storage template.

2. The method of claim 1, wherein, The storage template is a table. The generating the storage template based on the constant information comprises: generating a column name corresponding to the constant information; generating the table based on the column name.

3. The method of claim 2, wherein, The generating the table based on the column name comprises: generating a table creation statement corresponding to the log database based on the column name; creating the table in the log database based on the table creation statement.

4. The method of claim 3, wherein, The generating the table creation statement corresponding to the log database based on the column name comprises: generating the table creation statement corresponding to the log database based on the column name through a table creation statement model.

5. The method of claim 1, wherein, The generating the storage template based on the constant information comprises: in the case that a preset field is not contained in the constant information, generating the storage template based on the constant information and the preset field; wherein the preset field comprises at least one of a data source field or a time field, the data source field being used to represent a data source for generating the log data, and the time field being used to represent a time for generating the log data.

6. The method of claim 1, wherein, The constant information comprises a plurality of, and before the storing the variable information in the storage template, the method further comprises: verifying whether each of the constant information is contained in the storage template; The storing the variable information in the storage template comprises: in the case that each of the constant information is contained in the storage template, storing the variable information in the storage template.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: receiving a query request for the log data; converting the query request into a query statement corresponding to the storage template; querying the log data in the storage template based on the query statement.

8. The method of claim 7, wherein, The converting the query request into the query statement corresponding to the storage template comprises: converting the query request into the query statement corresponding to the storage template through a query statement model.

9. The method of claim 7, wherein, The storage template is a table, and the converting the query request into the query statement corresponding to the storage template comprises: determining key information from the query request, the key information comprising at least one of a time, a user identifier, or an entity to be queried; determining an identifier and a column name of the table corresponding to the key information; generating the query statement based on the identifier and the column name of the table.

10. A computing device, comprising: comprises: a processor and a memory; the processor is connected with the memory, the memory is used to store computer execution instructions, and the processor executes the computer execution instructions stored in the memory, so that the computing device implements the method in any one of claims 1-9.

Images