Log processing method and program product

Abstract

The embodiment of the invention discloses a log processing method and a program product. The method comprises the following steps: in response to a product log query request, determining a log query time period in the product log query request; determining a corresponding relationship between a preset query time period and a log query type, and determining a target query type corresponding to the log query time period according to the log query time period and the corresponding relationship; and in a log query system corresponding to the target query type, determining a target product log corresponding to the product log query request, and displaying the target product log. According to the technical scheme provided by the embodiment of the invention, the technical problems of relatively low efficiency and relatively poor accuracy caused by manual log query in related technologies are solved, the product logs can be quickly and accurately queried and processed, and the log query efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer processing technology, and in particular to a log processing method and a program product. Background Art

[0002] With the advancement of computer technology, the volume of product logs is increasing rapidly. As business system complexity increases exponentially, the volume of product logs is exploding. For example, the business systems of a large internet company may generate several terabytes or even tens of terabytes of log data daily. Therefore, querying these massive volumes of product logs is crucial.

[0003] In related technologies, relevant operation and maintenance personnel are often required to filter line by line in tens of thousands of nodes and massive logs. This is not only time-consuming and labor-intensive, but also prone to query errors due to the limitations of human operation, which in turn affects the accuracy of troubleshooting and business analysis. Summary of the Invention

[0004] In view of the above problems, the present invention provides a log processing method and program product to enable relatively fast and accurate query processing of product logs, thereby improving the efficiency and accuracy of log queries and, further, enhancing the accuracy of troubleshooting and business analysis.

[0005] According to one aspect of the present invention, a log processing method is provided, the method comprising:

[0006] In response to a product log query request, determine the log query period in the product log query request; determine the correspondence between the preset query period and the log query type, and determine the target query type corresponding to the log query period based on the log query period and the correspondence; in the log query system corresponding to the target query type, determine the target product log corresponding to the product log query request, and display the target product log.

[0007] According to another aspect of the present invention, a log processing device is provided. The device includes:

[0008] A query period determination module is used to determine the log query period in the product log query request in response to the product log query request; a query type determination module is used to determine the correspondence between the preset query period and the log query type, and determine the target query type corresponding to the log query period based on the log query period and the correspondence; a log query module is used to determine the target product log corresponding to the product log query request in the log query system corresponding to the target query type, and display the target product log.

[0009] According to another aspect of the present invention, an electronic device is provided, comprising:

[0010] one or more processors;

[0011] a storage device for storing one or more programs,

[0012] When the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method as described in any one of the embodiments of the present invention.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any log processing method of the present invention when executed.

[0014] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the computer program implements any log processing method of the present invention.

[0015] The technical solution of the embodiment of the present invention determines the log query period in the product log query request in response to the product log query request, and can provide a precise log period query range through the product log query request. The correspondence between the preset query period and the log query type is determined, and the target query type corresponding to the log query period is determined according to the log query period and the correspondence. Through the mapping rules of the query period and the query type, the intelligent matching of the storage layer and the query type can be achieved. In the log query system corresponding to the target query type, the target product log corresponding to the product log query request is determined, and the target product log is displayed. The technical solution of the embodiment of the present invention solves the technical problems in the related art of relying on manual log query, which have low efficiency and poor accuracy, and realizes that the product log can be queried and processed relatively quickly and accurately, thereby improving the efficiency and accuracy of log query.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flow chart of a log processing method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a log storage format provided by an embodiment of the present invention;

[0020] Figure 3 A flow chart of a log processing method provided by an embodiment of the present invention;

[0021] Figure 4 A flow chart of a log processing method provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of an interface for displaying log progress information provided by an embodiment of the present invention;

[0023] Figure 6 A flow chart of a log processing method provided by an embodiment of the present invention;

[0024] Figure 7 A flow chart of an optional embodiment of a log processing method provided by an embodiment of the present invention;

[0025] Figure 8 A schematic diagram of the structure of a log processing device provided by an embodiment of the present invention;

[0026] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0031] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0034] Figure 1 This is a flow chart of a log processing method provided in an embodiment of the present invention. This embodiment is applicable to situations where logs are processed. The method can be performed by a log processing device, which can be implemented in the form of hardware and / or software. The log processing device can be configured in an electronic device such as a computer or server.

[0035] like Figure 1 As shown, the method of this embodiment includes:

[0036] S110 : In response to a product log query request, determine a log query period in the product log query request.

[0037] The product log query request can be understood as a request for querying product logs. In an embodiment of the present invention, the product log query request can include at least a log query period. The log query period can be understood as the start and end time range for querying product logs. In an embodiment of the present invention, the time granularity of the log query period can support filtering by precision such as seconds, minutes, hours, days, weeks, and months, and needs to be selected based on actual needs, which is not specifically limited here. In an embodiment of the present invention, the log query period can be set according to actual needs. For example, the log query period can be from 2023-10-01 12:00:00 to 2023-11-01 15:00:00. In an embodiment of the present invention, the log query period can be a continuous period or a non-continuous period including multiple continuous periods. It should be noted that if the log query period is a non-continuous period including multiple continuous periods, each continuous period in the non-continuous period can be treated as a separate query period for product log query processing.

[0038] In an embodiment of the present invention, the product log query request may further include at least one of a product identifier, a log identifier, and an Internet Protocol address (IP) information. The product identifier can be used to distinguish different products. The log identifier can be used to distinguish different logs. The IP address information can be understood as a core identifier in network communications, which can be used to locate devices, transmit data, and ensure the orderly operation of the network.

[0039] Specifically, when a product log query request is received, the product log query request may be parsed to obtain a parsing result, and the log query period in the product log query request may be determined based on the parsing result.

[0040] In an embodiment of the present invention, there are multiple ways to obtain a product log query request. For example, a first interface can be displayed, wherein the first interface includes at least one configuration item of a log query condition, and in response to a query condition configuration operation for at least one of the configuration items, a product log query request is generated based on the query condition configuration operation. Alternatively, a second interface can be displayed, wherein the second interface includes a command box for inputting a product log query condition. A product log query request is generated based on the product log query condition inputted into the command box. Alternatively, audio data for product log query can be received through a third interface, and a product log query request can be generated based on the audio data. Optionally, generating a product log query request based on the audio data can be performed by converting the audio data into text data, thereby generating a product log query request based on the text data.

[0041] S120: Determine a correspondence between a preset query period and a log query type, and determine a target query type corresponding to the log query period according to the log query period and the correspondence.

[0042] Among them, the preset query period can be understood as a period pre-set for product log query. The number of preset query periods can be at least two. In an embodiment of the present invention, the method of obtaining at least two preset query periods may specifically include: displaying a log query period editing interface, wherein the log query period editing interface includes at least two period editing items. In response to the editing operation on at least two of the period editing items, at least two preset query periods are obtained. It should be noted that all preset query periods have no overlap and all preset query periods together constitute a complete time series. In an embodiment of the present invention, a complete time series may be a time series with the current moment as the end moment and before the current moment. The complete time series may include the current moment and the historical time interval before the current moment. Exemplarily, three query periods are pre-set, namely: [T -30天 , T0], [T -60天 , T -30天 ] and [T -180天 , T -60天 ].

[0043] In an embodiment of the present invention, the log query type can be understood as the type of log query. In an embodiment of the present invention, the correspondence between the preset query period and the log query type can be one-to-one or many-to-one. In other words, different preset query periods correspond to different log query types; or, multiple preset query periods correspond to the same log query type. The target query type can be understood as the log query type corresponding to the log query period. In an embodiment of the present invention, the target query type can be one or more log query types.

[0044] In the embodiment of the present invention, there are many ways to determine the correspondence between the preset query period and the log query type, which are not specifically limited here.

[0045] As an optional implementation of an embodiment of the present invention, determining the correspondence between the preset query period and the log query type includes: presetting at least two query periods, i.e., obtaining at least two preset query periods. For each preset query period, in response to a log query type setting operation for the preset query time, obtaining the log query type corresponding to the preset query period, thereby obtaining the relationship between the preset query period and the log query type.

[0046] As another optional implementation of an embodiment of the present invention, determining the correspondence between the preset query period and the log query type includes: determining the log query type. For each log query type, in response to a query period setting operation for the log query type, obtaining the preset query period corresponding to the log query type, so as to obtain the correspondence between the log query type and the preset query period.

[0047] Specifically, a correspondence between a preset query period and a log query type is determined. Furthermore, based on the log query period and the correspondence, the log query type corresponding to the log query period can be determined. Thus, the log query type corresponding to the log query period can be determined as the target query type corresponding to the target query period.

[0048] S130 : In a log query system corresponding to the target query type, determining a target product log corresponding to the product log query request, and displaying the target product log.

[0049] In an embodiment of the present invention, different log query types correspond to different log query systems. A log query system can be understood as a system for querying logs. In an embodiment of the present invention, a large number of product logs are stored in the log query system. The target product log can be understood as the product log corresponding to the product log query request. In an embodiment of the present invention, the number of target product logs can be one, two, or more. In the case of multiple target product logs, the target product logs can be product logs determined from different log query systems, or the target product logs can be product logs determined from the same log query system.

[0050] Specifically, based on the correspondence between log query types and log query systems, the log query system corresponding to the target query type is determined. Furthermore, the product log corresponding to the product log query request can be determined from the log query system corresponding to the target query type, thereby obtaining the target product log. The target product log can then be displayed according to a preset log sorting method. The preset log sorting method can be a reverse order sorting method or an ascending order sorting method.

[0051] In an embodiment of the present invention, the preset query period may include a first preset period and a second preset period. The end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period. In an embodiment of the present invention, the end time of the first preset period can be set according to actual needs, which is not specifically limited here. Exemplarily, the end time of the first preset period may be a time earlier than the preset period at the current moment; or, the end time of the first preset period is the current moment. For example, the first preset period may be [T -30天 , T0], which can be represented as a continuous time window within 30. Among them, T0 can be represented as the current moment. The second preset time period can be [-∞, T -30天 ], which can be expressed as a historical time interval 30 days ago. The log query type may include a first query type and a second query type. The first query type corresponds to the first preset time period, and the second query type corresponds to the second preset time period. Among them, the first query type can be understood as the query type corresponding to the first preset time period. The second query type can be understood as the query type corresponding to the second preset time period. It should be noted that different log query types correspond to different log query methods.

[0052] On the basis of the above embodiment, the determination of the target query type corresponding to the log query period based on the log query period and the corresponding relationship may include: when the log query period is within the first preset period, the target query type corresponding to the log query period may be determined as the first query type. When the log query period is within the second preset period, the target query type corresponding to the log query period may be determined as the second query type. When the start time of the log query period is within the first preset period and the end time of the log query period is within the second preset period, it may be determined that the target query type corresponding to the log query period includes the first query type and the second query type.

[0053] Specifically, when the log query period is within the first preset period, the first query type can be determined as the target query type corresponding to the log query period. When the log query period is within the second preset period, the second query type can be determined as the target query type corresponding to the log query period. When the start time of the log query period is within the first preset period and the end time of the log query period is within the second preset period, it can be determined that the target query types corresponding to the log query period include the first query type and the second query type.

[0054] In an embodiment of the present invention, the first query type corresponds to a hot query system, and the second query type corresponds to a cold query system, wherein the read and write speed of the hot query system is higher than that of the cold query system. In an embodiment of the present invention, the hot query system can be understood as a data storage system for storing product logs within a first preset time period. The cold query system can be understood as a data storage system for storing product logs within a second preset time period. In an embodiment of the present invention, the database storage system corresponding to the hot query system and the database storage system corresponding to the cold query system may be the same or different. Optionally, the hot query system may be a distributed search engine. The cold query system may be a distributed file system for storing large-scale data

[0055] As an optional implementation manner of an embodiment of the present invention, determining the target product log corresponding to the product log query request in the log query system corresponding to the target query type may include: when the target query type is the first query type, determining the product log corresponding to the product log query request in the hot query system and using it as the target product log.

[0056] As another optional implementation manner of an embodiment of the present invention, determining the target product log corresponding to the product log query request in the log query system corresponding to the target query type may include: when the target query type is the second query type, determining the product log corresponding to the product log query request in the cold query system and using it as the target product log.

[0057] As another optional implementation of the embodiment of the present invention, determining the target product log corresponding to the product log query request in the log query system corresponding to the target query type may include: when the target query type includes the first query type and the second query type, determining the product log corresponding to the product log query request in the hot query system, and determining the product log corresponding to the product log query request in the cold query system. Furthermore, based on the product logs determined by the hot query system and the product logs determined by the cold query system, the target product log corresponding to the product log query request may be obtained.

[0058] Based on the above embodiment, the method further includes: in response to a log storage request, obtaining at least one product log to be stored, and storing the product log to be stored in the hot storage system and the cold storage system respectively.

[0059] Among them, the log storage request can be understood as a request for storing the product log to be stored. The product log to be stored can be understood as a product log that needs to be stored. The product log to be stored can be a product log generated at the current moment and / or a product log received at the current moment. In an embodiment of the present invention, the preset query period can be set according to actual needs. In order to improve data query efficiency, in an embodiment of the present invention, a dual-write mode is adopted to store the product log to be stored in the hot storage system and the cold storage system respectively. Optionally, the hot storage system stores the product identifier and log identifier of the product log in the form of a key-value pair, and constructs a corresponding inverted index. The cold storage system organizes at least two log attribute information of the product log in reverse order, and stores the product log after the attribute information is organized in reverse order in a compressed storage manner.

[0060] In an embodiment of the present invention, the log content of the product log to be stored is written into the hot storage system, and the content of the product log to be stored can be written into the hot storage system according to the naming method of "product ID + log ID". In the hot storage system, an inverted index is constructed for the written full-text data, and the data is stored in a columnar manner in the form of k:v. Among them, different writing strategies can be used according to the data volume of different logs when dividing the index. For example, a separate index can be used for logs with a large amount of data to avoid generating a large number of small files and affecting the disk IO read and write performance. For logs with a small amount of data, multiple copies can be merged and arranged in order. The merging rules can refer to the grouping using the first letter of the log name, see Figure 2 In the embodiment of the present invention, the log content of the product log to be stored is written into the cold storage system, and the file path can be organized according to the product ID, time, and log ID (such as Figure 2Path information in the log). The data for that day is then organized in reverse order by "time," "IP," "log ID," and "detailed log." This means that at least two log attribute information of the product log are organized in reverse order. Optionally, the organized data can be stored in plain text to improve data compression performance.

[0061] Based on the above embodiment, after the product logs to be stored are stored in the hot storage system and the cold storage system, respectively, a program data storage period exceeding the corresponding preset storage period can be periodically deleted to reduce the storage capacity of the log storage system. The first data storage period of the hot storage system is shorter than the second data storage period of the cold storage system. The first and second storage periods end at the current time. The first and second storage periods can be set according to actual needs and are not specifically limited here. For example, the first storage period can be the last 30 days, and the second storage period can be the last 730 days.

[0062] To facilitate flexible configuration of storage periods for the log storage system, in an embodiment of the present invention, the method may further include: displaying a storage period configuration interface for the log storage system, wherein the storage period configuration interface includes a hot storage type and a cold storage type. In response to a storage period configuration operation for the hot storage type, a first storage period corresponding to the hot storage type is obtained. In response to a storage period configuration operation for the cold storage type, a second storage period corresponding to the cold storage type is obtained. The hot storage type corresponds to a hot query system, and the cold storage type corresponds to a cold query system.

[0063] The technical solution of the embodiment of the present invention determines the log query period in the product log query request in response to the product log query request, and can provide a precise log period query range through the product log query request. The correspondence between the preset query period and the log query type is determined, and the target query type corresponding to the log query period is determined according to the log query period and the correspondence. Through the mapping rules of the query period and the query type, the intelligent matching of the storage layer and the query type can be achieved. In the log query system corresponding to the target query type, the target product log corresponding to the product log query request is determined, and the target product log is displayed. The technical solution of the embodiment of the present invention solves the technical problems in the related art of relying on manual log query, which have low efficiency and poor accuracy, and realizes that the product log can be queried and processed relatively quickly and accurately, thereby improving the efficiency and accuracy of log query.

[0064] Figure 3A flow chart of a log processing method provided for an embodiment of the present invention, based on the aforementioned embodiment, optionally, determining the first product log corresponding to the product log query request in the thermal query system, and determining the first product log as the target product log, includes: determining the request parameters in the log query request, and generating a product log query statement based on the request parameters; executing the product log query statement in the thermal query system, obtaining the first product log corresponding to the product log query request, and determining the first product log as the target product log. Among them, the technical features that are the same or similar to those in the above embodiment are not repeated here. Figure 3 As shown, the method of this embodiment specifically includes:

[0065] S210 : In response to a product log query request, determine a log query period in the product log query request.

[0066] S220. Determine the correspondence between the preset query period and the log query type, and determine the target query type corresponding to the log query period based on the log query period and the correspondence; wherein the log query type includes a first query type and a second query type, the preset query period includes a first preset period and a second preset period, the first query type corresponds to the first preset period, the second query type corresponds to the second preset period, the first query type corresponds to a hot query system, and the second query type corresponds to a cold query system.

[0067] The end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period. The read and write speed of the hot query system is higher than that of the cold query system.

[0068] S230: When the target query type includes the first query type, determine the request parameters in the log query request, and generate a product log query statement based on the request parameters.

[0069] The request parameters may be understood as parameters in a log query request. Optionally, the request parameters may include at least a log query period. Furthermore, the request parameters may include at least one of a product identifier, a log identifier, and IP information. The product log query statement may be a database query statement generated based on the request parameters in the log query request and executable in a hot query system. The product log query statement may be used to retrieve product logs corresponding to the log query request from the hot query system.

[0070] Specifically, when the target query type is the first query type, the log query request can be parsed to determine the request parameters in the log query request. A product log query statement can then be generated based on the request parameters. The product log query statement is then executed in the hot query system to obtain a first product log corresponding to the product log query request, and the first product log is determined as the target product log.

[0071] In an embodiment of the present invention, generating a product log query statement based on the request parameters may include presetting a query statement generation template corresponding to a hot query system, wherein the query statement generation template is used to generate the product log query statement. Subsequently, the request parameters may be added to the query statement generation template to generate the product log query statement.

[0072] S240: Execute the product log query statement in the hot query system to obtain a first product log corresponding to the product log query request, determine the first product log as a target product log, and display the target product log.

[0073] The first product log may be understood as a product log obtained after executing the product log query statement in the hot query system.

[0074] Specifically, the product log query statement is executed in the hot query system to determine, based on the log query statement, a product log corresponding to the product log query request from the hot query system. This is a first product log. This first product log can then be determined as a target product log and displayed.

[0075] In an embodiment of the present invention, after executing the product log query statement in the hot query system, if it is determined that the number of product logs corresponding to the product log query request reaches a preset threshold, log data can be pulled in a rolling manner.

[0076] The technical solution of an embodiment of the present invention, when the target query type includes the first query type, determines the request parameters in the log query request and generates a product log query statement based on the request parameters; executes the product log query statement in the hot query system to obtain the first product log corresponding to the product log query request, determines the first product log as the target product log, and realizes the query function of hot data.

[0077] Figure 4A flow chart of a log processing method provided for an embodiment of the present invention, based on the aforementioned embodiment, optionally, in the log query system corresponding to the target query type, a target product log corresponding to the product log query request is determined, including: when the target query type is the second query type, a preset distributed cluster management technology is used to split the product log query request into multiple query task requests; in the cold query system, the second product log corresponding to each query task request is respectively determined, and the target product log is obtained based on the second product log. Among them, the technical features that are the same or similar to those in the above embodiment are not repeated here. Figure 4 As shown, the method of this embodiment specifically includes:

[0078] S310: In response to a product log query request, determine a log query period in the product log query request.

[0079] S320. Determine the correspondence between the preset query period and the log query type, and determine the target query type corresponding to the log query period based on the log query period and the correspondence; wherein the log query type includes a first query type and a second query type, the preset query period includes a first preset period and a second preset period, the first query type corresponds to the first preset period, the second query type corresponds to the second preset period, the first query type corresponds to a hot query system, and the second query type corresponds to a cold query system.

[0080] The end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period. The read and write speed of the hot query system is higher than that of the cold query system.

[0081] S330: When the target query type is the second query type, use a preset distributed cluster management technology to split the product log query request into multiple query task requests.

[0082] The term "preset distributed cluster management technology" can be understood as a pre-configured distributed cluster management technology. In embodiments of the present invention, the pre-configured distributed cluster management technology can be configured based on actual needs and is not specifically limited herein. For example, the pre-configured distributed cluster management technology may include Yarn cluster management technology. The query task request can be understood as a request obtained by splitting the product log query request.

[0083] Specifically, when the target query type is the second query type, the product log query request can be submitted to a preset distributed cluster, and the product log query request can be split and processed using the preset distributed cluster management technology. This can result in multiple split requests, i.e., multiple query task requests.

[0084] For example, the granularity of the log query period may be days, and the product log query request may be split according to the granularity of the log query period, thereby obtaining multiple query task requests, that is, multiple query tasks.

[0085] In an embodiment of the present invention, a preset distributed cluster includes a task-driving node and at least one task execution node; wherein the task-driving node is used to manage the task execution node. The task execution node can be used to query data, maintain heartbeats, and report the number of data items queried for the task to the task-driving node. The task-driving node splits the product log query request into multiple query task requests, and the query task requests are sent to the task execution node. It should be noted that, in an embodiment of the present invention, the correspondence between the task execution node and the query task request can be one-to-one or one-to-many.

[0086] In an embodiment of the present invention, the task-driving node uses a dynamic and balanced allocation method to send query task requests to the task execution nodes. Specifically, the task-driving node first determines the total number of tasks being executed by each task execution node through the heartbeat packets sent by the task execution nodes. For example, if the number of tasks to be processed by task execution node 1 at the current moment is 20, and the number of tasks to be processed by task execution node 2 at the current moment is 200, then tasks can be assigned to task execution node 1 first. In addition, the task-driving node will not assign tasks for consecutive dates to the same task execution node.

[0087] S340: Determine in the cold query system the second product logs corresponding to the respective query task requests, obtain the target product log based on the second product logs, and display the target product logs.

[0088] Specifically, for each task execution node, based on the query task request corresponding to the task execution node, a second product log corresponding to the query task request is determined in the cold query system. A target product log is obtained based on the second product log, and the target product log is displayed.

[0089] In an embodiment of the present invention, the task execution node, based on the query task request corresponding to the task execution node, specifically includes: extracting query parameters from the query task request, constructing a regular expression based on a finite state automaton based on the query parameters, and then calling a cold query system of the task execution node to execute the regular expression.

[0090] In an embodiment of the present invention, before displaying the target product log, the method further includes: determining the real-time number of the query task requests completed at the current moment and the total number of the query task requests, determining the query progress information of the query task requests at the current moment based on the real-time number and the total number, and displaying the query progress information; displaying the target product log includes: displaying the target product log in response to an event in which the real-time number is equal to the total number.

[0091] Among them, the real-time number can be understood as the number of query task requests that have been completed at the current moment. In an embodiment of the present invention, determining the real-time number of query task requests that have been completed at the current moment includes: determining the real-time number of query task requests that have been completed at the current moment based on the task execution information in the heartbeat packet sent by the task execution node and received by the task driving node. Optionally, the task execution information in the heartbeat packet may include the total number of tasks being executed by the task execution node and the amount of data that has been queried at the current moment.

[0092] In order to facilitate the rapid determination of the real-time number of query task requests completed at the current moment, in an embodiment of the present invention, the task-driving node can store the task execution information of the heartbeat packet sent by each task execution node in a preset data table. In addition, the task-driving node continuously monitors the heartbeat of the task execution node. If the heartbeat signal is not received for a preset number of consecutive times, the unfinished task of the task execution node can be re-added to the task scheduling queue. Among them, the task scheduling queue can be understood as a queue for storing tasks to be distributed.

[0093] In an embodiment of the present invention, based on the real-time quantity and the total quantity, determining the query progress information of the query task request at the current moment includes: calculating the ratio of the real-time quantity and the total quantity, and obtaining the query progress information of the query task request at the current moment based on the ratio (see Figure 5 ), which can achieve observability of the query process and help users understand the query progress.

[0094] In this embodiment of the present invention, each task corresponds to one day's worth of data. Since the data is already sorted in reverse chronological order when it is written, there is no need for secondary sorting during the query phase. The task driver node can simply merge the task data in chronological order from the end to the beginning.

[0095] The technical solution of an embodiment of the present invention is to split the product log query request into multiple query task requests by adopting a preset distributed cluster management technology when the target query type is the second query type; determine the second product log corresponding to each query task request in the cold query system, obtain the target product log based on the second product log, and realize the query function of cold data.

[0096] Figure 6 A flow chart of a log processing method provided for an embodiment of the present invention, based on the aforementioned embodiment, optionally, in the log query system corresponding to the target query type, determining the target product log corresponding to the product log query request, including: in the case where the target query type includes the first query type and the second query type, determining the third product log corresponding to the product log query request in the hot query system, and determining the fourth product log corresponding to the product log query request in the cold query system; obtaining the target product log based on the third product log and the fourth product log. Among them, the technical features that are the same or similar to those in the above embodiments are not repeated here. Figure 6 As shown, the method of this embodiment specifically includes:

[0097] S410 : In response to a product log query request, determine a log query period in the product log query request.

[0098] S420. Determine the correspondence between the preset query period and the log query type, and determine the target query type corresponding to the log query period based on the log query period and the correspondence; wherein the log query type includes a first query type and a second query type, the preset query period includes a first preset period and a second preset period, the first query type corresponds to the first preset period, the second query type corresponds to the second preset period, the first query type corresponds to a hot query system, and the second query type corresponds to a cold query system.

[0099] The end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period. The read and write speed of the hot query system is higher than that of the cold query system.

[0100] S430. When the target query type includes the first query type and the second query type, a third product log corresponding to the product log query request is determined in the hot query system, and a fourth product log corresponding to the product log query request is determined in the cold query system.

[0101] The third product log may be understood as the product log corresponding to the product log query request determined in the hot query system when the target query type includes the first query type and the second query type. The fourth product log may be understood as the product log corresponding to the product log query request determined in the cold query system when the target query type includes the first query type and the second query type.

[0102] S440: Obtain a target product log based on the third product log and the fourth product log, and display the target product log.

[0103] Specifically, the third product log and the fourth product log are arranged according to a preset product log arrangement order. This produces an arranged product log, i.e., a target product log, which is then displayed. In an embodiment of the present invention, the third product log and the fourth product log are integrated in reverse chronological order to produce an integrated product log, i.e., a target product log, which is then displayed.

[0104] The technical solution of an embodiment of the present invention solves the mixed query function of hot and cold data by determining, in the hot query system, a third product log corresponding to the product log query request, and determining, in the cold query system, a fourth product log corresponding to the product log query request, when the target query type includes the first query type and the second query type; and obtaining the target product log based on the third product log and the fourth product log.

[0105] The embodiment of the present invention provides an optional embodiment of a log processing method, and its specific implementation method can be found in the following embodiment. Among them, the technical features that are the same or similar to the above embodiment are not repeated here. Figure 7 The method of this embodiment specifically includes the following steps:

[0106] 1. Receive log selection query conditions input by the user through the web (front-end interface); wherein the log selection query conditions include log query period, product identifier, log identifier, and IP information.

[0107] 2. Call the gateway interface to send the log selection query conditions to the query agent. Through the query agent, based on the correspondence between the log query period and the log query system, determine the log query system corresponding to the log selection query conditions, and determine the target product log corresponding to the log selection query conditions from the log query system.

[0108] See also Figure 7If the log query period is within 30 days, the hot query system can be used for querying. If the log query period is before 30 days, the cold query system can be used for querying. If the log query period includes both the period within 30 days and the period before 30 days, the hot query system will be used for querying within 30 days, and the cold query system will be used for querying before 30 days.

[0109] The technical solution of the embodiment of the present invention can improve the efficiency of product log query by setting corresponding log query systems for different log query time periods.

[0110] Figure 8 A structural diagram of a log processing device provided by an embodiment of the present invention. Figure 8 As shown, the device includes: a query period determination module 410, a query type determination module 420, and a log query module 430. The query period determination module 410 is configured to, in response to a product log query request, determine the log query period in the product log query request; the query type determination module 420 is configured to determine a correspondence between a preset query period and a log query type, and determine a target query type corresponding to the log query period based on the log query period and the correspondence; and the log query module 430 is configured to, in a log query system corresponding to the target query type, determine a target product log corresponding to the product log query request and display the target product log.

[0111] The technical solution of the embodiment of the present invention determines the log query period in the product log query request in response to the product log query request, and can provide a precise log period query range through the product log query request. The correspondence between the preset query period and the log query type is determined, and the target query type corresponding to the log query period is determined according to the log query period and the correspondence. Through the mapping rules of the query period and the query type, the intelligent matching of the storage layer and the query type can be achieved. In the log query system corresponding to the target query type, the target product log corresponding to the product log query request is determined, and the target product log is displayed. The technical solution of the embodiment of the present invention solves the technical problems in the related art of relying on manual log query, which have low efficiency and poor accuracy, and realizes that the product log can be queried and processed relatively quickly and accurately, thereby improving the efficiency and accuracy of log query.

[0112] Optionally, the log query type includes a first query type and a second query type, the preset query period includes a first preset period and a second preset period, the first query type corresponds to the first preset period, the second query type corresponds to the second preset period, the end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period.

[0113] Optionally, the first query type corresponds to a hot query system, and the second query type corresponds to a cold query system, and the read and write speed of the hot query system is higher than that of the cold query system.

[0114] Optionally, the log query module 430 is used to determine the request parameters in the log query request when the target query type includes the first query type, and generate a product log query statement based on the request parameters; execute the product log query statement in the hot query system to obtain a first product log corresponding to the product log query request, and determine the first product log as the target product log.

[0115] Optionally, the log query module 430 is used to, when the target query type is the second query type, adopt a preset distributed cluster management technology to split the product log query request into multiple query task requests; determine the second product log corresponding to each query task request in the cold query system, and obtain the target product log based on the second product log.

[0116] Optionally, the log processing device further includes a progress display module. The progress display module is configured to determine, before displaying the target product log, the real-time number of completed query task requests and the total number of completed query task requests at the current moment, determine query progress information of the query task request at the current moment based on the real-time number and the total number, and display the query progress information; and the log query module 430 is configured to display the target product log in response to an event in which the real-time number equals the total number.

[0117] Optionally, the log query module 430 is used to determine a third product log corresponding to the product log query request in the hot query system when the target query type includes the first query type and the second query type, and to determine a fourth product log corresponding to the product log query request in the cold query system; and obtain the target product log based on the third product log and the fourth product log.

[0118] Optionally, the log processing device further includes a log storage module, configured to obtain at least one product log to be stored in response to a log storage request, and store the product log to be stored in the hot storage system and the cold storage system respectively.

[0119] Optionally, the hot storage system stores the product identification and log identification of the product log in the form of key-value pairs, and constructs a corresponding inverted index; the cold storage system organizes at least two log attribute information of the product log in reverse order, and stores the product log after the attribute information is organized in reverse order in a compressed storage manner.

[0120] The log processing device provided by the embodiment of the present invention can execute the log processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] It is worth noting that the various units and modules included in the above-mentioned log processing device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0122] Figure 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0123] like Figure 9 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0125] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the log processing method.

[0126] In some embodiments, the log processing method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the log processing method in any other appropriate manner (for example, by means of firmware).

[0127] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0133] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0134] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A log processing method, characterized in that: include: In response to a product log query request, determining a log query period in the product log query request; Determine a correspondence between a preset query period and a log query type, and determine a target query type corresponding to the log query period according to the log query period and the correspondence; In a log query system corresponding to the target query type, a target product log corresponding to the product log query request is determined, and the target product log is displayed.

2. The method according to claim 1, characterized in that The log query type includes a first query type and a second query type, and the preset query period includes a first preset period and a second preset period. The first query type corresponds to the first preset period, and the second query type corresponds to the second preset period. The end time of the second preset period is the start time of the first preset period, and the start time of the first preset period is earlier than the end time of the first preset period.

3. The method according to claim 2, characterized in that The first query type corresponds to a hot query system, and the second query type corresponds to a cold query system. The hot query system has a higher read and write speed than the cold query system.

4. The method according to claim 3, wherein determining, in a log query system corresponding to the target query type, a target product log corresponding to the product log query request comprises: In a case where the target query type includes the first query type, determining request parameters in the log query request, and generating a product log query statement based on the request parameters; The product log query statement is executed in the hot query system to obtain a first product log corresponding to the product log query request, and the first product log is determined as a target product log.

5. The method according to claim 3, wherein determining, in a log query system corresponding to the target query type, a target product log corresponding to the product log query request comprises: When the target query type is the second query type, a preset distributed cluster management technology is used to split the product log query request into multiple query task requests; In the cold query system, second product logs corresponding to the query task requests are respectively determined, and a target product log is obtained based on the second product logs.

6. The method according to claim 5, characterized in that Before displaying the target product log, the method further includes: Determine the real-time number of completed query task requests and the total number of query task requests at the current moment, determine query progress information of the query task request at the current moment based on the real-time number and the total number, and display the query progress information; The displaying of the target product log includes: In response to an event that the real-time quantity is equal to the total quantity, the target product log is displayed.

7. The method according to claim 3, wherein determining, in a log query system corresponding to the target query type, a target product log corresponding to the product log query request comprises: When the target query type includes the first query type and the second query type, determining a third product log corresponding to the product log query request in the hot query system, and determining a fourth product log corresponding to the product log query request in the cold query system; A target product log is obtained based on the third product log and the fourth product log.

8. The method according to claim 3, characterized in that The method further comprises: In response to the log storage request, at least one product log to be stored is obtained, and the product log to be stored is stored in the hot storage system and the cold storage system respectively.

9. The method according to claim 8, characterized in that The hot storage system stores the product identification and log identification of the product log in the form of a key-value pair, and constructs a corresponding inverted index; the cold storage system organizes at least two log attribute information of the product log in reverse order, and stores the product log after the log attribute information is organized in reverse order in a compressed storage manner.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the log processing method according to any one of claims 1 to 9.

Images