Log data processing method and device and electronic equipment

Abstract

The invention provides a log data processing method and device and electronic equipment, and relates to the technical fields of data processing, artificial intelligence, large models and the like. The method comprises the steps of obtaining original log data of a target task; wherein the target task is a task to be subjected to log analysis; preprocessing the original log data to obtain target log data; wherein the data scale of the target log data is lower than that of the original log data; determining an analysis reply corresponding to the target log data according to the target log data, the context information of the target task and the first large model; and generating an analysis result of the target task according to the analysis reply corresponding to the target log data.

Description

Technical Field

[0001] This disclosure relates to the technical fields of data processing, artificial intelligence, and large models, and in particular to a log data processing method, apparatus, and electronic device. Background Technology

[0002] Large model technology is powerful in natural language processing tasks. In addition to dialogue understanding and document understanding, it can also analyze task logs. For example, when the execution of a task is interrupted due to a sudden failure, the log data of the task can be analyzed using a large model to determine the cause of the failure.

[0003] However, due to the length and large volume of log data, the inherent input limitations of large models can affect the analysis results. Summary of the Invention

[0004] This disclosure provides a log data processing method, apparatus, and electronic device.

[0005] According to one aspect of this disclosure, a log data processing method is provided, the method comprising: acquiring raw log data of a target task; wherein the target task is a task to be log analyzed; preprocessing the raw log data to obtain target log data; wherein the data size of the target log data is smaller than the data size of the raw log data; determining the analysis response corresponding to the target log data based on the target log data, context information of the target task, and a first large model; and generating the analysis result of the target task based on the analysis response corresponding to the target log data.

[0006] According to another aspect of this disclosure, a log data processing apparatus is provided, the apparatus comprising: an acquisition module for acquiring raw log data of a target task; wherein the target task is a task to be log analyzed; a processing module for preprocessing the raw log data to obtain target log data; wherein the data size of the target log data is smaller than the data size of the raw log data; a determination module for determining an analysis response corresponding to the target log data based on the target log data, context information of the target task, and a first large model; and a generation module for generating an analysis result of the target task based on the analysis response corresponding to the target log data.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the log data processing method proposed above in this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing the computer to perform the log data processing method proposed above in this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the log data processing method proposed above in this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating a log data processing method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another log data processing method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another log data processing method provided in this embodiment of the disclosure; Figure 4 A schematic diagram illustrating the principle of a log data processing method provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present disclosure; Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] Large-scale modeling technology has demonstrated powerful capabilities in natural language processing tasks. Beyond common areas like dialogue understanding and general document understanding, practical testing has revealed its strong ability to understand and analyze specialized logs. For example, if the training of a large-scale model is interrupted due to a sudden failure, another large-scale model can be used to analyze the logs generated during the initial training to determine the cause of the failure and provide solutions. Compared to related technologies where experienced experts rely on reviewing extensive logs to pinpoint the cause and provide solutions, this approach is more efficient and offers more comprehensive coverage of long-tail scenarios (scenarios with low frequency, small scale, but extremely diverse and personalized characteristics).

[0014] However, in real-world applications, excessively long individual log entries or a large total number of log entries can easily lead to token overload issues in large models, which are difficult to avoid even when using large models adept at handling extremely long texts. Furthermore, since most logs exhibit highly similar content and low effective information density, directly inputting large models with a large volume of raw logs without any processing will result in poor analysis performance.

[0015] In related technologies, increasing the input token limit for large models can solve the problem of excessive input tokens. However, while this approach can increase the number of tokens that large models can process, it also slows down interface response speed, reduces response accuracy, and increases call costs. Moreover, large models are prone to missing excessively long contexts, and for input content such as logs with low effective information density, where log entries indicating errors / anomalies / faults / performance bottlenecks constitute only a very small proportion (high noise, low signal-to-noise ratio, sparse information), it is even more difficult to guarantee analysis quality. Furthermore, the amount of logs to be processed varies in different task scenarios, and there is still a possibility of exceeding the token limit.

[0016] Furthermore, the problem of excessive input tokens in large models can be solved by pruning the input content. However, while pruning logs can circumvent the input token limit, considering the uneven distribution of abnormal events in logs, the difficulty in interpreting a single log entry without context, and the need for inferences based on the correlation between multiple log entries for fault conditions, this approach often affects the effectiveness and accuracy of the analysis by disrupting the integrity of the log context.

[0017] To address the aforementioned issues, this disclosure provides a log data processing method, apparatus, and electronic device. By preprocessing the raw log data of the target task, a smaller target log data is obtained, which can accurately and efficiently remove redundant and irrelevant log data and compress the input content of the first large model. Based on the target log data, the context information of the target task, and the first large model, the analysis response corresponding to the target log data is determined. By combining the context information of the target task with the capabilities of the large model, accurate analysis of the target log data can be achieved, making the analysis response corresponding to the target log data highly consistent with the actual scenario of the target task. Furthermore, based on the analysis response corresponding to the target log data, the analysis result of the target task is determined, which can improve the accuracy and reliability of the analysis result.

[0018] The log data processing method, apparatus, and electronic device of this disclosure are described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating a log data processing method provided in an embodiment of the present disclosure.

[0020] like Figure 1 As shown, this log data processing method may include the following steps: Step S101: Obtain the raw log data of the target task.

[0021] It should be noted that the executing entity of the log data processing method in this embodiment can be a hardware device with data processing capabilities and / or the necessary software to drive the hardware device. Optionally, the executing entity may include a server, a user terminal, and other smart devices. Optionally, the user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, etc. Optionally, the server includes, but is not limited to, a network server, an application server, or a server of a distributed system, or a server combined with blockchain, etc. This embodiment does not impose specific limitations.

[0022] The target task is the task to be analyzed from the logs.

[0023] As an example, the target task can be any task whose execution was interrupted. Such tasks require log analysis to determine the cause of the interruption. For instance, the target task could be any large model training task that unexpectedly stopped during execution, requiring log analysis to determine the cause of the interruption.

[0024] The raw log data refers to the log data generated during the execution of the target task. The raw log data has not undergone any processing.

[0025] In some embodiments, raw log data of the target task within a target time period can be obtained. The target time period can be determined based on the time when log analysis of the target task is performed.

[0026] As an example, suppose the target task is interrupted at time t, and log analysis is required. In this case, the raw log data of the target task within a defined time period prior to time t can be obtained. The defined time period can be determined based on the specific scenario and past experience. For example, the raw log data of the target task within x hours prior to time t can be obtained.

[0027] Step S102: Preprocess the original log data to obtain target log data, wherein the size of the target log data is smaller than the size of the original log data.

[0028] Log data is characterized by low effective information density, with only a very small proportion of log entries indicating errors / anomalies / faults / performance bottlenecks (high noise, low signal-to-noise ratio, and sparse information). Therefore, it is necessary to streamline log data from multiple perspectives and eliminate redundant and invalid log data.

[0029] In some embodiments, different preprocessing methods can be applied to raw log data at different log levels. For example, Info-level log data is used to record normal operation data (such as service start / stop, received requests, user logins, database connection status, etc.). Therefore, Info-level log data may contain a large number of similar and repetitive log entries. For example, Info-level log data might look like this: 10:00:00 INFO Database connection pool: Currently active connections: 50 10:00:01 INFO Database connection pool: Currently active connections: 50 10:00:02 INFO Database connection pool: Currently active connections: 50 ... This allows for proportional sampling and compression of multiple log entries in Info-level log data that are highly similar except for their timestamps.

[0030] For example, Warn-level log data is used to record potentially abnormal operation data (an abnormal situation occurs, but the task can still continue to execute normally), and Error-level log data is used to record abnormal operation data (a failure occurs (such as operation failure, function unavailability, etc.), the task is interrupted and cannot continue to execute). Therefore, relevant logs that may be able to locate the cause of the task interruption can be extracted from Warn-level log data and / or Error-level log data.

[0031] Step S103: Based on the target log data, the context information of the target task, and the first major model, determine the analysis response corresponding to the target log data.

[0032] The first major model can be any major model, such as a large language model.

[0033] Among them, the context information of the target task records information related to the target task, which can help the first major model to better analyze the target log data.

[0034] As an example, if the target task is a large model training task, the context information of the target task can include how many nodes the large model training task used, how many training steps were executed, how long it took to run, GPU metrics, latency metrics, etc.

[0035] In some embodiments, a first major model can be used to analyze the target log data. To ensure the accuracy of the analysis response output by the first major model, it can combine the context information of the target task and relevant prompts to analyze the target log data and generate a corresponding analysis response. For example, the target log data, the context information of the target task, and relevant prompts can be input into the first major model to obtain the analysis response corresponding to the target log data output by the first major model.

[0036] The relevant prompts are used to constrain the content generated by the primary model. For example, the primary model can be constrained to output an analysis response in the order of describing the phenomenon, listing evidence (e.g., key logs), providing reasons, drawing conclusions, and offering suggested solutions, thereby improving the accuracy of the analysis response. Furthermore, the primary model can be required to provide possible causes in order of confidence index. The confidence index is a subjective rating given by the primary model based on the description of the phenomenon, evidence (e.g., key logs), and its internal knowledge, assessing the likelihood of each cause being true. This rating can be expressed as a percentage (e.g., 80%), probability (e.g., high / medium / low), or description (e.g., "very likely," "possible," "less likely").

[0037] As an example, the relevant prompts could be something like, "Describe the phenomenon you observed, list the key log data needed for the analysis of this phenomenon, analyze the possible causes, and sort the causes by confidence level from high to low. Finally, based on the aforementioned analysis, summarize the conclusions and provide suggested solutions."

[0038] The input to the first major model consists of target log data, target task context information, and relevant prompts, which may exceed the input limits of the first major model. Therefore, in some embodiments, if the target log data, target task context information, and relevant prompts exceed the input limits of the first major model, the target log data can be fragmented and input into the first major model separately.

[0039] When determining the number of shards, it is necessary to ensure that the target log data, target task context information, and related prompts for each shard do not exceed the input limit of the first major model.

[0040] The system can be sharded according to log length or number of log entries. Each shard needs to have a certain overlap (e.g., 5%, which can be determined based on the actual scenario and past experience) to ensure continuity (similar to a sliding window).

[0041] This can be done by sharding logs according to log nodes, with the aim of ensuring that logs from the same or similar nodes in the target log data are in the same shard.

[0042] Step S104: Generate the analysis results of the target task based on the analysis response corresponding to the target log data.

[0043] In some embodiments, if the target log data is not sharded, the analysis response corresponding to the target log data can be directly determined as the analysis result of the target task; if the target log data has been sharded, the analysis response corresponding to the target log data includes the analysis response corresponding to each shard. In this case, the analysis result of the target task can be generated based on the analysis response corresponding to each shard.

[0044] Among these, the second major model can be used to generate the analysis results for the target task based on the analysis responses corresponding to each segment.

[0045] The second major model can be any major model, such as a large language model.

[0046] To ensure the accuracy of the analysis results output by the second major model, the second major model can generate analysis results for the target task based on the analysis responses corresponding to each segment and related prompts.

[0047] The relevant prompts are used to indicate the relationship between the analysis responses corresponding to each shard. For example, the relevant prompts could be: each analysis response is generated based on a subset of node logs, or: each analysis response is generated based on logs from similar times.

[0048] The log data processing method provided in this disclosure preprocesses the original log data of the target task to obtain target log data with a smaller data size. It can accurately and efficiently remove redundant and irrelevant log data and compress the input content size of the first large model. Based on the target log data, the context information of the target task, and the first large model, it determines the analysis response corresponding to the target log data. It can combine the context information of the target task and the capabilities of the large model to achieve accurate analysis of the target log data, making the analysis response corresponding to the target log data highly consistent with the actual scenario of the target task. Based on the analysis response corresponding to the target log data, it determines the analysis result of the target task, which can improve the accuracy and reliability of the analysis result.

[0049] Figure 2 This is a flowchart illustrating another log data processing method provided in an embodiment of the present disclosure.

[0050] like Figure 2 As shown, this log data processing method may include the following steps: Step S201: Obtain the raw log data of the target task.

[0051] The target task is the task to be analyzed from the logs.

[0052] The explanation of step S201 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0053] Step S202: Preprocess the original log data according to the log level to obtain the target log data, wherein the size of the target log data is smaller than the size of the original log data.

[0054] In some embodiments, for multiple first logs with a first predetermined log level in the original log data, the multiple first logs can be compressed according to the target log content of the multiple first logs to obtain first candidate logs; wherein, the target log content includes log content excluding time content; for multiple second logs with a second predetermined log level in the original log data, the multiple second logs can be extracted according to historical logs with a second predetermined log level to obtain second candidate logs; and target log data is generated based on the first candidate logs and / or the second candidate logs.

[0055] The first setting level could be, for example, the Info level. The second setting level could be, for example, the Warn level and / or the Error level.

[0056] If a log entry is "10:00:00 INFO Database connection pool: Current active connections 50", then "10:00:00 INFO Database connection pool: Current active connections 50" is the log content of that log entry. Here, "10:00:00" is the time content of the log entry, "INFO" is the log level content of the log entry, and "Database connection pool: Current active connections 50" is the main content of the log entry.

[0057] Specifically, for logs at the first set level, compression processing is performed on log content excluding time information to accurately and efficiently remove redundant and irrelevant log data, reducing the data volume of the first candidate logs. For logs at the second set level, extraction processing is performed in conjunction with historical logs to accurately select logs with reference value to form the second candidate logs, improving the targeting and accuracy of log extraction. Finally, the target log data is generated based on the first candidate logs and / or the second candidate logs, achieving reasonable integration of log data at different levels, so that the target log data retains key core information while also ensuring data simplicity.

[0058] In some embodiments, multiple first logs can be clustered according to the target log content of multiple first logs to obtain at least one cluster; the first logs in the target cluster can be compressed to obtain a compression result; wherein, the target cluster is a cluster in at least one cluster where the number of first logs is higher than a set number threshold; and a first candidate log is determined based on the compression result and the uncompressed first logs.

[0059] As an example, for any first log entry, the feature vector of the first log entry can be determined based on the target log content of the first log entry; based on the feature vectors of multiple first log entries, multiple first log entries can be clustered to obtain at least one cluster.

[0060] The feature vector of the first log can be determined based on the target log content using any vectorization method. Vectorization methods include, but are not limited to, TF-IDF (Term Frequency - Inverse Document Frequency) and word embedding.

[0061] This can be achieved by using any clustering algorithm, based on the feature vectors of multiple first log entries, to cluster first log entries with similar target log content into the same cluster. Clustering algorithms include, but are not limited to, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and K-Means algorithms.

[0062] As another example, for any first log entry, the content of the target log entry can be replaced according to the set rules to obtain the templated content corresponding to the first log entry; wherein, the set rules are used to indicate that the set variables in the target log content are replaced with target placeholders; the templated content corresponding to the first log entry is hashed to obtain the fingerprint information of the first log entry; based on the fingerprint information of multiple first log entries, multiple first log entries are clustered to obtain at least one cluster.

[0063] One example of setting rules is replacing common variables in regular expression matching (such as timestamps, IP addresses, numeric IDs, variable parts of file paths, etc.) with a uniform placeholder (such as...). <timestamp> 、 <ip> 、 <number> 、 <path>For example, assuming the original log is [2023-10-27 08:30:25.123] INFO com.example.Service- Processing request from user 15378, result=SUCCESS, then the following templated content can be obtained: [ <timestamp>] INFO com.example.Service - Processing request from user <number>, result=SUCCESS.

[0064] Specifically, an arbitrary hash algorithm is used to calculate the hash value of the templated content corresponding to the first log, and this hash value is used as the fingerprint information of the first log. Hash algorithms include, but are not limited to, MD5 (Message-Digest Algorithm 5) and SHA-1 (Secure Hash Algorithm 1).

[0065] This method employs any clustering algorithm, clustering based on the fingerprint information of multiple first log entries to group first log entries with similar target log content into the same cluster. Clustering algorithms include, but are not limited to, the DBSCAN algorithm and the K-Means algorithm.

[0066] Specifically, the first log in the target cluster can be sampled and compressed according to a set ratio, and the sampled first log in the target cluster is used as the compression result.

[0067] Among them, the first log in the compressed result and the first uncompressed log can be identified as the first candidate log.

[0068] The logs in the first candidate log can be sorted according to their time content.

[0069] Specifically, clustering operations based on the target log content of multiple first logs can group first logs with similar target log content into the same cluster; targeted compression of target clusters with more than a set threshold of first logs can simplify the first logs; and combining the compression results with the uncompressed first logs to determine the first candidate logs can reduce the amount of data in high-frequency first logs while ensuring that sparse log information is not lost by retaining low-frequency uncompressed first logs. This provides a more refined and comprehensive data foundation for subsequent log analysis, problem investigation, and other operations, thereby improving processing efficiency and quality.

[0070] In some embodiments, multiple first log entries can also be filtered.

[0071] As an example, if a set of filtering modules is preset, multiple first logs can be filtered according to the preset set of filtering modules to delete the first logs of multiple first logs whose modules are in the set of filtering modules.

[0072] For example, assuming the preset set of filtering modules is: ["heartbeat", "monitoring"], multiple first log entries include: First Log 1: 10:00:00 INFO heartbeat module: System is running normally Log 1, 2: 10:00:01 INFO user module: User login successful ... Then log 1 will be filtered out, while log 2 will be retained.

[0073] Specifically, by using a set of preset filtering modules to perform targeted filtering operations on multiple first log entries, it is possible to accurately and efficiently delete log entries whose modules are within the set of filtering modules, thereby reducing the size of the log data.

[0074] In some embodiments, preset keywords and / or a vector knowledge base determined based on historical logs can be obtained; a first search is performed on multiple second logs based on the preset keywords to retrieve second logs whose content includes the preset keywords, thereby obtaining a first search result; and / or, based on the vectorized representation of the multiple second logs and the vectorized representation of historical logs stored in the vector knowledge base, a similarity score is determined for the multiple second logs, and a second search is performed on the multiple second logs based on the similarity score to retrieve a first predetermined number of second logs with the highest similarity score, thereby obtaining a second search result; and a second candidate log is determined based on the first search result and / or the second search result.

[0075] By acquiring preset keywords and a vector knowledge base determined based on historical logs, and performing a first search on multiple second logs based on the preset keywords, and a second search by calculating similarity scores using the vector knowledge base and the vectorized representations of multiple second logs, it is possible to accurately filter out second logs that meet specific conditions from multiple second logs. The first search directly matches keywords to ensure rapid location of relevant logs, while the second search uses vector similarity scores to mine potentially similar logs. The combination of the two can comprehensively and efficiently determine second candidate logs, providing accurate data support for subsequent log analysis, troubleshooting, etc., and improving analysis efficiency and accuracy.

[0076] Specifically, for any one of multiple second logs, the vector similarity or vector distance between the second log and the corresponding historical log can be determined based on the vectorized representation of the second log and the vectorized representation of any historical log stored in the vector knowledge base; and the similarity score of the second log can be determined based on the vector similarity or vector distance between the second log and the corresponding historical log.

[0077] As an example, in response to the second log having a similarity lower than a set similarity threshold and any historical log stored in the vector knowledge base, or the second log having a vector distance lower than a set distance threshold, the similarity score of the second log is determined to be a first set value; wherein, the first set value is used to indicate that the second log is not similar to any historical log stored in the vector knowledge base; in response to the second log having a vector similarity not lower than a set similarity threshold and at least one historical log stored in the vector knowledge base, the similarity score of the second log is determined based on the vector similarity not lower than the set similarity threshold, or, in response to the second log having a vector distance not lower than a set distance threshold and at least one historical log stored in the vector knowledge base, the similarity score of the second log is determined based on the vector distance not lower than the set distance threshold.

[0078] Specifically, for different vector similarity values, a corresponding similarity score can be preset, and then the mean value of vector similarity that is not lower than the set similarity threshold can be determined. The similarity score corresponding to the mean value is determined as the similarity score of the second log.

[0079] Similarly, for different vector distance values, a corresponding similarity score can be preset, and then the mean value of the vector distances that are not lower than the set distance threshold can be determined. The similarity score corresponding to the mean value can be determined as the similarity score of the second log.

[0080] In some embodiments, for any second log in the first search result or the second search result, the second log, and a predetermined number of second logs whose time content is adjacent to the time content of the second log, can be determined as second candidate logs.

[0081] As an example, each log entry in the first or second search result, along with several log entries before and after it, can be extracted to form a second candidate log. The number of these "several" log entries can be determined based on the actual scenario and past experience, such as 100 log entries, to ensure the complete interpretation of the context of a single abnormal log entry and the inference of the correlation between the abnormality and its cause.

[0082] Specifically, for any second log entry in either the first or second search result, it is combined with a specified number of second log entries that are temporally adjacent to it to be identified as a second candidate log entry. This helps to uncover richer and more accurate potential information and related clues, providing more complete and reliable data support for subsequent log analysis and improving the accuracy of the analysis results.

[0083] Step S203: Determine the analysis response corresponding to the target log data based on the target log data, the context information of the target task, and the first major model.

[0084] Step S204: Generate the analysis results of the target task based on the analysis response corresponding to the target log data.

[0085] The explanation of steps S203-S204 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0086] The log data processing method provided in this disclosure preprocesses the original log data according to the log level to obtain the target log data. It can perform different preprocessing operations on log data of different log levels based on the characteristics of the log data itself, and can efficiently and accurately simplify the log data while ensuring that the simplified log data has complete interpretability.

[0087] Figure 3 This is a flowchart illustrating another log data processing method provided in an embodiment of the present disclosure.

[0088] like Figure 3 As shown, this log data processing method may include the following steps: Step S301: Obtain the raw log data of the target task.

[0089] The target task is the task to be analyzed from the logs.

[0090] Step S302: Preprocess the original log data to obtain target log data, wherein the size of the target log data is smaller than the size of the original log data.

[0091] The explanation of steps S301-S302 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0092] Step S303: Obtain the first prompt text template, wherein the first prompt text template is used for the analysis and response generation process.

[0093] As an example, the first prompt text template is used to constrain the content generated by the first major model. For instance, it can constrain the first major model to output an analysis response in the order of describing the phenomenon, listing evidence (e.g., key logs), providing reasons, drawing conclusions, and offering suggested solutions, thereby improving the accuracy of the analysis response. Furthermore, the first major model can be required to provide possible reasons in order of confidence index. The confidence index is a subjective rating given by the first major model based on the description of the phenomenon, evidence (e.g., key logs), and its internal knowledge, assessing the likelihood of each reason being true. This rating can be expressed as a percentage (e.g., 80%), probability (e.g., high / medium / low), or description (e.g., "very likely," "possible," "less likely").

[0094] The first prompt text template could be something like: "Describe the phenomenon you observed, list the key log data required for the analysis of this phenomenon, analyze the possible causes, and sort the causes by confidence index from high to low. Finally, based on the above analysis, summarize the conclusions and provide suggested solutions."

[0095] Step S304: Determine the first prompt text based on the target log data, the context information of the target task, and the first prompt text template.

[0096] Among them, the context information of the target task records information related to the target task, which can help the first major model to better analyze the target log data.

[0097] As an example, if the target task is a large model training task, the context information of the target task can include how many nodes the large model training task used, how many training steps were executed, how long it took to run, GPU metrics, latency metrics, etc.

[0098] In some embodiments, the target log data, the context information of the target task, and the first prompt text template can be concatenated to obtain the first prompt text.

[0099] Step S305: Input the first prompt text into the first large model and obtain the analysis response corresponding to the target log data output by the first large model.

[0100] The first prompt text may exceed the input limit of the first large model. Therefore, in some embodiments, in response to the first prompt text exceeding the input limit of the first large model, the target log data is segmented to obtain at least one target log data segment; wherein, the second prompt text corresponding to any target log data segment does not exceed the input limit of the first large model, and the second prompt text is determined based on the corresponding target log data segment, the context information of the target task, and the first prompt text template; the second prompt text corresponding to at least one target log data segment is input into the first large model respectively to obtain the analysis response corresponding to at least one target log data segment output by the first large model.

[0101] The system can be sharded according to log length or number of log entries. Each shard needs to have a certain overlap (e.g., 5%, which can be determined based on the actual scenario and past experience) to ensure continuity (similar to a sliding window).

[0102] This can be done by sharding logs according to log nodes, with the aim of ensuring that logs from the same or similar nodes in the target log data are in the same shard.

[0103] Specifically, by segmenting the target log data that exceeds the input limit of the first major model, and generating a second prompt text that meets the input requirements and is based on the context information of the corresponding target log data segment, the target task, and the first prompt text template for each target log data segment, the input of the second prompt text into the first major model to obtain the corresponding analysis response, the problem of input exceeding the limit due to excessive data volume is solved. This ensures that the analysis of each target log data segment has a clear task orientation and context support, thus guaranteeing the execution effect of the first major model.

[0104] Step S306: Generate the analysis results of the target task based on the analysis response corresponding to the target log data.

[0105] In some embodiments, the analysis response corresponding to the target log data includes an analysis response corresponding to at least one target log data shard, thereby obtaining a second prompt text template; wherein, the second prompt text template is used for the generation and processing of analysis results; a third prompt text is determined based on the analysis response corresponding to at least one target log data shard and the second prompt text template; the third prompt text is input into a second large model to obtain the analysis results output by the second large model.

[0106] As an example, the second prompt text template is used to indicate the relationship between the analysis responses corresponding to each shard. For example, the second prompt text template could be: each analysis response is generated based on a subset of node logs, or: each analysis response is generated based on logs from similar times.

[0107] Specifically, by obtaining a second prompt text template specifically for generating and processing analysis results, a standardized and structured guidance framework can be provided for subsequent processing. Based on the second prompt text template, the third prompt text is accurately determined by combining the analysis response corresponding to at least one target log data shard, ensuring the completeness and relevance of the input content. After inputting the third prompt text into the second major model, the powerful data processing and intelligent analysis capabilities of the second major model can be fully utilized to efficiently obtain accurate, comprehensive, and practical analysis results.

[0108] In some embodiments, in response to the third prompt text exceeding the input limit of the second major model, the content of the analysis response corresponding to at least one target log data shard is extracted to obtain the corresponding summary content; based on the corresponding summary content and the second prompt template, a fourth prompt text is determined; the fourth prompt text is input into the second major model to obtain the analysis results output by the second major model.

[0109] In other words, when the third prompt text exceeds the input limit of the second major model, the content of the analysis response corresponding to each target log data segment can be extracted to obtain the summary content corresponding to each target log data segment. Then, based on the summary content corresponding to each target log data segment and the second prompt template, the fourth prompt text is determined and input into the second major model to obtain the analysis results output by the second major model.

[0110] Specifically, when the third prompt text exceeds the input limit of the second major model, the corresponding summary content is obtained by extracting the content of the analysis response corresponding to each target log data segment. This enables efficient compression of the analysis response corresponding to each target log data segment, reducing the data volume. Based on the summary content corresponding to each target log data segment and the second prompt template, the fourth prompt text is determined and input into the second major model to obtain the analysis results. This solves the problem of input exceeding the limit due to excessive data volume and improves the model's ability to handle complex inputs.

[0111] Further explanations of step S306 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0112] The log data processing method provided in this embodiment first obtains a first prompt text template for analysis response generation, then determines the first prompt text by combining the target log data and the context information of the target task, and inputs it into a first large model to obtain the analysis response. This provides the first large model with more accurate and comprehensive input content, enabling the first large model to better understand the log semantics and specific analysis objectives, thereby generating a more accurate, demand-responsive, and interpretable analysis response.

[0113] To clearly illustrate the above embodiments, examples are given below.

[0114] Figure 4 This is a schematic diagram illustrating the principle of a log data processing method provided in an embodiment of the present disclosure.

[0115] like Figure 4 As shown, this log data processing method may include the following steps: Step 1: Log preprocessing to obtain the target log data.

[0116] In this step, logs can be streamlined from multiple perspectives to reduce the input of the first major model while retaining the effective information in the logs.

[0117] Log preprocessing targets the raw log data of the target task, which can be the raw log data of the target task within a target time period. The target time period can be determined based on the time at which log analysis of the target task is performed. For example, assuming the target task is interrupted at time t, and log analysis is required at this time, the raw log data of the target task within a defined time period before time t can be obtained.

[0118] The specific steps of log preprocessing include: 1. Filter and compress Info-level logs. If a set of filtering modules is preset, then the Info level logs can be filtered according to the preset set of filtering modules to delete the Info level logs whose modules are in the set of filtering modules. Furthermore, for a large number of logs at the Info level that are highly similar in content except for timestamps, they are sampled and compressed proportionally to obtain the processing results for the Info level logs.

[0119] 2. Extract and process logs at the Warn and / or Error levels. It can obtain preset keywords determined based on historical logs at the Warn and / or Error levels, and extract logs containing preset keywords from the logs at the Warn and / or Error levels. It can obtain a vector knowledge base determined based on historical logs at the Warn and / or Error levels, vectorize the logs at the Warn and / or Error levels, determine the similarity score of the logs at the Warn and / or Error levels based on the vectorized representation of the logs at the Warn and / or Error levels, and the vectorized representation of the historical logs stored in the vector knowledge base, and extract the N logs with the highest similarity scores from the logs at the Warn and / or Error levels. Each extracted log entry, along with several preceding and following log entries, is used as the processing result for Warn and / or Error level logs. The number of these "several" entries can be determined based on the actual scenario and past experience, for example, 100 entries, to ensure the complete interpretation of the context of each abnormal log entry and the correlation inference of the cause of the abnormality.

[0120] Furthermore, the target log data can be generated based on the processing results of the Info level logs, and / or the processing results of the Warn level and / or Error level logs. For example, the processing results of the Info level logs, as well as the processing results of the Warn level and / or Error level logs, can be merged to obtain the target log data.

[0121] Step 2: The first major model generates the analysis response corresponding to the target log data. In this step, the target log data is input into the first main model, and the analysis response corresponding to the target log data output by the first main model is obtained. The target log data can be segmented based on the input limitations of the first main model. Furthermore, to ensure the accuracy of the analysis response output by the first main model, the input can also include the contextual information of the target task and the initial prompt text template.

[0122] In this context, sharding refers to dividing the target log data into several target log data shards if the target log data, the context information of the target task, and the first prompt text template exceed the input limit of the first major model.

[0123] When determining the number of shards, it is necessary to ensure that the target log data, target task context information, and related prompts for each shard do not exceed the input limit of the first major model.

[0124] The system can be sharded according to log length or number of log entries. Each shard needs to have a certain overlap (e.g., 5%, which can be determined based on the actual scenario and past experience) to ensure continuity (similar to a sliding window).

[0125] This can be done by sharding logs according to log nodes, with the aim of ensuring that logs from the same or similar nodes in the target log data are in the same shard.

[0126] Understandably, if the target log data, the context information of the target task, and the first prompt text template do not exceed the input limit of the first major model, then no sharding is required, and the data can be directly input into the first major model to obtain the analysis response corresponding to the target log data output by the first major model.

[0127] Among them, the context information of the target task records information related to the target task, which can help the first major model to better analyze the target log data.

[0128] As an example, if the target task is a large model training task, the context information of the target task can include how many nodes the large model training task used, how many training steps were executed, how long it took to run, GPU metrics, latency metrics, etc.

[0129] The first prompt text template is used to analyze the generation and processing of responses.

[0130] As an example, the first prompt text template is used to constrain the content generated by the first major model. For instance, it can constrain the first major model to output an analysis response in the order of describing the phenomenon, listing evidence (e.g., key logs), providing reasons, drawing conclusions, and offering suggested solutions, thereby improving the accuracy of the analysis response. Furthermore, the first major model can be required to provide possible reasons in order of confidence index. The confidence index is a subjective rating given by the first major model based on the description of the phenomenon, evidence (e.g., key logs), and its internal knowledge, assessing the likelihood of each reason being true. This rating can be expressed as a percentage (e.g., 80%), probability (e.g., high / medium / low), or description (e.g., "very likely," "possible," "less likely").

[0131] The first prompt text template could be something like: "Describe the phenomenon you observed, list the key log data required for the analysis of this phenomenon, analyze the possible causes, and sort the causes by confidence index from high to low. Finally, based on the above analysis, summarize the conclusions and provide suggested solutions."

[0132] Therefore, through step two, N analytical responses can be obtained (N≥1).

[0133] Step 3: Based on the analysis response corresponding to the target log data, generate the analysis results for the target task.

[0134] In this step, N analytical responses (N>1) can be merged and summarized.

[0135] Summary methods include: 1. Direct summary If the N analysis responses (N>1) + the second prompt text template do not exceed the input limit of the second largest model, then input the N analysis responses (N>1) + the second prompt text template into the second largest model and obtain the analysis results output by the second largest model.

[0136] 2. Summary If the N analytical responses (N>1) plus the second prompt text template exceed the input limit of the second major model, then the content of each of the N analytical responses (N>1) is extracted to obtain N summary contents (N>1). Then, the N summary contents (N>1) plus the second prompt text template are input into the second major model to obtain the analytical results output by the second major model.

[0137] The second prompt text template is used for generating and processing the analysis results.

[0138] As an example, the second prompt text template is used to indicate the relationship between the analysis responses corresponding to each shard. For example, the second prompt text template could be: each analysis response is generated based on a subset of node logs, or: each analysis response is generated based on logs from similar times.

[0139] In other words, regardless of the summary method, the relationship between multiple reports must be clearly stated in the merge summary processing prompt (e.g., each analysis response is generated based on some node logs, or each analysis response is generated based on logs from similar times) in order to facilitate the final decision.

[0140] It is understandable that if step two does not perform segmentation, then step two can yield one analysis response. In this case, there is no need to summarize, and the analysis response can be directly identified as the analysis result of the target task.

[0141] In summary, this approach can solve the problems of difficulty in ensuring task processing quality and exceeding the input limits of large models when directly inputting massive amounts of logs, as well as the problems caused by crudely pruning logs, such as disrupting the integrity of the log context and affecting the effectiveness and accuracy of task execution.

[0142] Figure 5 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present disclosure.

[0143] like Figure 5 As shown, the log data processing apparatus 500 of this embodiment includes an acquisition module 501, a processing module 502, a determination module 503, and a generation module 504.

[0144] The acquisition module 501 is used to acquire the raw log data of the target task; where the target task is the task to be analyzed by logs. Processing module 502 is used to preprocess the raw log data to obtain target log data; wherein the data size of the target log data is smaller than the data size of the raw log data. The determination module 503 is used to determine the analysis response corresponding to the target log data based on the target log data, the context information of the target task, and the first major model. The generation module 504 is used to generate the analysis results of the target task based on the analysis response corresponding to the target log data.

[0145] In one embodiment of this disclosure, the processing module 502 is further configured to: preprocess the raw log data according to the log level to obtain target log data.

[0146] In one embodiment of this disclosure, the processing module 502 is further configured to: compress multiple first logs with a log level of a first predetermined level in the original log data according to the target log content of the multiple first logs to obtain a first candidate log; wherein the target log content includes log content other than time content in the log content; extract multiple second logs with a log level of a second predetermined level in the original log data according to historical logs with a log level of the second predetermined level to obtain a second candidate log; and generate target log data according to the first candidate log and / or the second candidate log.

[0147] In one embodiment of this disclosure, the processing module 502 is further configured to: cluster the multiple first logs according to the target log content of the multiple first logs to obtain at least one cluster; compress the first logs in the target cluster to obtain a compression result; wherein, the target cluster is a cluster in at least one cluster where the number of first logs is higher than a set number threshold; and determine a first candidate log based on the compression result and the uncompressed first logs.

[0148] In one embodiment of this disclosure, the above-mentioned device further includes: a filtering module, configured to filter multiple first logs according to a preset set of filtering modules, so as to delete the first logs in the multiple first logs whose module information is within the set of filtering modules.

[0149] In one embodiment of this disclosure, the processing module 502 is further configured to: acquire preset keywords and / or a vector knowledge base determined based on historical logs; perform a first search on multiple second logs according to the preset keywords to retrieve second logs whose content includes the preset keywords from the multiple second logs, and obtain a first search result; and / or, determine a similarity score for multiple second logs based on the vectorized representation of the multiple second logs and the vectorized representation of historical logs stored in the vector knowledge base, and perform a second search on the multiple second logs according to the similarity score to retrieve a first predetermined number of second logs with the highest similarity score from the multiple second logs, and obtain a second search result; and determine a second candidate log based on the first search result and / or the second search result.

[0150] In one embodiment of this disclosure, the processing module 502 is further configured to: for any second log in the first search result or the second search result, determine the second log, and a second set number of second logs whose time information is adjacent to the time information of the second log as second candidate logs.

[0151] In one embodiment of this disclosure, the determining module 503 includes: a first obtaining unit, configured to obtain a first prompt text template; wherein the first prompt text template is used for the generation process of the analysis response; a first determining unit, configured to determine a first prompt text based on the target log data, the context information of the target task, and the first prompt text template; and a first processing unit, configured to input the first prompt text into a first large model and obtain the analysis response corresponding to the target log data output by the first large model.

[0152] In one embodiment of this disclosure, the first processing unit is further configured to: in response to the first prompt text exceeding the input limit of the first large model, perform segmentation processing on the target log data to obtain at least one target log data segment; wherein, the second prompt text corresponding to any target log data segment does not exceed the input limit of the first large model, and the second prompt text is determined based on the corresponding target log data segment, the context information of the target task, and the first prompt text template; input the second prompt text corresponding to at least one target log data segment into the first large model respectively, and obtain the analysis response corresponding to at least one target log data segment output by the first large model.

[0153] In one embodiment of this disclosure, the analysis response corresponding to the target log data includes analysis responses corresponding to at least one target log data shard; the generation module 504 includes: a second acquisition unit, used to acquire a second prompt text template; wherein the second prompt text template is used for the generation processing of analysis results; a second determination unit, used to determine a third prompt text based on the analysis responses corresponding to at least one target log data shard and the second prompt text template; and a second processing unit, used to input the third prompt text into a second large model and acquire the analysis results output by the second large model.

[0154] In one embodiment of this disclosure, the second processing unit is further configured to: in response to the third prompt text exceeding the input limit of the second large model, extract the content of the analysis response corresponding to at least one target log data shard to obtain the corresponding summary content; determine the fourth prompt text based on the corresponding summary content and the second prompt template; input the fourth prompt text into the second large model to obtain the analysis result output by the second large model.

[0155] The demonstration device provided in this embodiment preprocesses the original log data of the target task to obtain target log data with a smaller data size. It can accurately and efficiently remove redundant and irrelevant log data and compress the input content size of the first large model. Based on the target log data, the context information of the target task, and the first large model, it determines the analysis response corresponding to the target log data. It can combine the context information of the target task and the capabilities of the large model to achieve accurate analysis of the target log data, making the analysis response corresponding to the target log data highly consistent with the actual scenario of the target task. Based on the analysis response corresponding to the target log data, it determines the analysis result of the target task, which can improve the accuracy and reliability of the analysis result.

[0156] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0157] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0158] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0159] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on computer programs / instructions stored in read-only memory (ROM) 602 or loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0160] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606 such as keyboard, mouse, etc.; output unit 607 such as various types of monitors, speakers, etc.; storage unit 608 such as disk, optical disk, etc.; and communication unit 609 such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as log data processing methods. For example, in some embodiments, the log data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program / instructions may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program / instructions are loaded into RAM 603 and executed by the computing unit 601, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform log data processing methods by any other suitable means (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0167] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. The client-server relationship is created by computer programs / instructions running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0168] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in the disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this document does not impose any restrictions.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / number> < / timestamp> < / path> < / number> < / ip> < / timestamp>

Claims

1. A log data processing method, the method comprising: Obtain the raw log data of the target task; wherein, the target task is the task to be analyzed by logs. The original log data is preprocessed to obtain target log data; wherein the size of the target log data is smaller than the size of the original log data. Based on the target log data, the context information of the target task, and the first major model, determine the analysis response corresponding to the target log data; Based on the analysis response corresponding to the target log data, the analysis results of the target task are generated.

2. The method according to claim 1, wherein, The preprocessing of the original log data to obtain the target log data includes: The original log data is preprocessed according to the log level to obtain the target log data.

3. The method according to claim 2, wherein, The step of preprocessing the raw log data according to the log level to obtain the target log data includes: For multiple first logs with a first set log level in the original log data, the multiple first logs are compressed according to the target log content of the multiple first logs to obtain a first candidate log; wherein, the target log content includes the log content excluding the time content. For multiple second logs with a log level of the second set level in the original log data, extract and process the multiple second logs according to the historical logs with a log level of the second set level to obtain second candidate logs; The target log data is generated based on the first candidate log and / or the second candidate log.

4. The method according to claim 3, wherein, The step of compressing the multiple first logs based on their target log contents to obtain first candidate logs includes: Based on the target log content of the multiple first logs, the multiple first logs are clustered to obtain at least one cluster; The first log in the target cluster is compressed to obtain the compression result; wherein, the target cluster is the cluster in which the number of first logs in the at least one cluster is higher than a set number threshold; Based on the compression result and the uncompressed first log, the first candidate log is determined.

5. The method according to claim 2, wherein, The method further includes: Based on a preset set of filtering modules, the multiple first log entries are filtered to delete the first log entries whose modules are within the set of filtering modules.

6. The method according to claim 2, wherein, The step of extracting and processing multiple second logs based on historical logs at the second set log level to obtain second candidate logs includes: Obtain a preset keyword and / or vector knowledge base determined based on the historical logs; Based on the preset keywords, a first search is performed on the plurality of second logs to retrieve second logs whose log content includes the preset keywords, thereby obtaining the first search result; And / or, Based on the vectorized representations of the multiple second logs and the vectorized representations of the historical logs stored in the vector knowledge base, a similarity score is determined for the multiple second logs. Based on the similarity scores of the multiple second logs, a second retrieval is performed on the multiple second logs to retrieve a first predetermined number of second logs with the highest similarity scores from the multiple second logs, thereby obtaining the second retrieval results. The second candidate log is determined based on the first search result and / or the second search result.

7. The method according to claim 6, wherein, The step of determining the second candidate log based on the first search result and / or the second search result includes: For any second log in the first search result or the second search result, the second log, and a predetermined number of second logs whose time content is adjacent to the time content of the second log, are determined as the second candidate log.

8. The method according to claim 1, wherein, The step of determining the analysis response corresponding to the target log data based on the target log data, the context information of the target task, and the first major model includes: Obtain the first prompt text template; wherein, the first prompt text template is used to analyze the generation and processing of the response; The first prompt text is determined based on the target log data, the context information of the target task, and the first prompt text template; Input the first prompt text into the first large model, and obtain the analysis response corresponding to the target log data output by the first large model.

9. The method according to claim 8, wherein, The step of inputting the first prompt text into the first large model and obtaining the analysis response corresponding to the target log data output by the first large model includes: In response to the first prompt text exceeding the input limit of the first large model, the target log data is segmented to obtain at least one target log data segment; wherein, the second prompt text corresponding to any target log data segment does not exceed the input limit of the first large model, and the second prompt text is determined based on the corresponding target log data segment, the context information of the target task, and the first prompt text template; Input the second prompt text corresponding to the at least one target log data shard into the first large model, and obtain the analysis response corresponding to the at least one target log data shard output by the first large model.

10. The method according to claim 9, wherein, The analysis response corresponding to the target log data includes the analysis response corresponding to at least one target log data fragment; The step of generating the analysis result for the target task based on the analysis response corresponding to the target log data includes: Obtain the second prompt text template; wherein, the second prompt text template is used for the generation and processing of analysis results; Based on the analysis response corresponding to the at least one target log data shard and the second prompt text template, determine the third prompt text; Input the third prompt text into the second large model and obtain the analysis results output by the second large model.

11. The method according to claim 10, wherein, The step of inputting the third prompt text into the second large model and obtaining the analysis results output by the second large model includes: In response to the third prompt text exceeding the input limit of the second large model, the analysis response corresponding to the at least one target log data shard is extracted to obtain the corresponding summary content; Based on the corresponding summary content and the second prompt template, determine the fourth prompt text; Input the fourth prompt text into the second largest model and obtain the analysis results output by the second largest model.

12. A log data processing apparatus, the apparatus comprising: The acquisition module is used to acquire the raw log data of the target task; wherein, the target task is the task to be analyzed by logs. The processing module is used to preprocess the original log data to obtain target log data; wherein the size of the target log data is smaller than the size of the original log data. The determination module is used to determine the analysis response corresponding to the target log data based on the target log data, the context information of the target task, and the first major model; The generation module is used to generate the analysis results of the target task based on the analysis response corresponding to the target log data.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1-11.

15. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-11.

Images