Knowledge base updating method and device, equipment and storage medium
Through text processing and semantic association models, log text vectors are generated and the knowledge base is updated, which solves the problem of low knowledge base update accuracy in multilingual scenarios and realizes efficient update of the multilingual log analysis system.
Patent Information
- Application Number
- CN202510882485.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-10-17
AI Technical Summary
The existing technology has low accuracy in updating the knowledge base of the log analysis system in a multilingual scenario and cannot effectively support the needs of multilingual log analysis.
By obtaining the log text to be processed, using the text processing model to generate log text vectors, and using the semantic association model to calculate the vector distance, the historical text content in the preset knowledge base is updated based on the distance, and a differentiated processing method is used to process log texts from web pages and user input.
It improves the accuracy and efficiency of knowledge base updates in multilingual scenarios, ensures that multi-source and multilingual log texts are converted into semantic vectors of unified dimensions, and achieves accurate knowledge base updates.
Smart Images
Figure CN120804115A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and particularly relates to a knowledge base updating method and device, equipment and a storage medium. BACKGROUND
[0002] With the development of deep learning and artificial intelligence technology, the popularity of enterprise-level systems, distributed architectures and cross-border businesses has led to an explosive growth in log data, and the data form is becoming increasingly complex. Log analysis systems based on artificial intelligence (AI) have gradually become mainstream.
[0003] In the prior art, content updating is usually achieved by full fine-tuning and retrieval enhancement to generate external data sources, but the log analysis results lack complete context support and cannot accurately and effectively update the knowledge base of the log analysis system in a multilingual scenario.
[0004] Therefore, how to improve the accuracy of knowledge base updating in a multilingual scenario has become a technical problem that needs to be solved. SUMMARY
[0005] Embodiments of the present application provide a knowledge base updating method, device, equipment and storage medium to solve the problem of low accuracy of knowledge base updating in a multilingual scenario in the prior art.
[0006] In a first aspect, embodiments of the present application provide a knowledge base updating method, comprising:
[0007] obtaining log text of a log to be processed;
[0008] processing the log text through a text processing model to obtain a log text vector corresponding to the log text;
[0009] processing the historical text vector and the log text vector through a semantic association model according to the historical text vector and the log text vector in a preset knowledge base to obtain a vector distance between the historical text vector and the log text vector;
[0010] updating historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance.
[0011] In a possible implementation, the log text of the log to be processed is obtained, comprising:
[0012] obtaining a log to be processed;
[0013] verifying a uniform resource locator (URL) format of the log to be processed;
[0014] If the URL format is met, a web page pointed by the URL is parsed and incremental crawling is performed to obtain the log text;
[0015] If not, the log to be processed is preprocessed to obtain the log text; the preprocessing includes at least one of language detection processing, text cleaning processing, and text block processing.
[0016] In a possible implementation, the log text is processed by a text processing model to obtain a log text vector corresponding to the log text, including:
[0017] The log text is subjected to language detection by a language detection unit in the text processing model to obtain a language detection result;
[0018] The language detection result is subjected to vectorization processing by a vectorization unit in the text processing model to obtain the log text vector.
[0019] In a possible implementation, based on the vector distance, the historical text content in the preset knowledge base is updated, including:
[0020] According to the vector distance, it is determined whether the log text vector and the historical text vector are synonymous sentences;
[0021] If yes, based on the similarity of the log text vector and the historical text vector, the historical text content corresponding to the historical text vector is updated;
[0022] If not, the log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0023] In a possible implementation, based on the similarity of the log text vector and the historical text vector, the historical text content corresponding to the historical text vector is updated, including:
[0024] The similarity between the log text vector and the historical text vector is calculated;
[0025] When the similarity is greater than or equal to a preset similarity threshold, the historical text vector is replaced by the log text vector to update the historical text content corresponding to the historical text vector;
[0026] When the similarity is less than the preset similarity threshold, the log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0027] In a possible implementation, the method further includes:
[0028] In response to a monitoring trigger condition, the historical text content corresponding to the target log is obtained from the preset knowledge base;
[0029] Analyze the historical text content based on the preset log analysis model to obtain the analysis results of the target log;
[0030] Obtain user feedback on the analysis results of the target log;
[0031] Update the preset knowledge base based on feedback data.
[0032] In one possible implementation, historical text content is analyzed based on a preset log analysis model to obtain analysis results of the target log, including:
[0033] When the length of the historical text content exceeds a length threshold, the historical text content is divided into a plurality of text content fields based on a preset log field to obtain a text content sequence;
[0034] Input the text content sequence into the preset log analysis model to obtain the log response field and log analysis template;
[0035] Set the log ID of the target log;
[0036] Based on the log reply fields, log analysis template and log identifier, the analysis results of the target log are obtained.
[0037] In a second aspect, an embodiment of the present application provides a knowledge base updating device, comprising:
[0038] The acquisition module is used to obtain the log text of the log to be processed;
[0039] A processing module, configured to process the log text using a text processing model to obtain a log text vector corresponding to the log text;
[0040] The processing module is further used to process the historical text vector and the log text vector in the preset knowledge base through the semantic association model to obtain the vector distance between the historical text vector and the log text vector;
[0041] The updating module is used to update the historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance.
[0042] In a possible implementation, the acquisition module is specifically configured to:
[0043] Get the logs to be processed;
[0044] Verify the URL format of the log to be processed;
[0045] If the URL format is met, the web page pointed to by the URL is parsed and incrementally crawled to obtain the log text;
[0046] If not, the log to be processed is preprocessed to obtain log text; the preprocessing includes at least one of language detection processing, text cleaning processing, and text blocking processing.
[0047] In a possible implementation, the processing module is specifically configured to:
[0048] The language detection unit in the text processing model performs language detection on the log text to obtain a language detection result;
[0049] The vectorization unit in the text processing model performs vectorization processing on the language detection result to obtain a log text vector.
[0050] In a possible implementation, the updating module is specifically configured to:
[0051] According to the vector distance, it is determined whether the log text vector and the historical text vector are synonymous sentences;
[0052] If yes, the historical text content corresponding to the historical text vector is updated based on the similarity between the log text vector and the historical text vector;
[0053] If no, the log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0054] In a possible implementation, based on the similarity between the log text vector and the historical text vector, the updating module is specifically configured to:
[0055] The similarity between the log text vector and the historical text vector is calculated;
[0056] When the similarity is greater than or equal to a preset similarity threshold, the historical text vector is replaced by the log text vector to update the historical text content corresponding to the historical text vector;
[0057] When the similarity is less than the preset similarity threshold, the log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0058] In a possible implementation, the knowledge base updating apparatus further includes:
[0059] The acquisition module is further configured to acquire, in response to a monitoring trigger condition, historical text content corresponding to the target log from the preset knowledge base;
[0060] The processing module is further configured to analyze the historical text content based on a preset log analysis model to obtain an analysis result of the target log;
[0061] The acquisition module is further configured to acquire feedback data of the analysis result of the target log by the user.
[0062] The updating module is further configured to update the preset knowledge base based on the feedback data.
[0063] In a possible implementation, the processing module is specifically configured to:
[0064] When the length of the historical text content exceeds the length threshold, the historical text content is divided into a plurality of text content fields based on preset log fields, to obtain a text content sequence;
[0065] The text content sequence is input into a preset log analysis model to obtain a log reply field and a log analysis template;
[0066] The log identifier of the target log is set;
[0067] Based on the log reply field, the log analysis template and the log identifier, an analysis result of the target log is obtained.
[0068] In a third aspect, an electronic device is provided, which includes a processor and a memory connected with the processor in communication;
[0069] The memory stores computer-executable instructions;
[0070] The processor executes the computer-executable instructions stored in the memory to implement the method of the first aspect or any of the manners.
[0071] In a fourth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method of the first aspect or any of the manners.
[0072] In a fifth aspect, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium, at least one processor can read the computer program from the computer-readable storage medium, and the at least one processor executes the computer program to implement the method of the first aspect or any of the manners.
[0073] The knowledge base updating method, device, equipment and storage medium provided by the embodiments of the present application first acquire log text of a to-be-processed log. The to-be-processed log can come from different information sources, and the corresponding log text can contain different kinds of languages. The log text is processed by a text processing model to obtain a log text vector corresponding to the log text, so as to convert the multi-source multi-language log text into a unified dimension semantic vector. Then, according to a historical text vector in a preset knowledge base and the log text vector, a semantic association model is processed to accurately calculate the vector distance between the historical text vector and the log text vector. Finally, the historical text content corresponding to the historical text vector in the preset knowledge base is updated based on the vector distance, thereby improving the accuracy of knowledge base updating in a multi-language scenario. BRIEF DESCRIPTION OF DRAWINGS
[0074] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.
[0075] Figure 1 Flowchart of the knowledge base updating method provided by the embodiments of the present application Figure One ;
[0076] Figure 2 Processing flowchart of the semantic association model provided by the embodiments of the present application
[0077] Figure 3 Flowchart of the knowledge base updating method provided by the embodiments of the present application Figure Two ;
[0078] Figure 4 Flowchart of the knowledge base updating method provided by the embodiments of the present application Figure Three ;
[0079] Figure 5 Flowchart of the analysis of the target log provided by the embodiments of the present application
[0080] Figure 6 Structure diagram of the knowledge base updating device provided by the embodiments of the present application
[0081] Figure 7 Structure diagram of the electronic device provided by the embodiments of the present application.
[0082] The above-described drawings have shown the specific embodiments of the present application, and the following will have a more detailed description. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0083] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0084] Before introducing the embodiments of the present application, the application background of the embodiments of the present application is first explained:
[0085] With the development of deep learning and artificial intelligence technology, the popularity of enterprise-level systems, distributed architectures, and cross-border businesses makes log data show explosive growth, and the data form becomes more and more complex. AI-based log analysis systems have gradually become mainstream.
[0086] The traditional AI combined log analysis system can only analyze part of the log content, and the log analysis result lacks complete context support, and it is difficult to meet the fault positioning and content analysis needs in complex scenarios.
[0087] In the prior art, the way of full fine-tuning and retrieval augmented generation (RAG) external data source is adopted to realize content update. The full training mode needs to consume a large amount of computing resources, and faces the dilemma of being unable to update for a long time. When RAG external data source, due to the lack of effective data cleaning mechanism, dirty data will cause the model efficiency to decrease significantly. When processing logs in other languages in a Chinese environment, the accuracy of the log analysis system is significantly reduced, and it cannot meet the log analysis needs in a multi-language scenario.
[0088] In summary, how to improve the update accuracy of the knowledge base of the log analysis system in a multi-language scenario has become a technical problem that needs to be solved urgently.
[0089] In view of the technical problems existing in the prior art, the inventors of the present application have the following idea: for the problem of low knowledge base update accuracy, after obtaining the to-be-processed log text, a text processing model is used to generate a corresponding log text vector, and a semantic association model is used to calculate the distance between the vector and a historical text vector in a preset knowledge base, and the historical text content in the knowledge base is updated based on the distance, thereby improving the accuracy and efficiency of the knowledge base update. For log texts from different sources, different processing methods are adopted, log texts from web pages are parsed and incrementally crawled, and log texts from user input are subjected to language detection, text cleaning, and block splitting, thereby further improving the accuracy of the knowledge base update in a multi-language scenario.
[0090] Wherein, the unexplained part is disclosed by the following embodiments.
[0091] The technical solutions of the present application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments.
[0092] Figure 1 Flowchart of the knowledge base updating method provided by the embodiments of the present application Figure One As shown in Figure 1 The method can include the following steps:
[0093] Step 11, obtaining log text of a to-be-processed log.
[0094] In this step, the log analysis system obtains the log text of the to-be-processed log to provide data support for subsequent semantic analysis.
[0095] The log text of the to-be-processed log can include web links and user input text.
[0096] Specifically, step 11 can have the following implementation manner:
[0097] Step 1, obtaining a to-be-processed log.
[0098] In this implementation, the log analysis system receives the to-be-processed log through the multi-source data access layer, and then identifies the type of the input source of the to-be-processed log to provide a basis for subsequent differentiated processing.
[0099] The multi-source data access layer can receive various to-be-processed log types of input such as web pages, URLs, and user input text.
[0100] Step 2, verifying the URL format of the to-be-processed log, if the URL format is correct, parsing and incremental crawling the web page to obtain the log text.
[0101] In this implementation, the log analysis system starts the web page parsing module, first sends a hypertext transfer protocol (HTTP) request to obtain the web page source code, extracts the text content through a hyper text markup language (HTML) parser, filters irrelevant information such as advertisements and navigation bars, calls a tableless browser to simulate user interaction to obtain complete data for dynamically loaded web page content, only crawls newly added or modified log segments to reduce data processing volume, and finally generates log text that can be used for subsequent analysis.
[0102] Step 3, if the to-be-processed log does not conform to the URL format, preprocessing the to-be-processed log to obtain the log text.
[0103] The preprocessing includes at least one of language detection processing, text cleaning processing, and text chunking processing.
[0104] In this implementation, the log analysis system pre-processes the to-be-processed log when determining that the to-be-processed log comes from user input, and finally outputs log text meeting the requirement of vectorization processing.
[0105] In this implementation, the preprocessing process first identifies the language of the text through a language detection module to provide a basis for subsequent multilingual processing, and then performs text cleaning to remove garbled codes, duplicate lines, special symbols, and unify the text encoding format. For long text logs, a semantic chunking algorithm or a fixed-length chunking strategy (such as every 500 characters as a fixed block) is used to divide the text into segments suitable for model processing.
[0106] Step 12, processing the log text through the text processing model to obtain a log text vector corresponding to the log text.
[0107] In this step, the log analysis system processes the log text through the text processing model. First, the language type of the log text is identified through the multilingual detection module of the text processing model, and the corresponding vectorization model is dynamically called according to the language type to output a unified dimension semantic vector. Finally, all log texts are converted into structured vector representations.
[0108] Specifically, step 12 can include the following implementation ways:
[0109] Step 1, language detection of the log text through a language detection unit in the text processing model to obtain a language detection result.
[0110] In this implementation, the log analysis system calls the language detection unit in the text processing model to perform sliding window sampling on the log text, outputs the probability distribution of the text belonging to each language, and generates a language detection result containing language labels and their confidence levels, providing accurate language information for subsequent vectorization processing.
[0111] Step 2, vectorization processing of the language detection result through a vectorization unit in the text processing model to obtain a log text vector.
[0112] In this implementation, the vectorization unit in the text processing model is used to vectorize the language detection result to generate corresponding vector representations for text content in different languages. Then, through vector splicing or weighted fusion, a complete log text vector is generated to ensure effective expression of multilingual semantic information.
[0113] In one example, as Figure 2As shown, for Chinese text content, an enhanced version of a robust optimization bidirectional encoder (Bidirectional Encoder Representations from Transformers, BERT) pre-training model based on a full-word mask can be used.
[0114] In another example, as shown, for English text content, a Sentence-Bidirectional Encoder Representations from Transformers (Sentence-BERT) model is used to generate corresponding English text vectors through a twin network architecture and cosine similarity training, thereby improving the accuracy of short text semantic matching. Figure 2
[0115] It can be understood that for text content in other languages, a text processing model in the relevant language can also be used to vectorize the text content and generate a corresponding vector representation. The specific type of text processing model used in this embodiment is not limited.
[0116] Due to the difference in grammar between Chinese and English, in combination with Chinese grammar, the English input is segmented using a cross-sentence semantic distance calculation method before the log text vector is obtained through vectorization processing.
[0117] Step 13, according to the historical text vector and the log text vector in the preset knowledge base, processing through a semantic association model to obtain the vector distance between the historical text vector and the log text vector.
[0118] In this step, the semantic association model maps the historical text vector in the preset knowledge base and the log text vector to a unified semantic space, adaptively adjusts the embedding relationship of different language vectors through a dynamic mapping matrix, and then calculates the vector semantic distance between the historical text vector and the log text vector using a vector distance algorithm in the semantic association model.
[0119] Continuing the example of Figure 2 When the log text includes Chinese, English, and other languages, the Chinese vector, the English vector, and the other language vector are respectively input to the semantic association model as the log text vector, so that the semantic association model maps the historical text vector in the preset knowledge base and the log text vector to a unified semantic space, adaptively adjusts the embedding relationship of different language vectors through a dynamic mapping matrix, and thereby obtains the vector distance between the historical text vector and the log text vector.
[0120] The vector distance algorithm in the semantic association model may, for example, be a cosine similarity or Euclidean distance algorithm.
[0121] Step 14, updating the historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance.
[0122] In this step, as shown in Figure 2 , the semantic association model outputs the vector distance to the comparison deduplication module, and the log analysis system adopts a differentiated update strategy according to the difference in the vector distance, and updates the historical text content corresponding to the historical text vector in the preset knowledge base in a comparison deduplication manner.
[0123] Among them, the process of updating the preset knowledge base adopts a lightweight index, dynamically inserts new vectors into the RAG knowledge base, and avoids full reconstruction of the index.
[0124] The knowledge base updating method provided by the embodiment of the application first acquires log text of a log to be processed, then processes the log text through a text processing model to obtain a log text vector corresponding to the log text, processes the historical text vector and the log text vector in the preset knowledge base through a semantic association model to obtain a vector distance between the historical text vector and the log text vector, and finally updates historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance. The technical solution can convert multi-source multi-language log text into a unified dimension semantic vector, accurately calculate the vector distance between the historical text vector and the log text vector through the semantic association model, update the knowledge base based on the vector distance, and improve the accuracy of knowledge base updating in a multi-language scenario.
[0125] Figure 3 The flowchart of the knowledge base updating method provided by the embodiment of the application Figure Two , as shown in Figure 3 , step 14 can include the following steps:
[0126] Step 21, determining whether the log text vector and the historical text vector are synonymous sentences according to the vector distance.
[0127] In this step, the semantic association model in the log analysis system determines whether the log text vector and the historical text vector are synonymous sentences based on a preset threshold for the vector distance.
[0128] Among them, if the vector distance is less than the preset threshold, it is determined that the log text vector and the historical text vector are synonymous sentences, and if the vector distance is not less than the preset semantic similarity threshold, it is determined that the log text vector and the historical text vector are synonymous sentences.
[0129] In one example, the preset semantic similarity threshold can be set to 0.3. In actual application, the semantic similarity threshold can also be set to other values according to actual needs, and the specific value of the semantic similarity threshold is not limited in this embodiment.
[0130] Step 22, if the log text vector and the historical text vector are synonymous sentences, update the historical text content corresponding to the historical text vector based on the similarity of the log text vector and the historical text vector.
[0131] In this step, if the log text vector and the historical text vector are synonymous sentences, the similarity of the log text vector and the historical text vector is calculated by using the semantic correlation model, and the historical text content corresponding to the historical text vector is updated according to the similarity calculation result, so as to complete the purpose of updating the knowledge base.
[0132] In the calculation of similarity, for a single language vector, the cosine similarity formula is directly used to calculate the similarity between the log text vector and the historical text vector; for a cross-language vector, the vectors of different languages are projected into a unified semantic space through a dynamic mapping matrix, and then the cosine distance is calculated.
[0133] For example, the Chinese log vector and the English historical vector will first undergo alignment processing by the semantic correlation model to eliminate language differences before being measured. During the calculation process, the weight parameters are automatically adjusted according to the vector dimension and language type to ensure the accuracy of the similarity calculation.
[0134] Then, the size between the calculated similarity and the preset similarity threshold is judged. When the similarity is greater than or equal to the preset similarity threshold, the semantic correlation model determines that the semantics are repeated and performs an update operation, i.e., the historical text vector is replaced by the log text vector to update the historical text content corresponding to the historical text vector.
[0135] When the similarity is less than the preset similarity threshold, the log analysis system recognizes the log text as new knowledge, adds the vector to the index library through the incremental indexing algorithm, and stores the log text in the knowledge base, i.e., the log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0136] Step 23, if the log text vector and the historical text vector are not synonymous sentences, update the log text corresponding to the log text vector as new historical text content in the preset knowledge base.
[0137] In this step, if the log text vector and the historical text vector are not synonymous sentences, it means that the log text vector is new knowledge for the knowledge base, and the incremental update process is started. The log text corresponding to the log text vector is updated as new historical text content in the preset knowledge base.
[0138] The incremental updating procedure comprises the following steps: firstly, generating a unique knowledge unit identity (ID) for the newly added knowledge, and associating metadata such as language type and generation time; then, inserting the log text vector into the index library through a lightweight index component, so as to avoid resource consumption of full index reconstruction; finally, synchronizing the text content and the vector representation to the knowledge base storage system, forming a new key-value pair, and completing updating of the knowledge base.
[0139] The knowledge base updating method provided by the embodiments of the present application firstly determines whether the log text vector and the historical text vector are synonymous sentences according to the vector distance, and then updates the historical text content corresponding to the historical text vector based on the similarity of the log text vector and the historical text vector if the log text vector and the historical text vector are synonymous sentences, and updates the log text corresponding to the log text vector to the preset knowledge base as new historical text content if the log text vector and the historical text vector are not synonymous sentences. The technical solution analyzes the similarity of the log text vector and the historical text vector, updates the preset knowledge base, and improves the accuracy of knowledge base updating.
[0140] Figure 4 Flowchart of the knowledge base updating method provided by the embodiments of the present application Figure Three As shown in Figure 4 The method can further comprise the following steps:
[0141] Step 31: In response to a monitoring trigger condition, obtaining historical text content corresponding to a target log from a preset knowledge base.
[0142] In this step, the log analysis system starts a data acquisition process in response to a monitoring trigger condition. The system parses the characteristic tags (such as timestamp, error type, and source system) of the target log, constructs a query vector, and then quickly retrieves the historical text content corresponding to the target log in the knowledge base through a lightweight index.
[0143] The target log is a log with a problem, and the historical text content of the target log is the historical text content corresponding to the log with a problem (i.e., the log corresponding to the feedback result of the user who is not satisfied with the search result). The monitoring trigger condition includes a timing task (acquiring the historical text content corresponding to the target log according to a fixed period) and an event-driven condition (for example, the knowledge base uses user feedback abnormally, and there are three consecutive negative reviews).
[0144] In one possible implementation, if the historical text content corresponding to the target log is a compressed package, the historical text content is decompressed, and then the single files of the historical text content are processed one by one.
[0145] Step 32: Analyzing the historical text content based on a preset log analysis model to obtain an analysis result of the target log.
[0146] In this step, the log analysis system calls a preset log analysis model (such as a multi-turn dialogue analyzer based on a large language model) to process the historical text content, comprehensively analyzes the fields and templates of the historical text content, and then generates an analysis result of the target log based on the ID of the target log.
[0147] Further, step 32 can have the following implementation manner:
[0148] Step 1: when the length of the historical text content exceeds a length threshold, the historical text content is divided into a plurality of text content fields based on preset log fields, to obtain a text content sequence.
[0149] In this implementation, when the log analysis system determines that the length of the historical text content exceeds the length threshold, intelligent segmentation is adopted, the historical text content is split into a plurality of structured field sequences (i.e., text content fields) based on preset log fields, and a text content sequence is obtained.
[0150] Step 2: inputting the text content sequence into a preset log analysis model to obtain a log reply field and a log analysis template.
[0151] In this implementation, the log analysis system inputs the text content sequence into a preset log analysis model in field order to obtain a log reply field and a log analysis template corresponding to the text content.
[0152] The log analysis model is a large language model, the log reply field is a split keyword based on a question input by a user and an output answer, and the log analysis template is a sentence template based on the question input by the user and the output answer with the keyword removed.
[0153] In one example, the log reply field is: weather, umbrella. The log reply template is: today …… is not good, and needs …… to go out (i.e., content to be filled in the log reply field).
[0154] Step 3: setting a log identifier of the target log.
[0155] In this implementation, the log analysis system generates a unique log identifier for the target log as a unique index of the target log in the system, which will be embedded in the analysis result for subsequent knowledge base updating.
[0156] Step 4: obtaining an analysis result of the target log based on the log reply field, the log analysis template, and the log identifier.
[0157] In this implementation, the log analysis system fills the log reply field into the placeholder position of the corresponding log analysis template to form a structured report, and then generates an analysis result of the target log in combination with the log identifier.
[0158] The analysis result of the target log supports multiple output formats (such as JSON, HTML, PDF, etc.).
[0159] In a possible implementation, Figure 5 A schematic diagram of the analysis process of the target log provided by an embodiment of the present application is shown in Figure 5 The analysis process of the target log includes the following implementations.
[0160] Step 1, obtain historical text content corresponding to the target log.
[0161] Step 2, decompress the historical text content, and then process single log content one by one.
[0162] Step 3, determine whether the length of the historical text content exceeds a threshold value.
[0163] In this step, if the length of the historical text content exceeds the threshold value, Step 4 is performed, and if the length of the historical text content does not exceed the threshold value, Step 6 is performed.
[0164] Step 4, split the historical text content to obtain a text content sequence.
[0165] Step 5, input the text content sequence into a log analysis model to generate a log reply field and a log analysis template.
[0166] Step 6, input the historical text content into the log analysis model to generate the log reply field and the log analysis template.
[0167] Step 7, generate a log identifier for each single log content.
[0168] Step 8, obtain the analysis result of the target log according to the log reply field, the log analysis template, and the log identifier.
[0169] Step 33, obtain feedback data of a user on the analysis result of the target log.
[0170] In this step, feedback data of a knowledge base using user on the analysis result of the target log is collected through multiple channels (for example, embedding a scoring component on a knowledge base use page, a survey email, and obtaining a search failure rate), to provide quantitative basis for subsequent knowledge base updating and optimization.
[0171] In an example, the feedback data includes scoring data of 8 points input by the knowledge base using user on the use page, and the scoring reason is input in the opinion feedback box.
[0172] Step 34, update the preset knowledge base based on the feedback data.
[0173] In this step, if the user feedback analysis results are correct and complete, that is, the feedback data has a high score, the quality score of the knowledge unit searched by the user will be improved, and the retrieval weight of redundant content will be reduced. If the feedback data indicates that the analysis is wrong or incomplete, that is, the feedback data has a low score, the problem log generated during the user search process will be stored in the knowledge base, and the preset knowledge base will be updated based on the problem later.
[0174] The knowledge base update method provided in an embodiment of the present application retrieves historical text content corresponding to a target log from a preset knowledge base in response to a monitoring trigger condition. The method then analyzes the historical text content based on a preset log analysis model to obtain analysis results for the target log, obtains user feedback on the analysis results for the target log, and finally updates the preset knowledge base based on the feedback data. This technical solution dynamically retrieves historical log content from the knowledge base through a monitoring trigger mechanism and generates analysis results for the target log using a preset log analysis model, thereby improving the accuracy and professionalism of the log analysis results.
[0175] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.
[0176] Figure 6 This is a schematic diagram of the structure of the knowledge base updating device provided in the embodiment of the present application. Figure 6 As shown, the device includes:
[0177] An acquisition module 61 is used to acquire the log text of the log to be processed;
[0178] A processing module 62 is configured to process the log text using a text processing model to obtain a log text vector corresponding to the log text;
[0179] The processing module 62 is further used to process the historical text vector and the log text vector in the preset knowledge base through the semantic association model to obtain the vector distance between the historical text vector and the log text vector;
[0180] The updating module 63 is configured to update the historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance.
[0181] In a possible implementation, the acquisition module 61 is specifically configured to:
[0182] Get the logs to be processed;
[0183] Verify the URL format of the log to be processed;
[0184] If the URL format is met, the web page pointed to by the URL is parsed and incrementally crawled to obtain the log text;
[0185] If not, the to-be-processed log is preprocessed to obtain a log text; the preprocessing includes at least one of language detection processing, text cleaning processing, and text chunking processing.
[0186] In a possible implementation, the processing module 62 is specifically configured to:
[0187] perform language detection on the log text through a language detection unit in the text processing model to obtain a language detection result;
[0188] perform vectorization processing on the language detection result through a vectorization unit in the text processing model to obtain a log text vector.
[0189] In a possible implementation, the updating module 63 is specifically configured to:
[0190] determine, according to the vector distance, whether the log text vector and the historical text vector are synonymous sentences;
[0191] if yes, update historical text content corresponding to the historical text vector based on a similarity of the log text vector and the historical text vector;
[0192] if no, update, as new historical text content, log text corresponding to the log text vector to the preset knowledge base.
[0193] In a possible implementation, the updating module 63 is specifically configured to update the historical text content corresponding to the historical text vector based on the similarity of the log text vector and the historical text vector.
[0194] calculate a similarity between the log text vector and the historical text vector;
[0195] when the similarity is greater than or equal to a preset similarity threshold, replace the historical text vector with the log text vector to update the historical text content corresponding to the historical text vector;
[0196] when the similarity is less than the preset similarity threshold, update, as new historical text content, log text corresponding to the log text vector to the preset knowledge base.
[0197] In a possible implementation, the knowledge base updating apparatus further includes:
[0198] The acquisition module 61 is further configured to acquire, in response to a monitoring trigger condition, historical text content corresponding to a target log from a preset knowledge base;
[0199] The processing module 62 is further configured to analyze the historical text content based on a preset log analysis model to obtain an analysis result of the target log;
[0200] The acquisition module 61 is further used to obtain feedback data of the user's analysis results of the target log;
[0201] The updating module 63 is further configured to update the preset knowledge base based on the feedback data.
[0202] In a possible implementation, the processing module 62 is specifically configured to:
[0203] When the length of the historical text content exceeds a length threshold, the historical text content is divided into a plurality of text content fields based on a preset log field to obtain a text content sequence;
[0204] Input the text content sequence into the preset log analysis model to obtain the log response field and log analysis template;
[0205] Set the log ID of the target log;
[0206] Based on the log reply fields, log analysis template and log identifier, the analysis results of the target log are obtained.
[0207] The device provided in the embodiments of the present application can be used to execute the determination method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.
[0208] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, these modules can be fully or partially integrated together or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.
[0209] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor 71, a memory 72, and computer program instructions stored in the memory 72 and executable on the processor 71. When the processor 71 executes the computer program instructions, the method provided in any of the aforementioned embodiments is implemented.
[0210] Optionally, the above-mentioned components of the electronic device may be connected via a system bus.
[0211] The memory 72 can be a separate storage unit or can be integrated into the processor 71. The number of processors 71 is one or more.
[0212] It should be understood that the processor 71 can be a central processing unit (CPU), and can also be other general-purpose processors 71, digital signal processors 71 (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor 71 can be a microprocessor or can also be any conventional processor 71, etc. The steps of the method disclosed in the present application can be directly embodied as execution by the hardware processor 71 or by a combination of hardware and software modules in the processor 71.
[0213] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus. The memory 72 can include a random access memory 72 (RAM) and can also include a non-volatile memory 72 (NVM), such as at least one disk memory 72.
[0214] All or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a readable memory 72. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing memory 72 (storage medium) includes a read-only memory 72 (ROM), a RAM, a flash memory 72, a hard disk, a solid state disk, a magnetic tape (English: magnetic tape), a floppy disk (English: floppy disk), an optical disc (English: optical disc) and any combination thereof.
[0215] The electronic device provided by the embodiments of the present application can be used to execute the method provided by any one of the above-mentioned method embodiments, and has similar implementation principles and technical effects, which will not be described here.
[0216] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer instructions. When the computer instructions run on a computer, the computer executes the above method.
[0217] The computer readable storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0218] Optionally, the readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.
[0219] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer readable storage medium, and at least one processor can read the computer program from the computer readable storage medium. When the at least one processor executes the computer program, the above method can be realized.
[0220] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the claims that follow.
Claims
1. A knowledge base updating method, characterized in that: include: Get the log text of the log to be processed; Processing the log text using a text processing model to obtain a log text vector corresponding to the log text; According to the historical text vectors and the log text vectors in the preset knowledge base, processing is performed through a semantic association model to obtain a vector distance between the historical text vectors and the log text vectors; Based on the vector distance, the historical text content corresponding to the historical text vector in the preset knowledge base is updated.
2. The method according to claim 1, characterized in that The step of obtaining the log text of the log to be processed includes: Obtaining the log to be processed; Verify the uniform resource locator URL format of the log to be processed; If the URL format is met, the web page pointed to by the URL is parsed and incrementally crawled to obtain the log text; If not, preprocessing is performed on the log to be processed to obtain the log text; the preprocessing includes at least one of language detection processing, text cleaning processing, and text block processing.
3. The method according to claim 1, characterized in that The processing of the log text by a text processing model to obtain a log text vector corresponding to the log text includes: Performing language detection on the log text by a language detection unit in the text processing model to obtain a language detection result; The language detection result is vectorized by a vectorization unit in the text processing model to obtain the log text vector.
4. The method according to claim 1, wherein The updating of the historical text content in the preset knowledge base based on the vector distance includes: determining, based on the vector distance, whether the log text vector and the history text vector are synonymous sentences; If so, based on the similarity between the log text vector and the historical text vector, update the historical text content corresponding to the historical text vector; If not, the log text corresponding to the log text vector is updated into the preset knowledge base as new historical text content.
5. The method according to claim 4, characterized in that The updating of the historical text content corresponding to the historical text vector based on the similarity between the log text vector and the historical text vector includes: Calculating the similarity between the log text vector and the history text vector; When the similarity is greater than or equal to a preset similarity threshold, replacing the historical text vector with the log text vector to update the historical text content corresponding to the historical text vector; When the similarity is less than the preset similarity threshold, the log text corresponding to the log text vector is updated into the preset knowledge base as new historical text content.
6. The method according to claim 1, characterized in that The method further comprises: In response to a monitoring trigger condition, obtaining historical text content corresponding to the target log from the preset knowledge base; Analyze the historical text content based on a preset log analysis model to obtain an analysis result of the target log; Obtain user feedback on the analysis results of the target log; Based on the feedback data, the preset knowledge base is updated.
7. The method according to claim 6, characterized in that The analyzing the historical text content based on the preset log analysis model to obtain the analysis result of the target log includes: When the length of the historical text content exceeds a length threshold, dividing the historical text content into a plurality of text content fields based on a preset log field to obtain a text content sequence; Inputting the text content sequence into the preset log analysis model to obtain a log reply field and a log analysis template; Setting a log identifier of the target log; An analysis result of the target log is obtained based on the log reply field, the log analysis template and the log identifier.
8. A knowledge base updating device, characterized in that: include: The acquisition module is used to obtain the log text of the log to be processed; A processing module, configured to process the log text using a text processing model to obtain a log text vector corresponding to the log text; The processing module is further configured to process the historical text vector and the log text vector in a preset knowledge base through a semantic association model to obtain a vector distance between the historical text vector and the log text vector; An updating module is used to update the historical text content corresponding to the historical text vector in the preset knowledge base based on the vector distance.
9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.