Service knowledge base management system and method based on knowledge big model

Abstract

This invention discloses a service knowledge base management system and method based on a large knowledge model, specifically relating to the field of knowledge base management technology. It involves real-time monitoring and preprocessing of the knowledge base system's operational status data, using decision tree algorithms to extract features and identify key performance indicators, comprehensively analyzing abnormal changes and fluctuations in the knowledge base's hit rate and update frequency, accurately assessing the system's operational stability, and classifying the system state based on the assessment results. When the system is unstable, it monitors the performance and response effectiveness of the large knowledge model, using fuzzy logic to quantify its impact on user experience, thereby taking targeted optimization management measures. Through intelligent and data-driven methods, it improves the operational stability and response accuracy of the knowledge base system, optimizes the user experience, and enhances the overall system efficiency and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge base management, and in particular to a service knowledge base management system and method based on a knowledge big model. Background Art

[0002] Service knowledge base management based on knowledge big model refers to the intelligent management of service knowledge base using large-scale language model. This technology trains a large amount of text data through deep learning and natural language processing, and can understand and generate natural language text, so as to realize automatic classification, organization and retrieval of information in service knowledge base, and improve service efficiency and accuracy.

[0003] In practical applications, service knowledge base management based on knowledge big models can be widely used in customer service, technical support, online education and other fields. For example, in customer service, knowledge big models can automatically answer users' common questions and recommend relevant help documents or tutorials based on users' questions, greatly improving response speed and service quality. In the field of technical support, knowledge big models can help technicians quickly locate problems and provide solutions, reduce troubleshooting time, and improve user satisfaction.

[0004] The prior art has the following deficiencies:

[0005] In the existing technology, enterprises rely on knowledge big models to provide automated customer service and support, reduce reliance on manual customer service, and improve response speed and service quality. However, this reliance also means that once there is a problem with the knowledge base system, customer service will not be able to proceed normally. For example, technical failures or system downtime will cause the knowledge big model to fail to work properly, the customer service system will temporarily lose the automated answer function, and users will not be able to get timely help. In this case, during the service interruption, users may feel dissatisfied because they cannot get help in time, resulting in customer churn and negative reviews. At the same time, when the knowledge big model does not work, all customer issues need to be handled by manual customer service, which increases the workload of manual customer service and prolongs the waiting time of users. In addition, if there is a problem with the knowledge base system and the model performs poorly, it may affect the user experience. Poorly performing models may generate incorrect or irrelevant answers, misleading users, causing users to be unable to solve actual problems and reducing user satisfaction. Summary of the invention

[0006] The purpose of the present invention is to provide a service knowledge base management system and method based on a knowledge big model to solve the deficiencies in the background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a service knowledge base management method based on a knowledge big model, comprising the following steps;

[0008] S1: Obtain the operating status data of the knowledge base system in real time through monitoring tools, pre-process the collected operating status data, and organize and store them in time series;

[0009] S2: Extract features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identify performance indicators related to the health status of the knowledge base system;

[0010] S3: Monitor and analyze the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluate the operational stability of the knowledge base system based on the abnormal changes in the knowledge base hit rate and the fluctuation status of the knowledge base update frequency;

[0011] S4: according to the evaluation result of the running stability of the knowledge base system, the knowledge base system is divided into different running state categories, and the running state of the knowledge base system is divided into a stable running state and an unstable running state;

[0012] S5: When the running state of the knowledge base system is unstable, monitor the performance of the knowledge big model, judge the accuracy of the response generated by the knowledge big model, and evaluate the response effect of the model;

[0013] S6: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

[0014] In a preferred embodiment, the running status data of the knowledge base system at different times are feature extracted by a decision tree algorithm to identify performance indicators related to the health status of the knowledge base system, specifically:

[0015] Use monitoring tools to collect the running status data of the knowledge base system at different time points in real time. Each data point should include a timestamp and performance indicators. Sort the collected data in chronological order to ensure that the data at each time point has a corresponding timestamp. Store the data in a time series database for subsequent query and processing.

[0016] Normalize the collected raw data;

[0017] Use historical operating status data to train decision tree models;

[0018] Use the trained decision tree model to extract features from the running status data of the knowledge base system at different times;

[0019] The decision tree model automatically selects performance indicators that distinguish the health status of the system based on the characteristics of the data.

[0020] In a preferred embodiment, a knowledge base hit rate anomaly index is generated according to the abnormal change of the knowledge base hit rate within a fixed time period, and the method for obtaining the knowledge base hit rate anomaly index is:

[0021] Collect the knowledge base hit rate data within the time period s, and establish the corresponding time series set sk1, sk2, ..., sk i ,...,sk p ], sk i is the hit rate of the knowledge base at the i-th time point, i=1, 2, ..., p, and p is a positive integer greater than 0;

[0022] The initial knowledge base hit rate sk1 in the time series set is used as the initial EWMA value, and the exponentially weighted moving average formula is used to calculate the EWMA value at each time point. The specific calculation expression is: EWMA i =α*sk i +(1-α)*sk i-1 ; Among them, α is the smoothing coefficient, ranging from 0 to 1, EWMA i is the EWMA value at the current time point;

[0023] According to the current hit rate and EWMA i The difference between the values ​​is used to calculate the knowledge base hit rate anomaly at each time point. The specific calculation expression is: mk i =|sk i -EWMA i |; In the formula, mk i The knowledge base hit rate anomaly value at each time point is summed up for all time points to get the knowledge base hit rate anomaly index. The specific calculation expression is: Where SA is the knowledge base hit rate anomaly index.

[0024] In a preferred embodiment, the knowledge base update frequency fluctuation index is generated according to the knowledge base update frequency fluctuation state, and the method for obtaining the knowledge base update frequency fluctuation index is:

[0025] Collect the frequency data of knowledge base updates within a time period of t, select a window size N, and calculate the moving average update frequency at each time point. The specific calculation expression is: Where, CA t is the moving average update frequency at time t, N is the window size, x t-k is the number of updated entries at time tk;

[0026] In the same window, the standard deviation of each time point is calculated to reflect the fluctuation of the update frequency. The specific calculation expression is: In the formula, ES t is the standard deviation of the moving average update frequency at time t. The moving average and standard deviation are used to calculate the knowledge base update frequency fluctuation index. The specific calculation expression is: Where AW is the knowledge base update frequency fluctuation index.

[0027] In a preferred embodiment, the knowledge base hit rate anomaly index and the knowledge base update frequency fluctuation index are converted into a first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model predicts the operating stability value label of the knowledge base system with each group of first eigenvectors as the prediction target, and minimizes the sum of prediction errors for the operating stability value labels of all knowledge base systems as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The operating stability value of the knowledge base system is determined according to the model output results.

[0028] In a preferred embodiment, the acquired operational stability value of the knowledge base system is compared with a preset operational stability value reference threshold; if the operational stability value of the knowledge base system is less than the preset operational stability value reference threshold, the operational state of the knowledge base system is unstable, and the operational state of the corresponding knowledge base system is classified as an unstable operational state, and the system administrator needs to promptly check and adjust the knowledge base system; if the operational stability value of the knowledge base system is greater than or equal to the preset operational stability value reference threshold, the operational state of the knowledge base system is stable, and the operational state of the corresponding knowledge base system is classified as a stable operational state, and no intervention is required.

[0029] In a preferred embodiment, when the running state of the knowledge base system is an unstable running state, the performance of the knowledge big model is monitored and evaluated by the random forest model to judge the accuracy and effect of the response generated by the knowledge big model, specifically:

[0030] Collect the query questions submitted by users to the knowledge model in real time and record the responses generated by the knowledge model;

[0031] Monitor and collect system performance data when generating responses, and perform text feature extraction on user input and model responses;

[0032] Extracting conversational context features and system performance data;

[0033] Normalize the features extracted in real time;

[0034] Combine the extracted text features, conversation features, and system performance features into a feature vector, ensuring that the format of the feature vector is consistent with that used when training the model;

[0035] Input the feature vector constructed in real time into the trained random forest model;

[0036] Get the prediction results of the random forest model, which are the accuracy values ​​of the model responses.

[0037] In a preferred embodiment, the accuracy value of the acquired model response is compared with the accuracy value reference threshold. If the accuracy value is greater than or equal to the accuracy value reference threshold, the accuracy of the response generated by the large knowledge model is high, and a normal model response signal is generated. If the accuracy value is less than the accuracy value reference threshold, the accuracy of the response generated by the large knowledge model is low, and an abnormal model response signal is generated.

[0038] In a preferred embodiment, according to the operational stability of the knowledge base system and the response effect of the model, after using fuzzy logic, the degree of its impact on the user experience is graded, specifically:

[0039] The running stability value of the knowledge base system and the model response accuracy value are used as the input items of the fuzzy logic, and the degree of impact on user experience is used as the output item of the fuzzy logic;

[0040] Define the fuzzy set of the knowledge base system operation stability value and model response accuracy value, as well as the fuzzy set of the degree of impact on user experience;

[0041] Based on the running stability value of the knowledge base system and the model response accuracy value, a set of fuzzy rules are defined to determine the degree of impact on user experience;

[0042] The input items of knowledge base system operation stability value and model response accuracy value are converted into fuzzy sets through fuzzification function;

[0043] According to the fuzzy rules, calculate the applicability of each rule;

[0044] According to each fuzzy rule, calculate the fuzzy set of the impact degree of user experience;

[0045] Aggregate the results of all rules to obtain a comprehensive set of fuzzy outputs;

[0046] The comprehensive fuzzy output set is converted into a specific output value through the defuzzification method. According to the specific output value of the impact on user experience, it is divided into different levels. Different optimization management measures are taken according to the division results.

[0047] The present invention also provides a service knowledge base management system based on a knowledge big model, including a data acquisition module, a feature extraction module, a stability evaluation module, a stability classification module, a response effect evaluation module and an optimization management module;

[0048] Data acquisition module: obtains the operating status data of the knowledge base system in real time through monitoring tools, pre-processes the collected operating status data, and organizes and stores it in time series;

[0049] Feature extraction module: extracts features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identifies performance indicators related to the health status of the knowledge base system;

[0050] Stability evaluation module: monitors and analyzes the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluates the operational stability of the knowledge base system based on abnormal changes in the knowledge base hit rate and fluctuations in the knowledge base update frequency;

[0051] Stability classification module: according to the evaluation results of the running stability of the knowledge base system, it is divided into different running status categories, and the running status of the knowledge base system is divided into a stable running status and an unstable running status;

[0052] Response effect evaluation module: When the running state of the knowledge base system is unstable, the performance of the knowledge model is monitored to determine the accuracy of the responses generated by the knowledge model and evaluate the response effect of the model;

[0053] Optimization management module: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

[0054] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0055] 1. The present invention monitors the operating status data of the knowledge base system in real time, uses the decision tree algorithm to extract key performance indicators, comprehensively analyzes the abnormal changes and fluctuations in the hit rate and update frequency of the knowledge base, accurately evaluates the system operation stability, and divides it into two categories: stable and unstable. When the system is in an unstable state, the response accuracy of the knowledge model is further monitored, and the degree of its impact on the user experience is quantified through fuzzy logic, and corresponding optimization management measures are taken. The system's operating stability and response accuracy are effectively improved, the user experience is optimized, and the efficiency and reliability of the overall system are improved.

[0056] 2. The present invention can quickly respond and adjust when problems occur in the knowledge base system, avoid customer service interruptions caused by system downtime or failure, and reduce user dissatisfaction and customer churn. At the same time, through continuous monitoring and evaluation of the performance of the large knowledge model, it ensures that the model generates high-quality responses, reduces errors and irrelevant answers, and further improves user satisfaction. This multi-level monitoring and optimization strategy not only improves the reliability and stability of the knowledge base system, but also significantly improves the customer service quality and competitiveness of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0058] Figure 1 The figure is a flow chart of the method of the present invention.

[0059] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] See also Figure 1 As shown, the service knowledge base management method based on the knowledge big model described in this embodiment includes the following steps:

[0063] S1: Obtain the operating status data of the knowledge base system in real time through monitoring tools, pre-process the collected operating status data, and organize and store them in time series;

[0064] S2: Extract features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identify performance indicators related to the health status of the knowledge base system;

[0065] S3: Monitor and analyze the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluate the operational stability of the knowledge base system based on the abnormal changes in the knowledge base hit rate and the fluctuation status of the knowledge base update frequency;

[0066] S4: according to the evaluation result of the running stability of the knowledge base system, the knowledge base system is divided into different running state categories, and the running state of the knowledge base system is divided into a stable running state and an unstable running state;

[0067] S5: When the running state of the knowledge base system is unstable, monitor the performance of the knowledge big model, judge the accuracy of the response generated by the knowledge big model, and evaluate the response effect of the model;

[0068] S6: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

[0069] Among them, in S1, the operating status data of the knowledge base system is obtained in real time through the monitoring tool, and the collected operating status data is pre-processed and then sorted and stored in time series.

[0070] Install and configure monitoring tools on various servers and related components of the knowledge base system, such as Prometheus for data collection and Grafana for data visualization.

[0071] Monitoring tools continuously collect key performance indicators (KPIs) from the system, such as CPU usage, memory usage, disk I / O, network latency, and system response time. Data collection is ongoing to ensure real-time monitoring of the system's operating status.

[0072] The collected raw data may contain noise, outliers, or incomplete data. The data cleaning step involves removing these noise and outliers and filling in missing data to ensure the quality and accuracy of the data.

[0073] Convert data from different sources into a unified format and standardize the measurement units of the data so that data from different performance indicators can be compared and comprehensively analyzed.

[0074] The pre-processed data is sorted in chronological order, ensuring that each data point has a timestamp. This allows the data to be analyzed and processed according to the time dimension, making it easier to identify trends, cycles, and anomalies.

[0075] Store the sorted time series data in a database or data warehouse for subsequent query and analysis. Common storage systems include time series databases (such as Prometheus' own storage system) or other databases suitable for processing time series data.

[0076] S2: The decision tree algorithm is used to extract features from the running status data of the knowledge base system at different times, and the performance indicators related to the health status of the knowledge base system are identified.

[0077] Use monitoring tools (such as Prometheus) to collect real-time operating status data of the knowledge base system at different time points. Each data point should include a timestamp and key performance indicators such as CPU usage, memory usage, disk I / O, network latency, and system response time.

[0078] The collected data is sorted in chronological order to ensure that the data at each time point has a corresponding timestamp. The data is stored in a time series database for subsequent query and processing.

[0079] Clean the collected raw data to remove noise, outliers and missing data to ensure data quality and accuracy. Convert data from different sources into a unified format and standardize measurement units to make data of different performance indicators comparable.

[0080] Select decision tree models (such as CART, Classification and RegressionTrees) for feature extraction.

[0081] Use historical operating status data to train the decision tree model to ensure that the model can identify important features related to the system health status.

[0082] The trained decision tree model is used to extract features from the operating status data of the knowledge base system at different times.

[0083] The decision tree model automatically selects performance indicators that distinguish the health status of the system based on the characteristics of the data, such as system response time, CPU utilization, memory utilization, disk I / O, network latency, etc.

[0084] S3: Monitor and analyze the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluate the operational stability of the knowledge base system based on the abnormal changes in the knowledge base hit rate and the fluctuations in the knowledge base update frequency.

[0085] Monitor and analyze the changing trends of the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators within a fixed time period. The knowledge base hit rate refers to the proportion of valid answers that the knowledge base can provide when users query. A high hit rate means that the information in the knowledge base can meet most of the user's needs. Abnormal changes in the knowledge base hit rate refer to a significant change in the proportion of valid answers that the knowledge base system can provide within a fixed time period, usually manifested as a sudden drop or sharp fluctuation in the hit rate. This abnormal situation may reflect potential problems in the system, such as insufficient and inaccurate knowledge base content or system query function failures, which need to be detected and processed in a timely manner to ensure system stability and user satisfaction.

[0086] The knowledge base hit rate anomaly index is generated according to the abnormal changes in the knowledge base hit rate within a fixed time period. The method for obtaining the knowledge base hit rate anomaly index is:

[0087] Collect the knowledge base hit rate data within the time period s, and establish the corresponding time series set sk1, sk2, ..., sk i ,...,sk p ], sk i is the hit rate of the knowledge base at the i-th time point, i=1, 2, ..., p, and p is a positive integer greater than 0;

[0088] The initial knowledge base hit rate sk1 in the time series set is used as the initial EWMA value, and the exponentially weighted moving average formula is used to calculate the EWMA value at each time point. The specific calculation expression is: EWMA i =α*sk i +(1-α)*sk i-1 ; Among them, α is the smoothing coefficient, ranging from 0 to 1, EWMA i is the EWMA value at the current time point;

[0089] According to the current hit rate and EWMA i The difference between the values ​​is used to calculate the knowledge base hit rate anomaly at each time point. The specific calculation expression is: mk i =|sk i -EWMA i |; In the formula, mk i The knowledge base hit rate anomaly value at each time point is summed up for all time points to get the knowledge base hit rate anomaly index. The specific calculation expression is: Where SA is the knowledge base hit rate anomaly index.

[0090] The knowledge base hit rate anomaly index measures the degree of anomaly in the knowledge base system by comparing the difference between the current hit rate and the exponentially weighted moving average (EWMA) value. Specifically, the larger the anomaly index, the greater the deviation of the current hit rate from the historical trend, reflecting the recent significant fluctuations in the system. These fluctuations may be caused by sudden changes in the knowledge base content, failures in the system query function, or a sharp increase in external access.

[0091] When the knowledge base hit rate anomaly index is positive and large, it means that the current hit rate is significantly higher than the historical trend, which may indicate that the updated knowledge base content meets user needs more effectively, or that recent user query questions are more concentrated. In short, the larger the value of the knowledge base hit rate anomaly index, the lower the operational stability of the knowledge base system, and vice versa.

[0092] The knowledge base update frequency fluctuation state refers to the frequency of knowledge base content updates and their changes within a fixed time period, which is usually measured by counting the number of knowledge base updates at each time point. When the update frequency fluctuates significantly (for example, frequent increases or decreases), it may reflect the instability of the knowledge base content or abnormalities in the maintenance process. These fluctuations may affect the system performance and user experience.

[0093] The knowledge base update frequency fluctuation index is generated according to the knowledge base update frequency fluctuation state, and the method for obtaining the knowledge base update frequency fluctuation index is:

[0094] Collect the frequency data of knowledge base updates within a time period of t, such as the number of updated entries per day or per week, select a window size N (such as 7 days or 4 weeks), and calculate the moving average update frequency at each time point. The specific calculation expression is: Where, CA t is the moving average update frequency at time t, N is the window size, x t-k is the number of updated entries at time tk;

[0095] In the same window, the standard deviation of each time point is calculated to reflect the fluctuation of the update frequency. The specific calculation expression is: In the formula, ES t is the standard deviation of the moving average update frequency at time t. The moving average and standard deviation are used to calculate the knowledge base update frequency fluctuation index. The specific calculation expression is: Where AW is the knowledge base update frequency fluctuation index.

[0096] The larger the knowledge base update frequency fluctuation index is, the lower the stability of the knowledge base system operation is. The fluctuation index reflects the discrete degree of the knowledge base update frequency. When the fluctuation index is large, it means that the knowledge base update frequency is unstable, and there may be a large number of updates at times and a small number of updates at times. This instability may cause the knowledge base to be insufficiently updated in certain periods of time, unable to reflect the latest information in a timely manner, thus affecting the user's experience of obtaining accurate answers.

[0097] A stable knowledge base system usually means that the update frequency is relatively uniform, and the content can be continuously and regularly supplemented and corrected to ensure that users can obtain the latest and most comprehensive information at any time. On the contrary, a high volatility index indicates that the system update frequency is relatively irregular, which may be caused by poor management, uneven resource allocation, or other systemic problems. If it is not improved, users may encounter outdated or inaccurate information when the knowledge base is not updated in time, which will reduce their trust and satisfaction with the system.

[0098] Therefore, reducing the knowledge base update frequency fluctuation index should be an important goal in knowledge base management. By reasonably arranging update plans, allocating resources and optimizing management processes, the continuity and uniformity of knowledge base updates can be achieved, and the operational stability and reliability of the system can be improved, thereby better serving users and improving the overall user experience.

[0099] The knowledge base hit rate anomaly index and the knowledge base update frequency fluctuation index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model predicts the operation stability value label of the knowledge base system with each group of first eigenvectors as the prediction target, and minimizes the sum of prediction errors of the operation stability value labels of all knowledge base systems as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The operation stability value of the knowledge base system is determined according to the model output results.

[0100] The method for obtaining the operational stability value of the knowledge base system is as follows: from the first eigenvector training data of the trained machine learning model, obtain the corresponding function expression: CM=f1(SA, AW); wherein f1 is the output function of the model, SA is the knowledge base hit rate anomaly index, AW is the knowledge base update frequency fluctuation index, and CM is the operational stability value of the knowledge base system.

[0101] S4: according to the evaluation result of the running stability of the knowledge base system, the knowledge base system is divided into different running state categories, and the running state of the knowledge base system is divided into a stable running state and an unstable running state;

[0102] The obtained operation stability value of the knowledge base system is compared with the preset operation stability value reference threshold. If the operation stability value of the knowledge base system is less than the preset operation stability value reference threshold, it means that the operation state of the knowledge base system is unstable and abnormal conditions may occur. The operation state of the corresponding knowledge base system is classified as an unstable operation state, and the system administrator needs to check and adjust the knowledge base system in time. If the operation stability value of the knowledge base system is greater than or equal to the preset operation stability value reference threshold, it means that the operation state of the knowledge base system is stable. The operation state of the corresponding knowledge base system is classified as a stable operation state and no intervention is required.

[0103] S5: When the running state of the knowledge base system is unstable, monitor the performance of the knowledge big model, judge the accuracy of the response generated by the knowledge big model, and evaluate the response effect of the model;

[0104] When the running state of the knowledge base system is unstable, the performance of the knowledge model is monitored and evaluated through the random forest model to determine the accuracy and effect of the responses generated by the knowledge model. Specifically:

[0105] Collect the query questions submitted by users to the knowledge model in real time and record the responses generated by the knowledge model;

[0106] Monitor and collect system performance data when generating responses, such as response time, memory usage, etc. Perform text feature extraction on user input and model responses, such as TF-IDF, word vectors, n-grams, etc.

[0107] Extract contextual features of the conversation, such as conversation turns, reply length, tone and sentiment of the conversation, and extract system performance data, such as response time, memory usage, etc.

[0108] Normalize the features extracted in real time to ensure consistency with the training data. Check and handle missing values ​​in real-time data, using methods such as interpolation or filling the mean.

[0109] Combine the extracted text features, conversation features, and system performance features into a feature vector, ensuring that the feature vector format is consistent with that used when training the model.

[0110] The feature vector constructed in real time is input into the trained random forest model.

[0111] Get the prediction results of the random forest model, which are the accuracy values ​​of the model responses.

[0112] The accuracy value of the obtained model response is compared with the accuracy value reference threshold. If the accuracy value is greater than or equal to the accuracy value reference threshold, it means that the accuracy of the response generated by the large knowledge model is high and the model response effect is good. At this time, a normal model response signal is generated; if the accuracy value is less than the accuracy value reference threshold, it means that the accuracy of the response generated by the large knowledge model is low and the model response effect is poor. At this time, an abnormal model response signal is generated.

[0113] S6: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

[0114] The operation stability value (CM) of the knowledge base system and the model response accuracy value (A) are used as the input items of fuzzy logic, and the user experience impact degree (U) is used as the output item of fuzzy logic;

[0115] Define the fuzzy set of the knowledge base system operation stability value and the model response accuracy value, as well as the fuzzy set of the user experience impact degree (U), such as high, medium, and low;

[0116] Based on the running stability value of the knowledge base system and the model response accuracy value, a set of fuzzy rules are defined to determine the degree of impact on user experience. For example:

[0117] If CM is high and A is high, then U is good.

[0118] If CM is high and A is medium, then U is average.

[0119] If CM is high and A is low, then U is the difference.

[0120] If CM is medium and A is high, then U is good.

[0121] If CM is neutral and A is neutral, then U is general.

[0122] If CM is medium and A is low, then U is bad.

[0123] If CM is low and A is high, then U is normal.

[0124] If CM is low and A is medium, then U is poor.

[0125] If CM is low and A is low, then U is bad.

[0126] The input items knowledge base system operation stability value (CM) and model response accuracy value (A) are converted into fuzzy sets through fuzzification function.

[0127] The CM is fuzzified using three membership functions: low, medium and high.

[0128] A is fuzzified using three membership functions: low, medium and high.

[0129] According to the fuzzy rules, calculate the applicability of each rule. For example, for input CM = 0.6 and A = 0.7:

[0130] Calculate the membership of CM as low, medium and high.

[0131] Calculate the membership of A as low, medium, and high.

[0132] According to each fuzzy rule, the fuzzy set of the user experience impact degree (U) is calculated.

[0133] Aggregate the results of all rules to obtain a comprehensive fuzzy output set.

[0134] The comprehensive fuzzy output set is converted into a specific output value through a defuzzification method (such as the centroid method). For example, the user experience impact degree U=0.75U=0.75U=0.75 is obtained.

[0135] According to the specific output value of the impact on user experience, it is divided into different levels:

[0136] 0.0-0.3: Poor; 0.3-0.7: Average; 0.7-1.0: Good.

[0137] According to the division results, different optimization management measures are taken:

[0138] Poor: Conduct a comprehensive inspection and optimization, including improving the stability of the knowledge base update frequency, optimizing the accuracy of the knowledge model, and increasing manual intervention.

[0139] Average: Perform partial optimization, focus on problem areas with greater impact, and appropriately adjust the knowledge base update strategy and model training method.

[0140] Good: Maintain the current status, monitor and fine-tune regularly to ensure the system continues to operate stably.

[0141] In this embodiment, the running status data of the knowledge base system is acquired in real time through a monitoring tool, and the collected running status data is pre-processed and then sorted and stored in time series; the running status data of the knowledge base system at different times is feature extracted through a decision tree algorithm to identify performance indicators related to the health status of the knowledge base system; the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators are monitored and analyzed, and the running stability of the knowledge base system is evaluated according to the abnormal changes in the knowledge base hit rate and the fluctuations in the knowledge base update frequency; according to the evaluation results of the running stability of the knowledge base system, it is divided into different running status categories, and the running status of the knowledge base system is divided into a stable running status and an unstable running status; when the running status of the knowledge base system is an unstable running status, the performance of the knowledge big model is monitored to determine the accuracy of the response generated by the knowledge big model and to evaluate the response effect of the model; according to the running stability of the knowledge base system and the response effect of the model, after using fuzzy logic, the degree of its impact on the user experience is graded, and different optimization management measures are taken according to the classification results. Through intelligent and data-driven methods, the operational stability and response accuracy of the knowledge base system are improved, the user experience is optimized, and the efficiency and reliability of the overall system are improved.

[0142] Example 2

[0143] See also Figure 2 As shown, the service knowledge base management based on the knowledge big model described in this embodiment includes a data acquisition module, a feature extraction module, a stability evaluation module, a stability classification module, a response effect evaluation module and an optimization management module;

[0144] Data acquisition module: obtains the operating status data of the knowledge base system in real time through monitoring tools, pre-processes the collected operating status data, and organizes and stores it in time series;

[0145] Feature extraction module: extracts features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identifies performance indicators related to the health status of the knowledge base system;

[0146] Stability evaluation module: monitors and analyzes the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluates the operational stability of the knowledge base system based on abnormal changes in the knowledge base hit rate and fluctuations in the knowledge base update frequency;

[0147] Stability classification module: according to the evaluation results of the running stability of the knowledge base system, it is divided into different running status categories, and the running status of the knowledge base system is divided into a stable running status and an unstable running status;

[0148] Response effect evaluation module: When the running state of the knowledge base system is unstable, the performance of the knowledge model is monitored to determine the accuracy of the responses generated by the knowledge model and evaluate the response effect of the model;

[0149] Optimization management module: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

[0150] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0151] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0152] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0153] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0154] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A service knowledge base management method based on a knowledge big model, characterized by: The steps include: S1: Obtain the operating status data of the knowledge base system in real time through monitoring tools, pre-process the collected operating status data, and organize and store them in time series; S2: Extract features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identify performance indicators related to the health status of the knowledge base system; S3: Monitor and analyze the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluate the operational stability of the knowledge base system based on the abnormal changes in the knowledge base hit rate and the fluctuation status of the knowledge base update frequency; The step S3 comprises: generating a knowledge base hit rate anomaly index according to the abnormal change of the knowledge base hit rate within a fixed time period, and the method for obtaining the knowledge base hit rate anomaly index is: Collect the knowledge base hit rate data within the time period s, and establish the corresponding time series set. Take the initial knowledge base hit rate in the time series set as the initial EWMA value, use the exponentially weighted moving average formula to calculate the EWMA value at each time point, calculate the knowledge base hit rate anomaly value at each time point according to the difference between the current hit rate and the EWMA value, and sum up the knowledge base hit rate anomaly values ​​at all time points to obtain the knowledge base hit rate anomaly index; The knowledge base update frequency fluctuation index is generated according to the knowledge base update frequency fluctuation state. The method for obtaining the knowledge base update frequency fluctuation index is as follows: collect the frequency data of knowledge base updates within a time period, select a window size N, calculate the moving average update frequency at each time point, calculate the standard deviation at each time point in the same window to reflect the fluctuation of the update frequency, calculate the ratio of the moving average update frequency and the standard deviation at each time point, that is, calculate the knowledge base update frequency fluctuation index; S4: according to the evaluation result of the running stability of the knowledge base system, the knowledge base system is divided into different running state categories, and the running state of the knowledge base system is divided into a stable running state and an unstable running state; S5: When the running state of the knowledge base system is unstable, monitor the performance of the knowledge big model, judge the accuracy of the response generated by the knowledge big model, and evaluate the response effect of the model; S6: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

2. The service knowledge base management method based on the knowledge big model according to claim 1 is characterized by: The decision tree algorithm is used to extract features from the running status data of the knowledge base system at different times, and the performance indicators related to the health status of the knowledge base system are identified, specifically: Use monitoring tools to collect the running status data of the knowledge base system at different time points in real time. Each data point should include a timestamp and performance indicators. Sort the collected data in chronological order to ensure that the data at each time point has a corresponding timestamp. Store the data in a time series database for subsequent query and processing. Normalize the collected raw data; Use historical operating status data to train decision tree models; Use the trained decision tree model to extract features from the running status data of the knowledge base system at different times; The decision tree model automatically selects performance indicators that distinguish the health status of the system based on the characteristics of the data.

3. The service knowledge base management method based on the knowledge big model according to claim 1 is characterized by: The knowledge base hit rate anomaly index and the knowledge base update frequency fluctuation index are converted into the first eigenvector, and the first eigenvector is used as the input of the machine learning model. The machine learning model predicts the operation stability value label of the knowledge base system with each group of first eigenvectors as the prediction target, and minimizes the sum of prediction errors of the operation stability value labels of all knowledge base systems as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The operation stability value of the knowledge base system is determined according to the model output results.

4. The service knowledge base management method based on the knowledge big model according to claim 3 is characterized by: The obtained operation stability value of the knowledge base system is compared with a preset operation stability value reference threshold value. If the operation stability value of the knowledge base system is less than the preset operation stability value reference threshold value, the operation state of the knowledge base system is unstable, and the operation state of the corresponding knowledge base system is classified as an unstable operation state. The system administrator needs to check and adjust the knowledge base system in time. If the operation stability value of the knowledge base system is greater than or equal to a preset operation stability value reference threshold, the operation state of the knowledge base system is stable, and the operation state of the corresponding knowledge base system is classified as a stable operation state without intervention.

5. The service knowledge base management method based on the knowledge big model according to claim 4 is characterized by: When the running state of the knowledge base system is unstable, the performance of the knowledge model is monitored and evaluated through the random forest model to determine the accuracy and effect of the responses generated by the knowledge model. Specifically: Collect the query questions submitted by users to the knowledge model in real time and record the responses generated by the knowledge model; Monitor and collect system performance data when generating responses, and perform text feature extraction on user input and model responses; Extracting conversational context features and system performance data; Normalize the features extracted in real time; Combine the extracted text features, conversation features, and system performance features into a feature vector, ensuring that the format of the feature vector is consistent with that used when training the model; Input the feature vector constructed in real time into the trained random forest model; Get the prediction results of the random forest model, which are the accuracy values ​​of the model responses.

6. The service knowledge base management method based on the knowledge big model according to claim 5 is characterized by: The accuracy value of the obtained model response is compared with the accuracy value reference threshold. If the accuracy value is greater than or equal to the accuracy value reference threshold, the accuracy of the response generated by the knowledge model is high, and a normal model response signal is generated. If the accuracy value is less than the accuracy value reference threshold, the accuracy of the response generated by the knowledge model is low, and an abnormal model response signal is generated.

7. The service knowledge base management method based on the knowledge big model according to claim 6 is characterized by: According to the operational stability of the knowledge base system and the response effect of the model, after using fuzzy logic, the degree of its impact on the user experience is divided into levels, specifically: The running stability value of the knowledge base system and the model response accuracy value are used as the input items of the fuzzy logic, and the degree of impact on user experience is used as the output item of the fuzzy logic; Define the fuzzy set of the knowledge base system operation stability value and model response accuracy value, as well as the fuzzy set of the degree of impact on user experience; Based on the running stability value of the knowledge base system and the model response accuracy value, a set of fuzzy rules are defined to determine the degree of impact on user experience; The input items of knowledge base system operation stability value and model response accuracy value are converted into fuzzy sets through fuzzification function; According to the fuzzy rules, calculate the applicability of each rule; According to each fuzzy rule, calculate the fuzzy set of the impact degree of user experience; Aggregate the results of all rules to obtain a comprehensive set of fuzzy outputs; The comprehensive fuzzy output set is converted into a specific output value through the defuzzification method. According to the specific output value of the impact on user experience, it is divided into different levels. Different optimization management measures are taken according to the division results.

8. A service knowledge base management system based on a knowledge big model, used to implement the service knowledge base management method based on a knowledge big model according to any one of claims 1 to 7, characterized in that: It includes data acquisition module, feature extraction module, stability evaluation module, stability classification module, response effect evaluation module and optimization management module; Data acquisition module: obtains the operating status data of the knowledge base system in real time through monitoring tools, pre-processes the collected operating status data, and organizes and stores it in time series; Feature extraction module: extracts features from the running status data of the knowledge base system at different times through the decision tree algorithm, and identifies performance indicators related to the health status of the knowledge base system; Stability evaluation module: monitors and analyzes the knowledge base hit rate characteristics and knowledge base update frequency characteristics in the knowledge base system performance indicators, and evaluates the operational stability of the knowledge base system based on abnormal changes in the knowledge base hit rate and fluctuations in the knowledge base update frequency; Stability classification module: according to the evaluation results of the running stability of the knowledge base system, it is divided into different running status categories, and the running status of the knowledge base system is divided into a stable running status and an unstable running status; Response effect evaluation module: When the running state of the knowledge base system is unstable, the performance of the knowledge model is monitored to determine the accuracy of the responses generated by the knowledge model and evaluate the response effect of the model; Optimization management module: Based on the operational stability of the knowledge base system and the response effect of the model, fuzzy logic is used to classify the degree of impact on the user experience, and different optimization management measures are taken based on the classification results.

Images