Online service monitoring method and device and medium

Abstract

The embodiment of the invention provides an online service monitoring method and device and a medium. The method comprises the following steps: acquiring a plurality of user generated contents UGC published by a user in an online service; the online service is composed of a plurality of sub-services; for each UGC, based on a preset question database, identifying whether the UGC describes a service question, and under the condition that the UGC is identified as describing the service question, taking the UGC as a target UGC; and calling a first large language model to carry out natural language analysis and service problem positioning on the target UGC, and determining a problem sub-service corresponding to a service problem described in the target UGC.

Description

Technical Field

[0001] The present invention relates to the field of service monitoring, and in particular to an online service monitoring method, device and medium. Background Art

[0002] When monitoring user public opinion, the traditional method is to manually filter user public opinion feedback based on keywords and perform manual processing. This method mainly uses a program to filter user generated content (UGC, such as comments in the comment area, private messages, etc.) that meets the keyword conditions (such as hanging, crashing, etc.), and performs manual processing to determine whether there is a service problem. Alternatively, a monitoring method such as flow-based analysis is performed based on online requests. This method is mainly based on the success rate of online user requests, the HttpCode of the interface request response, the response time, and other characteristics. Statistics are performed, and the problems of online services are discovered using its statistical laws. Alternatively, based on the detection alarm of the automated test project, the main implementation method of this method is to perform automated test projects regularly through certain scheduling programs, and discover online service problems through automated test projects. The inventor found that the traditional methods each have different problems. The advantage of the method of directly manually filtering user public opinion feedback based on keywords and performing manual processing is that it can determine business-level problems, such as advertising dislocation, certain display problems, etc. The disadvantage is that it requires labor costs to be paid for manned duty, and the response speed is relatively slow. After the problem is discovered, the location problem also requires additional labor costs. The accuracy of content screening based on keywords is low, and it is easy to mix in a large amount of irrelevant content. The advantage of monitoring methods such as flow-based analysis based on online requests is that the processing process can be unattended and the detection speed is relatively fast, but the disadvantage is also relatively large. It cannot detect business errors, such as the disorder of client pictures, because its request success rate, response HttpCode and other indicators will not change. The advantage of the detection alarm method based on automated testing projects is that the problem is accurately located. The disadvantage is that because the automated testing project is executed on a scheduled basis and cannot be monitored online in real time, the problem is often discovered slowly, and there is no purpose during operation, consuming more resources.

[0003] In the process of implementing the present invention, the applicant discovered that the traditional online service problem location method has problems such as low execution efficiency and inability to locate business errors, which has become a technical problem that needs to be solved urgently in the prior art. Summary of the invention

[0004] The embodiments of the present invention provide an online service monitoring method, device and medium to solve the problems of low execution efficiency and inability to locate business errors in traditional online service problem locating methods.

[0005] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides an online service monitoring method, comprising:

[0006] An online service monitoring method, characterized by comprising:

[0007] Acquire multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services;

[0008] For each UGC, identifying whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, using the UGC as a target UGC;

[0009] The first language model is called to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC.

[0010] In a second aspect, an embodiment of the present invention provides an online service monitoring device, including:

[0011] A content acquisition unit, used to acquire multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services;

[0012] A qualitative identification unit, configured to identify, for each UGC, whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, use the UGC as a target UGC;

[0013] The problem determination unit is used to call the first language model to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the methods described above.

[0015] The above technical solution has the following beneficial effects: LLM is used to locate online service problems. LLM can be deployed online for automatic execution, so that it can replace manual supervision through automatic execution. Moreover, the natural language understanding ability of LLM can be used to locate service problems described in UGC. LLM can replace manual positioning of business problems. A large amount of user-generated content UGC can be collected online in real time in a short time. Before using the first language model to locate the problem sub-service, a large amount of user-generated content UGC obtained online in real time is first filtered, and the user-generated content UGC that does not describe the service problem or has a low possibility of describing the service problem is filtered out, and only the user-generated content UGC that has a high possibility of describing the service problem or is determined to describe the service problem is retained as the target UGC. Since usually, only a relatively small part of the large amount of user-generated content UGC will contain the service problem described by the user, so by pre-filtering to determine the target UGC, the number of UGCs to be analyzed and processed by the first language model can be significantly reduced in the subsequent steps, such as, significantly improving the execution efficiency, and significantly reducing the execution cost, while also realizing the automatic and real-time positioning of the sub-service with problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of an online service monitoring method according to one embodiment of the present invention;

[0018] Figure 2 This is an architectural diagram of an online service monitoring device according to one embodiment of the present invention;

[0019] Figure 3 It is a data flow diagram of an online service monitoring method according to one embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In view of various problems of traditional methods, the inventors proposed an online service monitoring method, device and medium through creative thinking. When conceiving the technology, the inventor creatively considered using LLM to locate online service problems. LLM can be deployed online for automatic execution, so that it can replace manual duty by automatic execution. Moreover, the natural language understanding ability of LLM can be used to locate the service problems described in UGC. LLM can replace manual positioning of business problems. In the process of invention and creation, the inventor also found that when users feedback problems, the effective information provided may be just a few simple words, such as "the pictures on Weibo are no longer visible", and the user UGC content generated by this is often difficult to accurately locate the specific service with problems. Therefore, only using LLM for processing can only obtain fuzzy fault service information. Therefore, some embodiments of the present invention also introduce an automated test project to detect specific fault points. The judgment information output by LLM is used to call the automated test of the module that may have problems for detection, which can improve the accuracy of problem discovery and reduce false alarms. After obtaining the problem sub-service located by LLM, the test case of the related problem sub-service can continue to be automatically called, and further according to the test results of the corresponding test case, it is accurately determined whether the sub-service actually has a problem, thereby improving the accuracy of locating the problem sub-service and solving the problem of not being able to detect in real time due to the timed call of the test case under the traditional technology. During the invention process, the inventors also found that due to the existence of a large number of UGCs with unreported service problems, when determining problem services from a large number of UGCs based on a large language model, the computing resource cost and usage cost are high, and the efficiency of determining problem services is low. The hardware cost required to use LLM (Large Language Model) to fully process online content UGC is very high. Therefore, the processing capacity of LLM limits the ability to directly use LLM to process the full amount of online UGC content. Some embodiments of the present invention also provide a method for using vectorization technology to pre-filter out some UGCs that do not or may not involve service problems, reduce the demand for LLM, and reduce costs. In some embodiments, the filtering method is further optimized, and the efficiency and effectiveness of filtering are improved by combining vectorization technology with LLM-based qualitative judgment to perform coarse filtering and fine filtering in layers.

[0022] First, as Figure 1 As shown, an embodiment of the present invention provides an online service monitoring method, comprising:

[0023] Step S10, obtaining multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services;

[0024] Step S11, for each UGC, identifying whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, taking the UGC as a target UGC;

[0025] Step S12, calling the first language model to perform natural language analysis and service problem location on the target UGC, and determining the problem sub-service corresponding to the service problem described in the target UGC.

[0026] In some embodiments, LLM is used to locate online service problems. LLM can be deployed online for automatic execution, so that it can replace manual duty by automatic execution. Moreover, the natural language understanding ability of LLM can be used to locate service problems described in UGC. LLM can replace manual positioning of business problems, and a large amount of user-generated content UGC can be collected online in real time in a short time. Under normal circumstances, only a relatively small part of the large amount of user-generated content UGC will contain service problems described by users. And each call to a large language model consumes more software and hardware resources and cost costs. Therefore, before using the first large language model to locate the problem sub-service, the embodiment of the present invention first filters a large amount of user-generated content UGC obtained online in real time, filters out user-generated content UGC that does not describe service problems or has a low possibility of describing service problems, and only retains user-generated content UGC that has a high possibility of describing service problems or is determined to describe service problems as target UGC. By pre-filtering to determine the target UGC, the number of UGCs to be analyzed and processed in subsequent steps, such as the first language model, can be significantly reduced, thereby significantly reducing the demand for software and hardware resources, significantly improving execution efficiency, and significantly reducing execution costs. At the same time, it also achieves automatic real-time positioning of problematic sub-services, avoiding the heavy workload, inefficiency and untimely response of manual processing of UGC. The speed of online UGC generation can easily exceed 1000qps (articles per second). Such a large amount of UGC is sent to LLM for processing, which is difficult for LLM to bear and requires a large amount of hardware costs. Through filtering, UGC that does not describe service problems, which accounts for 99.9% of the total UGC, can be filtered out. Taking LLM as GPT3.5turbo for testing, the current call fee is 0.00737 / thousand tokens, and a single UGC consumes about 0.2 thousand tokens on average. The UGC volume of a certain online service is calculated as 30 million per day. If LLM processes all UGC every day, it will cost about 44,000 yuan per day. After applying the embodiment of the present invention, the average daily fee is about 30 yuan, and the fee of the embodiment of the present invention is only 0.068% of the full processing fee. The fee reduction effect is very significant.

[0027] Further, for each UGC, identifying whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, using the UGC as a target UGC includes:

[0028] For each UGC, the correlation between the UGC and the question contents in the preset question database is calculated. When it is determined that the correlation meets the preset screening conditions, the UGC is identified as the target UGC that describes the service problem; wherein the question content refers to the historical UGC that describes at least one service problem.

[0029] In some embodiments, a large amount of user-generated content (UGC) will appear online in real time. If these user-generated content (UGC) are directly handed over to the large language model for processing, the execution cost and the consumption of computing resources will be very huge. The embodiment of the present invention determines the relevance of each UGC with the content of each question in the preset question database, and can pre-screen the target UGC that describes the service problem at a relatively low cost. Since the execution cost of the large language model is relatively high, after pre-screening, the UGC that obviously does not describe the service problem can be filtered out, and only the UGC that describes the service problem or is relatively likely or has a certain possibility of describing the service problem is submitted to the large language model for processing. The preset screening conditions are used to control the difficulty of the UGC obtained online being identified as the target UGC. For example, according to the specific situation of the online operation, the preset screening conditions can be adjusted so that more UGC obtained online is identified as the target UGC to avoid missing the service problems reported by users; the preset screening conditions can also be adjusted to make the conditions for identifying the UGC obtained online as the target UGC stricter, thereby reducing the number of UGCs identified as the target UGC and reducing the execution pressure on the subsequent steps.

[0030] The embodiments of the present invention have the following technical effects: the execution efficiency of content relevance analysis is usually high, and the consumption of computing device resources is small. Before using the first language model to locate more complex specific problem sub-services, the target UGC is determined by content relevance analysis, thereby significantly reducing the workload of the first language model in a manner of high execution efficiency and low resource consumption, thereby achieving the effect of improving efficiency and reducing costs.

[0031] Further, the calculating of the correlation between the UCG and each question content in the preset question database, and identifying the UCG as a target UGC describing a service question when it is determined that the correlation meets a preset screening condition, includes:

[0032] Vectorize the UGC to obtain a to-be-processed vector corresponding to the UGC;

[0033] Calculate the similarity between the vector to be processed and the question vector corresponding to each question content in the preset question database to obtain a similarity value between the vector to be processed and each question vector; wherein the preset question database includes question vectors corresponding to multiple question contents, and the multiple question contents in the preset question database are from a first historical UGC set, and each historical UGC in the first historical UGC set describes at least one service problem in a historical period;

[0034] Determine whether the maximum similarity value among all similarity values ​​is greater than or equal to a first preset similarity threshold;

[0035] When the maximum similarity value is greater than or equal to the first preset similarity threshold, the UGC is identified as a target UCG describing a service problem; otherwise, it is determined that the UGC does not describe a service problem.

[0036] In some embodiments, the execution efficiency of calculating the similarity between vectors is relatively high, and the execution cost of screening UGC by calculating the similarity values ​​between vectors is low and the efficiency is high.

[0037] Furthermore, the method for determining the first preset similarity threshold comprises:

[0038] Vectorizing each historical UGC in the second historical UGC set to obtain a corresponding historical user content vector; wherein each historical UGC in the second historical UGC set describes at least one service problem in a historical period, and the second historical UGC set has no intersection with the first historical UGC set;

[0039] For each historical user content vector, similarity calculation is performed between the historical user content vector and each question vector in the preset question database, and the maximum similarity value calculated is used as the target similarity value corresponding to the historical user content vector;

[0040] The target similarity values ​​corresponding to all historical user content vectors are sorted from high to low, and the target similarity value at a first preset sorting position is selected as the first preset similarity threshold.

[0041] Preferably, the distance between the first preset sorting position and the lowest sorting position is greater than the distance between the first preset sorting position and the highest sorting position.

[0042] In some embodiments, a similarity threshold is determined by comparing the similarity values ​​of two different sets of historical UGC sets, providing a method for obtaining a more accurate similarity threshold. For example, 100 similarity values ​​are obtained, and the 20th similarity value in the order of similarity values ​​from high to low is taken as the similarity threshold, so that the received UGC can be filtered more strictly to avoid UGC that does not describe service problems from being sent to subsequent steps for processing. Reduce the processing pressure of large models. The lowest ranking position is the position with the lowest target similarity value in the order of target similarity values; the highest ranking position is the position with the highest target similarity value in the order of target similarity values.

[0043] Further, for each UGC, identifying whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, using the UGC as a target UGC includes:

[0044] For each piece of UGC, the correlation between the UCG and each question content in the preset question database is calculated, and when it is determined that the correlation meets the preset preliminary selection condition, the UGC is used as the preliminary selection UGC; wherein the question content refers to the historical UGC that describes at least one service problem;

[0045] The second largest language model is called to perform natural language analysis on the preliminary selected UGC, and when it is determined based on the second largest language model that the preliminary selected UGC describes a service problem, the preliminary selected UGC is used as the target UGC.

[0046] In some embodiments, a large amount of user-generated content (UGC) will appear online in real time. If these user-generated content (UGC) are directly handed over to the large language model for processing, the execution cost and the consumption of computing resources will be very huge. The embodiment of the present invention determines the relevance of each UGC with the content of each question in the preset question database, and can pre-select the preliminary UGC that describes the service problem at a relatively low cost. Due to the high execution cost of the large language model, the UGC is preliminarily selected through correlation comparison and preset preliminary selection condition constraints. The preliminary selection conditions can be set broader to avoid missing the UGC that describes the service problem and filter out the UGC that obviously does not describe the service problem. After the preliminary selection, a large number of UGCs that do not have service problems can be filtered out. For the obtained preliminary UGCs, the second largest language model can be further used for more accurate qualitative screening, and the preliminary UGCs determined by the second largest language model to describe the service problem are used as the target UGC.

[0047] The first language model and the second language model can be the same language model instance or different language model instances, and can be deployed according to specific needs. For each UGC, before executing the step of calling the first language model to perform natural language analysis on the target UGC and determine the problem sub-service, the second language model is used to classify and identify the preliminary UGC, and divide it into two categories: target UGC that describes the user service problem and UGC that does not describe the user service problem. For the target UGC that describes the user service problem, it is necessary to further determine the problem sub-service before the step of calling the first language model to perform natural language analysis on the target UGC and determine the problem sub-service is allowed. Taking advantage of the fact that the classification problem requires fewer execution resources than determining the specific problem sub-service, the preliminary UGC that does not describe the service problem is filtered out in advance, and the number of target UGCs that call the first language model for analysis is reduced, which can improve the overall throughput and improve the overall execution efficiency. Preferably, the step of calculating the correlation between the UCG and each question content in the preset question database for each UGC, and when it is determined that the correlation meets the preset preliminary selection conditions, the UGC is used as the preliminary selection UGC, and the step of calling the second largest language model to perform natural language analysis on the preliminary selection UGC, and when it is determined based on the second largest language model that the preliminary selection UGC describes a service problem, the preliminary selection UGC is used as the target UGC, and the step of calling the first largest language model to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC, and the three steps are deployed as a pipeline execution mode to further improve the throughput. For example, first determine whether UGC1 meets the preliminary selection UGC conditions, determine the preliminary selection UGC1, and send the preliminary selection UGC1 to the second largest language model to determine whether it is the target UGC1. If it is the target UGC1, the target UGC1 is sent to the first largest language model to locate the problem sub-service. Every time UGC1 leaves the previous step, the new UGC2 is sent to the previous step, and so on. At this time, UGCC1 and UGCC2 are processed simultaneously in different steps of the embodiment of the present invention. Compared with the process of starting the next UGC only after each UGC is completely executed by the embodiment of the present invention, the execution mode of the pipeline can be deployed to increase the throughput exponentially.

[0048] The embodiment of the present invention has the following technical effects: through correlation calculation, the UGC obtained online that obviously does not describe the service problem is filtered out with a relatively broad preliminary selection condition, and the UGC suspected of describing the service problem is retained as the preliminary selection UGC as much as possible. This is the first layer of filtering, which is equivalent to rough filtering, and can significantly reduce the number of UGCs that need to be processed by the second largest language model. For the preliminary selected UGC, the second largest language model is used to qualitatively determine whether it describes the service problem. Qualitative judgment is simpler than locating specific problem sub-services and requires fewer resources. At the same time, the accuracy of the natural language analysis capability of the large language model can be used to improve the accuracy of determining the target UGC. This is the second layer of filtering, which is equivalent to precise filtering. After two layers of filtering, the workload of the first largest language model for locating specific problem sub-services is significantly reduced, thereby balancing the efficiency, cost and accuracy of problem sub-service locating, and achieving the effect of improving efficiency, reducing costs and improving accuracy. By utilizing the inherent knowledge reserve of the second largest language model, the preliminary UGC is accurately classified in advance to determine the target UGC that describes the service problem. Therefore, the first largest language model only needs to perform a more detailed problem sub-service determination operation on the target UGC that describes the service problem, which can significantly reduce the workload of the first largest language model in processing the target UGC. Because processing the target UGC to determine the problem sub-service requires more computing resources such as execution time and memory compared to qualitatively distinguishing whether a service problem is described, the hierarchical processing implemented in the embodiment of the present invention can significantly reduce the demand for computing resources, improve execution efficiency, and thus increase the throughput per unit time.

[0049] Preferably, for each piece of UGC, the correlation between the UCG and each question content in the preset question database is calculated, and when it is determined that the correlation meets the preset preliminary selection condition, the UGC is used as the preliminary selection UGC; wherein the question content refers to the historical UGC that describes at least one service problem, including:

[0050] Vectorize the UGC to obtain a to-be-processed vector corresponding to the UGC;

[0051] Calculate the similarity between the vector to be processed and the question vector corresponding to each question content in the preset question database to obtain a similarity value between the vector to be processed and each question vector; wherein the preset question database includes question vectors corresponding to multiple question contents, and the multiple question contents in the preset question database are from a first historical UGC set, and each historical UGC in the first historical UGC set describes at least one service problem in a historical period;

[0052] Determine whether the maximum similarity value among all similarity values ​​is greater than or equal to a second preset similarity threshold;

[0053] When the maximum similarity value is greater than or equal to the second preset similarity threshold, the UGC is identified as a preliminary UCG describing a service problem; otherwise, it is determined that the UGC does not describe a service problem.

[0054] In some embodiments, the execution efficiency of calculating the similarity between vectors is relatively high, and the execution cost of performing preliminary selection of UGC by calculating the similarity values ​​between vectors is low and the efficiency is high.

[0055] Furthermore, the method for determining the second preset similarity threshold comprises:

[0056] Vectorizing each historical UGC in the second historical UGC set to obtain a corresponding historical user content vector; wherein each historical UGC in the second historical UGC set describes at least one service problem in a historical period, and the second historical UGC set has no intersection with the first historical UGC set;

[0057] For each historical user content vector, similarity calculation is performed between the historical user content vector and each question vector in the preset question database, and the maximum similarity value calculated is used as the target similarity value corresponding to the historical user content vector;

[0058] The target similarity values ​​corresponding to all historical user content vectors are sorted from high to low, and the target similarity value at a second preset sorting position is selected as the second preset similarity threshold.

[0059] Preferably, the distance between the second preset sorting position and the lowest sorting position is smaller than the distance between the second preset sorting position and the highest sorting position.

[0060] In some embodiments, by comparing the similarity values ​​of two different sets of historical UGC collections, a similarity threshold is determined, providing a method for obtaining a more accurate similarity threshold. For example, 100 similarity values ​​are obtained, and the 80th similarity value sorted from high to low is taken as the similarity threshold, because the original 100 pieces of content are selected materials related to feedback issues, and the similarity itself should be high in theory, but because the vectorization model is a pre-trained model, the similarity calculated for some content may be low. For example, taking the lowest similarity value (the 100th similarity value) will cause the filtering effect of this step to decrease, and taking the 50th or higher similarity value may cause the similarity requirement to be too high to filter out user-generated content that should not be filtered. According to statistical data and the 2 / 8 theorem, preferably, the similarity value of the 80th sequence number position is determined as the similarity threshold. The lowest sorting position is the position with the lowest target similarity value in the sorting of the target similarity value; the highest sorting position is the position with the highest target similarity value in the sorting of the target similarity value.

[0061] Further, calling the second largest language model to perform natural language analysis on the preliminary selected UGC, and in the case where it is determined based on the second largest language model that the preliminary selected UGC describes a service problem, using the preliminary selected UGC as the target UGC, includes:

[0062] Generate a second question text, and input the second question text into the second largest language model; the second question text is used to inquire whether the primary UGC describes a service problem, and requires the second largest language model to answer "yes" or "no";

[0063] Receive a second answer text responded by the second language model to the second question text;

[0064] When the second answer text is "yes", it is determined that the preliminary UGC describes the service problem, and the preliminary UCG is used as the target UCG; otherwise, it is determined that the preliminary UGC does not describe the service problem.

[0065] The following is an explanation using a specific embodiment: the second largest language model determines whether the preliminary UGC feedbacks service problems (or faults). Specifically, a large language model such as LLM such as GPT3.5 or GPT4 can be used to determine whether the preliminary UGC is a feedback service problem. An example of the second prompt statement: "Weibo is an App similar to Twitter. You are now a customer service staff of Weibo. You have received the following feedback information from users. Now please determine whether the following feedback information indicates that there is a problem with the Weibo service. If so, please answer yes. If not, please answer no. Please do not output any other text except yes and no. The following is the user's feedback information: xxx". If the model answers yes, continue to execute other steps to further locate the problem sub-service. If the answer is no, the preliminary UGC does not describe a service problem and does not need to be further processed. Among them, xxx is the preliminary UGC.

[0066] Further, calling the first language model to perform natural language analysis and service problem location on the target UGC to determine the problem sub-service corresponding to the service problem described in the target UGC includes:

[0067] Generate a first question text, and input the first question text into the first language model; the first question text is used to inquire about the problem sub-service corresponding to the service problem described in the target UGC, and the first question text includes the service names and corresponding function descriptions of multiple sub-services constituting the online service;

[0068] Receive a first answer text in response to the first question text by the first language model; the first answer text includes a service name corresponding to the service problem described in the target UGC;

[0069] According to the service name specified in the first answer text, the problem sub-service corresponding to the service problem described in the target UGC is determined.

[0070] The following is an explanation using a specific embodiment: the first language model is used to determine what sub-service module the target UGC feedback content involves. Specifically, an LLM (large language model) such as GPT3.5 or GPT4 can be used as the first language model to determine which sub-services the target UGC reflects may have problems. An example of the first prompt sentence is: "Weibo is an App similar to Twitter. You are now a Weibo engineer and need to determine what service has problems based on user feedback. This is the user's feedback information: xxx. The following content is the description information of each service and corresponding function of Weibo: yyy. Finally, please tell me what service has problems. Please tell me the service name. Do not output other information. If there are multiple possibilities, directly separate the service names with commas." Among them, xxx is the content of the target UGC, and yyy is the content of the description information of each sub-service and the corresponding function. That is, it is equivalent to the description of at least one sub-service name and its corresponding function. In an embodiment of the present invention, by providing a description of at least one sub-service name and its corresponding function in the first prompt sentence, the scope of the sub-services explored by the first language model can be limited, the difficulty of the first language model generating answers can be reduced, and the accuracy of generating answers can be improved.

[0071] Further, the service name corresponding to the service problem described in the target UGC in the first answer text includes one or more;

[0072] The determining, according to the service name specified in the first answer text, the problem sub-service corresponding to the service problem described in the target UGC includes:

[0073] For one or more sub-services corresponding to one or more service names in the first answer text, respectively execute the automated test cases corresponding to each sub-service to generate a test result corresponding to each sub-service;

[0074] The sub-service with abnormal test results is used as the problem sub-service corresponding to the service problem described in the target UGC.

[0075] In some embodiments, the target UGC is generated by the user, and the target UGC contains a description of the problem reported by the user for one or more sub-services. The first language model is a model that can understand natural language and give answer information for question information; for example, it includes but is not limited to various versions of GPT, ChatGLM, etc. The first language model can understand the semantic content of natural language, and use the first language model to perform natural language analysis on the target UGC, so that the large language model automatically analyzes the problem sub-services reported in the target UGC, thereby replacing manual analysis, significantly improving the recognition efficiency of the problem sub-service and shortening the total time consumed for problem service processing. After obtaining the problem sub-service, the automated test case corresponding to the problem sub-service is automatically called, and the two operations of automatically discovering the problem sub-service based on the first language model and automatically testing and confirming the problem sub-service are seamlessly connected. Finally, according to the test results of each problem sub-service, it can be seen that the test case of the problem sub-service has passed and which one has not passed the test, so that one or more problem sub-services that have not passed the test can be used as the problem sub-service corresponding to the service problem described in the target UGC, and the sub-service with problems can be reported to the operation and maintenance personnel through online communication methods including but not limited to SMS, email, and / or WeChat. The whole process is completed automatically without the participation of technical personnel.

[0076] The embodiments of the present invention have the following technical effects: utilizing the natural language analysis and understanding capabilities of a large language model to perform natural language analysis on target UGC, thereby replacing manual analysis and identification, realizing automated analysis and identification of target UGC, and transferring the identification and positioning of problem services from manual offline processing to online automated processing. After locating the problem sub-service, functional testing of the problem sub-service is performed by executing corresponding test cases. The sub-service with problems can be completely and accurately determined through the test results. The entire problem sub-service determination process is automatically processed online without manual intervention. Compared with traditional manual identification and positioning methods, this technical solution has a fast response. After receiving UGC, the large language model can be called online in real time to start analyzing and understanding UGC. There is no technical The response delay problem caused by the physiological work and rest cycle of personnel; this technical solution has high analysis and processing efficiency, and the knowledge reserve threshold required for technical personnel is low. It is based on the automatic analysis of the online large language model, making full use of the higher execution efficiency of computing equipment and the inherent knowledge reserve of the large language model, and the analysis and processing efficiency of UGC is higher. The offline manual analysis is transferred to the online large language model analysis, and seamless connection with automated testing is achieved. The connection time from determining the problem sub-service to starting to execute the corresponding automated test is reduced to milliseconds, while the traditional connection time from manually determining the problem sub-service to starting to execute the corresponding automated test case is at least minutes, and even requires hours or days to enter the automated testing stage due to the process transfer involving the executors. This technical solution has high accuracy in locating the problem service, automatically calling the corresponding test case for the problem sub-service, and automatically screening the problem sub-service, so as to accurately locate the sub-service with problems through the test results, while achieving timely response and efficient processing of UGC, it can accurately locate the sub-service with problems.

[0077] Furthermore, the method further comprises:

[0078] The multiple UGCs obtained from the online service are added to the content queue for caching, so that each UGC is taken out from the content queue one by one to perform subsequent steps such as identifying whether the UGC describes a service problem.

[0079] In some embodiments, multiple UGCs are received through an online content queue, and UGCs are taken out from the content queue one by one and provided to subsequent steps for processing, thereby achieving decoupling and isolation between content producers and content consumers, as well as the function of content caching.

[0080] Second, as Figure 2 As shown, an embodiment of the present invention provides an online service monitoring device, comprising:

[0081] The content acquisition unit 200 is used to acquire multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services;

[0082] A qualitative identification unit 201 is used to identify, for each UGC, whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, use the UGC as a target UGC;

[0083] The problem determination unit 202 is used to call the first language model to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC.

[0084] Furthermore, the qualitative identification unit 201 is specifically used for:

[0085] For each UGC, the correlation between the UGC and the question contents in the preset question database is calculated. When it is determined that the correlation meets the preset screening conditions, the UGC is identified as the target UGC that describes the service problem; wherein the question content refers to the historical UGC that describes at least one service problem.

[0086] Furthermore, the qualitative identification unit 201 specifically includes:

[0087] A first UGC vectorization module, used for vectorizing the UGC to obtain a to-be-processed vector corresponding to the UGC;

[0088] A first similarity determination module is used to calculate the similarity between the vector to be processed and the question vector corresponding to each question content in the preset question database, so as to obtain a similarity value between the vector to be processed and each question vector; wherein the preset question database includes question vectors corresponding to multiple question contents respectively, and the multiple question contents in the preset question database are from a first historical UGC set, and each historical UGC in the first historical UGC set describes at least one service problem in a historical period;

[0089] A first similarity comparison module, used to determine whether a maximum similarity value among all similarity values ​​is greater than or equal to a first preset similarity threshold;

[0090] The first target UGC determination module is used to identify the UGC as a target UGC that describes a service problem when the maximum similarity value is greater than or equal to the first preset similarity threshold; otherwise, determine that the UGC does not describe a service problem.

[0091] Furthermore, the device further includes: a first preset similarity threshold determination module configured to:

[0092] Vectorizing each historical UGC in the second historical UGC set to obtain a corresponding historical user content vector; wherein each historical UGC in the second historical UGC set describes at least one service problem in a historical period, and the second historical UGC set has no intersection with the first historical UGC set;

[0093] For each historical user content vector, similarity calculation is performed between the historical user content vector and each question vector in the preset question database, and the maximum similarity value calculated is used as the target similarity value corresponding to the historical user content vector;

[0094] The target similarity values ​​corresponding to all historical user content vectors are sorted from high to low, and the target similarity value at a first preset sorting position is selected as the first preset similarity threshold.

[0095] Preferably, the distance between the first preset sorting position and the lowest sorting position is greater than the distance between the first preset sorting position and the highest sorting position.

[0096] Furthermore, the qualitative identification unit 201 includes:

[0097] A preliminary selection module is used to calculate the correlation between the UCG and each question content in the preset question database for each UGC, and when it is determined that the correlation meets the preset preliminary selection condition, use the UGC as the preliminary selection UGC; wherein the question content refers to the historical UGC that describes at least one service problem;

[0098] The selection module is used to call the second largest language model to perform natural language analysis on the preliminary selected UGC, and when it is determined based on the second largest language model that the preliminary selected UGC describes a service problem, the preliminary selected UGC is used as the target UGC.

[0099] Furthermore, the preliminary selection module includes:

[0100] A second UGC vectorization module, used to vectorize the UGC to obtain a to-be-processed vector corresponding to the UGC;

[0101] A second similarity determination module is used to calculate the similarity between the vector to be processed and the question vector corresponding to each question content in the preset question database, so as to obtain a similarity value between the vector to be processed and each question vector; wherein the preset question database includes question vectors corresponding to multiple question contents respectively, and the multiple question contents in the preset question database are from a first historical UGC set, and each historical UGC in the first historical UGC set describes at least one service problem in a historical period;

[0102] A second similarity comparison module, used to determine whether the maximum similarity value among all similarity values ​​is greater than or equal to a second preset similarity threshold;

[0103] The preliminary UGC determination module is used to identify the UGC as a preliminary UGC that describes the service problem when the maximum similarity value is greater than or equal to the second preset similarity threshold; otherwise, determine that the UGC does not describe the service problem.

[0104] Furthermore, the device further includes: a second preset similarity threshold determination module configured to:

[0105] Vectorizing each historical UGC in the second historical UGC set to obtain a corresponding historical user content vector; wherein each historical UGC in the second historical UGC set describes at least one service problem in a historical period, and the second historical UGC set has no intersection with the first historical UGC set;

[0106] For each historical user content vector, similarity calculation is performed between the historical user content vector and each question vector in the preset question database, and the maximum similarity value calculated is used as the target similarity value corresponding to the historical user content vector;

[0107] The target similarity values ​​corresponding to all historical user content vectors are sorted from high to low, and the target similarity value at a second preset sorting position is selected as the second preset similarity threshold.

[0108] Preferably, the distance between the second preset sorting position and the lowest sorting position is smaller than the distance between the second preset sorting position and the highest sorting position.

[0109] Furthermore, the selection module includes:

[0110] A second questioning module is used to generate a second question text and input the second question text into the second largest language model; the second question text is used to inquire whether the primary UGC describes a service problem and requires the second largest language model to answer "yes" or "no";

[0111] A second answer acquisition module, configured to receive a second answer text replied by the second largest language model to the second question text;

[0112] The second target UGC determination module is used to determine that the preliminary UGC describes a service problem and use the preliminary UCG as the target UCG when the second answer text is "yes", otherwise, determine that the preliminary UGC does not describe a service problem.

[0113] Further, the problem determination unit 202 includes:

[0114] A first questioning module is used to generate a first question text and input the first question text into the first language model; the first question text is used to inquire about the problem sub-service corresponding to the service problem described in the target UGC, and the first question text includes the service names and corresponding function descriptions of multiple sub-services constituting the online service;

[0115] A first answer acquisition module is used to receive a first answer text replied by the first language model to the first question text; the first answer text includes a service name corresponding to the service problem described in the target UGC;

[0116] A problem sub-service determination module is used to determine the problem sub-service corresponding to the service problem described in the target UGC according to the service name specified in the first answer text.

[0117] Further, the service name corresponding to the service problem described in the target UGC in the first answer text includes one or more;

[0118] The problem sub-service determination module is configured as follows:

[0119] For one or more sub-services corresponding to one or more service names in the first answer text, respectively execute the automated test cases corresponding to each sub-service to generate a test result corresponding to each sub-service;

[0120] The sub-service with abnormal test results is used as the problem sub-service corresponding to the service problem described in the target UGC.

[0121] Furthermore, the device also includes:

[0122] The content cache unit is used to add the multiple UGCs obtained from the online service to the content queue for caching, so as to take out each UGC from the content queue one by one and trigger the qualitative identification unit 201.

[0123] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0124] As for the device, readable storage medium and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0125] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.

[0126] Glossary:

[0127] Online services: Different from the test environment, online services refer to the servers that actually provide Internet services to users and the programs running on them. In reality, online services are usually not a single program, but a complex data stream composed of dozens or hundreds of different programs. Because discovering and locating online problems has always been a difficult task in operation and maintenance work.

[0128] Online public opinion: refers to UGC (User Generated Content) content such as comments and private messages posted by users in real time online. For example, when Alibaba Cloud crashed, a large number of users posted on Weibo that Alipay could not be opened.

[0129] Test automation project: refers to the program used to test whether the service is working properly, which is often composed of different test cases corresponding to different test points. Generally speaking, these cases will be organized into different modules. This form of organization is related to the division of online services. For example, Weibo, the private messages and group chats in the Weibo message box are services provided by two different program groups, so they are divided into two modules: private message automation testing and group chat automation testing. This division method strengthens the pertinence of the test automation project. When different services need to be tested, different test automation projects can be run to avoid wasting time and resources by running all of them.

[0130] Traffic analysis: When online services are running, a large number of user requests will be sent to the server, and the server will accumulate a large amount of data. For example, if the request received is an http request, then these http requests will have characteristic data such as success rate, time consumption, response code, etc. Statistical analysis can be performed on these data. When certain characteristic data fluctuates, it may mean that there is a problem with the online service.

[0131] Text vectorization: A mathematical method that can transform a piece of text into a high-dimensional vector. The purpose of vectorization is to make the vectors of texts with similar meanings closer. In this way, when you have a piece of text, you can judge whether the text has similar meanings to other texts based on this "distance" information. The development of this vectorization technology relies on the progress of machine learning NLP (Neuro-Linguistic Programming) technology. There are currently a variety of models that can provide related functions, such as chatglm-embedding and gpt4-embedding. They are actually by-products of training these large models.

[0132] Cosine similarity: an algorithm for calculating the "distance" of vectors, similar to Euclidean distance, etc. The cosine similarity algorithm used in the embodiment of the present invention can convert the distance between two vectors into a cosine value between 0 and 1, which is easier to define the boundary values ​​that are considered to be similar in meaning than methods such as Euclidean distance.

[0133] PostgreSQL: An object-relational database. It is mainly used here as a vector database after installing a plug-in. That is, it can store one field whose value is a high-dimensional matrix. It also supports the use of various "distance" algorithms in the database to calculate the distance of the matrix for select operations.

[0134] LLM: Large Language Model refers to pre-trained language models such as GPT3.5, GPT4, ChatGLM, etc., which have billions to hundreds of billions of parameters.

[0135] Pre-training: refers to the model being trained in advance with a large amount of content, and the embodiments of the present invention do not adjust or retrain when they are used. Models such as the GPT series are also pre-trained models. The advantage of pre-training models is that they avoid the high training costs of the user in exchange for the general problem-solving ability of the model.

[0136] like Figure 3 As shown, it is a data flow diagram of an online service monitoring method according to one embodiment of the present invention. Figure 3 The embodiments of the present invention are described.

[0137] Step S31, filling the online UGC content queue: the online UGC content will be written into a UGC content queue, which contains the microblogs, comments, private messages, etc. sent by users;

[0138] Step S32, vectorization of the content to be processed: read a piece of UGC content (content to be processed) from the UGC content queue, and convert the UGC into a (text vector) high-dimensional vector, i.e., the vector to be processed, through an Embedding model such as Chatglm-embedding or gpt4-embedding;

[0139] Step S33, calculate the cosine similarity between the content to be processed and the content in the database (PostgreSQL vector database): calculate the cosine similarity value between the text vector (the vector to be processed) converted in the previous step and the vector of each content in the vector database, and take the maximum cosine similarity value from all the cosine similarity values;

[0140] Step S34, whether the maximum cosine similarity value is greater than σ: determine whether the maximum cosine similarity value is greater than σ (preset similarity threshold). If it is not greater than σ, it means that the content to be processed is not similar to any preset content in the vector database, and the content (to be processed) is discarded. If it is greater than σ, it means that the content to be processed is similar to the content corresponding to the vector in the vector database corresponding to the maximum cosine similarity value, and it is necessary to continue to perform public opinion processing on the content to be processed;

[0141] The content in the vector database is 100 pieces of online user feedback information or similar content about various problems on Weibo that were manually screened in advance. These contents were vectorized in advance and stored in the vector database. When judging whether the similarity is greater than σ, the source of σ is to select 100 pieces of online user feedback information or similar content about various problems on Weibo, and after vectorization, calculate the cosine similarity with the 100 pieces of content in the vector database, take the maximum value each time, and then sort them from high to low by similarity value, and take the 80th value as σ;

[0142] Step S35, the model determines whether the content is a feedback service failure: through a large language model such as LLM such as GPT3.5 or GPT4, it is determined whether the previously read content to be processed is a feedback service problem. Prompt example: "Weibo is an App similar to Twitter. You are now a customer service staff of Weibo. You have received the following feedback information from users. Now please determine whether the following feedback information indicates that there is a problem with the Weibo service. If so, please answer yes. If not, please answer no. Please do not output any other text except yes and no. The following is the user's feedback information: xxxx". If the model answers yes, it will continue to execute. If it answers no, the content to be processed will be discarded and no further processing is required;

[0143] Step S36, the model determines which module the user feedback content belongs to: through LLM such as GPT3.5 or GPT4, the read content reflects which services may have problems. Prompt example: "Weibo is an App similar to Twitter. You are now a Weibo engineer and need to determine which service has problems based on user feedback. This is the user's feedback information: xxxxxx. The following content is the description information of Weibo's various services and corresponding functions: xxxxxxxxxxxxxxxxxxxxxxxxxx. Finally, please tell me what service has problems. Please tell me the service name. Do not output other information. If there are multiple possibilities, separate the service names directly with commas." Optionally, through step S38, a notification reminder is sent to DingTalk, WeChat, etc., so that the operation and maintenance personnel are notified of the problematic services.

[0144] Step S37, test service scheduling: If LLM determines that some services have problems, it will call the automated test project of the corresponding service, and after the automated test project is completed, execute step S38 to notify the development and operation personnel of the results through channels such as DingTalk and WeChat. For example, the automated test cases of modules A, B, and C are scheduled in the figure.

[0145] The embodiment of the present invention has the following technical effects: compared with the traditional keyword-based online user feedback information filtering method, the embodiment of the present invention adopts text vectorization technology to solve the problem that keyword-based filtering is not accurate enough and is prone to false alarms or non-reporting; compared with the traditional manual interpretation of user feedback information, the embodiment of the present invention adopts pre-trained LLM to replace manual interpretation of content, which can reduce the cost of manual duty and be on call 24 hours a day; after using LLM to determine that there is indeed a problem with the online service, the embodiment of the present invention also uses LLM to make a preliminary judgment on the service that caused the problem, and verifies the problem through automated testing engineering, which improves the accuracy of problem feedback and reduces the total time of problem troubleshooting. The embodiment of the present invention utilizes the natural language processing ability of LLM to realize the processing of online user feedback information without human attendance. At the same time, LLM can undertake the work of classifying user feedback information, and can preliminarily judge the back-end service with problems based on the functional information affected by the back-end service, and use automated testing engineering to accurately check. The entire process does not require human intervention and achieves a high degree of automation. The embodiments of the present invention improve the ability to discover online service problems, shorten the time to discover online problems, effectively identify user feedback information, avoid manual supervision and manual intervention, and realize automatic feedback information processing, thereby solving the problem of inaccurate problem point judgment in traditional manual processing solutions based on online user feedback and the inaccurate keyword-based content filtering method. It also solves the problem that the LLM calling capacity is generally several times per second and cannot directly process thousands of UGC contents per second of online services for content interpretation.

[0146] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0147] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0148] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0149] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including". In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0150] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0151] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0152] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0153] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0154] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An online service monitoring method, characterized in that: include: Acquire multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services; For each UGC, identifying whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, using the UGC as a target UGC; The first language model is called to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC.

2. The online service monitoring method according to claim 1, characterized in that: The step of identifying, for each UGC, whether the UGC describes a service problem based on a preset problem database, and taking the UGC as a target UGC when the UGC is identified as describing a service problem, includes: For each UGC, the correlation between the UGC and the question contents in the preset question database is calculated. When it is determined that the correlation meets the preset screening conditions, the UGC is identified as the target UGC that describes the service problem; wherein the question content refers to the historical UGC that describes at least one service problem.

3. The online service monitoring method according to claim 2, characterized in that: The calculating the correlation between the UCG and each question content in the preset question database, and identifying the UCG as a target UGC describing a service question when it is determined that the correlation meets a preset screening condition, includes: Vectorize the UGC to obtain a to-be-processed vector corresponding to the UGC; Calculate the similarity between the vector to be processed and the question vector corresponding to each question content in the preset question database to obtain a similarity value between the vector to be processed and each question vector; wherein the preset question database includes question vectors corresponding to multiple question contents, and the multiple question contents in the preset question database are from a first historical UGC set, and each historical UGC in the first historical UGC set describes at least one service problem in a historical period; Determine whether the maximum similarity value among all similarity values ​​is greater than or equal to a first preset similarity threshold; When the maximum similarity value is greater than or equal to the first preset similarity threshold, the UGC is identified as a target UCG describing a service problem; otherwise, it is determined that the UGC does not describe a service problem.

4. The online service monitoring method according to claim 3, characterized in that: The method for determining the first preset similarity threshold comprises: Vectorizing each historical UGC in the second historical UGC set to obtain a corresponding historical user content vector; wherein each historical UGC in the second historical UGC set describes at least one service problem in a historical period, and the second historical UGC set has no intersection with the first historical UGC set; For each historical user content vector, similarity calculation is performed between the historical user content vector and each question vector in the preset question database, and the maximum similarity value calculated is used as the target similarity value corresponding to the historical user content vector; The target similarity values ​​corresponding to all historical user content vectors are sorted from high to low, and the target similarity value at a first preset sorting position is selected as the first preset similarity threshold.

5. The online service monitoring method according to claim 1, characterized in that: The step of identifying, for each UGC, whether the UGC describes a service problem based on a preset problem database, and taking the UGC as a target UGC when the UGC is identified as describing a service problem, includes: For each piece of UGC, the correlation between the UCG and each question content in the preset question database is calculated, and when it is determined that the correlation meets the preset preliminary selection condition, the UGC is used as the preliminary selection UGC; wherein the question content refers to the historical UGC that describes at least one service problem; The second largest language model is called to perform natural language analysis on the preliminary selected UGC, and when it is determined based on the second largest language model that the preliminary selected UGC describes a service problem, the preliminary selected UGC is used as the target UGC.

6. The online service monitoring method according to claim 5, characterized in that: The calling of the second largest language model to perform natural language analysis on the preliminary selected UGC, and when it is determined based on the second largest language model that the preliminary selected UGC describes a service problem, using the preliminary selected UGC as the target UGC, includes: Generate a second question text, and input the second question text into the second largest language model; the second question text is used to inquire whether the primary UGC describes a service problem, and require the second largest language model to answer "yes" or "no"; Receive a second answer text responded by the second language model to the second question text; When the second answer text is "yes", it is determined that the preliminary UGC describes the service problem, and the preliminary UCG is used as the target UCG; otherwise, it is determined that the preliminary UGC does not describe the service problem.

7. The online service monitoring method according to claim 1, characterized in that: The calling of the first language model to perform natural language analysis and service problem location on the target UGC, and determining the problem sub-service corresponding to the service problem described in the target UGC, includes: Generate a first question text, and input the first question text into the first language model; the first question text is used to inquire about the problem sub-service corresponding to the service problem described in the target UGC, and the first question text includes the service names and corresponding function descriptions of multiple sub-services constituting the online service; Receive a first answer text in response to the first question text by the first language model; the first answer text includes a service name corresponding to the service problem described in the target UGC; According to the service name specified in the first answer text, the problem sub-service corresponding to the service problem described in the target UGC is determined.

8. The online service monitoring method according to claim 7, characterized in that: The first answer text includes one or more service names corresponding to the service problem described in the target UGC; The determining, according to the service name specified in the first answer text, the problem sub-service corresponding to the service problem described in the target UGC includes: For one or more sub-services corresponding to one or more service names in the first answer text, respectively execute the automated test cases corresponding to each sub-service to generate a test result corresponding to each sub-service; The sub-service with abnormal test results is used as the problem sub-service corresponding to the service problem described in the target UGC.

9. An online service monitoring device, characterized in that: include: A content acquisition unit, used to acquire multiple pieces of user-generated content (UGC) published by users in an online service; wherein the online service is composed of multiple sub-services; A qualitative identification unit, configured to identify, for each UGC, whether the UGC describes a service problem based on a preset problem database, and if the UGC is identified as describing a service problem, use the UGC as a target UGC; The problem determination unit is used to call the first language model to perform natural language analysis and service problem location on the target UGC, and determine the problem sub-service corresponding to the service problem described in the target UGC.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images