Method, apparatus, device, storage medium and program product for data processing

Abstract

According to embodiments of the present disclosure, methods, apparatuses, devices, storage media, and program products for data processing are provided. The method includes: in response to receiving an analysis indication of a running of an agent, obtaining interaction data associated with the agent, the interaction data including at least a plurality of questions collected by the agent and corresponding reply information; based on an analysis strategy, obtaining a set of analysis results on the plurality of questions in the interaction data, the set of analysis results including at least a set of labels for the plurality of questions, the set of labels indicating reply quality of the agent for the plurality of questions; and presenting a visualization content for the set of analysis results.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for data processing. Background Technology

[0002] With the rapid development of computer technology, intelligent agents have gradually become one of the key architectures for building intelligent systems. In some scenarios, an intelligent agent is an intelligent entity capable of perceiving its environment, making autonomous decisions, and executing tasks. The concept encompasses a wide range of fields, from simple software modules to complex autonomous robots. With the widespread application of intelligent agent platforms, the demand for analyzing intelligent agent operational data is also increasing. Summary of the Invention

[0003] In a first aspect of this disclosure, a data processing method is provided. The method includes: in response to receiving an analysis instruction for the operation of an agent, acquiring interaction data associated with the agent, the interaction data including at least multiple questions and corresponding response information collected by the agent; based on an analysis strategy, acquiring a set of analysis results regarding the multiple questions in the interaction data, the set of analysis results including at least a set of labels for the multiple questions, the set of labels indicating the quality of the agent's responses to the multiple questions; and presenting visualizations of the set of analysis results.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: an interactive data acquisition module configured to acquire interactive data associated with the agent in response to receiving an analysis instruction to operate an agent, the interactive data including at least multiple questions and corresponding responses collected by the agent; an analysis result acquisition module configured to acquire a set of analysis results regarding the multiple questions in the interactive data based on an analysis strategy, the set of analysis results including at least a set of labels for the multiple questions, the set of labels indicating the quality of the agent's responses to the multiple questions; and a visualization content presentation module configured to present visualization content for the set of analysis results.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown; Figure 2 A schematic diagram of an example architecture capable of implementing this solution according to some embodiments of this disclosure is shown; Figure 3 A flowchart illustrating a data processing procedure according to some embodiments of the present disclosure is shown; Figure 4 A block diagram of an apparatus for data processing according to some embodiments of the present disclosure is shown; and Figure 5 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0010] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0011] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0012] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0013] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0014] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0019] As briefly mentioned earlier, with the widespread application of intelligent agent platforms, the demand for analyzing intelligent agent operational data is increasing. In intelligent agent operational analysis scenarios, it is typically necessary to determine the relevance between user issues and intelligent agent functions, and then perform data statistics and analysis based on the results. Conventionally, intelligent agent platforms rely heavily on professional intervention and basic data statistics at the operational analysis level. For example, professionals need to manually filter irrelevant issues, determine whether each user issue falls within the scope of the intelligent agent's responsibilities, and manually label the issue categories. This approach is labor-intensive and cannot automate the processing of large amounts of data. Furthermore, this approach can lead to incomplete labeling of events transferred to human intervention, resulting in a lack of analysis on customer service resource allocation. Consequently, with existing technologies, intelligent agent operational data is scattered across multiple systems (e.g., user issue logs, customer service ERP systems, etc.).

[0020] Furthermore, semantic matching can be performed using models to analyze the agent's operational data. However, this approach is prone to misjudgment in vertical domains (e.g., service scenarios), potentially leading to the omission of invalid questions. Conversely, question filtering can also be performed using fixed rules (e.g., keyword filtering). However, fixed rules struggle to cover complex semantic scenarios (e.g., the implicit association between user questions and prompts), thus reducing the flexibility of the judgment.

[0021] In view of this, embodiments of the present disclosure provide a data processing method. The method, in response to receiving an analysis instruction for the operation of an agent, acquires interaction data associated with the agent, the interaction data including at least multiple questions and corresponding response information collected by the agent. Then, based on an analysis strategy, it acquires a set of analysis results regarding the multiple questions in the interaction data, the set of analysis results including at least a set of labels for the multiple questions, the set of labels indicating the quality of the agent's responses to the multiple questions. Finally, it presents visualizations of the set of analysis results.

[0022] In this way, the problems collected by the agent are automatically analyzed based on the analysis strategy, and tags indicating the quality of the reply are generated, solving the technical problems that the identification of irrelevant problems and problem classification require manual processing item by item, with low efficiency and inability to handle large-scale data. In this way, the relevance between the user's problem and the agent's function can be automatically judged, and the cost is reduced. At the same time, by presenting the visual content of the analysis results, the problem that operation data is scattered in multiple systems and needs to be manually integrated can be solved, so that the operation data of the agent can be analyzed in real-time linkage, and multi-dimensional data monitoring can be achieved.

[0023] Figure 1 FIG. shows a schematic diagram of an exemplary environment 100 in which embodiments of the present disclosure can be implemented. In this exemplary environment 100, the server 130 can establish a communication connection with one or more electronic devices (collectively or individually referred to as electronic devices 110). At least one application 120 can be installed in the electronic device 110. The user 140 can interact with the application 120 via the electronic device 110 and / or an attached device of the electronic device 110. The user 140 can be referred to as a management personnel or an operation personnel.

[0024] In some embodiments, the user 140 can input a problem or a request to the application 120 in the electronic device 110 through the interface 150. The application 120 can communicate with the server 130 to obtain relevant data or processing results, and present the response content to the user 140 through the interface 150. The exemplary environment 100 can be used to implement the dynamic problem relevance judgment of the agent platform, where the application 120 on the electronic device 110 can receive the problem input from the user 140, and combine the natural language processing model and the rule engine provided by the server 130 to perform problem relevance judgment, and finally display the judgment result and relevant operation data to the user 140 through the interface 150.

[0025] In some embodiments, the application 120 can be any suitable application that can provide question-and-answer services. In Figure 1 the environment 100, if the application 120 is in an active state, the terminal device 110 can present the page 150 of the application 120. The page 150 can include various pages that the application 120 can provide, such as information interaction pages, query pages, search pages, result presentation pages, and so on.

[0026] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0027] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0028] In this paper, an agent refers to a system capable of autonomous control based on machine learning models. An agent, for example, can make decisions and autonomously execute actions based on machine learning models to achieve preset goals or complete preset tasks. An agent can be an automated program that understands the user's intent and can utilize models or invoke tools to complete various types of tasks. In some contexts, examples of agents may include, but are not limited to, bots, chatbots, digital avatars, intelligent customer service, and digital assistants. Alternatively, an agent can also be an intelligent role implemented based on a machine learning model. An agent can process user requests based on generative models (e.g., language models, multimodal models) to perform specified types of tasks.

[0029] In some scenarios, intelligent agents can be represented as virtual avatars or physical entities to interact with users. Intelligent agents can possess intelligent dialogue and information processing capabilities, responding to user queries and providing appropriate answers. In some examples, intelligent agents can be presented in a graphical form within the interactive interface, such as as avatars, animated characters, or other visual representations.

[0030] In this paper, the intelligent agent can be deployed locally on electronic device 110 or remotely. In the case of remote deployment, electronic device 110 can directly invoke the intelligent system, or it can invoke the intelligent system via server 130.

[0031] The examples will continue to be described with reference to the accompanying drawings. In the following description, the examples will be primarily described with respect to electronic device 110. It should be understood that the actions described with respect to electronic device 110 can also be performed by an agent deployed at electronic device 110, or by an agent in collaboration with its server (e.g., server 130).

[0032] For ease of understanding, the following will refer to Figure 2 This disclosure describes the data processing scheme used in this publication. Figure 2 A schematic diagram of an example architecture 200 capable of implementing the present solution according to some embodiments of the present disclosure is shown.

[0033] like Figure 2 As shown, the example architecture 200 includes a data acquisition module 210, an analysis module 230, and an optimization module 240. The electronic device 110, through the data acquisition module 210, can acquire and analyze interaction data associated with the agent to determine a set of analysis results corresponding to this interaction data. In some examples, the interaction data includes at least multiple questions and corresponding responses collected by the agent. A set of analysis results may include a first question list, which may include questions with a first label (e.g., referred to as "invalid questions"). In some embodiments, the first label indicates that the quality of responses to multiple questions is below a threshold level. For example, a question with a first label may indicate a question that the agent cannot answer or answers inaccurately.

[0034] In some embodiments, a set of analysis results may include a second list of questions, which may include questions with a second label (e.g., referred to as "valid questions"). The second label indicates that the quality of the response is greater than a threshold level. For example, a question with a second label may indicate that the agent has answered an accurate question. A set of analysis results may include multiple question types corresponding to different questions. A set of analysis results may include the number of questions with a first label, such as the number of invalid questions.

[0035] In some embodiments, the electronic device 110 can present a visualization of the analysis results corresponding to these interactive data. For example, the electronic device 110 can present the analysis results on a first interface (e.g., an operations dashboard). This allows the user 140 (who may be referred to as a manager or operations staff) to view the analysis results on the first interface. Furthermore, the electronic device 110 can further optimize the intelligent agent based on the analysis results.

[0036] Electronic device 110 can input these invalid questions into analysis module 230. In analysis module 230, electronic device 110 can optimize the knowledge base corresponding to the agent. For example, if the knowledge base does not contain the corresponding knowledge points for a given question, then the question is invalid. In this scenario, electronic device 110 can add the corresponding knowledge points to the knowledge base, thereby optimizing the knowledge base. Furthermore, electronic device 110 can further optimize the agent based on the analysis results in optimization module 240.

[0037] Specifically, in box 211, if electronic device 110 receives an analysis instruction for the operation of an agent, it acquires interaction data associated with that agent. The interaction data includes at least multiple questions and corresponding responses collected by the agent. In some embodiments, electronic device 110 can integrate and acquire interaction data from multiple data sources. These data sources may include user question logs, which include, but are not limited to, question text and session identifiers. These data sources may include agent response records, such as response content and the type of tool used when answering questions. Alternatively / additionally, these data sources may also include, but are not limited to, organizational structure information (e.g., member identifiers, department information), events transferred to human agents (e.g., customer service identifiers), and multi-system data such as user feedback. In some embodiments, the interaction data may also include fields such as the data source (e.g., communication tools or web pages) and user identifier. These fields can be used for subsequent operational report analysis and multi-dimensional data filtering.

[0038] In some embodiments, the electronic device 110 can extract feature data from the interaction data to construct a structured dataset for subsequent analysis. For example, for a particular question, the electronic device 110 can use a model to analyze the intent of the question text, the corresponding entity (e.g., employee ID, date), and conversation context information (e.g., whether it is a series of questions), source, and other feature data.

[0039] After receiving the interactive data, the example architecture 200 can proceed to boxes 221 and 201. In box 221, the electronic device 110 performs a classification operation on each of the multiple questions in the interactive data. In some examples, the electronic device 110 may utilize an agent to analyze the type of each question. In box 222, the electronic device 110 can determine the question type to which each of the multiple questions belongs.

[0040] Accordingly, in box 201, electronic device 110 can determine whether each of these questions is relevant to the agent's capabilities. For example, for a particular question, electronic device 110 analyzes, based on interaction data, whether the task indicated by the question falls within the agent's capabilities. For ease of discussion, the following description uses one of the multiple questions as an example to illustrate how the analysis result for that question is determined. It should be understood that this is merely exemplary, and other questions in the multiple questions can be handled in the same way. In box 212, if electronic device 110 determines that the question is not relevant to the agent's capabilities, then example architecture 200 proceeds to box 212. In box 212, electronic device 110 identifies the question as an irrelevant question (an example of a question with a first label) and determines the proportion of irrelevant questions among these questions.

[0041] If electronic device 110 determines that the question is relevant to the agent's capabilities, then example architecture 200 proceeds to box 213. In box 213, electronic device 110 determines whether the agent has answered the question. If electronic device 110 determines that the agent has not answered the question, then example architecture 200 proceeds to box 214. In box 214, electronic device 110 adds the question to the list of unanswered questions (questions in the list of unanswered questions are examples of questions with a first label).

[0042] If electronic device 110 determines that the agent has answered the question, then example architecture 200 proceeds to box 215. In box 215, electronic device 110 determines whether to transfer the question to a human agent. If electronic device 110 determines that the question is transferred to a human agent, then example architecture 200 proceeds to box 216. In box 215, electronic device 110 adds the question to the human agent transfer list (the questions in the human agent transfer list are examples of questions with the first label).

[0043] If electronic device 110 determines that the question has not been transferred to a human answer, example architecture 200 proceeds to box 217. In box 217, electronic device 110 can determine whether the agent's answer to the question is accurate. If electronic device 110 determines that the agent's answer to the question is accurate, example architecture 200 proceeds to box 218. In box 218, electronic device 110 updates the answer accuracy (this question is an example of a question with a second label). If electronic device 110 determines that the agent's answer to the question is inaccurate, example architecture 200 proceeds to box 219. In box 219, electronic device 110 can add the question to the inaccurate question list (the questions in the inaccurate question list are examples of questions with a first label).

[0044] In some embodiments, at the analysis module 230, the electronic device 110 can update the knowledge base that the agent needs to access based on a question with a first label (e.g., a question that the agent cannot answer or answers inaccurately). In block 231, the electronic device 110 acquires questions with the first label obtained via the data acquisition module 210. For ease of discussion, the following description uses one of a number of questions with the first label as an example to illustrate how the knowledge base can be updated based on that question. It should be understood that this is merely exemplary, and other questions with the first label can also update the knowledge base in the following manner. In block 232, the electronic device 110 can determine whether the knowledge base contains the knowledge needed to solve the question.

[0045] If electronic device 110 determines that the knowledge base does not contain the knowledge needed to solve the problem, then example architecture 200 proceeds to box 233. In box 233, electronic device 110 adds the knowledge corresponding to the problem to the knowledge base. If electronic device 110 determines that the knowledge base contains the knowledge needed to solve the problem, then example architecture 200 proceeds to box 234. In box 234, electronic device 110 can determine whether the knowledge in the knowledge base regarding solving the problem is outdated.

[0046] If electronic device 110 determines that the knowledge for solving the problem is outdated, example architecture 200 proceeds to box 235. In box 235, electronic device 110 updates the knowledge in the knowledge base. If electronic device 110 determines that the knowledge for solving the problem is not outdated, example architecture 200 proceeds to box 236. In box 236, electronic device 110 determines whether the knowledge for solving the problem conflicts with 3 currently used to solve the problem in the knowledge base. If electronic device 110 determines that there is a conflict, example architecture 200 proceeds to box 237. In box 237, electronic device 110 resolves the conflict. For example, electronic device 110 may update the knowledge currently used to solve the problem in the knowledge base to the knowledge actually used to solve the problem.

[0047] If electronic device 110 determines that there is no conflict between the two, the example architecture 200 proceeds to block 238. In block 238, electronic device 110 treats the problem as a negative problem sample, which can be used to adjust the overall performance and problem-solving capabilities of the agent.

[0048] At the optimization module 240, the electronic device 110 can perform various detection operations on negative problem samples to resolve these negative problem samples, thereby improving the performance and problem-solving capabilities of the agent. These operations may include, for example, data switching detection 241, recall problem detection 242, sorting problem detection 243, summarizing problem detection 244, and negative problem sample resolution retest 245.

[0049] This enables a complete processing flow from data collection and problem analysis to system optimization, thereby supporting the operational optimization decisions of the intelligent agent platform.

[0050] The preceding text describes an example procedure disclosed herein for data processing. The following text details how to obtain analytical results for multiple questions.

[0051] In embodiments of this disclosure, electronic device 110 obtains a set of analysis results regarding multiple questions in interaction data based on an analysis strategy. The set of analysis results includes at least a set of labels for the multiple questions, indicating the quality of the agent's responses to the multiple questions. In some embodiments, the set of labels may include a first label and a second label. The first label indicates that the quality of the responses to the multiple questions is below a threshold level, for example, indicating that the question is invalid or the response is inaccurate. The second label indicates that the quality of the responses is above a threshold level, for example, indicating that the question is valid or the response is accurate.

[0052] In some embodiments, the analysis strategy may indicate at least one of the following: a machine learning model, preset rules (which are pre-configured), or a combination of a machine learning model and preset rules. This flexible configuration of the analysis strategy allows for adaptation to different business scenario requirements.

[0053] The following section will describe in detail the specific implementation methods for obtaining analysis results based on different analysis strategies.

[0054] In some embodiments, where the analysis strategy instructs the machine learning model, the electronic device 110 can use the machine learning model to perform semantic analysis on the problem in order to determine the relevance of the problem to the function of the agent.

[0055] In some embodiments, the machine learning model may include a Natural Language Processing (NLP) model. The electronic device 110 can train the machine learning model based on historical question samples with historical labels and corresponding prompts from the agent. In some examples, the electronic device 110 can utilize a contrastive learning framework to train the machine learning model. For example, the electronic device 110 acquires the agent's prompts and identifies them as standard samples. Correspondingly, the electronic device 110 acquires the user's question and identifies it as a contrast sample. Further, the electronic device 110 determines a loss function based on the standard and contrast samples. Then, the loss function is used to narrow the vector distance between related samples. For example, the prompt "How to check salary?" and the user question "When is this month's salary paid?" are aligned in vector space. This enhances the machine learning model's semantic understanding capability within a vertical domain.

[0056] In some embodiments, the machine learning model can be built upon a pre-trained model (such as the BERT model) and its semantic alignment capability can be optimized through contrastive learning. During training, the electronic device 110 can use historically labeled "valid / invalid" question samples, with the agent's prompt words as positive examples, to train the machine learning model. This enhances the model's semantic understanding of a specific domain.

[0057] In some embodiments, the electronic device 110 can utilize a machine learning model to obtain the label corresponding to the question, as well as the corresponding confidence level. For example, the machine learning model can output confidence levels and classification results based on the question text and prompt words. In some examples, the electronic device 110 can quantize and compress the machine learning model, thereby improving the inference speed of the machine learning model.

[0058] In some embodiments, the electronic device 110 can perform matching analysis on questions using preset rules indicated by the analysis strategy. In this scenario, the preset rules can be configured to be enabled to participate in the relevance judgment of the questions. In some embodiments, the user 140 can configure the preset rules to be enabled or disabled. When the preset rules are configured to be enabled, the electronic device 110 can generate analysis results for multiple questions based on the machine learning model and the preset rules.

[0059] In some embodiments, preset rules can be configured via a second configuration interface. In some embodiments, the electronic device 110 can receive preset rules in natural language form through the second configuration interface. For example, the electronic device 110 can provide a configuration interface for user 140 to configure preset rules. User 140 can input preset rules in this configuration interface. Natural language may include, but is not limited to, text and speech. In some examples, a rule logic parser can convert natural language descriptions into executable conditions.

[0060] In some embodiments, user 140 can configure preset rules in a visual editing manner on a second configuration interface (e.g., a rule definition interface). For example, electronic device 110 can provide user 140 with a visual editor for visual editing in the second configuration interface. For example, user 140 can create preset rules by dragging and dropping connections between components, nodes, and modules using objects (e.g., a mouse) in the second configuration interface. For example, user 140 can configure preset rules such as "(condition 1 OR condition 2) AND condition 3" in this way. In some embodiments, electronic device 110 can employ a preset engine to match questions in real time and output Boolean results. For example, a preset engine (e.g., Drools) is a tool that allows preset rules (e.g., if…then… or any other appropriate rule) to run independently and be flexibly modified.

[0061] In some embodiments, preset rules may include the scope of tasks performed by the agent. For example, user 140 may configure the tasks the agent can perform, such as processing only administrative-related queries, only guidance-related queries, or only attendance-related queries, etc. Preset rules may also include keywords corresponding to the agent. For example, user 140 may configure keywords corresponding to the agent in a second configuration interface. For instance, assuming the agent is used to handle administrative issues, user 140 may configure keywords related to administrative matters (e.g., expense reimbursement, invoices, workstations, office supplies, etc.). Additionally / alternatively, preset rules may also include character matching rules, such as matching regular expressions.

[0062] In the embodiments of this disclosure, the electronic device 110 can obtain analysis results based on a machine learning model indicated by an analysis strategy and preset rules. Thus, by employing a dual-mode judgment mechanism, dynamic fusion of semantic understanding and business rules can be achieved. This dual-mode judgment mechanism can solve the problems of poor model adaptability in vertical domains and incomplete coverage of static rules, improving the flexibility and accuracy of judgment.

[0063] In some embodiments, if the electronic device 110 determines that the confidence level output by the machine learning model is lower than a confidence threshold, it can further utilize preset rules to obtain a set of analysis results corresponding to multiple questions. For ease of understanding, the following description uses the first question among multiple questions as an example to illustrate how to obtain the analysis results corresponding to the first question. Of course, the analysis results for the other questions among the multiple questions can also be obtained in the following manner.

[0064] In some embodiments, the electronic device 110, based on a machine learning model, can obtain a first analysis result corresponding to a first question. The first analysis result includes a first label for the first question and a first confidence level corresponding to the first label. The first confidence level indicates the credibility of the first label. Further, if the electronic device 110 determines that the first confidence level is less than a confidence threshold, it can obtain a second analysis result corresponding to the first question based on preset rules.

[0065] As an example, if electronic device 110 determines that the confidence level of the machine learning model's output is lower than a confidence threshold (e.g., 70% or any other appropriate threshold), it can trigger a pre-defined rule check to obtain the analysis result corresponding to the problem. In this way, by utilizing the machine learning model and pre-defined rules, the accuracy of the judgment can be improved. This threshold-triggered mechanism allows for secondary verification using rules when the machine learning model is uncertain, thereby enhancing the overall reliability of the judgment.

[0066] In some embodiments, the electronic device 110 can also obtain a set of analysis results corresponding to multiple questions based on the weights of a machine learning model and preset rules. In some embodiments, the electronic device 110 determines a first weight corresponding to the machine learning model based on historical analysis results obtained through the machine learning model. Accordingly, the electronic device 110 can determine a second weight corresponding to the preset rules based on historical analysis results obtained through preset rules. Then, the electronic device 110 can obtain a set of analysis results using the machine learning model and preset rules according to the first weight and the second weight.

[0067] As an example, electronic device 110 can use a weighted voting method to obtain a set of analysis results. Electronic device 110 can set the weight of the machine learning model (e.g., 60%, or any other appropriate weight) based on the accuracy of historical analysis results output by the machine learning model. Electronic device 110 can set the weight of the preset rule (e.g., 40%, or any other appropriate weight) based on the accuracy of historical analysis results obtained through preset rules. Furthermore, electronic device 110 can generate a final label based on the weight of the machine learning model and the weight of the preset rule. For example, assuming the machine learning model returns a probability value of 0.85 and the rule matching result is 1 (successful match), then electronic device 110 can use a weighted mode to calculate: 0.6 × 0.85 + 0.4 × 1 = 0.91, and generate a "valid question" label based on this weighted result.

[0068] In some embodiments, the first weight and the second weight can be dynamically adjusted based on historical accuracy. For example, weight values ​​can be automatically assigned based on the accuracy performance of the machine learning model and preset rules on historical data, so that modules with higher accuracy receive greater weight.

[0069] In some embodiments, the analysis strategy can be configured via a first configuration interface. In scenarios where the analysis strategy instructs a machine learning model and preset rules, the electronic device 110 can receive first configuration information via the first configuration interface. The first configuration information indicates output rules for a third analysis result and a fourth analysis result, wherein the third and fourth analysis results are inconsistent. The third analysis result is output via a machine learning model, and the fourth analysis result is obtained via preset rules.

[0070] As an example, in the event of a conflict between the results of a machine learning model and preset rules, electronic device 110 can employ a conflict resolution strategy. For instance, user 140 can pre-configure first configuration information on a first configuration interface, such as prioritizing the analysis results output by preset rules. Alternatively / additionally, the first configuration information can indicate that the analysis results of modules with higher historical accuracy (e.g., machine learning models and / or preset rules) should be prioritized. In this way, the flexibility and interpretability of the judgment can be improved.

[0071] In this way, by adopting a dual-mode judgment mechanism, the problems of models being prone to misjudgment in vertical domains and being unable to flexibly adapt to business rules can be solved. Correspondingly, it can also solve the problems of fixed rules being unable to cover complex semantic scenarios and unable to dynamically integrate with model results, thereby improving the flexibility and accuracy of judgment.

[0072] The above describes how to obtain a set of analytical results. The following section will describe in detail how to present this set of analytical results.

[0073] In embodiments of this disclosure, electronic device 110 may present visualizations of a set of analysis results.

[0074] In some embodiments, the electronic device 110 can present visualizations of a set of analysis results on a second interface. On the second interface, the electronic device 110 can present a set of analysis results in the form of a form. As an example, the electronic device 110 can write the set of analysis results into the corresponding fields of a form (e.g., a report) (e.g., whether the question is valid, question category, or any other appropriate field) and correlate it with existing metrics (such as conversion rate to human intervention, effective response rate).

[0075] In some embodiments, the visualization content may indicate metrics such as the number of valid questions, question category distribution, source distribution, and referral rate to human intervention. In some examples, user 140 can perform operations on the corresponding fields entered in the form on the second interface, such as viewing, filtering, and editing. For example, user 140 can filter data according to dimensions such as source, category, and validity. In some examples, electronic device 110 can generate visualization charts based on this set of analysis results. Visualization charts include, but are not limited to, funnel charts (e.g., percentage of valid questions) and pie charts (e.g., source distribution).

[0076] In some embodiments, metrics such as the number of effective issues, issue classification distribution, source distribution, and rate of manual intervention can be synchronized periodically through data pipelines (e.g., T+1 day synchronization), while log details can be updated in real time so that users can make immediate decisions.

[0077] Thus, by presenting the visual content of the analysis results, the problem of scattered operation data in multiple systems can be solved, thereby enabling real-time linkage analysis of operation data and multi-dimensional data monitoring.

[0078] In some embodiments, a set of analysis results may include at least one problem with a first label, where the first label indicates that the response quality for the at least one problem is less than a threshold level. In response to determining that the first label indicates that the response quality for the at least one problem is less than the threshold level, at least one of the following may be performed based on the at least one problem: updating the prompt words of the agent or updating the preset rules indicated by the analysis strategy.

[0079] As an example, assume that user 140 filters the "invalid question" category through the second interface and discovers high-frequency invalid questions (e.g., how to apply for leave). In such a scenario, user 140 can adjust the agent prompt words or add new preset rules (e.g., add the prompt word leave). In some embodiments, the electronic device 110 can re-judge the problem according to the updated preset rules and update the analysis results presented in the second interface. Thus, continuous improvement of the agent platform can be achieved, constantly enhancing the accuracy of problem handling and the user experience.

[0080] In summary, the automated analysis of the problems collected by the agent based on the analysis strategy and the generation of labels indicating the response quality solve the technical problems of manual processing of each item for irrelevant problem identification and problem classification, low efficiency, and inability to handle large-scale data. In this way, the relevance between user problems and agent functions can be automatically judged, and the cost is reduced. At the same time, by presenting the visual content of the analysis results, the problem of scattered operation data in multiple systems that needs to be manually integrated can be solved, so that the operation data of the agent can be analyzed in real-time linkage and multi-dimensional data monitoring can be achieved.

[0081] Figure 3 FIG. 300 shows a flowchart of a method 300 for data processing according to some embodiments of the present disclosure. Method 300 can be implemented in environment 100. For example, method 300 can be implemented at server 130 or electronic device 110.

[0082] In block 310, the electronic device 110, in response to receiving an analysis instruction for the operation of the agent, obtains interaction data associated with the agent, where the interaction data at least includes a plurality of problems collected by the agent and corresponding response information.

[0083] In block 320, the electronic device 110, based on the analysis strategy, obtains a set of analysis results for the plurality of problems in the interaction data, where the set of analysis results at least includes a set of labels for the plurality of problems, and the set of labels indicates the response quality of the agent for the plurality of problems.

[0084] In box 330, electronic device 110 presents visualizations of a set of analysis results.

[0085] In some embodiments, the analysis strategy indicates at least one of the following: a machine learning model, preset rules, the preset rules being pre-configured, or a machine learning model and preset rules.

[0086] In some embodiments, the preset rule is configured to be enabled.

[0087] In some embodiments, the analysis strategy instructs a machine learning model and preset rules, and obtaining a set of analysis results includes: for a first question among multiple questions, obtaining a first analysis result corresponding to the first question based on the machine learning model, the first analysis result including a first label of the first question and a first confidence level corresponding to the first label, the first confidence level indicating the credibility of the first label; and in response to the first confidence level being less than a confidence level threshold, obtaining a second analysis result corresponding to the first question based on preset rules.

[0088] In some embodiments, the analysis strategy instructs a machine learning model and preset rules, and obtaining a set of analysis results includes: determining a first weight corresponding to the machine learning model based on historical analysis results obtained through the machine learning model; determining a second weight corresponding to the preset rules based on historical analysis results obtained through the preset rules; and obtaining a set of analysis results using the machine learning model and preset rules based on the first weight and the second weight.

[0089] In some embodiments, the analysis strategy is configured via a first configuration interface, and the method further includes: in response to the analysis strategy instructing a machine learning model and preset rules, receiving first configuration information via the first configuration interface, the first configuration information instructing output rules for a third analysis result and a fourth analysis result, the third analysis result and the fourth analysis result being inconsistent, the third analysis result being output via a machine learning model, and the fourth analysis result being obtained via preset rules.

[0090] In some embodiments, preset rules are configured via a second configuration interface; wherein the preset rules are configured in at least one of the following forms: natural language or visual editing.

[0091] In some embodiments, the preset rules include at least one of the following: the scope of the tasks performed by the agent, the keywords corresponding to the agent, or character matching rules.

[0092] In some embodiments, the machine learning model is trained based on historical question samples with historical labels and corresponding prompts for the agent.

[0093] In some embodiments, a set of analysis results includes at least one question with a first label, and the method further includes: in response to determining that the first label indicates that the quality of the response to the at least one question is less than a threshold level, performing at least one of the following based on the at least one question: updating the agent's prompt words; updating the agent's knowledge base; and updating the preset rules indicated by the analysis strategy.

[0094] In some embodiments, a set of labels includes a first label indicating that the quality of responses to multiple questions is less than a threshold level; and a set of labels includes a second label indicating that the quality of responses is greater than a threshold level.

[0095] In some embodiments, a set of analysis results may further include at least one of the following: question types corresponding to multiple questions, the number of questions with a first label among the multiple questions, a first question list, the first question list including questions with the first label, or a second question list, the second question list including questions with a second label.

[0096] Figure 4 A block diagram of an apparatus 400 for data processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as a server 130 or an electronic device 110, or may be included in a server 130 or an electronic device 110.

[0097] like Figure 4 As shown, device 400 includes an interaction data acquisition module 410, configured to acquire interaction data associated with the agent in response to receiving an analysis instruction for the agent's operation. The interaction data includes at least multiple questions and corresponding responses collected by the agent. Device 400 also includes an analysis result acquisition module 420, configured to acquire a set of analysis results regarding the multiple questions in the interaction data based on an analysis strategy. The set of analysis results includes at least a set of labels for the multiple questions, indicating the quality of the agent's responses to the multiple questions. Device 400 also includes a visualization content presentation module 430, configured to present visualization content based on the set of analysis results.

[0098] In some embodiments, the analysis strategy indicates at least one of the following: a machine learning model, preset rules, the preset rules being pre-configured, or a machine learning model and preset rules.

[0099] In some embodiments, the preset rule is configured to be enabled.

[0100] In some embodiments, the analysis strategy indicates a machine learning model and preset rules, and the analysis result acquisition module 420 is further configured to, for a first question among multiple questions, acquire a first analysis result corresponding to the first question based on the machine learning model, the first analysis result including a first label of the first question and a first confidence level corresponding to the first label, the first confidence level indicating the credibility of the first label; and in response to the first confidence level being less than a confidence level threshold, acquire a second analysis result corresponding to the first question based on preset rules.

[0101] In some embodiments, the analysis strategy indicates a machine learning model and preset rules, and the analysis result acquisition module 420 is further configured to determine a first weight corresponding to the machine learning model based on historical analysis results obtained through the machine learning model; determine a second weight corresponding to the preset rules based on historical analysis results obtained through the preset rules; and acquire a set of analysis results using the machine learning model and preset rules based on the first weight and the second weight.

[0102] In some embodiments, the analysis strategy is configured via a first configuration interface, and the device 400 further includes a configuration information receiving module configured to receive first configuration information via the first configuration interface in response to the analysis strategy instructing the machine learning model and preset rules. The first configuration information instructs output rules for the third analysis result and the fourth analysis result. The third analysis result and the fourth analysis result are inconsistent. The third analysis result is output via the machine learning model, and the fourth analysis result is obtained via the preset rules.

[0103] In some embodiments, preset rules are configured via a second configuration interface; wherein the preset rules are configured in at least one of the following forms: natural language or visual editing.

[0104] In some embodiments, the preset rules include at least one of the following: the scope of the tasks performed by the agent, the keywords corresponding to the agent, or character matching rules.

[0105] In some embodiments, the machine learning model is trained based on historical question samples with historical labels and corresponding prompts for the agent.

[0106] In some embodiments, a set of analysis results includes at least one question with a first label, and the apparatus 400 further includes an update module configured to, in response to determining that the first label indicates that the quality of the response to the at least one question is less than a threshold level, perform at least one of the following based on the at least one question: update the agent's prompt words; update the agent's knowledge base; update the preset rules indicated by the analysis strategy.

[0107] In some embodiments, a set of labels includes a first label indicating that the quality of responses to multiple questions is less than a threshold level; and a set of labels includes a second label indicating that the quality of responses is greater than a threshold level.

[0108] In some embodiments, a set of analysis results may further include at least one of the following: question types corresponding to multiple questions, the number of questions with a first label among the multiple questions, a first question list, the first question list including questions with the first label, or a second question list, the second question list including questions with a second label.

[0109] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0110] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 5 The device 500.

[0111] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose computing device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0112] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0113] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0114] The communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0115] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0116] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0117] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0119] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0121] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: In response to receiving an analysis instruction for the operation of the agent, the agent acquires interaction data associated with the agent, the interaction data including at least multiple questions and corresponding response information collected by the agent; Based on the analysis strategy, a set of analysis results is obtained regarding multiple questions in the interaction data. The set of analysis results includes at least a set of labels for the multiple questions, and the set of labels indicates the quality of the agent's response to the multiple questions. as well as Presents visualizations of the set of analysis results.

2. The method of claim 1, wherein the analysis strategy indicates at least one of the following: Machine learning models Preset rules, which are pre-configured, or The machine learning model and the preset rules.

3. The method according to claim 2, wherein the preset rule is configured to be enabled.

4. The method according to claim 2, wherein the analysis strategy instructs the machine learning model and the preset rules, and obtaining the set of analysis results includes: Regarding the first of the aforementioned problems, Based on the machine learning model, a first analysis result corresponding to the first question is obtained. The first analysis result includes a first label of the first question and a first confidence level corresponding to the first label. The first confidence level indicates the credibility of the first label. as well as In response to the first confidence level being less than the confidence threshold, a second analysis result corresponding to the first question is obtained based on the preset rule.

5. The method according to claim 2, wherein the analysis strategy instructs the machine learning model and the preset rules, and obtaining the set of analysis results includes: Based on the historical analysis results obtained through the machine learning model, the first weight corresponding to the machine learning model is determined; Based on the historical analysis results obtained through the preset rules, the second weight corresponding to the preset rules is determined; as well as Based on the first weight and the second weight, the set of analysis results are obtained using the machine learning model and the preset rules.

6. The method of claim 2, wherein the analysis strategy is configured via a first configuration interface, and the method further comprises: In response to the analysis strategy instructing the machine learning model and the preset rules, The system receives first configuration information via the first configuration interface. The first configuration information indicates the output rules for the third analysis result and the fourth analysis result. The third analysis result and the fourth analysis result are inconsistent. The third analysis result is output by the machine learning model, and the fourth analysis result is obtained by the preset rules.

7. The method according to claim 2, wherein the preset rule is configured via a second configuration interface; in, The preset rules are configured in at least one of the following forms: natural language or visual editing.

8. The method according to claim 7, wherein the preset rule includes at least one of the following: The scope of tasks performed by the intelligent agent. The keywords corresponding to the intelligent agent, or Character matching rules.

9. The method according to claim 2, wherein the machine learning model is trained based on historical question samples with historical labels and prompt words corresponding to the agent.

10. The method of claim 1, wherein the set of analysis results includes at least one question having a first label, and the method further includes: In response to determining that the first label indicates that the quality of the response to the at least one question is less than a threshold level, perform at least one of the following based on the at least one question: Update the prompt words of the agent; Update the knowledge base of the intelligent agent; Update the preset rules indicated by the analysis strategy.

11. The method of claim 1, wherein the set of labels includes a first label indicating that the quality of responses to the plurality of questions is less than a threshold level; Furthermore, the set of labels includes a second label that indicates the quality of the response is greater than a threshold level.

12. The method of claim 11, wherein the set of analytical results further comprises at least one of the following: The question types corresponding to the multiple questions The number of questions that have the first label among the multiple questions. The first question list includes questions that have the first tag, or The second question list includes questions that have the second tag.

13. A data processing apparatus, comprising: An interaction data acquisition module is configured to acquire interaction data associated with the agent in response to receiving an analysis instruction for the operation of the agent, the interaction data including at least multiple questions and corresponding response information collected by the agent; The analysis result acquisition module is configured to acquire a set of analysis results about multiple questions in the interaction data based on an analysis strategy. The set of analysis results includes at least a set of labels for the multiple questions, and the set of labels indicates the quality of the agent's response to the multiple questions. as well as The visualization content presentation module is configured to present visualization content for the set of analysis results.

14. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processor.

15. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 12.

Images