Answer monitoring method and device

The answer monitoring method and device address the issue of incorrect responses by verifying answer templates and data integrity, ensuring accurate and user-friendly interactions.

CN113742462BActive Publication Date: 2024-12-06BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010698895.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-20
Publication Date
2024-12-06
Estimated Expiration
2040-07-20

AI Technical Summary

Technical Problem

Existing systems lack real-time monitoring and verification of answer templates, leading to incorrect or garbled responses due to unverified data, which affects user experience.

Method used

Implement a method and device for answer monitoring that includes template determination, real-time data verification, and fallback mechanisms to ensure correct and intelligible responses by checking data type, protocol format, and content completeness before rendering.

Benefits of technology

Ensures accurate and user-friendly answers by filtering out incorrect or incomplete responses, providing real-time preview and fallback options to enhance user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113742462B_ABST
    Figure CN113742462B_ABST
Patent Text Reader

Abstract

The present invention discloses an answer monitoring method and device, and relates to the field of computer technology. A specific implementation of the method includes: determining an answer template corresponding to a consulting question, and based on the component corresponding to the answer template, calling an external interface to obtain real-time data corresponding to the question; when the real-time data passes the protocol format verification, replacing the variables in the answer template with the real-time data to obtain a rendered answer; determining the data type of the rendered answer, and checking the rendered answer based on the verification method corresponding to the data type to perform an answer return operation corresponding to the verification result. This implementation checks the correctness of the real-time data before answer assembly and rendering, and checks the correctness of the answer before pushing the answer to the user after assembly, so as to replace it when an abnormality is found, solving the problem that the existing abnormal answers cannot be filtered.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an answer monitoring method and device. Background Art

[0002] The answer is displayed in text form through the operation platform configuration and saved in the knowledge base. When the dialogue response is triggered, the intelligent response system renders the answer and displays it to the user. In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:

[0003] 1. Operation personnel cannot perceive the correctness and display effect of the template. They can only know whether the template is usable and whether the rendering effect meets expectations after selecting the template and filling in and replacing the variables in the template with real data in the response process. There is no intuitive and real-time response effect presentation in the whole process.

[0004] 2. Currently, when the answer is rendered, it will be pushed directly to the gateway, and then pushed to the user by the gateway. However, if the data acquisition fails or the data is abnormal, it will not be discovered and intercepted in real time, and only garbled and wrong answers will be pushed to the user, affecting the user experience. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides an answer monitoring method and device, which can at least solve the problem of unverified answers and answer templates in the prior art.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for monitoring an answer is provided, comprising:

[0007] Determine an answer template corresponding to the consulting question, and based on the component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0008] When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer;

[0009] The data type of the rendered answer is determined, and the rendered answer is verified based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

[0010] Optionally, before determining the answer template corresponding to the consultation question, the method further includes:

[0011] In response to a selection operation of an answer template, obtaining simulation data through a component corresponding to the answer template to verify whether the data type of the simulation data is consistent with the predetermined data type of the answer template; wherein the simulation data includes input parameters and output parameters;

[0012] If they are consistent, the variables in the answer template are filled and replaced with the simulated data to obtain a rendered template answer;

[0013] The template answer is parsed into data in a predetermined format and rendered on a page for preview verification, so as to confirm that the answer template verification has passed after receiving a preview confirmation message.

[0014] Optionally, determining an answer template corresponding to the consultation question includes:

[0015] Identify the intent category to which the consulting question belongs by an intent classification method, and determine a rule tree corresponding to the intent classification; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set;

[0016] Starting from the root node of the rule tree, using the component set component to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions;

[0017] If it is consistent, the answer to be rendered corresponding to the root node is determined, and the answer template of the answer to be rendered is obtained. Otherwise, the above judgment steps are repeated until all nodes are traversed.

[0018] Optionally, the method further includes:

[0019] When the real-time data fails to pass the protocol format verification, or multiple return results in the components corresponding to the answer template are empty, a fallback answer corresponding to the intent classification is obtained and displayed.

[0020] Optionally, the data type of the rendered answer is the same as the data type of the real-time data;

[0021] In the case where the real-time data passes the protocol format verification, before replacing the variables in the answer template with the real-time data, it also includes: if the data field of the real-time data is text, determining whether the text field of the real-time data is empty, and if not empty, determining that the real-time data passes the protocol format verification;

[0022] The rendering answer is checked based on a verification method corresponding to the data type to execute an answer return operation corresponding to the verification result, including: detecting abnormal characters on the rendering answer, and returning the rendering answer if the detection result is that the abnormal characters do not exist, otherwise replacing the rendering answer with a predetermined fallback answer and returning it.

[0023] Optionally, the data type of the rendered answer is the same as the data type of the real-time data;

[0024] In the case where the real-time data passes the protocol format verification, before replacing the variables in the answer template with the real-time data, the method further includes: if the data field of the real-time data is non-text, determining whether the data field of the real-time data can be parsed into a predetermined format, and if so, determining that the real-time data passes the protocol format verification;

[0025] The rendering answer is checked based on the verification method corresponding to the data type to execute the answer return operation corresponding to the verification result, including: performing a protocol check and a content incompleteness check on the rendering answer, and returning the rendering answer if the check result is passed, otherwise replacing the rendering answer with a predetermined fallback answer and returning it.

[0026] Optionally, the method further includes:

[0027] If there is only one component that returns an empty result, a similar answer to the rendered answer is calculated based on a similarity model, and the rendered answer is replaced with the similar answer and returned.

[0028] To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided an answer monitoring device, comprising:

[0029] A template determination module is used to determine an answer template corresponding to the consulting question, and based on the component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0030] A data verification module, used for replacing the variables in the answer template with the real-time data to obtain a rendered answer when the real-time data passes the protocol format verification;

[0031] The answer verification module is used to determine the data type of the rendered answer, and to verify the rendered answer based on a verification method corresponding to the data type, so as to perform an answer return operation corresponding to the verification result.

[0032] Optionally, a template verification module is also included, which is used to:

[0033] In response to a selection operation of an answer template, obtaining simulation data through a component corresponding to the answer template to verify whether the data type of the simulation data is consistent with the predetermined data type of the answer template; wherein the simulation data includes input parameters and output parameters;

[0034] If they are consistent, the variables in the answer template are filled and replaced with the simulated data to obtain a rendered template answer;

[0035] The template answer is parsed into data in a predetermined format and rendered on a page for preview verification, so as to confirm that the answer template verification has passed after receiving a preview confirmation message.

[0036] Optionally, the template determination module is used to:

[0037] Identify the intent category to which the consulting question belongs by an intent classification method, and determine a rule tree corresponding to the intent classification; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set;

[0038] Starting from the root node of the rule tree, using the component set component to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions;

[0039] If it is consistent, the answer to be rendered corresponding to the root node is determined, and the answer template of the answer to be rendered is obtained. Otherwise, the above judgment steps are repeated until all nodes are traversed.

[0040] Optionally, the device further includes an abnormality protection module, which is used to:

[0041] When the real-time data fails to pass the protocol format verification, or multiple return results in the components corresponding to the answer template are empty, a fallback answer corresponding to the intent classification is obtained and displayed.

[0042] Optionally, the data type of the rendered answer is the same as the data type of the real-time data;

[0043] The data verification module is used to: if the data field of the real-time data is text, determine whether the text field of the real-time data is empty, and if it is not empty, determine that the real-time data passes the protocol format verification;

[0044] The answer verification module is used to: perform abnormal character detection on the rendered answer, and return the rendered answer if the detection result is that the rendered answer does not exist; otherwise, replace the rendered answer with a predetermined fallback answer and return it.

[0045] Optionally, the data type of the rendered answer is the same as the data type of the real-time data;

[0046] The data verification module is used to: if the data field of the real-time data is non-text, determine whether the data field of the real-time data can be parsed into a predetermined format, and if so, determine that the real-time data passes the protocol format verification;

[0047] The answer verification module is used to perform a protocol check and a content incomplete check on the rendered answer, and return the rendered answer if the check result is passed; otherwise, the rendered answer is replaced with a predetermined fallback answer and returned.

[0048] Optionally, the device further comprises a similar answer module, which is used to: if there is only one component that returns an empty result, calculate a similar answer to the rendered answer based on a similarity model, replace the rendered answer with the similar answer and return it.

[0049] To achieve the above objective, according to another aspect of an embodiment of the present invention, an answer monitoring electronic device is provided.

[0050] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned answer monitoring methods.

[0051] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned answer monitoring methods is implemented.

[0052] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: when configuring the answer template, the operator provides a real-time preview function to ensure the display effect and correctness of the answer template, and monitors the answer before the answer is pushed, so that when abnormal answers are found, they are filtered and replaced based on the bottom-up answer, avoiding the bad experience of empty answers, garbled codes, high-risk answers, etc.

[0053] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0055] Figure 1 is a main flow chart of an answer monitoring method according to an embodiment of the present invention;

[0056] Figure 2 is a flow chart of an optional answer monitoring method according to an embodiment of the present invention;

[0057] Figure 3 is a flow chart of another optional answer monitoring method according to an embodiment of the present invention;

[0058] Figure 4 is a flow chart of another optional answer monitoring method according to an embodiment of the present invention;

[0059] Figure 5 is a flow chart of another optional answer monitoring method according to an embodiment of the present invention;

[0060] Figure 6 It is a specific implementation diagram of the monitoring answer template;

[0061] Figure 7 is a schematic diagram of main modules of an answer monitoring device according to an embodiment of the present invention;

[0062] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0063] Fig. 9 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0065] See also Figure 1 , which shows a main flow chart of an answer monitoring method provided by an embodiment of the present invention, comprising the following steps:

[0066] S101: Determine an answer template corresponding to a consulting question, and based on a component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0067] S102: When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer;

[0068] S103: Determine the data type of the rendered answer, and verify the rendered answer based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

[0069] In the above implementation, for step S101, the intelligent dialogue system pre-establishes the corresponding relationship between the answer template and the component. After determining the answer template for processing the consultation question, based on the component corresponding to the answer template (encapsulation of the external interface (microservice)), the real-time data is obtained by calling the external interface. For example, if the consultation question is where is my express delivery, the real-time data obtained needs to call the logistics interface to obtain the corresponding logistics information.

[0070] It should be noted that different answer templates use different components to obtain data, and the same component will obtain different data results for different input parameters, such as Order 1 and Order 2 have their own details. Ultimately, different answer templates render different answers, and the same answer template can also render different answers for different user questions.

[0071] For step S102, after acquiring the real-time data and before integrating the real-time data with the answer template, this solution will verify the protocol format of the real-time data. For details, see the following Figure 3 The description shown is not described in detail here.

[0072] There are variable placeholders in the answer template, and the data source is the component. Only when the real-time data protocol format verification is passed, the real-time data obtained by the component will be parsed to replace the corresponding variables in the answer template, and a system of protocol fields will be supplemented, completing a simple rendering and obtaining the rendered answer.

[0073] For step S103, after the preliminary protocol format verification is passed, further verification will be performed. Before the answer is pushed to the user, parsing verification will be performed based on the data type of the answer; wherein the data type of the answer is usually the same as the data type of the real-time data.

[0074] For example, we detect abnormal characters in text answers, such as jimiResult (a field in the answer, which will appear in the text answer only when an abnormality occurs), garbled characters such as \&, and some high-risk sensitive words. If an abnormality is detected, we replace the text answer with a fixed phrase, such as "JIMI is still learning, please try to describe your problem in other ways?"

[0075] However, for non-text answers, we will perform protocol checks. For example, for graphic templates, we will check whether the corresponding position in the template (picture address field) contains a picture link and whether text content appears. For logistics templates, we will check whether there is a corresponding logistics number and logistics information. In addition, we can also check whether the content is incomplete so as to replace the fixed words.

[0076] The method provided in the above embodiment is to solve the problem that abnormal answers cannot be filtered out. This solution checks the correctness of real-time data before answer assembly and rendering, and checks the correctness of answers before pushing them to users after assembly, so as to replace them when abnormalities are found.

[0077] See also Figure 2 , shows a flow chart of an optional answer monitoring method according to an embodiment of the present invention, comprising the following steps:

[0078] S201: Identify the intent category to which the consulting question belongs by means of intent classification, and determine a rule tree corresponding to the intent category; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set;

[0079] S202: Starting from the root node of the rule tree, using the component set component to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions;

[0080] S203: If it is consistent, determine the answer to be rendered corresponding to the root node, and obtain the answer template of the answer to be rendered, otherwise repeat the above judgment steps until all nodes are traversed;

[0081] S204: Based on the component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0082] S205: When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer;

[0083] S206: Determine the data type of the rendered answer, and verify the rendered answer based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

[0084] In the above embodiment, for steps S204 to S206, see Figure 1 The description of steps S101 to S103 is not repeated here.

[0085] In the above implementation, for steps S201 to S203, different users ask different questions, so multiple intent classifications are pre-set to analyze the intent of the user after receiving the question, such as modifying an order, logistics inquiry, etc., using NLU (Natural Language Understanding).

[0086] Furthermore, before the user asks a question (including the next question), question prediction can be performed based on user information. This prediction method is an existing means and will not be described in detail here.

[0087] In this solution, a corresponding rule tree is set for each intent classification. Each rule tree starts from a root node and is set with at least one node, and corresponding components and predetermined conditions are pre-set for each node.

[0088] In actual operation, the user's current inquiry question may cover less information, so it is necessary to obtain input parameters in combination with its context. Then, starting from the root node, use the corresponding components to obtain input parameters from the inquiry question and context, and judge whether the input parameters meet the predetermined conditions of the root node. For example, the root node is an order list, and the predetermined condition is list format. If the input parameters do not meet the predetermined conditions of the root node, continue to judge from the child nodes of the root node.

[0089] The method provided in the above embodiment accurately identifies the intention of the user's consultation question based on intent classification, so that the corresponding answer template can be accurately obtained based on the intention, thereby improving the integration of real-time data and answer templates and enhancing the accuracy of the answers provided to users.

[0090] See also Figure 3 , shows a schematic flow chart of another optional answer monitoring method according to an embodiment of the present invention, comprising the following steps:

[0091] S301: Determine an answer template corresponding to the consulting question, and based on the component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0092] S302: If the data field of the real-time data is text, determine whether the text field of the real-time data is empty; if not, determine that the real-time data passes protocol format verification;

[0093] S303: If the data field of the real-time data is non-text, determining whether the data field of the real-time data can be parsed into a predetermined format, and if so, determining that the real-time data passes the protocol format verification;

[0094] S304: When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer; wherein the data type of the rendered answer is the same as the data type of the real-time data;

[0095] S305: Detect abnormal characters on the rendered answer. If the detection result is that the abnormal characters do not exist, return the rendered answer; otherwise, replace the rendered answer with a predetermined fallback answer and return the answer;

[0096] S306: Perform a protocol check and a content incomplete check on the rendering answer. If the check result is passed, the rendering answer is returned; otherwise, the rendering answer is replaced with a predetermined fallback answer and returned.

[0097] In the above embodiment, for steps S301 and S304, see Figure 1 The description of steps S101 and S102 is not repeated here.

[0098] In the above implementation, for steps S302 and S305, it is determined whether the type field of the real-time data is text. If so, it is determined that the real-time data is text data, and it is verified whether its content field is not empty. If it is not empty, it means that the real-time data contains real data and passes the protocol format verification.

[0099] However, if the verification fails, a fallback text response related to the user intent classification will be directly returned. Intent classification means that after receiving the user's question, the dialogue system will identify the user intent classification through NLU, and a fallback response will be pre-configured for each intent classification, and pre-loaded after the intent classification is identified.

[0100] For text-based real-time data, after it is merged with the answer template, the type of the rendered answer is also text-based. Before the rendered answer is pushed to the gateway (or user), it is checked for abnormal characters, such as jimiResult, \& and other garbled characters, as well as some high-risk sensitive words. If an abnormality is detected, a fixed-word replacement is performed for this text answer, and the fallback text corresponds to the intent classification.

[0101] For steps S303 and S306, it is determined whether the type field of the real-time data is template. If so, the real-time data is determined to be non-text data, and its data field is verified to be parsable as json, etc. If it can be parsed as json, it is determined that the real-time data passes the protocol format verification. However, if the verification fails, a fallback text response related to the user intent classification is directly returned.

[0102] For non-text answers, we will perform protocol checks and content incompleteness checks (including checks on some data types). For example, for graphic templates, we will verify whether the image address field is filled with the image type. For example, for logistics templates, we will check whether there is a corresponding logistics number and logistics information. If the check is abnormal, we will also perform a fixed replacement.

[0103] The method provided in the above embodiment checks the real-time data and rendered answers of text and non-text types, and pushes the fallback answer to the user when an abnormality is detected, so as to avoid the bad experience of the user caused by empty answers, garbled characters, high-risk answers, etc.

[0104] See also Figure 4 , shows a schematic flow chart of another optional answer monitoring method according to an embodiment of the present invention, comprising the following steps:

[0105] S401: Identify the intent category to which the consulting question belongs by an intent classification method, and determine a rule tree corresponding to the intent classification; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set;

[0106] S402: Starting from the root node of the rule tree, using the component set component to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions;

[0107] S403: If it is in compliance, then determine the answer to be rendered corresponding to the root node, and obtain the answer template of the answer to be rendered; otherwise, repeat the above determination steps until all nodes are traversed;

[0108] S404: Based on the components corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question, and record the number of components whose return results are empty;

[0109] S405: Verify the protocol format of the real-time data;

[0110] S406: When the real-time data fails to pass the protocol format verification or multiple return results in the components corresponding to the answer template are empty, obtaining and displaying a fallback answer corresponding to the intent classification;

[0111] S407: When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer;

[0112] S408: Determine the data type of the rendered answer, and verify the rendered answer based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result;

[0113] S409: If there is only one component that returns an empty result, a similar answer to the rendered answer is calculated based on a similarity model, and the rendered answer is replaced with the similar answer and returned.

[0114] In the above embodiment, for steps S401 to S403, see Figure 2 For the description of steps S201 to S203, see step S405. Figure 3 For the description shown, steps S407 and S408 can be found in Figure 1 The description of steps S102 and S103 is not repeated here.

[0115] In the above implementation, for steps S406 and S409, one answer template may correspond to multiple components, and some components may have empty results when calling external interfaces.

[0116] 1) If two or more components return empty results, the answer is considered unavailable. In this case, a business fallback response will be made directly, and a fallback answer corresponding to the intent classification will be returned. For non-text answers, if the field to be verified does not return data, the answer is also considered invalid or empty.

[0117] 2) If only a single component returns an empty result (for text-based answers), the similarity model is used to calculate the answer that is similar to the rendered answer, the original rendered answer is discarded, and replaced with the calculated similar answer.

[0118] Different answer templates have different numbers and template structures. The field names and data types are determined when the template is defined. The data value in the rendered data structure is parsed into the same JSON data body, and the legitimacy of its required fields is verified. When configuring non-text templates, operators will simultaneously configure a text response as a fallback response when the rendered answer is abnormal. For text templates, a fallback response will be configured based on the business (i.e., intent classification). There is also a case where the fallback answer is calculated based on text similarity.

[0119] The method provided in the above embodiment considers the abnormal situations of components, real-time data and rendered answers respectively, so as to provide corresponding answers for different abnormalities, and avoid the bad experience brought to users by empty answers, garbled codes, high-risk answers, etc.

[0120] See also Figure 5 , shows a schematic flow chart of an optional answer monitoring method according to an embodiment of the present invention, comprising the following steps:

[0121] S501: In response to a selection operation of an answer template, obtaining simulation data through a component corresponding to the answer template to verify whether a data type of the simulation data is consistent with a predetermined data type of the answer template; wherein the simulation data includes input parameters and output parameters;

[0122] S502: If they are consistent, the variables in the answer template are filled and replaced with the simulation data to obtain a rendered template answer;

[0123] S503: Parse the template answer into data in a predetermined format and render it on a page for preview verification, so as to confirm that the answer template verification has passed after receiving a preview confirmation message.

[0124] In the above implementation, for step S501, in addition to the verification before pushing the answer to the user, the operator can also monitor the correctness and display effect from the source when configuring the answer template (ie, the template for displaying the answer).

[0125] R&D personnel write answer templates (variables need to be filled in) in the background and save them to the knowledge base, such as graphic templates, video templates, logistics information templates, etc. Operation personnel select the required template from the template list and obtain the corresponding simulation data based on the template. It should be noted that simulation data is currently mainly generated by manual simulation when defining the template.

[0126] The role of the component is to obtain data, which is essentially the encapsulation of a series of microservices, such as obtaining order details through order numbers, querying logistics tracks through logistics order numbers, etc. The encapsulated components are exposed to the answer template for the answer template to obtain data and complete the answer assembly.

[0127] After obtaining the simulated data, the template verification phase will begin. The operation platform will merge the template with the simulated data to verify whether the required fields are missing and whether the data format meets the predetermined requirements. Different templates verify different fields and may require different data types. For example, the data format for the graphic template verification is a picture. The verification here is mainly to ensure that the data format meets the template requirements and to prepare for the next step of parsing the merged template.

[0128] For step S502, if the verification result is consistent, a function calculation (such as Velocity template rendering) is called to merge the simulation data and the answer template to fill the simulation data into the correct position in the template. However, if the verification result is inconsistent, the R&D personnel need to be prompted to make repair adjustments.

[0129] For step S503, after the template and simulation data are integrated, the real-time display module is called to parse the integrated template answer (including the answer template + simulation data) into JSON format data, which is then rendered on the page for display so that the operator can verify the display effect and correctness. Similarly, when the analysis fails, the R&D personnel will be prompted to make repairs and adjustments.

[0130] Take the container template (i.e. the container of templates) as an example, which carries a combination of multiple templates. Taking the Guess What You Want template and the Quick Card template as examples, the templates defined by the R&D staff are as follows:

[0131]

[0132]

[0133] Among them, "problem-predProblem.data" is the data required for front-end display. The actual production environment is mainly provided by components. When verifying the template effect, the operator mainly verifies this part of the simulated data.

[0134] See also Figure 6As shown in the figure, the first box is the specific question predicted by the user after question prediction, and the specific format is List. The second box is the obtained quick card. In order to reduce the user's effort, the maximum question category that the user may ask can be calculated based on the user portrait. The container template combines these two answer templates into one.

[0135] After R&D saves this template to the knowledge base, operations personnel can see the options of this template on the operations platform. After simulating several specific questions that users want to ask and their desired intentions, they can preview the template effects in real time.

[0136] The method provided in the above embodiment addresses the shortcomings and deficiencies of the prior art. Before the answer template goes online, the correctness of the answer template is verified by means of simulated data, and after filling in the simulated data, the template preview effect is displayed on the operation platform based on a real-time preview method, which facilitates the verification of the display effect.

[0137] The method provided in the embodiment of the present invention verifies the answer template before it goes online, and checks the answer before pushing it to the user. If an abnormal answer is found (including component returns empty, component returns garbled characters, internal exceptions when rendering the answer), an early warning will be issued, and a correction will be made according to different business / intent classifications. If the correction fails, a fallback response will be returned.

[0138] See also Figure 7 , which shows a schematic diagram of main modules of an answer monitoring device 700 provided in an embodiment of the present invention, including:

[0139] The template determination module 701 is used to determine the answer template corresponding to the consulting question, and based on the component corresponding to the answer template, call the external interface to obtain the real-time data corresponding to the question;

[0140] A data verification module 702 is used to replace the variables in the answer template with the real-time data to obtain a rendered answer when the real-time data passes the protocol format verification;

[0141] The answer verification module 703 is used to determine the data type of the rendered answer, and verify the rendered answer based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

[0142] The implementation device of the present invention further includes a template verification module 704 (not shown in the figure), which is used to:

[0143] In response to a selection operation of an answer template, obtaining simulation data through a component corresponding to the answer template to verify whether the data type of the simulation data is consistent with the predetermined data type of the answer template; wherein the simulation data includes input parameters and output parameters;

[0144] If they are consistent, the variables in the answer template are filled and replaced with the simulated data to obtain a rendered template answer;

[0145] The template answer is parsed into data in a predetermined format and rendered on a page for preview verification, so as to confirm that the answer template verification has passed after receiving a preview confirmation message.

[0146] In the implementation device of the present invention, the template determination module 701 is used to:

[0147] Identify the intent category to which the consulting question belongs by an intent classification method, and determine a rule tree corresponding to the intent classification; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set;

[0148] Starting from the root node of the rule tree, using the component set component to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions;

[0149] If it is consistent, the answer to be rendered corresponding to the root node is determined, and the answer template of the answer to be rendered is obtained. Otherwise, the above judgment steps are repeated until all nodes are traversed.

[0150] The implementation device of the present invention further includes an abnormality protection module 705 (not shown in the figure), which is used to:

[0151] When the real-time data fails to pass the protocol format verification, or multiple return results in the components corresponding to the answer template are empty, a fallback answer corresponding to the intent classification is obtained and displayed.

[0152] In the implementation device of the present invention, the data type of the rendered answer is the same as the data type of the real-time data;

[0153] The data verification module 702 is used to: if the data field of the real-time data is text, determine whether the text field of the real-time data is empty, and if it is not empty, determine that the real-time data passes the protocol format verification;

[0154] The answer verification module 703 is used to: perform abnormal character detection on the rendered answer, and return the rendered answer if the detection result is that the rendered answer does not exist; otherwise, replace the rendered answer with a predetermined fallback answer and return it.

[0155] In the implementation device of the present invention, the data type of the rendered answer is the same as the data type of the real-time data;

[0156] The data verification module 702 is used to: if the data field of the real-time data is non-text, determine whether the data field of the real-time data can be parsed into a predetermined format, and if so, determine that the real-time data passes the protocol format verification;

[0157] The answer verification module 703 is used to perform a protocol check and a content incomplete check on the rendering answer, and return the rendering answer if the check result is passed; otherwise, replace the rendering answer with a predetermined fallback answer and return it.

[0158] In the implementation device of the present invention, the device further includes a similar answer module 706 (not shown in the figure), which is used to:

[0159] If there is only one component that returns an empty result, a similar answer to the rendered answer is calculated based on a similarity model, and the rendered answer is replaced with the similar answer and returned.

[0160] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0161] Figure 8 An exemplary system architecture 800 is shown in which embodiments of the present invention may be applied.

[0162] like Figure 8 As shown, the system architecture 800 may include terminal devices 801, 802, 803, a network 804 and a server 805 (just an example). The network 804 is used to provide a medium for communication links between the terminal devices 801, 802, 803 and the server 805. The network 804 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0163] The user can use the terminal devices 801, 802, 803 to interact with the server 805 through the network 804 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 801, 802, 803.

[0164] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0165] The server 805 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 801 , 802 , and 803 (only as an example).

[0166] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the device is generally set in the server 805 .

[0167] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0168] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0169] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0170] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0171] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0172] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a template determination module, a data verification module, and an answer verification module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the answer verification module may also be described as a "verification answer module".

[0175] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0176] Determine an answer template corresponding to the consulting question, and based on the component corresponding to the answer template, call an external interface to obtain real-time data corresponding to the question;

[0177] When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer;

[0178] The data type of the rendered answer is determined, and the rendered answer is verified based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

[0179] According to the technical solution of the embodiment of the present invention, when configuring the answer template, the operator provides a real-time preview function to ensure the display effect and correctness of the answer template, and monitors the answer before the answer is pushed, so that when abnormal answers are found, they can be filtered and replaced based on the backup answer to avoid the bad experience of empty answers, garbled codes, high-risk answers, etc.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for monitoring an answer, characterized in that: include: Determining an answer template corresponding to a consulting question, comprising: identifying the intent classification to which the consulting question belongs by an intent classification method, and determining a rule tree corresponding to the intent classification; wherein the rule tree comprises at least one node, and each node corresponds to a predetermined condition and a component set; starting from the root node of the rule tree, using the components in the component set to obtain input parameters from the consulting question and context, and determining whether the input parameters meet the predetermined conditions; if so, determining the answer to be rendered corresponding to the root node, and obtaining the answer template of the answer to be rendered, otherwise repeating the above judgment steps until all nodes are traversed; Based on the component corresponding to the answer template, calling an external interface to obtain real-time data corresponding to the question; When the real-time data passes the protocol format verification, the variables in the answer template are replaced with the real-time data to obtain a rendered answer; The data type of the rendered answer is determined, and the rendered answer is verified based on a verification method corresponding to the data type to perform an answer return operation corresponding to the verification result.

2. The method according to claim 1, characterized in that Before determining the answer template corresponding to the consultation question, the method further includes: In response to a selection operation of an answer template, obtaining simulation data through a component corresponding to the answer template to verify whether the data type of the simulation data is consistent with the predetermined data type of the answer template; wherein the simulation data includes input parameters and output parameters; If they are consistent, the variables in the answer template are filled and replaced with the simulated data to obtain a rendered template answer; The template answer is parsed into data in a predetermined format and rendered on a page for preview verification, so as to confirm that the answer template verification has passed after receiving a preview confirmation message.

3. The method according to claim 1, characterized in that: The method further comprises: When the real-time data fails to pass the protocol format verification, or multiple return results in the components corresponding to the answer template are empty, a fallback answer corresponding to the intent classification is obtained and displayed.

4. The method according to claim 1, characterized in that: The data type of the rendered answer is the same as the data type of the real-time data; In the case where the real-time data passes the protocol format verification, before replacing the variables in the answer template with the real-time data, it also includes: if the data field of the real-time data is text, determining whether the text field of the real-time data is empty, and if not empty, determining that the real-time data passes the protocol format verification; The rendering answer is checked based on a verification method corresponding to the data type to execute an answer return operation corresponding to the verification result, including: detecting abnormal characters on the rendering answer, returning the rendering answer if the detection result is that the abnormal characters do not exist, otherwise replacing the rendering answer with a predetermined fallback answer and returning it.

5. The method according to claim 1, characterized in that The data type of the rendered answer is the same as the data type of the real-time data; In the case where the real-time data passes the protocol format verification, before replacing the variables in the answer template with the real-time data, the method further includes: if the data field of the real-time data is non-text, determining whether the data field of the real-time data can be parsed into a predetermined format, and if so, determining that the real-time data passes the protocol format verification; The rendering answer is checked based on the verification method corresponding to the data type to execute the answer return operation corresponding to the verification result, including: performing a protocol check and a content incompleteness check on the rendering answer, and returning the rendering answer if the check result is passed, otherwise replacing the rendering answer with a predetermined fallback answer and returning it.

6. The method according to claim 1, characterized in that The method further comprises: If there is only one component that returns an empty result, a similar answer to the rendered answer is calculated based on a similarity model, and the rendered answer is replaced with the similar answer and returned.

7. An answer monitoring device, characterized in that: include: A template determination module is used to determine an answer template corresponding to a consulting question, including: identifying the intent classification to which the consulting question belongs by an intent classification method, and determining a rule tree corresponding to the intent classification; wherein the rule tree includes at least one node, and each node corresponds to a predetermined condition and a component set; starting from the root node of the rule tree, using the components in the component set to obtain input parameters from the consulting question and context, and judging whether the input parameters meet the predetermined conditions; if so, determining the answer to be rendered corresponding to the root node, and obtaining the answer template of the answer to be rendered, otherwise repeating the above judgment steps until all nodes are traversed; Based on the component corresponding to the answer template, calling an external interface to obtain real-time data corresponding to the question; A data verification module, used for replacing the variables in the answer template with the real-time data to obtain a rendered answer when the real-time data passes the protocol format verification; The answer verification module is used to determine the data type of the rendered answer, and to verify the rendered answer based on a verification method corresponding to the data type, so as to perform an answer return operation corresponding to the verification result.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.