Intelligent interaction method and system based on end-to-end multi-modal instruction understanding

Abstract

This application provides an intelligent interaction method and system based on end-to-end multimodal instruction understanding, relating to the field of terminal technology. The method includes: acquiring input information from a peer device; analyzing the input information and pre-set dialogue goals using an intelligent agent to obtain response content for the input information; the intelligent agent is trained based on Reinforcement Learning with Human Feedback (RLHF), where a reward model in RLHF is used to evaluate the response content output by the intelligent agent in terms of dialogue quality and task completion; and sending the response content to the peer device. In this way, the intelligent agent can generate response content based on pre-set dialogue goals. During the dialogue, relevant element information about the dialogue goals can be obtained, resulting in highly logical and high-quality responses.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an intelligent interaction method and system based on end-to-end multimodal instruction understanding. Background Technology

[0002] Mobile phones, tablets, and other electronic devices can communicate with callers through intelligent agents (e.g., call assistants, voice assistants) in scenarios where it is inconvenient to answer calls (e.g., in a meeting room, while the user is driving) or where the user does not wish to answer calls (e.g., telemarketing calls, harassing calls).

[0003] In a possible design, the agent relies on preset fixed templates (e.g., fixed text and feature words) to recognize the caller's intent and automatically respond. For example, the agent can automatically generate a fixed response such as "He is in a meeting" upon recognizing fixed text such as "Who is this?" and feature words such as "the owner's name".

[0004] However, the agent lacks flexibility. Summary of the Invention

[0005] This application provides an intelligent interaction method and system based on end-to-end multimodal instruction understanding, applicable to the field of terminal technology. It helps improve the flexibility and generalization ability of intelligent agents.

[0006] In a first aspect, embodiments of this application propose an intelligent interaction method based on end-to-end multimodal instruction understanding. The method includes: acquiring input information from a peer device; analyzing the input information and pre-set dialogue goals using an intelligent agent to obtain response content for the input information; the intelligent agent is trained based on Reinforcement Learning with Human Feedback (RLHF), where a reward model in RLHF is used to evaluate the response content output by the intelligent agent based on a first metric and a second metric; the first metric reflects dialogue quality, and the second metric reflects the completion status of the dialogue goal (also known as task completion); and sending the response content to the peer device.

[0007] In this way, the reward model can evaluate from two dimensions: dialogue quality and task completion, enabling the agent to communicate efficiently and with high quality according to the dialogue goal and obtain information related to the dialogue goal.

[0008] The intelligent agent can respond based on pre-set dialogue goals, revolving around a specific task (such as booking tickets or collecting information), thus improving the practicality and efficiency of the dialogue. The agent is trained using RLHF and evaluated from two dimensions—"dialogue quality" and "task completion"—using a reward model. This allows the agent to communicate efficiently and with high quality according to the dialogue goal, obtaining relevant information and providing more accurate and efficient services. Furthermore, this method is applicable to various dialogue scenarios (e.g., customer service, voice assistants, automated call answering).

[0009] Optionally, the first indicator includes any one or more of the following: relevance, dialogue logic, information retrieval completeness, naturalness, and conciseness; the second indicator includes any one or more of the following: inquiry completion rate and goal completion rate; wherein, relevance is used to characterize the degree of connection between the response content and the dialogue goal; dialogue logic is used to characterize the clarity of the language logic of the response content; information retrieval completeness is used to characterize the comprehensiveness of the elements corresponding to the dialogue goal during the dialogue process; naturalness is used to characterize the fluency of the dialogue process; conciseness is used to characterize the refinement of the response content; inquiry completion rate is used to characterize the completion rate of the inquiry items in the dialogue goal during the dialogue process; goal completion rate is used to characterize the completion rate of the goal items in the dialogue goal during the dialogue process.

[0010] This allows for quality assessment of responses from multiple dimensions, making the reward model's quality evaluation criteria clearer and more operational. It also enables task completion assessment of responses from multiple dimensions, further clarifying and making the reward model's task completion evaluation criteria more practical. After training based on this reward model, the agent can improve the accuracy of its responses.

[0011] Optionally, the reward model's evaluation of the response content is obtained by weighting the indicators in the first indicator and the indicators in the second indicator.

[0012] In this way, by weighting multiple indicators, scattered and multi-dimensional evaluation results can be integrated into a single, comparable evaluation score, simplifying preference judgment in the RLHF training process. In addition, by adjusting the weights of different indicators, the reward model's preferences can be customized for different application scenarios (such as customer service scenarios that focus more on efficiency or social scenarios that focus more on naturalness), making the solution more adaptable.

[0013] Optionally, before weighted calculation, the reward model is also used to normalize the indicators in the first indicator and the indicators in the second indicator.

[0014] In this way, indicators of different dimensions and magnitudes are normalized, so that different indicators can be weighted and calculated on a unified scale. This reduces the situation where certain indicators dominate the entire evaluation result due to their large numerical range, and improves the fairness and scientific nature of weighted summation.

[0015] Optionally, the weights of each indicator in the second indicator are greater than the weights of each indicator in the first indicator.

[0016] In this way, by giving higher weight to the task completion index, the agent is clearly guided to prioritize how to advance and complete the dialogue goal when generating responses, thereby improving the practicality and effectiveness of the agent in task-oriented dialogues.

[0017] Optionally, when the query response rate is greater than zero, the query completion rate is negatively correlated with the query response rate; when the query response rate is zero, the query completion rate is zero; wherein, the query response rate is used to characterize the ratio between the number of queries in the dialogue target appearing in a single response and the number of queries in the dialogue target; the ratio between the number of queries for at least one element and the number of queries in the response content; and / or, when the target response rate is greater than zero, the target completion rate is negatively correlated with the target response rate; when the target response rate is zero, the target response rate is zero; wherein, the target response rate is used to characterize the ratio between the number of target items in the dialogue target appearing in a single response and the number of target items in the dialogue target; the ratio between the number of target items in the dialogue target and the number of target queries in the response content.

[0018] Thus, if a single response includes multiple queries, the query completion rate decreases; if a single response includes multiple target items, the target completion rate decreases; a penalty is added to responses with multiple targets / multiple queries to reduce the number of queries or target items.

[0019] Optionally, for the same dialogue objective, the reward model is trained based on at least two responses corresponding to the dialogue objective.

[0020] In this way, by training based on multiple responses under the same goal, the reward model is provided with direct and high-quality comparative learning samples, enabling it to learn human preference judgments faster and more accurately, thereby training a reward model with stronger discriminative ability.

[0021] Optionally, at least two responses are obtained by processing the dialogue objective based on different types of large models.

[0022] In this way, by integrating responses generated by different types of large models, we can obtain samples with varying styles, strategies, and focuses, enriching the diversity of training data. This helps the trained reward model to have better generalization ability and to more fairly evaluate "good" responses of different styles.

[0023] Optionally, the reward model is trained based on the first response content and the second response content; the first response content is the highest-rated response content among at least two responses corresponding to the dialogue target; the second response content is the lowest-rated response content among at least two responses corresponding to the dialogue target.

[0024] In this way, by selecting the "best" and "worst" responses as training pairs, the reward model is provided with the most significantly different samples in the learning space, enabling the model to learn the decision boundary between "good" and "bad" most effectively, thereby accelerating model convergence and improving its discriminative ability.

[0025] Optionally, the difference between the rating corresponding to the first response and the rating corresponding to the second response is greater than or equal to the second threshold.

[0026] By setting a threshold for the difference in scores, sample pairs with consistent opinions and very clear preferences among human annotators are selected. This reduces the use of ambiguous and difficult-to-distinguish sample pairs for training, decreases noise, and further improves the accuracy and reliability of the reward model training.

[0027] Optionally, the agent is obtained based on training data, which includes: dialogue target, dialogue text, and elements.

[0028] In this way, by explicitly including "dialogue goals, dialogue text, and elements" in the training data, the agent can learn the complete mapping relationship from goals to elements, and then to specific dialogues, making the training process more targeted.

[0029] Optionally, the dialogue target is derived from the call recording.

[0030] Compared to manually constructed data, dialogue targets derived from real call recordings are more consistent with user expression habits and actual conversation logic. Thus, using real-world call recordings as training samples improves the agent's ability to understand users' true intentions, expression methods, and language habits. Training with this data enhances the model's accuracy and generalization ability in realistic, complex, and varied call environments.

[0031] Secondly, embodiments of this application provide an RLHF training method for an intelligent agent. This method can be applied to electronic devices, servers, or any training device.

[0032] The method includes: using an agent to analyze input information and pre-set dialogue goals to obtain response content to the input information; using the reward model in RLHF to evaluate the response content output by the agent based on a first metric and a second metric to obtain an evaluation result; adjusting the agent based on the evaluation result; the first metric is used to reflect the dialogue quality, and the second metric is used to reflect the completion of the dialogue goal (also known as task completion).

[0033] Optionally, the first indicator includes any one or more of the following: relevance, dialogue logic, information retrieval completeness, naturalness, and conciseness; the second indicator includes any one or more of the following: inquiry completion rate and goal completion rate; wherein, relevance is used to characterize the degree of connection between the response content and the dialogue goal; dialogue logic is used to characterize the clarity of the language logic of the response content; information retrieval completeness is used to characterize the comprehensiveness of the elements corresponding to the dialogue goal during the dialogue process; naturalness is used to characterize the fluency of the dialogue process; conciseness is used to characterize the refinement of the response content; inquiry completion rate is used to characterize the completion rate of the inquiry items in the dialogue goal during the dialogue process; goal completion rate is used to characterize the completion rate of the goal items in the dialogue goal during the dialogue process.

[0034] Optionally, the reward model in RLHF is used to evaluate the response content of the agent's output in terms of dialogue quality and task completion, and the evaluation result is obtained, including: evaluating the response content of the agent's output to obtain each indicator in the first indicator and each indicator in the second indicator; and obtaining the evaluation result by weighting the indicators in the first indicator and each indicator in the second indicator.

[0035] Optionally, before weighted calculation, the method further includes: normalizing each indicator in the first indicator and each indicator in the second indicator.

[0036] Optionally, the weights of each indicator in the second indicator are greater than the weights of each indicator in the first indicator.

[0037] Optionally, when the query response rate is greater than zero, the query completion rate is negatively correlated with the query response rate; when the query response rate is zero, the query completion rate is zero; wherein, the query response rate is used to characterize the ratio between the number of query items in the dialogue target appearing in a single response content and the number of query items in the dialogue target, and the ratio between the number of query items for at least one element and the number of queries in the response content; and / or, when the target response rate is greater than zero, the target completion rate is negatively correlated with the target response rate; when the target response rate is zero, the target response rate is zero; wherein, the target response rate is used to characterize the ratio between the number of target items in the dialogue target appearing in a single response content and the number of target items in the dialogue target, and the ratio between the number of target items in the dialogue target and the number of target queries in the response content. Optionally, for the same dialogue target, the reward model is trained based on at least two response contents corresponding to the dialogue target.

[0038] Optionally, at least two responses are obtained by processing the dialogue objective based on different types of large models.

[0039] Optionally, the reward model is trained based on the first response content and the second response content; the first response content is the highest-rated response content among at least two responses corresponding to the dialogue target; the second response content is the lowest-rated response content among at least two responses corresponding to the dialogue target.

[0040] Optionally, the difference between the rating corresponding to the first response and the rating corresponding to the second response is greater than or equal to the second threshold.

[0041] Thirdly, embodiments of this application provide a method for training a reward model. This method can be applied to electronic devices, servers, or any training device.

[0042] The method includes: generating multiple response contents for the same dialogue target; inputting each response content into a reward model to obtain each evaluation result; generating a first loss based on each evaluation result and the preset evaluation result corresponding to each response content; and optimizing the reward model according to the first loss until the first loss is less than or equal to a first value.

[0043] Fourthly, embodiments of this application provide a method for generating training data.

[0044] The method includes: converting call recordings into dialogue text; extracting information and elements from the dialogue text to obtain the caller's inquiry, the receiver's target, the receiver's inquiry, and elements corresponding to the dialogue text; extracting a summary from the dialogue text to obtain a dialogue summary; comparing the dialogue summary with a vertical domain to obtain the domain corresponding to the dialogue text; and generating a dialogue target based on the domain of the dialogue text, the receiver's target, and the receiver's inquiry.

[0045] Fifthly, embodiments of this application provide an intelligent interactive device, which can be an electronic device, or a chip or chip system within an electronic device. The intelligent interactive device may include a display unit and a processing unit. When the intelligent interactive device is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to enable the electronic device to implement an intelligent interactive method described in the first aspect or any possible implementation of the first aspect. When the intelligent interactive device is an electronic device, the processing unit may be a processor. The intelligent interactive device may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to enable the electronic device to implement an intelligent interactive method described in the first aspect or any possible implementation of the first aspect. When the intelligent interactive device is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to enable the electronic device to implement an intelligent interactive method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit inside the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0046] In a sixth aspect, embodiments of this application provide an electronic device including a processor and a memory, the memory being used to store computer execution instructions, and the processor being used to execute the computer execution instructions stored in the memory to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0047] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0048] Eighthly, embodiments of this application provide a computer program product including a computer program, which, when run, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0049] Ninthly, this application provides a chip or chip system including at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a circuit, the at least one processor being used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0050] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0051] In a tenth aspect, this application provides an interactive system comprising: an electronic device and a peer device; the peer device is used to send input information to the electronic device; the electronic device is used to implement the method described in the first aspect or any possible implementation thereof.

[0052] It should be understood that the second to tenth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0054] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0056] Figure 4 A flowchart illustrating an intelligent interaction method based on end-to-end multimodal instruction understanding provided in an embodiment of this application;

[0057] Figure 5 A schematic diagram of the structure of an RLHF provided in an embodiment of this application;

[0058] Figure 6 A schematic diagram of an RLHF training process provided in an embodiment of this application;

[0059] Figure 7 A flowchart illustrating a training method for a reward model provided in an embodiment of this application;

[0060] Figure 8 A schematic diagram of a call recording processing flow provided for an embodiment of this application;

[0061] Figure 9 This is a schematic diagram of the structure of a training data generation model provided in an embodiment of this application. Detailed Implementation

[0062] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0063] 1. Large Language Models (LLM)

[0064] A large language model is a model based on machine learning and natural language processing techniques. This model learns the human ability to understand and generate language by training on large amounts of text data.

[0065] 2. Reinforcement learning from human feedback (RLHF)

[0066] RLHF is a large model fine-tuning technique used to optimize language models (LMs) with human feedback. RLHF enables language models trained on general text corpora to align with complex human values.

[0067] 3. Reward Model (RM)

[0068] A reward model is a model used for human preference calibration. This model takes a series of texts and returns a scalar reward that should numerically represent human preference. The RM can be an end-to-end LM or a modular system of output rewards (e.g., the model sorts the outputs and converts the sorting into a reward).

[0069] 4. Reinforcement learning (RL)

[0070] RL is a branch of machine learning where an agent learns by interacting with its environment. RL is a goal-oriented learning process where the agent is not told what action to take; instead, it learns from the consequences of its actions.

[0071] 5. Proximal Policy Optimization (PPO) Algorithm

[0072] The PPO algorithm is a trust region optimization (TRO) algorithm that uses gradient constraints to ensure that update steps do not compromise the stability of the learning process.

[0073] Proximal Policy Optimization (PPO) uses the ratio of New Policy to Old Policy to limit the update magnitude of New Policy. To make the policy gradient less sensitive to a slightly larger Step Size, two main variations of the PPO algorithm can be used: Proximal Policy Optimization Penalty (PPO-penalty) and Proximal Policy Optimization Clipping (PPO-clip). PPO-penalty utilizes KL divergence constraints.

[0074] The formula for PPO penalty is expressed as follows: .

[0075] The formula for PPO-clip loss is expressed as follows: , .

[0076] in, Indicate the importance of sampling between the old and new strategies. This represents the action at that the policy takes in state st. This represents the advantage function. Since the objective function involves importance sampling, and when performing importance sampling... Cannot be with The difference is too great, so a constraint needs to be added during model training. This constraint is similar to a regularization term. and The KL divergence of the output action is used to measure and The degree of similarity. During training, what is learned... and The more similar, the better.

[0077] 6. Other terms

[0078] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0079] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0080] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0081] In this embodiment, "send" and "receive" indicate the direction of signal transmission. For example, "send information to the first device" can be understood as the destination of the information being the first device, which may include direct transmission via the air interface or indirect transmission via the air interface by other units or modules. "Receive information from the second device" can be understood as the source of the information being the second device, which may include direct reception from the second device via the air interface or indirect reception from the second device via the air interface by other units or modules. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface.

[0082] In other words, sending and receiving can be done between devices, such as between a first device and a second device; or it can be done within a device, such as between components, modules, chips, software modules, or hardware modules within a device via a bus, wiring, or interface.

[0083] It is understandable that information may undergo necessary processing, such as encoding and modulation, before being sent from the source to the destination. Similarly, the destination, upon receiving information from the source, can also perform corresponding processing, such as decoding and demodulation, to interpret the valid information from the source. Similar expressions in this application can be understood in a similar way and will not be elaborated further.

[0084] 7. Electronic equipment

[0085] The electronic devices in this application embodiment may include handheld devices with communication functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, terminal devices in 5G networks, or terminal devices in future evolved public land mobile networks (PLMNs), etc., and the embodiments of this application are not limited to these.

[0086] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0087] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0088] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0089] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0090] For example, Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, the scene includes: first device 101 and second device 102.

[0091] Both the first device 101 and the second device 102 are electronic devices with communication functions. The electronic devices can be described in the corresponding descriptions above, and will not be repeated here. The type of the first device 101 can be the same as or different from that of the second device 102; no specific limitation is made here.

[0092] The first device 101 may be equipped with an intelligent agent. The intelligent agent is used to engage in dialogue with the second device 102 based on the set target content; it can also be understood that the intelligent agent can proactively ask questions or respond to inquiries raised by the second device 102 based on the target content and the dialogue situation.

[0093] In this embodiment, the intelligent agent can be applied to both telephone answering and telephone calling scenarios. That is, the roles of the first device 101 and the second device 102 during a call depend on the scenario. In an incoming call scenario, the first device 101 can be the called party (also known as the receiver), and the second device 102 can be the calling party (also known as the incoming caller); in a calling scenario, the first device 101 can be the calling party, and the second device 102 can be the called party.

[0094] For example, in a call scenario, when the first device 101 receives a call request from the second device 102, the first device 101 can respond to an operation to instruct automatic answering by calling an agent and automatically answering the call through the agent, and then conversing with the second device 102. The operation to instruct automatic answering can be a user's voice instruction to answer automatically, or a pressing operation on an automatic answering control, etc., and is not specifically limited here.

[0095] In some embodiments, when the first device 101 receives a call request, the intelligent agent in the first device 101 can also automatically identify the category of the caller; if the category of the caller is a preset category (e.g., intermediary, food delivery, harassment call), it can automatically answer the call and talk to the caller.

[0096] In this embodiment, the first device 101 and the second device 102 can establish a communication connection. This communication connection can be of any form, such as any form of wireless communication connection or any form of wired communication connection. For example, a wireless communication connection can include the following forms: wireless fidelity (WIFI) communication connection, cellular communication connection, Bluetooth communication connection, near-field communication connection, ultra-wide band (UWB) communication connection, and infrared transmission communication connection, etc. A wired communication connection can include the following forms: Ethernet communication connection, fiber optic communication connection, and universal serial bus (USB) connection, etc.

[0097] In this embodiment of the application, the first device 101 and the second device 102 may establish a communication connection through protocols such as Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol over Secure Socket Layer (HTTPS), and this embodiment of the application does not impose any restrictions on this.

[0098] The first device 101 and the second device 102 can also achieve P2P communication through any form of communication connection. For example, the first device 101 and the second device 102 can achieve P2P communication through wireless transmission, such as Bluetooth transmission, ultra-wide band (UWB) communication, or infrared transmission. The first device 101 and the second device 102 can also achieve P2P communication through wired transmission.

[0099] In one possible design, the agent in the first device 101 communicates with the second device 102 through a manually defined dialogue template.

[0100] In some embodiments, the dialogue template includes fixed text and feature words. By recognizing the feature words in the input statement, a corresponding preset response can be triggered, thereby achieving automated dialogue interaction.

[0101] For example, the agent can preset a dialogue template in the form of: "[Who are you?] What can I do for [the owner's name]?" In this template, "Who are you?" is a fixed text part, and "the owner's name" and "what can I do for" are feature words. When the agent recognizes content that matches the feature words, it can generate and output a preset fixed reply such as "He is in a meeting and will call you back later".

[0102] This approach, based on pre-defined templates and keyword matching, can handle simple question-and-answer scenarios with a high degree of standardization, enabling basic peer-to-peer functionality. However, its response logic relies on fixed text patterns, resulting in low flexibility.

[0103] In other embodiments, the dialogue template includes: a dialogue intent and at least one template fragment, which includes one or more of the following: slots, feature words, and fixed text. The slots and feature words each correspond to a dictionary, which includes multiple words of the same type. The slots describe key information for realizing the dialogue intent, the feature words describe features for expressing the dialogue intent, and the fixed text includes words that must be included to identify the expression of the dialogue intent.

[0104] In this way, a single dialogue template can recognize multiple different expressions of the same dialogue intent, which can improve the flexibility of the agent to some extent.

[0105] However, dialogue templates have limited coverage and cannot exhaustively represent the diverse expressions in natural language. If the user uses a synonym, elliptical sentence, or colloquial expression not covered by the template, the agent may not be able to correctly recognize the user's intent, leading to interaction failure.

[0106] In view of this, embodiments of this application provide an intelligent interaction method and system based on end-to-end multimodal instruction understanding. The intelligent agent can generate response content based on pre-defined dialogue objectives. This response content can be used to ask questions about elements related to the dialogue objective, or to respond to inquiries from the other end.

[0107] Dialogue goals are used to indicate user intent. Taking a phone call as an example, dialogue goals can include: the main domain (e.g., a vertical domain), subdomains, target items, and queries. Target items can be understood as specific responses that need to be completed; queries can be understood as inquiries made to obtain elements from the dialogue goals.

[0108] For example, the pattern of the dialogue goal could be "When the domain calls, if the subdomain, reply with xxx, and ask a question about xxx".

[0109] In this embodiment, "domain" can be understood as the main category or scenario of the call conversation, and can be used as a first-level filtering condition to identify the basic type of incoming call. Examples include e-commerce retail, real estate services, transportation, education and training, financial services, social entertainment, lifestyle services, healthcare, customer service, business negotiations, sales consultation, and emergency contacts.

[0110] A subdomain can be understood as a more specific sub-segment within a domain, and can be used as a second-level precise match to implement refined dialogue strategies. For example, if the domain is customer service, the subdomain could be complaint handling, product consultation, after-sales support, etc.

[0111] For example, taking education and training, healthcare, and service promotion as examples, the dialogue objectives could be as follows: "When education and training calls to confirm an appointment, reply that the class time is fine, ask about the teacher's fees, how many classes are held per day, and inquire about the class location"; "When healthcare calls to discuss medical examination fees, reply that the medical examination fee is too cheap, and ask for a more upscale package, inquire about the examination items, and inquire about the specific time of the examination"; "When service promotion calls to coordinate a car test drive appointment, reply that you are currently busy, but may have time to look at cars next weekend, and are willing to accept a test drive arrangement at the company entrance at noon, and inquire whether a test drive can be arranged at the company entrance at noon."

[0112] In this way, relevant information about the dialogue objective can be obtained during the dialogue process, resulting in highly logical and high-quality responses.

[0113] For example, taking a call scenario where the dialogue goal is "When a real estate service calls, if recommending a house in area A, ask questions about the house price, location, and nearby amenities," the relevant elements of the dialogue goal could be "price, address, and amenities," etc. Adaptively, after receiving input information from the second device, the agent can ask questions about price, address, and amenities.

[0114] For example, the dialogue process can be as follows:

[0115] "Second device: Hello, this is a real estate company. We have a few properties available for you to take a look at."

[0116] First device: Hello, I am the smart assistant. The owner is busy right now. Where is the house located?

[0117] Second equipment: Area A;

[0118] First Equipment: Okay, could you please tell me the exact address and what the price is?

[0119] Second equipment: XXXXX.

[0120] To better understand the embodiments of this application, the structure of the electronic device according to the embodiments of this application is described below. For example, Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0121] like Figure 2 As shown, the electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a user identification module card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a fingerprint sensor 180H, a proximity sensor 180G, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0122] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0123] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the intelligent interaction method in the embodiments of this application.

[0124] For example, processor 110 can utilize an agent to generate response content based on pre-set dialogue objectives. This response content is used to respond to queries from the peer device, and / or to query the peer device for elements related to the dialogue objectives. The agent is trained based on dialogue text with different elements set.

[0125] Electronic devices can implement display functions through GPUs, displays 194, and application processors. A GPU is a microprocessor for image processing, connecting the displays 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0127] For example, Figure 3This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the runtime and system libraries, and the kernel layer.

[0128] The application layer can include a series of application packages.

[0129] The application package may include: camera, calendar, map, phone, music, settings, video, and intelligent agent, etc. In this embodiment, the intelligent agent is used to generate response content based on pre-set dialogue objectives.

[0130] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 3 As shown, the application framework layer may include a window manager, an activity manager service (AMS), a resource manager, a notification manager, a content provider, and a view system.

[0131] The window manager is used to manage window applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0132] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, and more.

[0133] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0134] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0135] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0136] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part is the Android core library. The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0137] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0138] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0139] The kernel layer is the layer between hardware and software. It includes at least display drivers, audio drivers, Wi-Fi drivers, and sensor drivers. The hardware includes at least a processor, display screen, Wi-Fi module, and sensors.

[0140] Understandable, Figure 3 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components, or combine some components, or split some components, or arrange the components differently; this application does not impose any limitations.

[0141] For example, Figure 4 A flowchart illustrating an intelligent interaction method based on end-to-end multimodal instruction understanding provided in this application embodiment is shown below. Figure 4 As shown, the method includes:

[0142] S401. Obtain input information from the peer device.

[0143] Input information is used to indicate the vertical domain to which the dialogue belongs. Input information may include: voice data, and / or visual environment data captured by the image acquisition device of the peer device, etc. This application does not specifically limit this.

[0144] S402. The intelligent agent analyzes the input information and the pre-set dialogue target to obtain the response content.

[0145] In this embodiment, the agent is trained using Reinforcement Learning High-Frequency Learning (RLHF) that incorporates human feedback. The reward model in RLHF is used to evaluate the agent's responses in terms of quality and completion of the dialogue objective.

[0146] S403, Send a reply to the peer device.

[0147] In summary, the reward model evaluates dialogue quality and task completion from two dimensions, enabling agents to communicate efficiently and with high quality according to the dialogue objective and obtain information related to the dialogue objective.

[0148] For ease of understanding, the following will be combined with Figures 5 to 7 Explain RLHF and the reward model.

[0149] It's important to note that RLHF is a complex concept involving multiple models and different training phases, typically comprising the following three phases: Phase 1: Pre-training a Model (LM). An LM can be trained using classic pre-training objectives. Specifically, it can be trained using a supervised fine-tuning (SFT) dataset. For example, using GPT-3 as the LM, GPT-3 can be fine-tuned based on the collected SFT dataset, following the training method of GPT-3. Phase 2: Training the reward model (RM). Specifically, the RM receives a series of texts and returns a scalar reward, the value of which corresponds to a person's preference. Regarding the selection of the RM, it can be another finely tuned LM or an LM trained from scratch based on preference data.

[0150] Phase 3 involves proximal policy optimization (PPO) of the reward model. Specifically, the large language model is updated and optimized based on the feedback from the reward model. This phase can also be understood as fine-tuning the large language model.

[0151] In short, the RLHF process uses a reward model to score the responses generated by the policy model (i.e., the agent in this embodiment) to obtain reward values; with the goal of maximizing the reward value, the parameters of the policy model are updated using a policy gradient algorithm. The updated policy model can then obtain responses with higher rewards.

[0152] The following is combined with Figures 5 to 7 The structure and process of RLHF are explained.

[0153] For example, Figure 5 This is a schematic diagram of the structure of an RLHF provided in an embodiment of this application. Figure 5 As shown, RLHF includes: policy model (policy LM), supervised fine-tuning (SFT), reward model (RM), generalized advantage estimation (GAE), and value model.

[0154] The policy model is used to generate responses based on the dialogue objective (also known as the prompt). This policy model corresponds to the agent mentioned earlier. During RLHF training, the policy model is the model that RLHF training aims to produce, and its gradients are updated during PPO training. The supervised fine-tuning model is used to fine-tune the pre-trained policy model before RLHF, enabling the policy model to engage in dialogue.

[0155] The reward model is used to evaluate responses and obtain a reward value. This reward value indicates the quality of the response. The reward model does not update gradients during PPO training.

[0156] In this embodiment of the application, the reward model can evaluate the response content output by the agent from two aspects: quality and task completion, and obtain quality evaluation results and task evaluation results.

[0157] It should be understood that evaluation results (e.g., quality evaluation results, task evaluation results, etc.) can be represented by scores, ratings, or other arbitrary methods, without specific limitations here. For example, a reward model can score the response output of an agent from two aspects: quality and task completion, to obtain a quality score (e.g., mos_score) and a task score (e.g., task_score).

[0158] In this application embodiment, the quality aspect may include one or more of the following indicators: correlation, dialogue logic, information acquisition, naturalness, and simplicity. In some embodiments, the quality score may be obtained by averaging the values ​​of each indicator, calculating the weights, or any other method, without specific limitations here.

[0159] Relevance is used to characterize the degree of connection between the response content and the dialogue goal. For example, content relevance is defined as: whether the agent's response is directly related to the caller's statement or question, and whether it provides a reasonable response based on the caller's statement. A scoring scale of 1 to 5 is used, and the specific scoring criteria can be as follows:

[0160] 1 point: The agent's response is irrelevant to the caller's input context, the information is irrelevant, and the answer is not related to the question. 2 points: Somewhat irrelevant. The agent is generally in a conversation with the caller in the same scenario, but there is a confusion of roles in the response, or the agent makes a promise that it should not make as an agent. 3 points: Partially relevant. The agent is generally in a conversation with the caller in the same scenario, but the agent misunderstands the caller's meaning or the caller's intended focus, or the agent guides the caller to perform an action that is outside the scope of the call but is relatively reasonable. 4 points: Mostly relevant. Provides additional useful information, and the agent guides the caller to perform an action that is not entirely in line with business requirements but is relatively reasonable. 5 points: Fully relevant. Directly responds to the caller's content and effectively responds to the caller's content according to business requirements; all content in the response is consistent with the current context.

[0161] In this way, setting relevance metrics can reduce instances where responses deviate from the dialogue objective, ensuring that interactive content revolves around the dialogue objective and improving the effectiveness of the dialogue.

[0162] Dialogue logic is used to characterize the clarity of the language logic of the response content; for example, the definition of dialogue logic rationality can be: whether all the responses provided by the agent throughout the dialogue process are clear, coherent, and effectively convey relevant information. A reasonable response should ensure the consistency, coherence, and relevance of the information.

[0163] Using a rating scale of 1 to 5, the specific rating criteria are as follows: 1 point: The agent's responses contain multiple obvious logical contradictions. The agent's responses violate basic logical rules. The user cannot understand the agent's intention, and the entire dialogue is devoid of information. 2 points: The agent's responses contain multiple inconsistent viewpoints or pieces of information. The responses are partially relevant but lack necessary contextual support, making them difficult for the user to understand. The agent's responses are logically disordered. 3 points: The agent's responses are generally consistent, but occasionally contain small logical jumps, affecting the overall fluency; the responses are relevant to the user's questions, but sometimes lack in-depth reasoning or explanation, leading to unclear understanding and a less smooth transition between dialogues. 4 points: The agent's responses are largely consistent, with clear logical relationships, effective information delivery, and a smooth overall dialogue; the responses are closely related to the user's questions, effectively supporting the topic of the dialogue and promoting understanding. 5 points: All of the agent's responses are completely consistent, logically rigorous, and convey information clearly and effectively; the overall dialogue is very smooth; the responses are not only relevant but also guide the dialogue deeper, promoting better understanding and communication, and improving the user experience. The agent actively inquires for relevant information and guides the dialogue to its conclusion.

[0164] In this way, setting up the logic of the dialogue can reduce logical confusion in the dialogue process and improve the coherence of the interactive content in terms of narrative flow and structural hierarchy.

[0165] Information acquisition completeness is used to characterize the comprehensiveness of the element information corresponding to the dialogue goal during the dialogue process; information acquisition completion degree can also be called the degree of proactive inquiry of information.

[0166] For example, the definition of proactive inquiry information degree can be: whether the intelligent agent can proactively initiate effective information inquiry in order to complete information collection when the caller does not hang up the phone.

[0167] Taking a sales-related scenario as an example, a scoring scale of 1 to 5 can be used. The specific scoring criteria are as follows: 1 point: In a multi-turn dialogue, the agent does not proactively ask any questions in any round, relying entirely on information provided by the caller. Furthermore, the caller does not provide any useful information, resulting in incomplete information gathering. 2 points: In a multi-turn dialogue, the agent does not proactively ask any questions in any round, but the caller provides some useful information. Information gathering is insufficient, and important information may be missed. 3 points: In a multi-turn dialogue, the agent proactively asks relevant questions only in a few rounds, resulting in incomplete information gathering and a failure to deeply understand the caller's needs. 4 points: In a multi-turn dialogue, the agent proactively asks relevant questions in most rounds, guiding the dialogue and collecting information, resulting in relatively comprehensive information gathering. 5 points: In a multi-turn dialogue, the agent proactively asks multiple relevant questions in most rounds, guiding the dialogue and collecting information, ensuring comprehensive and effective information gathering.

[0168] Understandably, for scenarios such as express delivery and food delivery, the scoring criteria for proactively inquiring about information may include: asking for delivery location information when the delivery location is unknown, without requiring other information to be asked.

[0169] In this way, setting an information completeness index can reduce the omission of essential information corresponding to the dialogue objective.

[0170] Naturalness is used to characterize the fluency of dialogue and the naturalness of interaction. It can reflect whether the language expression is close to real interpersonal communication habits. For example, the definition of naturalness can be: whether the agent's response is fluent and natural, whether it is close to spoken language, whether it conforms to the daily communication habits of Chinese people, whether it does not require the use of formal or professional vocabulary, but prefers to use everyday spoken vocabulary, while allowing a certain amount of interjections and pauses.

[0171] Using a rating scale of 1 to 5, the specific rating criteria are as follows: 1 point: The language is very unnatural, unclear, and the vocabulary or sentence structure used gives the impression of a "wrong number." 2 points: The language is not clear enough; some parts are understandable, but some vocabulary or sentence structures are misunderstood, thus affecting the natural and smooth development of subsequent communication. 3 points: The language is somewhat forced; the vocabulary or sentence structure used is too generic. It conveys the meaning but is not appropriate for the actual context. The content of the reply is understood, but it sounds a bit off. 4 points: The language is basically fluent. The written reply is awkward, but the voice reply is fine. 5 points: The language is fluent and natural, in line with the current context. The reply is smooth both visually and aurally.

[0172] In this way, setting the naturalness index can make the response content closer to human communication habits, reduce mechanical and stiff expressions, and enhance the affinity of the interaction.

[0173] Simplicity characterizes the conciseness of a response, reflecting the efficiency of information transmission and the clarity of its structure. For example, simplicity can be defined as: whether the agent's response is concise and clear, without redundant information.

[0174] Using a rating scale of 1 to 5, the specific rating criteria can be as follows: 1 point: Very lengthy, contains a lot of unnecessary information, avoids repetition of sentence segments with the same semantic information, is not concise enough, does not conform to human conversation terminology, avoids completely identical statements between multiple rounds of conversation, allows for similar meanings but with a progression in emotion or tone. 2 points: Not concise enough, lengthy information, contains unnecessary expressions, contains a lot of redundant information (sentences expressing the same meaning appearing repeatedly in a single response) or repetitive descriptions of the caller. 3 points: Moderately concise, some redundant expressions, does not conform to the logic of normal communication, or excessive use of interjections, resulting in an unnatural reading experience. 4 points: Mostly concise, occasionally redundant. Redundant information is acceptable, with minimal redundancy. 5 points: Very concise, direct information. All key information is present, there is no redundancy. As long as all sentences contain key information without redundancy, even long sentences can receive a score of 5. It is not recommended to score solely based on sentence length.

[0175] In this way, setting the conciseness index can reduce redundant and verbose expressions during the conversation, making the response content get straight to the point and improving the efficiency of information transmission.

[0176] In summary, the quality of responses can be evaluated from multiple dimensions, making the quality assessment criteria of the reward model clearer and more operational. After training based on this reward model, the agent can improve the accuracy of its responses.

[0177] Task completion metrics may include one or more of the following: query completion rate (ask_completion_rate) and goal completion rate (goal_completion_rate). In some embodiments, the task score may be obtained by averaging the values ​​of each metric, calculating weights, or any other method, without specific limitations here.

[0178] The query completion rate is used to characterize the completion rate of query items in the dialogue objective during the dialogue process; the objective completion rate is used to characterize the completion rate of objective items in the dialogue objective during the dialogue process.

[0179] Understandably, setting a completion rate allows control over the completion of queries (e.g., efficiency, completeness) within the dialogue objective during the interaction process. Thus, the completion rate can be used to assess and guide the dialogue process towards its objective, enabling the objective to be achieved with greater completeness and efficiency within appropriate dialogue rounds.

[0180] Setting a target completion rate allows control over the completion of target items (e.g., efficiency, completeness) within the dialogue objective during the interaction process. In this way, the target completion rate can be used to assess and guide the dialogue process towards the objective, thereby achieving the dialogue objective with greater completeness and efficiency within appropriate dialogue rounds.

[0181] This allows for the evaluation of goal achievement from multiple dimensions, making the evaluation criteria for goal achievement in the reward model clearer and more operational. After training based on this reward model, the agent can improve the accuracy and comprehensiveness of acquiring dialogue goal-related information.

[0182] The value model is used to predict the total reward value obtained by the policy model from the current state (e.g., the dialogue goal and the partial responses already generated) to the end of the dialogue. This prediction can be used to determine the gap between the generation steps and the expected outcome.

[0183] The Gaussian Advantage (GAE) is used to derive an "Advantage" value based on the reward value output by the reward model and the predicted value output by the value model. This Advantage value is used to evaluate the gap between the response content and the average level.

[0184] In this way, training the agent through RLHF can make the agent more closely resemble human goals, desires, and needs when performing tasks, thereby improving the accuracy and relevance of the agent's output responses.

[0185] The following is combined with Figure 6 right Figure 5 The training process of RLHF is illustrated below. For example, Figure 6 This is a schematic diagram of an RLHF training process provided in an embodiment of this application. Figure 6 As shown, the process includes:

[0186] S601, Sampling.

[0187] Specifically, a batch of dialogue targets is taken from the training dataset. The dialogue targets (also known as cue words) are input into the policy model to obtain the response content (also known as the answer).

[0188] S602, Assessment

[0189] Specifically, each input reward model (hint, answer) is evaluated to obtain an immediate reward. Additionally, the value model evaluates each state in the generation process (the context when generating the t-th token) and provides a predicted value.

[0190] S603, computational advantages.

[0191] Using GAE, the advantage value is calculated for each generation step by combining the immediate reward sequence provided by the reward model and the predictions of the value model. The advantage value measures "how much better it is to take a certain action (generate a certain response) than the average performance in a given state".

[0192] S604, Strategy Model Update

[0193] The PPO algorithm is used to update the parameters of Policy LM based on the advantage value. Understandably, if generating a response brings a positive advantage (i.e., the result is better than expected), the probability of generating that response in the future increases; conversely, it decreases.

[0194] Furthermore, the value model can be updated. This allows the value model's predictions to more closely approximate actual cumulative rewards, making its future predictions more accurate.

[0195] S605, Constraints and Stability.

[0196] Specifically, the updated strategy model must comply with the KL divergence penalty.

[0197] In this way, the updated policy model (e.g., Policy LM) cannot deviate too far from the original SFT model (or the policy model from the previous step), ensuring the readability and consistency of the policy model's output.

[0198] The above Figure 5 and Figure 6 The training process of RLHF is explained below. Figure 7 and Figure 8 The training process of the reward model is explained.

[0199] For example, Figure 7 This is a flowchart illustrating a training method for a reward model provided in an embodiment of this application. Figure 7 As shown, the process includes:

[0200] S701. Generate multiple response responses for the training dialogue target.

[0201] In some embodiments, the dialogue target can be input into multiple large models to obtain the corresponding response text for each model. The large models can be Llama-3.2-8B-Instruct, GPT4o, claude3.7, Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-72B-Instruct, qwen3_8B_base_sft, Qwen3-1.7B, Qwen3-8B, etc., and are not specifically limited here.

[0202] Understandably, if a large model does not typically perform SFT on the training dialogue target, it can use a general dialogue target (prompt) for sampling; if a large model does perform SFT, it can use the dialogue target sampled during training.

[0203] In this way, by generating responses using large models of different types, training samples with varying styles, strategies, and focuses can be obtained. When the reward model is subsequently trained using these multiple responses, it can be exposed to a wider range of "good" and "bad" examples, reducing bias caused by limited training data and improving the universality of the reward model's evaluation criteria.

[0204] S702. Manually evaluate each response to obtain the preset evaluation results for each response.

[0205] In this embodiment, the evaluation can be conducted from two aspects: dialogue quality and task completion. The corresponding indicators for dialogue quality and task completion can be found in the descriptions above, and are not specifically limited here.

[0206] In some embodiments, the evaluation results may be expressed in the form of scores, grades or other arbitrary forms, without specific limitations here.

[0207] The evaluation result can be obtained by comprehensively calculating based on the above indicators. The comprehensive calculation can use a weighted method or any other method, and there are no specific restrictions here.

[0208] In some embodiments, the reward model evaluates the response content by weighting the various metrics of dialogue quality and task completion. The weights for each metric can be the same or different; no specific limitation is made here.

[0209] In this way, by weighting the various indicators, the evaluation results from multiple dimensions can be integrated into a unified score, simplifying the calculation. Furthermore, the relative importance of different indicators can be adjusted for different application scenarios (for example, customer service scenarios may place more emphasis on task completion, while companion robots may place more emphasis on naturalness), making it widely applicable.

[0210] In some embodiments, the weights of each metric in task completion are higher than the weights of each metric in dialogue quality. This increases the contribution of task completion, guiding the agent to prioritize task achievement during model training, reducing instances where the agent deviates from the dialogue objective, and improving the agent's usability and reliability.

[0211] Understandably, if the sub-evaluation results corresponding to each indicator are represented by scores, then the sub-evaluation results can be normalized before comprehensive calculation. This reduces calculation bias caused by differences in units and value ranges, helping the reward model to converge more stably and faster, thereby improving training efficiency.

[0212] Normalization methods can include Min-Max, Z-score, sigmoid functions, etc., and no specific limitation is made here.

[0213] S703. Input each response into the reward model to obtain each evaluation result and the first loss. The first loss is related to the evaluation result and the preset evaluation result.

[0214] For example, as described above Figure 5 Taking the RHLF process shown as an example, the evaluation result corresponding to the reward model can be:

[0215] ;in, Indicates the state Next, actions (behavioral sequences). In contrast to actions (behavioral sequences) A better probability. The sigmoid function is used to convert the score value into a range of 0. The probability value of 1.

[0216] For example, as described above Figure 5 Taking the RHLF process shown as an example, the first loss corresponding to the reward model can be: .in, This represents the value of the loss function. For the expectation, This is the initial state. and These are two different action sequences, which can also be understood as equal-length trajectories randomly downsampled from the initial policy, containing a state sequence (si, si+1, ... si+n-1) or a state-action pair sequence (si, ai, si+1, ai+2, ... si+n-1, ai+n-1) of length n. For reward models; Parameters adjusted during the training of the reward model; Data distribution for the initial strategy;

[0217] It should be understood that, with Superior For example, based on the loss function of the above reward model, a reward can be obtained that can score the entire trajectory. Reinforcement learning is performed by comparing and contrasting the trajectory scores.

[0218] In this way, the reward model can compare multiple different responses under the same goal, enabling comparative learning. This helps the reward model learn more precisely the subtle differences between "good responses" and "bad responses," thus making its evaluation criteria more accurate and robust.

[0219] S704. Optimize the reward model based on the first loss until the first loss is less than or equal to the first threshold.

[0220] This ensures that the evaluation results of the reward model match the preset evaluation results.

[0221] In some embodiments, the first loss can be calculated from the two best and worst responses among multiple responses to optimize the reward model.

[0222] In this way, by selecting the highest and lowest rated responses as training samples, the distance between "good" and "bad" in the reward model's representation space can be widened, accelerating the training process of the reward model and improving its accuracy.

[0223] In some embodiments, if the evaluation results corresponding to the response content are similar, the sample can be discarded, or the response content corresponding to the sample can be adjusted.

[0224] For example, if the evaluation result is the sum of the normalized mean mos_score and the mean task_score, then the difference between the evaluation results corresponding to the two responses is less than or equal to the second threshold. This allows adjustment of the response corresponding to the worst evaluation result.

[0225] In some embodiments, the second threshold can be a fixed value, the maximum value among the 20% of the smallest differences in the total data, or any value. This application does not specifically limit the method for determining the second threshold.

[0226] In some embodiments, the response content corresponding to the worst evaluation result can be replaced with the response content corresponding to the worst evaluation result of other dialogue targets.

[0227] This allows for greater contrast, enhancing the discriminative power between training samples and improving the learning efficiency of the reward model. The rating difference threshold filters out ambiguous or poorly defined comparison samples. Controlling the sample pairs used for contrastive learning to have significant learning value makes the reward model training more efficient and the learned discrimination boundaries clearer.

[0228] Based on the above embodiments, some of the reply content can be replaced with longer reply content. This makes the reward model more biased towards shorter replies, making the reply content more concise and reducing redundancy.

[0229] In some embodiments, 20% of the data may be selected, and the rejected data may be replaced with longer responses. This application does not specify a particular percentage for the replaced data.

[0230] Based on the above embodiments, the number of target items and the number of query items can also be controlled to reduce the situation where a single response includes too many target items or too many query items.

[0231] In one possible implementation, the query completion rate is negatively correlated with the query response rate. The query response rate is used to characterize the number of queries in a single response during a dialogue. Thus, when there are too many queries, the query completion rate decreases, reducing the likelihood of an excessive number of queries in a single session.

[0232] For example, when the query response rate is greater than or equal to a first preset threshold, the query completion rate can be negatively correlated with the query response rate. This allows for control over the number of queries in a single response, reducing the likelihood of a single response containing too many target items or too many queries.

[0233] In some embodiments, the query response rate can be the average number of queries in a single response during a conversation. For example, the query response rate can be the ratio between the number of queries and the number of responses. Thus, when there are too many queries, a higher average number results in a lower query completion rate, reducing the likelihood of too many queries in a single session.

[0234] In other embodiments, the query response rate can be the ratio between the number of queries from the dialogue target appearing in a single response and the total number of queries from the dialogue target. This allows for optimization of a single response, reducing the likelihood of a single response containing too many queries.

[0235] Understandably, the criteria for completing a query can be determined using a large model or any other method. For example, a template used for large model scoring could be as follows:

[0236] Task Description

[0237] # Dialogue

[0238] {dialogue}

[0239] # Inquiry items for the caller

[0240] Target 1: xxx

[0241] Objective 2: xxx

[0242] # Output

[0243] Question 1: Yes or No

[0244] Query 2: "Yes or No" adaptive, the query response rate can be the ratio between the number of queries that are answered "yes" in a single response (also known as the number of queries marked as completed) and the total number of queries.

[0245] For example, the first preset threshold can be zero. When the query response rate is greater than zero, the query completion rate is negatively correlated with the query response rate; when the query response rate is zero, the query completion rate is zero. The formula can be as follows:

[0246] The expression `ask_completion_rate=1-ask_rate(response)` is used if `ask_rate(response)` > 0, and `ask_completion_rate=0` if `ask_rate(response)` = 0. Here, `ask_completion_rate` represents the query completion rate, and `ask_rate(response)` represents the query response rate.

[0247] Thus, if a single response includes multiple queries, the query completion rate decreases; if a single response includes multiple target items, the target completion rate decreases; a penalty is added to responses with multiple targets / multiple queries to reduce the number of queries or target items.

[0248] In one possible implementation, the goal completion rate is negatively correlated with the goal response rate. The goal response rate is used to characterize the number of goal items in a single response during a dialogue. Thus, when there are too many goal items, the goal completion rate decreases, reducing the likelihood of an excessive number of goal items in a single instance.

[0249] For example, when the target response rate is greater than or equal to a second preset threshold, the target completion rate can be negatively correlated with the target response rate. This allows for control over the number of target items in a single response, reducing the likelihood of a single response containing too many target items.

[0250] In some embodiments, the target response rate can be the average number of target items in a single response during the dialogue. For example, the target response rate can be the ratio between the number of target items and the number of responses. Thus, when there are too many target items, a higher average number results in a lower target completion rate, reducing the likelihood of too many target items in a single response.

[0251] In other embodiments, the target response rate can be the ratio between the number of target items appearing in a single response and the total number of target items in the dialogue objective. This allows for optimization of a single response, reducing the likelihood of a single response including too many target items.

[0252] Understandably, the criteria for achieving the target item can be determined using a large model or any other method. For example, a template used for large model scoring could be as follows:

[0253] "You are a dialogue task receiver goal completion judge, mainly responsible for determining whether the receiver's goal has been completed. Below is a dialogue history and the receiver's latest reply. Please use the dialogue history to determine whether each goal item in the receiver's goals has been completed in the receiver's latest reply."

[0254]

definition

[0255] The receiver's goal: In a conversation, the receiver engages in the dialogue to achieve their own goal; this is the objective the receiver needs to accomplish during the conversation.

[0256]

enter

[0257] [Dialogue with History]

[0258] {dialogue}

[0259] [Latest response from the caller]

[0260] {response}

[0261] [Target of the recipient]

[0262] {goals}

[0263]

Require

[0264] You need to read and understand the conversation history. Based on the conversation history, please determine whether each goal item in the receiver's goals has been addressed in the receiver's latest response. The output format is as follows; please avoid any unnecessary output.

[0265] Target x: Yes or No. Adaptively, the task response rate can be the ratio between the number of targets that are "yes" and the total number of target items (elements).

[0266] For example, the second preset threshold can be zero. When the target response rate is greater than zero, the target completion rate is negatively correlated with the target response rate; when the target response rate is zero, the target completion rate is zero. The formula can be as follows:

[0267] The function `goal_completion_rate = 1 - goal_rate(response)` is defined if `goal_rate(response)` > 0, and `goal_completion_rate = 0` if `goal_rate(response)` = 0. Here, `goal_completion_rate` represents the goal completion rate, and `goal_rate(response)` represents the goal response rate.

[0268] Thus, if a single response includes multiple objectives, the objective completion rate decreases; a penalty is added to responses with multiple objectives or multiple objective items to reduce the number of objective items.

[0269] The above embodiments illustrate the training of the reward model. The training samples can also be obtained by processing call recording data.

[0270] Compared to manually constructed data, dialogue targets derived from real call recordings are more consistent with user expression habits and actual conversation logic. Thus, using real-world call recordings as training samples improves the agent's ability to understand users' true intentions, expression methods, and language habits. Training with this data enhances the model's accuracy and generalization ability in realistic, complex, and varied call environments.

[0271] The following is combined with Figure 8 and Figure 9 The process of converting a recorded call into a dialogue target is explained.

[0272] For example, Figure 8 This is a schematic diagram illustrating a call recording process provided in an embodiment of this application. Figure 8 As shown, the process includes:

[0273] S801. Convert the call recording data into dialogue text labeled with the dialogue characters.

[0274] Dialogue roles are used to distinguish different tasks. For example, they can be the caller and the receiver. No specific restrictions are set here.

[0275] S802. Generate the caller's target, caller's inquiry, receiver's target, and receiver's inquiry based on the dialogue text, and supplement any missing entity elements. Detect the task completion rate for the generated targets and inquiries, and regenerate the targets and inquiries based on whether the task completion rate meets the requirements.

[0276] S803. Generate a dialogue summary based on the dialogue text, calculate the similarity between the summary and the list of vertical domains, and use the top M vertical domains and the dialogue text to generate the main domain and sub-domains corresponding to the dialogue.

[0277] S804. Generate a dialogue target based on the generated main domain, subdomain, target item, and query item.

[0278] In this way, the recordings can be converted into dialogue goals used to train the agent.

[0279] For example, Figure 9 This is a schematic diagram of the structure of a training data generation model provided in an embodiment of this application. For example... Figure 9 As shown, the generative model can be achieved by combining nine LLM models.

[0280] The following is combined with Figure 9 The process of converting call recording data into dialogue targets is described. This process may include steps one through four.

[0281] Step 1: Rewrite the call recording into dialogue text that distinguishes different roles.

[0282] Specifically, the call recording data is converted into recorded text using the ASR model. Then, the recorded text is converted into dialogue-style text using the LLM1 model, which distinguishes different roles.

[0283] In some embodiments, the same recorded text can be processed multiple times using the LLM1 model to obtain multiple dialogue texts. For example, with two processing steps, dialogue text 1 and dialogue text 2 can be obtained. In this way, multiple dialogue texts can be processed subsequently, improving robustness.

[0284] Step 2: Quality screening of the dialogue text.

[0285] Specifically, the dialogue text is input into the LLM2 model to obtain the filtering results.

[0286] Adaptively, if the filtering results meet the preset conditions, proceed to step three of the dialogue text processing. If the filtering results do not meet the preset conditions, the dialogue text is manually modified. The modified dialogue text can then proceed to step three of the processing.

[0287] Taking the screening results as a scoring example, the preset condition can be greater than or equal to the third threshold. The third threshold can be 0.5, 0.8, or any value; there is no specific limitation here.

[0288] For example, the LLM2 model can perform quality screening based on relevance, usability, accuracy, sentiment, dialogue logic, and coherence. Relevance characterizes the degree of connection between contexts. Usability characterizes the richness of the response information. Accuracy characterizes the precision of the information. Clarity characterizes whether the response is clear and easy to understand.

[0289] In this embodiment of the application, quality screening can also be performed based on dimensions such as words, sentences, and word order. No specific limitations are set here.

[0290] Step 3: Extract the query items, target items (also known as target response items), and entity elements from the dialogue text.

[0291] For example, the LLM3 model can be used to extract the query items and target items from the dialogue text; the LLM4 model can be used to extract the elements from the dialogue text, obtaining the caller's elements and the receiver's elements.

[0292] In some embodiments, the query items and target items can be further extracted using the LLM5 model to obtain the caller's target item, the caller's query item, the receiver's target item, and the receiver's query item, etc.

[0293] By extracting data multiple times, the accuracy of both the query and target items can be improved.

[0294] If the feature information meets preset condition 2, the extraction ends; if the feature information does not meet preset condition 2, step three is repeated until the feature information meets preset condition 2 or the number of repetitions N is reached. The number of repetitions N can be 5, 4, or any value, and is not specifically limited here.

[0295] Preset condition 2 may include: the task completion rate reaching the fourth threshold and / or the role not matching. The task completion rate can be referred to the corresponding explanation above, and is not specifically limited here. The fourth threshold can be 0.5, 0.6, or any value, and is not specifically limited here.

[0296] In some embodiments, the determination of preset condition 2 can be achieved through the LLM6 model.

[0297] Step 4: Vertical Matching

[0298] Specifically, the LLM7 model is used to extract summaries from the dialogue text. The extracted summaries are then compared with a list of vertical regions to obtain the top M similarity-ranked vertical regions. M can be 10, 5, or any other value; no specific limitation is made here.

[0299] Input the top M vertical domains with the highest similarity ranking and the dialogue text into the LLM8 model to obtain the main domain and subdomain corresponding to the dialogue text.

[0300] Step 5: Generate Dialogue Objectives

[0301] Input the main domain, subdomain, target item, and query item into the LLM9 model to obtain the dialogue target.

[0302] It should be understood that if the agent is applied to the receiver, the main domain, subdomain, receiver's target item, and receiver's query item can be input into the LLM9 model to obtain the corresponding dialogue target for the receiver. If the agent is applied to the caller, the main domain, subdomain, caller's target item, and caller's query item can be input into the LLM9 model to obtain the corresponding dialogue target for the caller.

[0303] Thus, the LLM model described above can realize the conversion process from call recording text to custom dialogue text and dialogue templates, including dialogue role conversion, target and query conversion, vertical and sub-vertical conversion, and dialogue target generation.

[0304] It should be understood that, in addition to the LLM model mentioned above, the above-mentioned process can also be achieved through any other model with the same function or in any other way that can achieve the same function, without any specific limitations here.

[0305] It should be understood that the above-described model training process can be performed in an electronic device or in a server. In some embodiments, the model training process can also be performed jointly by a server and an electronic device. For example, the server can undertake the main computational work, and the electronic device can undertake the secondary computational work; or, the server can undertake the secondary computational work, and the electronic device can undertake the main computational work. This application embodiment does not limit this.

[0306] A server can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed file system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0307] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0308] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0309] The methods provided in the embodiments of this application have been described above. The apparatus for performing the above methods, provided in the embodiments of this application, is described below. Those skilled in the art will understand that the methods and apparatus can be combined and referenced together, and the related apparatus provided in the embodiments of this application can perform the steps in the above list sorting method.

[0310] The method provided in this application can be applied to electronic devices with communication functions. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-described related descriptions, which will not be repeated here.

[0311] This application provides an electronic device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the electronic device to perform the above-described method.

[0312] This application provides an interactive system, which includes: an electronic device and a peer device; the peer device is used to send input information to the electronic device; the electronic device is used to implement the above method.

[0313] This application provides a chip. The chip includes a processor, which calls a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0314] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0315] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0316] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0317] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0318] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. An intelligent interaction method based on end-to-end multimodal instruction understanding, characterized in that, include: Obtain input information from the peer device; An intelligent agent analyzes the input information and pre-set dialogue objectives to obtain response content for the input information. The dialogue objectives indicate user intent and include a main domain, sub-domain, target item, and query item. The intelligent agent is trained based on Reinforcement Learning High-Frequency Learning (RLHF) incorporating human feedback. The reward model in RLHF is used to evaluate the response content output by the intelligent agent based on a first metric and a second metric. The first metric reflects dialogue quality, and the second metric reflects the completion of the dialogue objectives. The first metric includes any one or more of the following: relevance, dialogue logic, information retrieval completeness, naturalness, and conciseness. The second indicator includes inquiry completion rate and target completion rate, wherein the relevance is used to characterize the degree of association between the response content and the dialogue target; the dialogue logic is used to characterize the clarity of the language logic of the response content; the information acquisition completeness is used to characterize the comprehensiveness of the elements corresponding to the dialogue target during the dialogue process; the naturalness is used to characterize the fluency of the dialogue process; the conciseness is used to characterize the refinement of the response content; the inquiry completion rate is used to characterize the completion rate of the inquiry items in the dialogue target during the dialogue process; and the target completion rate is used to characterize the completion rate of the target items in the dialogue target during the dialogue process. Send the reply content to the peer device; When the query response rate is greater than zero, the query completion rate is negatively correlated with the query response rate; when the query response rate is zero, the query completion rate is zero; wherein, the query response rate is used to characterize the ratio between the number of queries from the dialogue target appearing in a single response and the number of queries from the dialogue target. When the target response rate is greater than zero, the target completion rate is negatively correlated with the target response rate; when the target response rate is zero, the target response rate is zero; wherein, the target response rate is used to characterize the ratio between the number of target items in the dialogue target appearing in a single response and the number of target items in the dialogue target.

2. The method according to claim 1, characterized in that, The reward model's evaluation result of the response content is obtained by weighting the indicators in the first indicator and the indicators in the second indicator.

3. The method according to claim 2, characterized in that, Before weighted calculation, the reward model is also used to normalize the indicators in the first indicator and / or the indicators in the second indicator.

4. The method according to claim 2 or 3, characterized in that, The weights of each indicator in the second indicator are greater than the weights of each indicator in the first indicator.

5. The method according to any one of claims 1-3, characterized in that, For the same dialogue objective, the reward model is trained based on at least two responses corresponding to the dialogue objective.

6. The method according to claim 5, characterized in that, The at least two responses are obtained by processing the dialogue target based on different types of large models.

7. The method according to claim 6, characterized in that, The reward model is trained based on the content of the first reply and the content of the second reply; The first response is the response with the highest rating among at least two responses corresponding to the dialogue target; The second response is the response with the lowest score among at least two responses corresponding to the dialogue target.

8. The method according to claim 7, characterized in that, The difference between the score corresponding to the first response and the score corresponding to the second response is greater than or equal to the second threshold.

9. The method according to any one of claims 1-3, characterized in that, The intelligent agent is obtained based on training data, which includes: dialogue target, dialogue text, and elements.

10. The method according to claim 9, characterized in that, The dialogue objective is derived from the conversion of call recordings.

11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

13. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-10.

15. An interactive system, characterized in that, include: Electronic devices and peer devices; The peer device is used to send input information to the electronic device; The electronic device is used to implement the method as described in any one of claims 1-10.

Images