Conversation generation method and device based on virtual object, equipment and storage medium

Abstract

The embodiment of the invention provides a dialogue generation method and device based on a virtual object, equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the field of financial science and technology. The method comprises the steps of generating initial reply content for an initial question text based on a preset dialogue generation model, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter; dialogue feedback data and target feedback reply content matched with the dialogue feedback data are obtained; updating the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter; generating prediction feedback reply content based on a preset dialogue generation model and the target learning rate parameter; and on the basis of the target learning rate parameter, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, updating the initial model parameter to obtain a current dialogue generation model. According to the embodiment of the invention, the behavior and style of the virtual object for generating the dialogue content can be flexibly adjusted, and the dialogue reply accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, is applicable to the field of financial technology, and in particular, to a method and apparatus, device, and storage medium for generating a dialogue based on a virtual object. Background Art

[0002] Artificial Intelligence (AI)-based character dialogue refers to the technology of using AI-generated virtual characters to simulate human conversation behavior. In addition, AI-based virtual character dialogue can be applied to task scenarios such as virtual assistants in financial technology, customer service systems, and game character dialogues. The core of virtual object-based character dialogue lies in its ability to simulate human language communication, by analyzing user input and generating appropriate responses, so as to conduct continuous multi-round dialogues with the target object.

[0003] At present, when generating character dialogues based on virtual objects, related technologies usually respond to questions input by the target object through preset dialogue templates, rules or learned dialogue strategies. However, since the dialogue behaviors (such as the reply text generated based on the questions input by the target object) and dialogue styles of these virtual characters usually use fixed parameters set in the initial training stage, the dialogue content replied by the virtual object is not accurate enough and lacks flexibility, making it difficult to better adapt to the changing dialogue needs of the target object. Therefore, how to propose a dialogue generation method based on virtual objects that can flexibly adjust the dialogue behavior and dialogue style of virtual objects and improve the accuracy of dialogue responses has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device, equipment, and storage medium for generating dialogues based on virtual objects, which can flexibly adjust the behavior and style of the generated dialogue content of the virtual objects and improve the accuracy of dialogue responses.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a dialogue based on a virtual object, the method comprising:

[0006] Get the target object's initial question text;

[0007] Performing dialogue generation on the initial question text based on a preset dialogue generation model to obtain initial answer content, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter, and the initial learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the initial answer content by the preset dialogue generation model;

[0008] Acquire the dialogue feedback data fed back by the target object in response to the initial reply content, and acquire the target feedback reply content matching the dialogue feedback data;

[0009] Update the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter;

[0010] Performing dialogue generation on the dialogue feedback data based on the preset dialogue generation model and the target learning rate parameter to obtain predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model;

[0011] Based on the target learning rate parameter, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, the initial model parameters are updated to obtain a current dialogue generation model, and dialogue generation is performed according to the current dialogue generation model.

[0012] In some embodiments, the updating of the initial model parameters based on the target learning rate parameter, the dialogue feedback data, the target feedback reply content, and the predicted feedback reply content to obtain the current dialogue generation model includes:

[0013] Acquire a predicted feedback reply sequence from the predicted feedback reply content;

[0014] constructing an initial loss function based on the predicted feedback reply sequence, the dialogue feedback data and the initial model parameters;

[0015] Construct a target reward function based on dialogue feedback data;

[0016] Performing function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value;

[0017] Constructing a model loss function based on the initial loss function, the target reward value, and a preset reward weight of the target reward value;

[0018] Model parameters are updated based on the model loss function, the target learning rate parameter, and the initial model parameter to obtain the current dialogue generation model.

[0019] In some embodiments, constructing an initial loss function based on the predicted feedback reply sequence, the dialogue feedback data, and the initial model parameters includes:

[0020] Constructing a time step interval based on a target time step matched by the prediction feedback reply sequence, wherein the target time step is an interval endpoint of the time step interval;

[0021] Obtaining a sequence of predicted feedback reply intervals from the predicted feedback reply content based on the time step intervals;

[0022] Performing time step prediction based on the predicted feedback reply sequence, the predicted feedback reply interval sequence, the dialogue feedback data and the initial model parameters to obtain a predicted output probability of the target time step, wherein the predicted output probability is used to indicate a matching degree of the predicted feedback reply sequence predicted by the preset dialogue generation model based on the target time step;

[0023] The predicted output probability is logarithmically calculated based on the target time step and the total number of preset time steps to obtain the initial loss function.

[0024] In some embodiments, the updating of model parameters based on the model loss function, the target learning rate parameter, and the initial model parameter to obtain the current dialogue generation model includes:

[0025] Performing gradient calculation on the initial model parameters based on the model loss function to obtain a gradient loss function;

[0026] Performing function adjustment on the gradient loss function based on the target learning rate to obtain an adjusted loss function;

[0027] Performing model parameter updates on the initial model parameters based on the adjusted loss function to obtain target model parameters;

[0028] The preset dialogue generation model is adjusted based on the target model parameters to obtain the current dialogue generation model.

[0029] In some embodiments, the preset dialogue generation model includes a preset strategy network and a preset value network, and the step of adjusting the preset dialogue generation model based on the target model parameters to obtain the current dialogue generation model includes:

[0030] extracting a cumulative reward value from the target model parameters;

[0031] Performing gradient calculation on the cumulative reward value to obtain a cumulative reward gradient value;

[0032] Extracting initial strategy parameters and initial value parameters from the initial model parameters;

[0033] Performing parameter updates on the initial strategy parameters based on the cumulative reward gradient value and the target learning rate parameter to obtain target strategy parameters;

[0034] Performing parameter updates on the initial value parameter based on the cumulative reward gradient value and the target learning rate parameter to obtain a target value parameter;

[0035] Adjusting the parameters of the preset strategy network based on the target strategy parameters to obtain an adjusted strategy network;

[0036] Adjusting the parameters of the preset value network based on the target value parameters to obtain an adjusted value network;

[0037] The current dialogue generation model is constructed based on the adjusted policy network and the adjusted value network.

[0038] In some embodiments, performing function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value includes:

[0039] Calculating the similarity between the target feedback reply content and the predicted feedback reply content to obtain a feedback reply similarity value;

[0040] Performing feedback data detection on the dialogue feedback data based on a preset feedback text detection model to obtain an object feedback score;

[0041] Performing emotion detection on the dialogue feedback data based on the object feedback score to obtain an object emotion score;

[0042] The target reward value is obtained by performing a weighted calculation on the feedback reply similarity value, the object feedback score and the object emotion score.

[0043] In some embodiments, updating the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter includes:

[0044] extracting the dialogue interaction time and dialogue interaction frequency from the dialogue feedback data;

[0045] Performing interaction time detection on the dialogue feedback data based on the dialogue interaction time to obtain a dialogue interaction time score;

[0046] Performing interaction frequency detection on the dialogue feedback data based on the dialogue interaction frequency to obtain a dialogue interaction frequency score;

[0047] Performing feedback score calculation on the object emotion score, the conversation interaction time score, and the conversation interaction frequency score to obtain an implicit feedback score;

[0048] Calculating the difference between the preset expected feedback score of the target object and the implicit feedback score to obtain a feedback score difference;

[0049] The initial learning rate parameter is updated based on the feedback score difference to obtain the target learning rate parameter.

[0050] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a virtual object-based dialogue generation device, the system comprising:

[0051] A first acquisition module is used to acquire the initial question text of the target object;

[0052] A first generation module is used to generate a dialogue for the initial question text based on a preset dialogue generation model to obtain an initial answer content, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter, and the initial learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the initial answer content;

[0053] A second acquisition module is used to acquire the dialogue feedback data fed back by the target object to the initial reply content, and acquire the target feedback reply content matching the dialogue feedback data;

[0054] A learning rate parameter updating module, used to update the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter;

[0055] a second generation module, configured to perform dialogue generation on the dialogue feedback data based on the preset dialogue generation model and the target learning rate parameter to obtain predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model;

[0056] A parameter updating module is used to update the initial model parameters based on the target learning rate parameters, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, to obtain a current dialogue generation model, and to generate a dialogue according to the current dialogue generation model.

[0057] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements any method proposed in the first aspect of the embodiment of the present application.

[0058] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application further proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method proposed in the first aspect of the embodiments of the present application.

[0059] The virtual object-based dialogue generation method, apparatus, device, and storage medium proposed in the embodiments of the present application obtain an initial question text of a target object, and generate a dialogue for the initial question text based on a preset dialogue generation model to obtain an initial reply content, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter, and the initial learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the initial reply content; further, dialogue feedback data fed back by the target object to the initial reply content is obtained, and target feedback reply content matching the dialogue feedback data is obtained; further, the initial learning rate parameter is updated based on the dialogue feedback data to obtain a target learning rate parameter; dialogue generation is performed on the dialogue feedback data based on the preset dialogue generation model and the target learning rate parameter to obtain a predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the predicted feedback reply content; further, the initial model parameters are updated based on the target learning rate parameter, the dialogue feedback data, the target feedback reply content, and the predicted feedback reply content to obtain a current dialogue generation model, and dialogue generation is performed according to the current dialogue generation model. Compared with the related art that generates dialogues through preset dialogue templates, rules or learned dialogue strategies, it is easy for the dialogue content replied by the virtual object to be inaccurate and lack flexibility, making it difficult to better adapt to the target object's changing dialogue needs. When generating dialogues, the present application can update the learning rate parameters of the preset dialogue generation model according to the dialogue feedback data of the target object on the reply content, thereby adjusting the content style and content behavior of the reply content generated by the preset dialogue generation model, and adjusting the preset dialogue generation model in combination with the updated learning rate parameters to obtain the current dialogue generation model, so as to generate dialogues for subsequent question texts. Therefore, the present application can flexibly adjust the behavior and style of the generated dialogue content of the virtual object and improve the accuracy of the dialogue reply. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a method for generating a dialogue based on a virtual object provided in an embodiment of the present application;

[0061] Figure 2 yes Figure 1 A flowchart of the specific method of step S140;

[0062] Figure 3 yes Figure 1 A flowchart of the specific method of step S160;

[0063] Figure 4 yes Figure 3 A flowchart of the specific method of step S320;

[0064] Figure 5 yes Figure 3 A flowchart of the specific method of step S340;

[0065] Figure 6 yes Figure 3 A flowchart of the specific method of step S360;

[0066] Figure 7 yes Figure 6 A flowchart of the specific method of step S640;

[0067] Figure 8 It is a module structure block diagram of a virtual object-based dialogue generation device provided in an embodiment of the present application;

[0068] Fig. 9 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] First, some nouns involved in this application are analyzed:

[0073] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. Natural language processing involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0075] Large Language Model (LLM): It is an artificial intelligence technology based on deep learning. It learns the grammar, semantics and contextual information of the language by training on a large amount of text data, so that it can model natural language text.

[0076] Artificial Intelligence (AI)-based character dialogue refers to the technology of using AI-generated virtual characters to simulate human conversational behavior. An AI character dialogue system is a computer program that can simulate human conversational behavior, usually driven by NLP and machine learning technologies. These systems generate conversation content through pre-trained language models and exhibit specific character traits and behaviors when interacting with objects. In addition, dialogues based on AI virtual characters can be applied to task scenarios such as virtual assistants in financial technology, customer service systems, and game character dialogues to provide personalized and humanized interactive experiences. The core of character dialogues based on virtual objects lies in its ability to simulate human language communication, by analyzing user input and generating appropriate responses, so as to conduct continuous multi-round dialogues with the target object.

[0077] At present, when generating character dialogues based on virtual objects, related technologies usually respond to questions input by the target object through preset dialogue templates, rules or learned dialogue strategies. However, since the dialogue behaviors (such as the reply text generated based on the questions input by the target object) and dialogue styles of these virtual characters usually adopt fixed parameters set in the initial training stage, the dialogue content replied by the virtual object is not accurate enough and lacks flexibility, making it difficult to better adapt to the changing dialogue needs of the target object. In addition, when dealing with emotionally rich or highly personalized dialogues, the existing systems often show the weakness of lack of anthropomorphism and cannot meet the expectations of the object for natural interaction. In this way, the AI ​​character dialogue system adopted by the related technology has obvious deficiencies in flexibility, naturalness and meeting the personalized needs of the object. Therefore, how to propose a dialogue generation method based on virtual objects that can flexibly adjust the dialogue behavior and dialogue style of virtual objects and improve the accuracy of dialogue responses has become a technical problem that needs to be solved urgently.

[0078] Based on this, the virtual object-based dialogue generation method, device, equipment, and storage medium provided in the embodiments of the present application can flexibly adjust the behavior and style of the virtual object's generated dialogue content and improve the accuracy of dialogue responses.

[0079] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0080] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] The virtual object-based dialogue generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The virtual object-based dialogue generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements a virtual object-based dialogue generation method, etc., but is not limited to the above forms.

[0082] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0083] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as the object's reply information, the object's historical data, and the object's identity information, the object's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the sensitive personal information of the object, the object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the object's separate permission or separate consent, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0084] Please refer to Figure 1 , Figure 1It is an optional flowchart of the virtual object-based dialogue generation method provided in an embodiment of the present application. In some embodiments of the present application, the virtual object-based dialogue generation method of the embodiment of the present application includes but is not limited to steps S110 to S160.

[0085] Step S110, obtaining the initial question text of the target object;

[0086] Step S120, performing dialogue generation on the initial question text based on a preset dialogue generation model to obtain initial answer content;

[0087] Step S130, obtaining dialogue feedback data fed back by the target object in response to the initial reply content, and obtaining target feedback reply content matching the dialogue feedback data;

[0088] Step S140, updating the initial learning rate parameter based on the dialogue feedback data to obtain the target learning rate parameter;

[0089] Step S150, performing dialogue generation on the dialogue feedback data based on a preset dialogue generation model and a target learning rate parameter to obtain predicted feedback reply content;

[0090] Step S160, based on the target learning rate parameters, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, the initial model parameters are updated to obtain the current dialogue generation model, and dialogue generation is performed according to the current dialogue generation model.

[0091] In steps S110 to S160 of some embodiments, the present application can generate dialogue content through a preset dialogue generation model, and adjust the model's output of the content behavior and content style of the AI ​​character through dialogue feedback data fed back by the target object to the initial reply content. In other words, the present application can automatically optimize the preset dialogue generation model during the interaction process, so that the AI ​​character can better adapt to the personalized needs and preferences of the target object, thereby providing a more natural and coherent interactive experience.

[0092] In step S110 of some embodiments, the initial question text refers to the question information input by the target object in the application system with virtual object dialogue. For example, in the insurance field of financial technology, the target object is a user who consults insurance products (such as social insurance, medical insurance, etc.), then the target object can exchange information with the intelligent robot (i.e., the virtual object set in the insurance system) in the insurance system, and the information input by the target object is equivalent to the initial question text. At this time, the initial question text may include the demand questions that the target object wants to consult, the target object's reply content, etc., without specific limitation.

[0093] In step S120 of some embodiments, the preset dialogue generation model is an artificial intelligence technology specially designed to simulate the interactive process of human dialogue. This model can understand and generate natural language to achieve effective communication with the object. The preset dialogue generation model used in this application refers to a pre-defined model that can generate dialogue responses based on the input question text.

[0094] Among them, the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter. The initial learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the initial reply content, that is, the initial learning rate parameter is a hyperparameter that controls the parameter update speed during the model training process. During the learning process, the learning rate determines the magnitude of the model weight update. The initial model parameter refers to the parameter setting obtained by the preset dialogue generation model after the previous training is completed, and these parameters determine the current behavior and performance of the preset dialogue generation model. The content style associated with the initial reply content refers to the style of the reply content currently generated by the preset dialogue generation model, such as formal, informal, humorous, etc. The content behavior associated with the initial reply content refers to the behavior of the reply content currently generated by the preset dialogue generation model, such as directly answering questions, providing suggestions, asking for more information, etc. In addition, the degree of adjustment refers to how the initial learning rate parameter affects the model's adjustment of the reply content style. Moreover, a higher initial learning rate parameter may result in a larger change in the style of the model, while a lower initial learning rate parameter may result in more subtle adjustments.

[0095] It is understandable that when the preset dialogue generation model processes the initial question text input by the target object, it will generate the reply content according to the preset initial learning rate parameters and initial model parameters. The initial learning rate parameters are used to influence how the model adjusts its style and behavior of generating replies, that is, it determines the sensitivity or amplitude of the model's adjustment of style and behavior when generating replies, so as to control the model's response to the input question when generating replies, so that it is more in line with the expected dialogue style and behavior.

[0096] It should be noted that when generating the reply content, the preset dialogue generation model can first understand the initial question text entered by the target object, which usually involves the understanding of the grammar, semantics and context of the language. Furthermore, the preset dialogue generation model can maintain the state of the dialogue, understand the flow of the dialogue, and decide what actions should be taken at each stage of the dialogue. In order to make the dialogue coherent, the preset dialogue generation model can record the historical dialogue between the target object and the virtual object, and construct smooth, natural and relevant reply content. Among them, the preset dialogue generation model adopted in this application can be constructed based on the model structure of deep learning, such as LLM, Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), Transformer, etc., which are not specifically limited here.

[0097] It should be noted that when the preset dialogue generation model has not been adjusted, the initial learning rate parameter of the present application can be a fixed parameter, and the parameter can be set to any value between 0.001 and 0.01, and can be selected through cross-validation or experiments to ensure the stability of the preset dialogue generation model in the initial stage.

[0098] In step S130 of some embodiments, further, after the target object obtains the initial reply content, the present application can collect and analyze the dialogue feedback data of the target object, that is, user feedback. The dialogue feedback data at this time can be positive, negative or neutral, and the dialogue feedback data can be obtained through direct input of the target object (such as ratings, comments) or through indirect interaction data (such as user's emotional changes and response time, etc.), etc., to obtain the evaluation, rating, further question text or other forms of response of the target object. The target feedback reply content refers to the matching reply content queried in the preset content library based on the object feedback data, as the reply content expected by the target object.

[0099] It should be noted that in actual applications, the dialogue generation system can actively or passively collect feedback from the target object on the performance of the AI ​​character after each interaction or within a specific time interval, that is, obtain dialogue feedback data, so as to understand the user's satisfaction and acceptance of the initial response content. Furthermore, the behavior and dialogue style of the AI ​​character can be dynamically adjusted based on these object feedback data, enhancing the naturalness and flexibility of the character's interaction.

[0100] In step S140 of some embodiments, further, the present application can update the initial learning rate parameter based on the dialogue feedback data to obtain the target learning rate parameter. That is, the present application can use the dialogue feedback data as the basis for adjusting the learning rate parameter. The target learning rate parameter is the updated new learning rate parameter, which reflects the adjustment made by the preset dialogue generation model according to the dialogue feedback data, so that the preset dialogue generation model can better adapt to the preferences and needs of the target object and improve the quality and relevance of the dialogue. In addition, by lowering or increasing the learning rate, the system can control the speed of updating the model parameters, thereby affecting the model's adaptation speed to new information and the effect of dialogue generation.

[0101] For example, if user feedback shows that the response generated by the conversation is not accurate or relevant enough, the system may increase the learning rate to adjust the model parameters more quickly and improve the accuracy of the response. On the contrary, if the response is good enough, the system may reduce the learning rate to avoid over-adjustment and maintain the stability of the model. It can be seen that too large a learning rate may cause the model to oscillate near the optimal solution, or in extreme cases, cause the model to diverge. Although a learning rate that is too small can ensure that the model eventually converges, it will greatly reduce the speed of model training, but may cause the model to fall into a local optimal solution. In this way, by accurately adjusting the initial learning rate parameters, the preset conversation generation model can be automatically optimized and the user experience can be improved.

[0102] Please refer to Figure 2 , Figure 2 1 is a flowchart of a specific method of step S140 provided in an embodiment of the present application. In some embodiments of the present application, step S140 may specifically include but is not limited to steps S210 to S260.

[0103] Step S210, extracting the dialogue interaction time and the dialogue interaction frequency from the dialogue feedback data;

[0104] Step S220, performing interaction time detection on the dialogue feedback data based on the dialogue interaction time to obtain a dialogue interaction time score;

[0105] Step S230, performing interaction frequency detection on the dialogue feedback data based on the dialogue interaction frequency to obtain a dialogue interaction frequency score;

[0106] Step S240, calculating the feedback score of the object emotion score, the dialogue interaction time score and the dialogue interaction frequency score to obtain an implicit feedback score;

[0107] Step S250, performing a difference calculation on the preset expected feedback score and the implicit feedback score of the target object to obtain a feedback score difference;

[0108] Step S260, updating the initial learning rate parameter based on the feedback score difference to obtain the target learning rate parameter.

[0109] In step S210 of some embodiments, when updating the learning rate parameter, the present application can determine whether the current reply content reaches the expected result of the target object by introducing implicit feedback scoring to dynamically adjust the learning rate parameter. Specifically, the dialogue interaction time and dialogue interaction frequency between the target object and the dialogue system can be extracted from the dialogue feedback data. Among them, if the target object spends a long time in the dialogue process with the dialogue system, it can be indicated that the target object has a high degree of acceptance of the reply content generated by the dialogue system, otherwise it indicates that the target object is dissatisfied with the generated reply content. The frequency of dialogue between the target object and the dialogue system can also reflect implicit feedback. For example, a higher dialogue frequency indicates that the target object is interested in the reply content generated by the system, and a lower frequency means that the generated reply content does not meet expectations. In this way, these data can be used to analyze the target object's participation and the fluency of the dialogue.

[0110] In step S220 of some embodiments, the present application may further evaluate the time taken by the target object to respond to the system, i.e., the dialogue interaction time. For example, the dialogue interaction time indicates how often the target object replies to the reply content. A score may be further given based on this time, i.e., the dialogue interaction time score, which reflects the timeliness of the target object's response.

[0111] In step S230 of some embodiments, the present application may further provide a score for the conversation interaction frequency, namely, a conversation interaction frequency score, which reflects the activity and participation of the target object.

[0112] In step S240 of some embodiments, the present application may further perform feedback score calculation on the object emotion score, conversation interaction time score, and conversation interaction frequency score to obtain an implicit feedback score.

[0113] It should be noted that the object emotion score is a score determined after emotion detection of the dialogue feedback data. By analyzing the emotional information of the target object (such as facial expression, tone, text emotion analysis, etc.), the emotional state of the target object during the dialogue can be judged. If the emotion tends to be negative, it means that the target object is not satisfied with the content of the dialogue response, otherwise it means that the target object is satisfied. The specific calculation process of the object emotion score will be explained in the subsequent embodiments and will not be repeated here.

[0114] It should be noted that the specific calculation process of the implicit feedback score obtained by weighted calculation of the object emotion score, dialogue interaction time score and dialogue interaction frequency score is as follows: Implicit feedback score = w1*object emotion score + w2*dialogue interaction time score + w3*dialogue interaction frequency score. Among them, w1 is the emotion analysis weight corresponding to the object emotion score, which indicates the influence of the object emotion score determined by the emotional state (positive or negative) of the target object on the implicit feedback score; w2 is the interaction time weight corresponding to the dialogue interaction time score, which indicates the influence of the dialogue interaction time score on the implicit feedback score; w3 is the interaction time weight corresponding to the dialogue interaction time score, which indicates the influence of the dialogue interaction frequency score on the implicit feedback score.

[0115] In step S250 of some embodiments, the preset expected feedback score is a preset target object's expected feedback score for the generated dialogue content, which is equivalent to a score threshold, for example, a high score corresponds to a smaller learning rate change, and a low score corresponds to a larger learning rate change. In this way, the present application can compare the preset expected feedback score with the actual implicit feedback score and calculate the difference, which can be used to evaluate the difference between the system performance and the target object's expectations.

[0116] In step S260 of some embodiments, the initial learning rate parameter may be further adjusted according to the feedback score difference to obtain a new target learning rate parameter. If the difference is large, it may mean that the model needs to be adjusted faster to meet the expectations of the target object, and the learning rate parameter may be increased; if the difference is small, the model may need to be adjusted more stably to avoid overreaction, and the learning rate parameter may be reduced.

[0117] It should be noted that the process of calculating the target learning rate parameter can be shown in the following formula 1:

[0118]

[0119] Among them, α represents the target learning rate parameter, α0 represents the initial learning rate parameter, and β represents the preset adjustment coefficient, which is used to control the sensitivity of the learning rate parameter change and can be adjusted according to actual needs; f target represents the preset expected feedback score, f actual Represents an implicit feedback rating.

[0120] In step S150 of some embodiments, the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model based on the adjusted learning rate parameter. That is, the application can use the adjusted learning rate parameter to regenerate the predicted feedback reply content corresponding to the dialogue feedback data to predict the reply that the target object may expect.

[0121] In step S160 of some embodiments, the present application may further combine the target learning rate parameter, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, adjust the initial model parameters, and generate a current dialogue generation model. Then, the current dialogue generation model is used to continue dialogue generation, improve the accuracy and adaptability of the model, and generate a reply that better meets the expectations of the target object.

[0122] In the above embodiment, the present application can analyze the collected object feedback through natural language processing technology, and understand the emotions, needs and preferences of user feedback through a large language model. Then, a reinforcement learning method is used to optimize the generation parameters of the preset dialogue generation model, so that the behavior and dialogue style of the AI ​​character can better adapt to the personalized needs of users.

[0123] In some embodiments, the present application may further optimize the model by optimizing the loss function of the preset dialogue generation model, that is, the loss function at this time may include a reward signal of user feedback.

[0124] Please refer to Figure 3 , Figure 3 1 is a flowchart of a specific method of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to steps S310 to S360.

[0125] Step S310, obtaining a predicted feedback reply sequence from the predicted feedback reply content;

[0126] Step S320, constructing an initial loss function based on the predicted feedback response sequence, the dialogue feedback data and the initial model parameters;

[0127] Step S330, constructing a target reward function based on the dialogue feedback data;

[0128] Step S340, performing function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value;

[0129] Step S350, constructing a model loss function based on the initial loss function, the target reward value and the preset reward weight of the target reward value;

[0130] Step S360, updating the model parameters based on the model loss function, the target learning rate parameters and the initial model parameters to obtain the current dialogue generation model.

[0131] In step S310 of some embodiments, the predicted feedback reply content of the present application refers to the reply content of a predicted feedback reply sequence containing T (T is a positive integer) time steps, and the predicted feedback reply sequence may include a series of elements for representing words, phrases or smaller language units for subsequent analysis and calculation.

[0132] In step S320 of some embodiments, an initial loss function may be constructed using the predicted feedback reply sequence, the dialogue feedback data, and the initial model parameters. This function is used to evaluate the difference between the model's prediction and the target object's actual feedback and guide the adjustment of the model parameters.

[0133] Please refer to Figure 4 , Figure 4 4 is a flowchart of a specific method of step S320 provided in an embodiment of the present application. In some embodiments of the present application, step S320 may specifically include but is not limited to steps S410 to S440.

[0134] Step S410, constructing a time step interval based on the target time step matched by the predicted feedback reply sequence;

[0135] Step S420, obtaining a predicted feedback reply interval sequence from the predicted feedback reply content based on the time step interval;

[0136] Step S430, performing time step prediction based on the predicted feedback reply sequence, the predicted feedback reply interval sequence, the dialogue feedback data and the initial model parameters to obtain the predicted output probability of the target time step;

[0137] Step S440, performing logarithmic calculation on the predicted output probability based on the target time step and the total number of preset time steps to obtain an initial loss function.

[0138] In step S410 of some embodiments, the time step matched by the predicted feedback reply sequence may be recorded as the target time step, for example, the predicted feedback reply sequence y t The corresponding target time step is time step t. At this time, the time step interval constructed based on the target time step is [1, t-1], and the target time step is the interval endpoint of the time step interval.

[0139] In step S420 of some embodiments, the present application may obtain a predicted feedback reply interval sequence from the predicted feedback reply content based on the time step interval, which includes a feedback reply sequence corresponding to each time step in the time step interval.

[0140] In step S430 and step S440 of some embodiments, further, the present application may perform time step prediction based on the predicted feedback reply sequence, the predicted feedback reply interval sequence, the dialogue feedback data and the initial model parameters to obtain the predicted output probability of the target time step, and the predicted output probability is used to indicate the degree of matching of the predicted feedback reply sequence predicted by the preset dialogue generation model based on the target time step. Further, the present application may perform logarithmic calculation on the predicted output probability based on the target time step and the total number of preset time steps to obtain an initial loss function. Among them, the specific calculation process of the present application to determine the predicted output probability of the target time step is shown in Formula 2 below:

[0141]

[0142] In Formula 2, L1(θ) represents the initial loss function, T represents the total number of preset time steps, and y t represents the prediction feedback reply sequence, y 1:t-1 represents the predicted feedback response interval sequence, x represents the dialogue feedback data, θ represents the initial model parameters, P(y t |y 1:t-1 ,x;θ) represents the time step y before the given 1:t-1 (i.e. from y1 to y t ) and input x, generate the current time step y t Here P is the probability distribution, i.e. the predicted output probability at the target time step.

[0143] In step S330 of some embodiments, the present application may further construct a target reward function based on the dialogue feedback data, which is used to evaluate the performance of the dialogue system and provide positive or negative feedback to motivate the model to develop in a better direction.

[0144] In step S340 of some embodiments, the present application may further perform function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value. In other words, the present application may determine the target reward value based on the object feedback data, and adjust the model in combination with the target reward value, so that the generated dialogue of the model is more in line with the expectations of the target object.

[0145] Please refer to Figure 5 , Figure 5 3 is a flowchart of a specific method of step S340 provided in an embodiment of the present application. In some embodiments of the present application, step S340 may specifically include but is not limited to steps S510 to S540.

[0146] Step S510, calculating the similarity between the target feedback reply content and the predicted feedback reply content to obtain a feedback reply similarity value;

[0147] Step S520, performing feedback data detection on the dialogue feedback data based on a preset feedback text detection model to obtain an object feedback score;

[0148] Step S530, performing emotion detection on the dialogue feedback data based on the object feedback score to obtain the object emotion score;

[0149] Step S540, performing weighted calculation on the feedback reply similarity value, the object feedback score and the object sentiment score to obtain a target reward value.

[0150] In step S510 of some embodiments, when determining the target reward value, the present application may consider the object feedback score and the sentiment positivity, that is, the higher the object feedback score, the more positive the sentiment analysis result, and the greater the reward value. Specifically, a similarity calculation method (such as cosine similarity, Jaccard similarity, or edit distance, etc.) can be used to quantify the similarity between the target feedback reply content and the predicted feedback reply content to obtain a feedback reply similarity value.

[0151] In step S520 of some embodiments, further, a preset feedback text detection model may be used to analyze the conversation feedback data. This model may be based on natural language processing technology to identify key features of the conversation feedback data and give a score to evaluate the quality or relevance of the conversation feedback data.

[0152] In step S530 of some embodiments, the present application may further utilize sentiment analysis technology to further analyze the sentiment tendency in the dialogue feedback data, such as positive, negative, or neutral, according to the object feedback score, and quantify an object sentiment score.

[0153] In step S540 of some embodiments, in order to comprehensively consider the similarity of the responses, the quality of the feedback, and the emotional tendency, the application may perform weighted calculation on the feedback response similarity value, the object feedback score, and the object emotional score to obtain a target reward value. This weighted calculation assigns different weights to each factor according to its importance, and ultimately obtains a target reward value. This reward value reflects the overall performance of the dialogue system in the dialogue interaction.

[0154] In step S350 of some embodiments, the present application may combine the initial loss function, the target reward value, and the preset reward weight to construct a model loss function. This function comprehensively considers the accuracy of model prediction and the performance of the dialogue system, and is used to guide the optimization of model parameters. The specific calculation process of the model loss function can be seen in the following formula 3:

[0155]

[0156] In Formula 3, L(θ) represents the model loss function, λ represents the preset reward weight, Represents the target reward function, which is used to evaluate the predicted feedback response content of the preset dialogue generation model The target reward value between the actual target feedback response content y, and the specific form of the reward function can be flexibly adjusted according to the actual task scenario without limitation. It means that the weighted value of the target reward function is subtracted from the initial loss function, indicating that if the value of the target reward function is higher (that is, the model's prediction is better), then this part will reduce the total loss, thereby encouraging the model to learn better parameters.

[0157] In step S360 of some embodiments, further, the present application may use the model loss function and the target learning rate parameters to update the initial model parameters, and this process involves gradient descent or other optimization algorithms, the purpose of which is to reduce model loss, improve model performance, and generate more accurate dialogue responses.

[0158] In the above embodiments, the present application can use object feedback data to optimize model parameters through reinforcement learning, and gradually adjust the dialogue content generated by the model to make it more coherent and in line with user expectations in long-term interactions. For example, when the preset dialogue generation model is a model built based on LLM, the present application dynamically optimizes the model by introducing user feedback signals, so that the generated dialogue content is more in line with user expectations and improves the user experience. In addition, the present application can not only process directly input object feedback data, but also combine indirect data such as sentiment analysis, usually through long-term continuous learning to achieve multi-dimensional optimization of AI character performance, flexibly adjust the behavior and style of the generated dialogue content of the virtual object, and improve the accuracy of dialogue responses.

[0159] Please refer to Figure 6 , Figure 6 3 is a flowchart of a specific method of step S360 provided in an embodiment of the present application. In some embodiments of the present application, step S360 may specifically include but is not limited to steps S610 to S640.

[0160] Step S610, performing gradient calculation on the initial model parameters based on the model loss function to obtain a gradient loss function;

[0161] Step S620, adjusting the gradient loss function based on the target learning rate to obtain an adjusted loss function;

[0162] Step S630, updating the initial model parameters based on the adjusted loss function to obtain target model parameters;

[0163] Step S640: adjusting the preset dialogue generation model based on the target model parameters to obtain a current dialogue generation model.

[0164] In step S610 of some embodiments, when adjusting the model, the present application may use a gradient descent algorithm to derive the model loss function with respect to the initial model parameters to obtain a gradient. This gradient represents the rising direction of the loss function at the current parameter point, and its opposite direction is the direction of parameter update.

[0165] In step S620 of some embodiments, the gradient is further multiplied by the target learning rate parameter to obtain an adjusted gradient, that is, an adjusted loss function. This adjusted loss function will be used to update the model parameters, and the learning rate determines the magnitude of the update.

[0166] In step S630 and step S640 of some embodiments, the original initial model parameters may be further added with the adjusted gradient (or subtracted with the gradient, depending on the implementation of the optimization algorithm) to obtain updated model parameters, i.e., target model parameters, to find parameter values ​​that minimize the loss function. The target model parameters are used to replace the original model parameters to obtain a new, optimized preset dialogue generation model, i.e., the current dialogue generation model. This model should be able to provide better performance in subsequent dialogue generation tasks. Based on this, the calculation process of determining the target model parameters in this application can be shown in the following formula 4:

[0167]

[0168] In Formula 4, θ new represents the target model parameters, θ old represents the initial model parameters, represents the adjusted loss function, and α represents the target learning rate parameter.

[0169] In the above embodiments, the present application can continuously iterate the parameter optimization process so that the preset dialogue generation model can gradually learn better parameter configurations, thereby improving its performance in dialogue generation tasks. The present application can achieve adaptive updates and reinforcement learning, that is, the present application can use reinforcement learning to enable the model to not only pay attention to feedback in the current dialogue, but also to how to optimize the overall strategy in multiple rounds of interaction to adapt to the long-term preferences and needs of different users. In addition, the reinforcement learning of the present application introduces a reward mechanism that allows the model to learn how to continuously improve in future dialogues based on user feedback, thereby forming a more intelligent and personalized dialogue strategy.

[0170] It should be noted that the preset dialogue generation model includes a preset strategy network and a preset value network. The preset strategy network is used to determine what kind of dialogue response the system should generate in each dialogue state. The preset value network is used to estimate the long-term return of the current state and action, so as to determine whether it is worth taking a certain action in a certain state.

[0171] Please refer to Figure 7 , Figure 7 4 is a flowchart of a specific method of step S640 provided in an embodiment of the present application. In some embodiments of the present application, step S640 may specifically include but is not limited to steps S710 to S780.

[0172] Step S710, extracting the accumulated reward value from the target model parameters;

[0173] Step S720, performing gradient calculation on the cumulative reward value to obtain a cumulative reward gradient value;

[0174] Step S730, extracting initial strategy parameters and initial value parameters from the initial model parameters;

[0175] Step S740, updating the initial strategy parameters based on the accumulated reward gradient value and the target learning rate parameter to obtain the target strategy parameters;

[0176] Step S750, updating the initial value parameter based on the accumulated reward gradient value and the target learning rate parameter to obtain the target value parameter;

[0177] Step S760, adjusting the parameters of the preset strategy network based on the target strategy parameters to obtain an adjusted strategy network;

[0178] Step S770, adjusting the parameters of the preset value network based on the target value parameters to obtain an adjusted value network;

[0179] Step S780, constructing a current dialogue generation model based on the adjusted policy network and the adjusted value network.

[0180] In step S710 and step S720 of some embodiments, during the model generation dialogue process, the present application may record the reward value obtained at each step of the model and accumulate the reward value. Further, the cumulative reward value may be derived with respect to the model parameters to obtain a cumulative reward gradient value, which indicates how the model parameters affect the accumulation of rewards, i.e., determines the rate of change of the cumulative reward value relative to the model parameters.

[0181] In step S730 of some embodiments, in reinforcement learning, since the preset dialogue generation model of the present application includes a preset strategy network (determining the choice of action) and a preset value network (evaluating the value of the current state), the parameters of these two parts can be extracted from the initial model parameters to obtain the initial strategy parameters and the initial value parameters.

[0182] In steps S740 to S770 of some embodiments, further, the present application may use the cumulative reward gradient value and the target learning rate parameter to adjust the initial policy parameters in order to obtain higher rewards in subsequent dialogue generation. Similarly, the cumulative reward gradient value and the target learning rate parameter may be used to adjust the initial value parameter to improve the accuracy of the preset value network's assessment of the state value. Further, the present application may apply the updated target policy parameters to the preset policy network for parameter adjustment to obtain an adjusted policy network, and apply the target value parameters to the preset value network for parameter adjustment to obtain an adjusted value network.

[0183] In step S780 of some embodiments, the present application may construct a new current dialogue generation model by integrating the adjusted policy network and the adjusted value network, and this model should be able to provide better performance in subsequent dialogue generation tasks.

[0184] It should be noted that, according to the rewards of user feedback, the reinforcement learning algorithm of this application will adjust the policy network so that the next dialogue generation is more in line with user expectations. This application can use the policy gradient method to update the preset policy network. In addition, the value network estimates the "long-term reward" of the current dialogue state. When the model generates a dialogue and obtains user feedback, the system will not only consider the immediate reward, but also estimate the possible rewards in the future under this state. Therefore, the preset value network of this application can be updated using the Q-learning algorithm, and the updated value network helps the model better judge when to generate a more detailed response and when to simplify the answer, thereby improving the long-term satisfaction of users.

[0185] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

[0186] A method for generating a dialogue based on a virtual object provided in an embodiment of the present application can continuously update the policy network and the value network according to multiple rounds of user feedback, so that the AI ​​character can adapt to user needs. In other words, the policy updates after each dialogue will be accumulated into the long-term policy optimization, so that the model can not only improve the quality of dialogue generation in the short term, but also gradually adapt to the personalized needs of users through the reinforcement learning framework. The present application can realize the dynamic adjustment of the dialogue generation model, which means that the model can make an immediate response to user feedback in the short term, that is, the behavior of generating dialogue can be adjusted according to the feedback of the user in each interaction. This adjustment is an optimization for the current dialogue, and the goal is to immediately improve the quality of dialogue generated by the model and make the output of the model more in line with user needs. In other words, the present application can be adjusted in the short term, and the model will quickly adjust the parameters according to user feedback after each dialogue, so as to generate a response that is more in line with user expectations in the next round of dialogue.

[0187] Please refer to Figure 8 , Figure 8 : is a schematic diagram of the module structure of a virtual object-based dialogue generation device provided in an embodiment of the present application. In some embodiments of the present application, the virtual object-based dialogue generation device may specifically include:

[0188] The first acquisition module 810 is used to acquire the initial question text of the target object;

[0189] A first generation module 820 is used to generate a dialogue for the initial question text based on a preset dialogue generation model to obtain an initial answer content. The preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter. The initial learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the initial answer content.

[0190] The second acquisition module 830 is used to acquire the dialogue feedback data fed back by the target object to the initial reply content, and acquire the target feedback reply content matching the dialogue feedback data;

[0191] A learning rate parameter updating module 840 is used to update the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter;

[0192] A second generation module 850 is used to generate a dialogue on the dialogue feedback data based on a preset dialogue generation model and a target learning rate parameter to obtain predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model;

[0193] The parameter updating module 860 is used to update the initial model parameters based on the target learning rate parameters, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, obtain the current dialogue generation model, and perform dialogue generation according to the current dialogue generation model.

[0194] It should be noted that the virtual object-based dialogue generation device of the embodiment of the present application is used to execute the above-mentioned virtual object-based dialogue generation method, and the virtual object-based dialogue generation device of the embodiment of the present application corresponds to the aforementioned virtual object-based dialogue generation method. For the specific training process, please refer to the aforementioned virtual object-based dialogue generation method, which will not be described one by one here.

[0195] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for generating a dialogue based on a virtual object according to the embodiment of the present application is implemented.

[0196] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0197] Combine the following Fig. 9 The electronic device according to the embodiment of the present application is introduced in detail.

[0198] Please refer to Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, including:

[0199] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0200] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the virtual object-based dialogue generation method of the embodiment of this application;

[0201] Input / output interface 930, used to implement information input and output;

[0202] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0203] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0204] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0205] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for generating a dialogue based on a virtual object according to the embodiment of the present application is implemented.

[0206] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0207] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0208] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0209] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0211] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0212] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0213] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0216] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0217] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a dialogue based on a virtual object, characterized in that: The method comprises: Get the target object's initial question text; Performing dialogue generation on the initial question text based on a preset dialogue generation model to obtain initial answer content, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter, and the initial learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the initial answer content by the preset dialogue generation model; Acquire the dialogue feedback data fed back by the target object in response to the initial reply content, and acquire the target feedback reply content matching the dialogue feedback data; Update the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter; Performing dialogue generation on the dialogue feedback data based on the preset dialogue generation model and the target learning rate parameter to obtain predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model; Based on the target learning rate parameter, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, the initial model parameters are updated to obtain a current dialogue generation model, and dialogue generation is performed according to the current dialogue generation model.

2. The method according to claim 1, characterized in that: The updating of the initial model parameters based on the target learning rate parameter, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content to obtain the current dialogue generation model includes: Acquire a predicted feedback reply sequence from the predicted feedback reply content; constructing an initial loss function based on the predicted feedback reply sequence, the dialogue feedback data and the initial model parameters; Construct a target reward function based on dialogue feedback data; Performing function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value; Constructing a model loss function based on the initial loss function, the target reward value, and a preset reward weight of the target reward value; Model parameters are updated based on the model loss function, the target learning rate parameter, and the initial model parameter to obtain the current dialogue generation model.

3. The method according to claim 2, characterized in that The constructing an initial loss function based on the predicted feedback reply sequence, the dialogue feedback data and the initial model parameters comprises: Constructing a time step interval based on a target time step matched by the prediction feedback reply sequence, wherein the target time step is an interval endpoint of the time step interval; Obtaining a sequence of predicted feedback reply intervals from the predicted feedback reply content based on the time step intervals; Performing time step prediction based on the predicted feedback reply sequence, the predicted feedback reply interval sequence, the dialogue feedback data and the initial model parameters to obtain a predicted output probability of the target time step, wherein the predicted output probability is used to indicate a matching degree of the predicted feedback reply sequence predicted by the preset dialogue generation model based on the target time step; The predicted output probability is logarithmically calculated based on the target time step and the total number of preset time steps to obtain the initial loss function.

4. The method according to claim 2, characterized in that: The updating of model parameters based on the model loss function, the target learning rate parameter and the initial model parameter to obtain the current dialogue generation model includes: Performing gradient calculation on the initial model parameters based on the model loss function to obtain a gradient loss function; Performing function adjustment on the gradient loss function based on the target learning rate to obtain an adjusted loss function; Performing model parameter updates on the initial model parameters based on the adjusted loss function to obtain target model parameters; The preset dialogue generation model is adjusted based on the target model parameters to obtain the current dialogue generation model.

5. The method according to claim 4, characterized in that The preset dialogue generation model includes a preset strategy network and a preset value network, and the model adjustment of the preset dialogue generation model based on the target model parameters to obtain the current dialogue generation model includes: extracting a cumulative reward value from the target model parameters; Performing gradient calculation on the cumulative reward value to obtain a cumulative reward gradient value; Extracting initial strategy parameters and initial value parameters from the initial model parameters; Performing parameter updates on the initial strategy parameters based on the cumulative reward gradient value and the target learning rate parameter to obtain target strategy parameters; Performing parameter updates on the initial value parameter based on the cumulative reward gradient value and the target learning rate parameter to obtain a target value parameter; Adjusting the parameters of the preset strategy network based on the target strategy parameters to obtain an adjusted strategy network; Adjusting the parameters of the preset value network based on the target value parameters to obtain an adjusted value network; The current dialogue generation model is constructed based on the adjusted policy network and the adjusted value network.

6. The method according to any one of claims 2 to 5, characterized in that: The performing function calculation on the target feedback reply content and the predicted feedback reply content based on the target reward function to obtain a target reward value includes: Calculating the similarity between the target feedback reply content and the predicted feedback reply content to obtain a feedback reply similarity value; Performing feedback data detection on the dialogue feedback data based on a preset feedback text detection model to obtain an object feedback score; Performing emotion detection on the dialogue feedback data based on the object feedback score to obtain an object emotion score; The target reward value is obtained by performing a weighted calculation on the feedback reply similarity value, the object feedback score and the object emotion score.

7. The method according to claim 6, characterized in that The updating of the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter includes: extracting the dialogue interaction time and dialogue interaction frequency from the dialogue feedback data; Performing interaction time detection on the dialogue feedback data based on the dialogue interaction time to obtain a dialogue interaction time score; Performing interaction frequency detection on the dialogue feedback data based on the dialogue interaction frequency to obtain a dialogue interaction frequency score; Performing feedback score calculation on the object emotion score, the conversation interaction time score, and the conversation interaction frequency score to obtain an implicit feedback score; Calculating the difference between the preset expected feedback score of the target object and the implicit feedback score to obtain a feedback score difference; The initial learning rate parameter is updated based on the feedback score difference to obtain the target learning rate parameter.

8. A virtual object-based dialogue generation device, characterized in that: The device comprises: A first acquisition module is used to acquire the initial question text of the target object; A first generation module is used to generate a dialogue for the initial question text based on a preset dialogue generation model to obtain an initial answer content, wherein the preset dialogue generation model is provided with an initial learning rate parameter and an initial model parameter, and the initial learning rate parameter is used to indicate the degree of adjustment of the preset dialogue generation model to the content style and content behavior associated with the initial answer content; A second acquisition module is used to acquire the dialogue feedback data fed back by the target object to the initial reply content, and acquire the target feedback reply content matching the dialogue feedback data; A learning rate parameter updating module, used to update the initial learning rate parameter based on the dialogue feedback data to obtain a target learning rate parameter; a second generation module, configured to perform dialogue generation on the dialogue feedback data based on the preset dialogue generation model and the target learning rate parameter to obtain predicted feedback reply content, wherein the target learning rate parameter is used to indicate the degree of adjustment of the content style and content behavior associated with the predicted feedback reply content by the preset dialogue generation model; A parameter updating module is used to update the initial model parameters based on the target learning rate parameters, the dialogue feedback data, the target feedback reply content and the predicted feedback reply content, to obtain a current dialogue generation model, and to generate a dialogue according to the current dialogue generation model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images