Method and apparatus for optimizing an open-loop robot model based on online feedback

Abstract

The application discloses an opening robot model optimization method and device based on online feedback, comprising: acquiring a real-time collected visitor statement, inputting the visitor statement into a current version of the opening robot model, outputting a guide statement, acquiring a corresponding feedback signal and basic information of the visitor after the visitor receives the guide statement, and determining the emotional tendency of the guide statement; using an improved PPO algorithm to perform reinforcement learning on the training process of the current version of the opening robot model, in the improved PPO algorithm, the feedback signal and the emotional tendency are used to calculate the score in the reward function, and the next version of the opening robot model is trained; the next version of the opening robot model is tested until the opening robot model passes the test and the parameter is updated. The application solves the problems of low utilization efficiency of user feedback, great dependence of the opening robot model on artificial intervention and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent dialogue, and more specifically to an optimization method and apparatus for an open-mouth robot model based on online feedback. Background Technology

[0002] Talking robots are primarily used in the initial stages of human-computer dialogue. They anticipate the visitor's intentions and guide them to speak through specific dialogue. Currently, mainstream talking robot technologies mainly include the following categories:

[0003] 1. Rule-based dialogue system: responds to user input through preset rules and templates.

[0004] 2. Retrieval-based dialogue system: Retrieves the most similar answer from a pre-prepared database based on the user's input.

[0005] 3. Generative Dialogue System: Based on a large language model, it generates dialogue based on prompts and context.

[0006] 4. Hybrid Dialogue System: Combining the advantages of the above technologies, different response strategies are selected for different scenarios.

[0007] These systems are typically deployed to production environments after initial training and are maintained and updated through manual intervention.

[0008] Despite significant progress in existing open-mouth robot technology, the following problems and drawbacks still exist:

[0009] 1. Lack of continuous self-improvement capability: Most systems lack an automated continuous learning mechanism after deployment, and cannot automatically obtain feedback from actual use and optimize, resulting in no improvement in system performance over time.

[0010] 2. Inefficient use of feedback: Even if user feedback is collected, existing systems lack effective mechanisms to automatically transform this feedback into model improvements, usually requiring manual intervention and retraining.

[0011] 3. Lack of specificity in the improvement process: System updates are often global and it is difficult to make precise optimizations for specific customer problems or specific scenarios.

[0012] 4. There are fundamental defects in the training data acquisition process, stemming from both the data and the environment, which result in training efficiency and effectiveness falling far short of the ideal. Summary of the Invention

[0013] The purpose of this application is to propose an optimization method and apparatus for an open-mouth robot model based on online feedback to address the aforementioned technical problems. This method effectively solves the technical problems of existing open-mouth robot models, such as the lack of an effective feedback mechanism, inaccurate optimization, poor adaptability, and low training efficiency and poor results due to data and environmental limitations. This provides a new technical path for the development of intelligent dialogue systems.

[0014] In a first aspect, the present invention provides an optimization method for an open-mouth robot model based on online feedback, comprising the following steps:

[0015] The interaction logs in the historical database are obtained and the historical average opening rate of visitors in each time period is calculated to construct a mapping relationship between time periods and historical average opening rates.

[0016] The system acquires visitor statements and basic information collected online in real time. It inputs the visitor statements and the first prompt words constructed based on the basic information into the open robot model currently deployed in the online inference engine, outputs the prompting statement, determines the sentiment of the prompting statement, and obtains the corresponding feedback signals of the visitor after receiving the prompting statement. The system then combines the visitor statements, basic information, sentiment, prompting statement, and access time and stores them in the experience data pool.

[0017] The current version of the open-mouth robot model is selected based on the same open-mouth robot model currently deployed in the online inference engine. An improved PPO algorithm is used for reinforcement learning training. In the improved PPO algorithm, the current version of the open-mouth robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The historical average opening rate is determined based on the time period and mapping relationship of the access time of the experience data in the experience data pool, and the sampling weights are calculated. Based on the sampling weights, all positive samples from the experience data in the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain training data. Visitor statements, guiding statements, and first prompt words constructed based on the corresponding basic information in the training data are input into the Actor network and Critic network, respectively, to obtain the outputs of the Actor network and Critic network. The score in the reward function is calculated based on the feedback signal and sentiment tendency. The reward function is calculated based on the score in the reward function and the KL divergence between the outputs of the Actor network and Critic network. The TD error is calculated using the reward function. The advantage function is calculated using the TD error after multiple steps. The loss function is calculated using the advantage function. The next version of the open-mouth robot model is trained based on the loss function.

[0018] Test the next version of the open-mouth robot model. If the test fails, roll back to the previous N-step version and repeat the above steps for training until the open-mouth robot model passes the test. If the test passes, update the parameters of the next version of the open-mouth robot model to the open-mouth robot model in the online inference engine.

[0019] As a preferred option, the sampling weight is inversely proportional to the historical average aperture ratio.

[0020] Preferably, the score in the reward function is calculated based on feedback signals and sentiment tendencies, specifically including:

[0021] If the feedback signal indicates that the visitor outputs a reply after receiving the guidance statement, the score corresponding to the feedback signal is 1; if the feedback signal indicates that the visitor does not output a reply after receiving the guidance statement, the score corresponding to the feedback signal is 0.

[0022] If the sentiment of the guiding statement is positive, the corresponding score is 0.5; if the sentiment of the guiding statement is neutral, the corresponding score is 0; if the sentiment of the guiding statement is negative, the corresponding score is -0.5.

[0023] The score in the reward function is the sum of the score corresponding to the feedback signal and the score corresponding to the sentiment tendency.

[0024] As a preferred option, the next version of the opening robot model will be tested, specifically including:

[0025] The next version of the open-mouth robot model is compared with the current version of the open-mouth robot model to obtain the test results. If the next version of the open-mouth robot model is better than the current version of the open-mouth robot model in the test results, the test is determined to pass; otherwise, the test is determined to fail.

[0026] As a preferred approach, the emotional tone of the guiding statement is determined, specifically including:

[0027] Construct and train an emotion classification model to obtain a trained emotion classification model;

[0028] A second cue word is constructed to guide the judgment of the sentiment tendency of the guiding statement. The guiding statement and the second cue word are input into a trained sentiment classification model to obtain the corresponding sentiment tendency.

[0029] As a preferred option, basic information includes search source, search keywords, operating system, source province / city, and / or source operator.

[0030] Secondly, the present invention provides an optimization device for an opening robot model based on online feedback, comprising:

[0031] The mapping relationship establishment module is configured to obtain interaction logs from the historical database and calculate the historical average opening rate of visitors in each time period in order to build a mapping relationship between the time period and the historical average opening rate.

[0032] The data processing module is configured to acquire visitor statements and basic information collected online in real time, input visitor statements and first prompt words constructed based on basic information into the open robot model currently deployed in the online inference engine, output guidance statements, determine the sentiment of guidance statements, acquire the corresponding feedback signals of visitors after receiving guidance statements, and store them in the experience data pool in combination with visitor statements, basic information, sentiment, guidance statements and access time.

[0033] The training module is configured to use the same open-mouth robot model as the currently deployed online inference engine as the current version of the open-mouth robot model and perform reinforcement learning training using an improved PPO algorithm. In the improved PPO algorithm, the current version of the open-mouth robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The historical average opening rate is determined according to the time period and mapping relationship of the access time of the experience data in the experience data pool, and the sampling weight is calculated. According to the sampling weight, all positive samples in the experience data of the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain training data. The visitor's statement, the guiding statement, and the first prompt word constructed based on the corresponding basic information in the training data are respectively input into the Actor network and the Critic network to obtain the output of the Actor network and the output of the Critic network. The score in the reward function is calculated according to the feedback signal and sentiment tendency. The reward function is calculated according to the score in the reward function and the KL divergence between the output of the Actor network and the output of the Critic network. The TD error is calculated using the reward function. The advantage function is calculated using the TD error after multiple steps. The loss function is calculated using the advantage function. The next version of the open-mouth robot model is trained based on the loss function.

[0034] The testing module is configured to test the next version of the open-mouth robot model. If the test fails, it will roll back to the previous N-step version and repeat the above steps for training until the open-mouth robot model passes the test. If the test passes, the parameters of the next version of the open-mouth robot model will be updated in the online inference engine's open-mouth robot model.

[0035] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0036] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0037] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The optimization method of the opening robot model based on online feedback proposed in this invention can adapt to a scenario where the historical average opening rate changes periodically (non-stationary) with time and other external environmental factors based on reinforcement learning algorithm. It effectively utilizes the rare positive samples generated in the "difficult period" to improve the positive samples sampled when the historical average opening rate is low, thereby avoiding model performance fluctuations and frequent training interruptions or rollbacks caused by drastic changes in data distribution, and finally achieving robust and efficient online model optimization.

[0040] (2) The optimization method of the open-mouth robot model based on online feedback proposed in this invention provides a self-improvement mechanism based on online feedback, which enables the open-mouth robot model to continuously learn from real-time interaction and optimize its response strategy. Therefore, the dialogue quality can be improved with the increase of usage time.

[0041] (3) The optimization method for the open-mouth robot model based on online feedback proposed in this invention provides an efficient feedback collection and processing mechanism. It adopts an improved PPO algorithm, which uses the feedback signals provided by visitors and the emotional tendency of the guiding statements to calculate the score in the reward function. It can automatically analyze and utilize various types of user feedback to guide the optimization of the model, solve the problem of low efficiency in utilizing user feedback, and can perform targeted optimization for the specific needs of different scenarios and different user groups. It can realize the automated training and improvement of the open-mouth robot model, reduce the dependence on manual intervention, and improve the efficiency of system maintenance and optimization. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1This is a flowchart illustrating an optimization method for an open-mouth robot model based on online feedback, as an embodiment of this application.

[0044] Figure 2 A flowchart illustrating an optimization method for an open-mouth robot model based on online feedback, as an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of an optimization device for an open-mouth robot model based on online feedback, as an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] Figure 1 An embodiment of this application illustrates an optimization method for an open-mouth robot model based on online feedback, comprising the following steps:

[0049] S1. Obtain the interaction logs from the historical database and calculate the historical average opening rate of visitors in each time period to construct the mapping relationship between the time period and the historical average opening rate.

[0050] Specifically, in the embodiments of this application, reinforcement learning is applied to the opening robot model to achieve continuous self-optimization. However, in real-world online environments, positive samples that provide positive feedback (i.e., user feedback signals indicating opening) are extremely rare, with an average opening rate of only about 7% to 8%. This means that over 90% of the training data consists of negative samples, making it difficult for the model to effectively learn from the vast amount of negative samples and acquire scarce successful experiences. Data analysis clearly indicates that the key performance indicator (opening rate) of the opening robot model exhibits a dynamic change over time. For example, the opening rate is higher between 8 PM and midnight, but significantly lower at 6 or 7 AM. Existing PPO algorithms and their sampling strategies typically assume a fixed environment and fail to consider or adapt to such periodic changes. When the opening robot model learns from data from "difficult periods" (such as early morning), its performance is prone to fluctuations and declines due to the scarcity of positive samples, frequently triggering the system's automatic rollback mechanism. Conventional training methods passively and indiscriminately accept empirical data from all time periods with severely uneven data distribution, failing to proactively address and compensate for dynamic environmental changes at the algorithmic level. This is the root cause of training instability and inefficiency. Therefore, it is necessary to address the technical problems of unstable training processes and low convergence efficiency exposed by existing online reinforcement learning methods when facing reward sparsity and environmental dynamism.

[0051] Therefore, before training begins, a massive amount of interaction logs are read from the historical database. Statistical analysis is performed on the historical dialogues in the interaction logs, calculating the average historical speaking rate for each time period using a preset time unit (e.g., hourly). The average historical speaking rate is defined as the ratio of the number of dialogues in which a visitor spoke within each time period to the total number of historical dialogues within that time period. For example, if there are 10 historical dialogues in a time period, but only one of them has a visitor speaking, then the average historical speaking rate for that time period is 1 / 10. The analysis results of the interaction logs—that is, the mapping relationship from "time period" to "average historical speaking rate"—can be further stored as an environment status statistics table for quick querying, facilitating subsequent retrieval.

[0052] S2: Obtain visitor statements and basic information collected online in real time. Input the visitor statements and the first prompt words constructed based on the basic information into the open robot model currently deployed in the online inference engine. Output the guiding statement, determine the sentiment of the guiding statement, obtain the corresponding feedback signal after the visitor receives the guiding statement, and store it in the experience data pool in combination with the visitor statement, basic information, sentiment, guiding statement and access time.

[0053] In a specific embodiment, determining the sentiment of the guiding statement includes:

[0054] Construct and train an emotion classification model to obtain a trained emotion classification model;

[0055] A second cue word is constructed to guide the judgment of the sentiment tendency of the guiding statement. The guiding statement and the second cue word are input into a trained sentiment classification model to obtain the corresponding sentiment tendency.

[0056] In a specific embodiment, the basic information includes the search source, search keywords, operating system, source province / city, and / or source operator.

[0057] For details, please refer to Figure 2 Visitors enter the page through search terms and open a chat window. The chatbot model sends prompts to guide the visitor to speak further, keeping the conversation going. Therefore, the chatbot model is typically deployed on an online inference engine to generate prompts and interact with visitors. All experience data generated during the interaction, including visitor visit time, model-generated prompts, and user feedback data, is recorded and stored in an experience data pool. Feedback data includes explicit and implicit feedback data. Explicit feedback data includes visitor feedback signals and basic information. Feedback signals are determined by whether the visitor responds to the prompts output by the chatbot model, including both "speaking" and "not speaking" signals. A visitor responding to the prompts indicates "speaking," while not responding indicates "not speaking." Visitor basic information includes search source, search keywords, operating system, province / city of origin, and mobile carrier. This basic information requires data cleaning and format conversion to remove noise and outliers, and to convert data from different sources into a unified format and standard. The above basic information is helpful in constructing the first prompt word to improve the quality of the guiding statements output by the speaking robot model. Using search keywords, source province / city, and other information as auxiliary information to construct the first prompt word helps the guiding statements output by the speaking robot model to better encourage visitors to speak, thus increasing the speaking rate. Implicit feedback data includes the sentiment tendency of the guiding statements output by the speaking robot model, which is divided into three types: positive, negative, and neutral. The sentiment tendency can also be used together with the feedback signal as a reward signal to guide the training and optimization of the speaking robot model. After collecting a sufficient amount of empirical data in the empirical data pool through the above methods, training data can be sampled from the empirical data pool.

[0058] Specifically, the embodiments of this application employ an improved version of the existing Proximal Policy Optimization (PPO) algorithm within the Human Feedback-Based Reinforcement Learning (RLHF) framework, and apply it to the training and optimization process of a talking robot model. In the improved PPO algorithm, a rollout is first performed, where collected visitor statements and a first prompt word constructed based on basic information are input into the current version of the talking robot model. The current version of the talking robot model generates a guiding statement based on the visitor statement and the prompt word. The current version of the talking robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. A reward model is used to evaluate the visitor statement and the guiding statement to obtain the corresponding reward function.

[0059] Furthermore, the sentiment tendency of the guiding statements output by the speaking robot model can be predicted using a trained sentiment classification model. Specifically, the sentiment classification model includes dictionary-based sentiment classification models, machine learning-based sentiment classification models, or deep learning-based sentiment classification models, and their specific structure and approach are not limited here.

[0060] S3: The same open-mouth robot model as the one currently deployed in the online inference engine is used as the current version of the open-mouth robot model, and the improved PPO algorithm is used for reinforcement learning training. In the improved PPO algorithm, the current version of the open-mouth robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The historical average opening rate is determined according to the time period and mapping relationship of the access time of the experience data in the experience data pool, and the sampling weight is calculated. According to the sampling weight, all positive samples in the experience data of the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain the training data. The visitor's statement, the guiding statement, and the first prompt word constructed based on the corresponding basic information in the training data are respectively input into the Actor network and the Critic network to obtain the output of the Actor network and the output of the Critic network. The score in the reward function is calculated according to the feedback signal and sentiment tendency. The reward function is calculated according to the score in the reward function and the KL divergence between the output of the Actor network and the output of the Critic network. The TD error is calculated using the reward function. The advantage function is calculated using the TD error after multiple steps. The loss function is calculated using the advantage function. The next version of the open-mouth robot model is trained based on the loss function.

[0061] In a specific embodiment, the sampling weight is inversely proportional to the historical average aperture ratio.

[0062] Specifically, the currently deployed open-mouth robot model on the online inference engine can provide corresponding guidance statements for visitor-proposed statements online. Simultaneously, an open-mouth robot model identical to the one currently deployed on the online inference engine is used as the current version of the open-mouth robot model for training and optimization. One innovation of this application's embodiments lies in the fact that, when preparing to begin a training iteration, training data is not directly and uniformly sampled from the experience data pool. Instead, the sampling weights corresponding to the experience data for each time period are calculated based on the mapping relationship between time periods and historical average open-mouth rates. These sampling weights are then used to weight the positive samples in the experience data for each time period.

[0063] All empirical data in the empirical data pool includes positive and negative samples. Positive samples are empirical data where the feedback signal indicates an open or closed loop, while negative samples are empirical data where the feedback signal indicates a closed or closed loop. For each piece of empirical data in the empirical data pool, its access time (curEnterTime) is first parsed to determine the time period in which it was generated. All empirical data within that time period are then obtained, and each piece of empirical data is categorized as either a positive or negative sample. Next, using this time period as an index, the corresponding historical average opening rate P is queried from the mapping relationship between the time period and the historical average opening rate established in step S1. hist According to the query P hist A sampling weight W is calculated for all positive samples of all empirical data within this time period. This sampling weight is inversely proportional to the historical average aperture ratio, for example, W∝1 / P. hist This means that positive samples from empirical data generated in periods with lower historical average opening rates are assigned higher sampling weights. Based on the calculated weights, all positive samples from the empirical data pool for that period are weighted and sampled evenly, forming a batch of training data. Therefore, a higher proportion of positive samples are sampled from empirical data in periods with lower historical average opening rates. This method collects more empirical data with open feedback signals compared to the average sampling method. The above dynamic sampling process makes the positive and negative samples of each batch of training data more balanced. The more balanced training data constructed through the above dynamic sampling is used to train and update the parameters of the Actor and Critic networks, making better use of the positive samples corresponding to periods with low historical average opening rates.

[0064] In one example, the formula for calculating the sampling weight is:

[0065] W = B / P hist ;

[0066] Where B represents the sampling coefficient, P hist The historical average aperture rate is determined by the time period in which the access time of the experience data in the experience data pool is located and the mapping relationship. W represents the sampling weight.

[0067] The embodiments of this application can proactively identify and compensate for "difficult" time periods in the environment. For rare positive samples generated during periods with low historical average opening rates, their sampling weights are increased to ensure that positive samples are not overwhelmed, enabling the model to be trained sufficiently and effectively in all time periods. By reconstructing the training data, the embodiments of this application provide the model with a relatively balanced and stable learning environment at all times. This fundamentally avoids a significant decline in model performance caused by drastic fluctuations in the original data distribution, directly reducing the number of times automatic rollback is triggered, allowing the training process to proceed continuously and efficiently, and solving the problem of training essentially stalling in the prior art. Unlike solutions that adjust only after problems occur (such as increased loss), the embodiments of this application utilize prior knowledge of the environment to proactively and predictively intervene before problems occur. This forward-looking design makes the training process more stable and controllable. Because the model is robustly trained in all environmental states, it can learn more universal strategies without developing "bias" for certain specific time periods. Therefore, the average performance of the finally trained model over the entire time period (such as the total aperture ratio) is expected to be substantially improved, surpassing the level of only minor improvement achieved in existing technologies.

[0068] In a specific embodiment, the score in the reward function is calculated based on the feedback signal and sentiment tendency, specifically including:

[0069] If the feedback signal indicates that the visitor outputs a reply after receiving the guidance statement, the score corresponding to the feedback signal is 1; if the feedback signal indicates that the visitor does not output a reply after receiving the guidance statement, the score corresponding to the feedback signal is 0.

[0070] If the sentiment of the guiding statement is positive, the corresponding score is 0.5; if the sentiment of the guiding statement is neutral, the corresponding score is 0; if the sentiment of the guiding statement is negative, the corresponding score is -0.5.

[0071] The score in the reward function is the sum of the score corresponding to the feedback signal and the score corresponding to the sentiment tendency.

[0072] Specifically, in the embodiments of this application, feedback signals and sentiment tendencies are used as scores in the reward function. Visitor statements, guidance statements, and first prompt words constructed based on basic information are input into the Actor network and Critic network, respectively, yielding the corresponding outputs of the Actor network and Critic network. The remaining parts are the same as the traditional PPO algorithm. Therefore, feedback information can be effectively utilized to supervise and optimize the training process of the current version of the talking robot model. In one embodiment, if the feedback signal is "open," meaning the visitor responds to the guidance statement, the corresponding score is set to 1; if the feedback signal is "not open," meaning the visitor does not respond to the guidance statement, the corresponding score is set to 0; and if the sentiment tendency is positive, the corresponding score is set to 0.5; if the sentiment tendency is negative, the corresponding score is set to -0.5; if the sentiment tendency is neutral, the corresponding score is 0. The score corresponding to the feedback signal and the score corresponding to the sentiment tendency are added together to obtain the score in the reward function. Then, the KL divergence between the outputs of the Actor network and the Critic network is calculated. Combining the score with the KL divergence yields the reward function. This reward function is further combined with the value function before and after the update to calculate the TD error. The advantage function is calculated using the TD error after multiple steps. The value function loss is calculated using the advantage function and the value function. The policy gradient loss is calculated using the advantage function. In the optimization phase, the value function loss is combined with the policy specificity loss to obtain the total loss function. The parameters of the Actor network and the Critic network are optimized with the goal of minimizing this total loss function, thus obtaining the next version of the open-mouth robot model.

[0073] The speaking robot model in the embodiments of this application is trained based on a pre-trained large language model. During its training process, one version of the speaking robot model is first deployed on a cloud server as an online inference engine. At the same time, based on the feedback data generated by this version of the speaking robot model during its interaction with visitors, another version of the speaking robot model is further trained. Instead of training one of the versions of the speaking robot model that is already running online, resource isolation is achieved, ensuring service stability.

[0074] S4. Test the next version of the open-mouth robot model. If the test fails, roll back to the previous N steps and repeat the above steps for training until the open-mouth robot model passes the test. If the test passes, update the parameters of the next version of the open-mouth robot model to the open-mouth robot model in the online inference engine.

[0075] In a specific embodiment, testing the next version of the opening robot model includes:

[0076] The next version of the open-mouth robot model is compared with the current version of the open-mouth robot model to obtain the test results. If the next version of the open-mouth robot model is better than the current version of the open-mouth robot model in the test results, the test is determined to pass; otherwise, the test is determined to fail.

[0077] Specifically, in the embodiments of this application, after training the next version of the speaking robot model, testing is required to verify the optimization effect. The specific process includes: comparing the next version of the speaking robot model with the current version, collecting performance metrics for both models to further evaluate the optimization effect. If the test passes, the next version of the speaking robot model is updated to the online inference engine's speaking robot model via hot update without service interruption. That is, the next version of the speaking robot model is first deployed on a small scale, and after confirming the effect, the deployment scope is gradually expanded until it completely replaces the currently deployed speaking robot model on the online inference engine. If the test fails, the version is rolled back, data is collected again, and the above steps are repeated for training. During deployment, the performance of the next version of the speaking robot model is continuously monitored, and real-time fine-tuning and emergency rollback are performed based on the monitoring results. Specifically, a parameter log can be constructed to record the parameters of the speaking robot model at different training steps. If a test fails or an emergency rollback occurs, the parameters of the previous N steps are retrieved to roll back the speaking robot model. The above steps can be repeated continuously to optimize the model and improve system performance.

[0078] Specifically, embodiments of this application apply reinforcement learning to online dialogue systems to achieve continuous self-optimization. This system can automatically obtain real-time feedback data (including feedback signals, i.e., whether a user has responded) from online services, automatically process data and train models, and integrate automated testing and deployment mechanisms. This mechanism can perform A / B testing on newly trained models; if the new model's performance meets the standards, a non-stop "hot update" is performed; if it does not meet the standards, it is automatically "rolled back" to the old version, thus ensuring the basic stability and continuity of the online service from an engineering perspective. During model training, rich basic information (such as IP source, search keywords, browser, etc.) is used to construct prompts to generate more personalized guidance.

[0079] The embodiments of this application, through a closed-loop feedback mechanism, can continuously learn user preferences and needs. The optimized model can more accurately understand user intent and provide more precise responses. Real-world testing shows an approximately 20% increase in the response rate. Furthermore, automated feedback collection and analysis reduce manual intervention, precise responses lower the workload of delivery personnel, system maintenance costs decrease by approximately 20%, while service quality improves. It can quickly adapt to new user needs and business scenarios, and performance continuously improves with usage time, avoiding the performance degradation issues of traditional systems over time.

[0080] Further reference Figure 3 As an implementation of the methods shown in the above figures, this application provides an embodiment of an optimization device for an open-mouth robot model based on online feedback. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0081] This application provides an optimization device for an opening robot model based on online feedback, comprising:

[0082] The mapping relationship establishment module 1 is configured to obtain the interaction logs in the historical database and calculate the historical average opening rate of visitors in each time period in order to build a mapping relationship between the time period and the historical average opening rate.

[0083] Data processing module 2 is configured to acquire visitor statements and basic information collected online in real time, input visitor statements and first prompt words constructed based on basic information into the open robot model currently deployed in the online inference engine, output guiding statements, determine the sentiment of guiding statements, acquire the corresponding feedback signals of visitors after receiving guiding statements, and store them in the experience data pool in combination with visitor statements, basic information, sentiment, guiding statements and access time.

[0084] Training module 3 is configured to use the same open-mouth robot model as the currently deployed online inference engine as the current version of the open-mouth robot model and perform reinforcement learning training using an improved PPO algorithm. In the improved PPO algorithm, the current version of the open-mouth robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The historical average opening rate is determined according to the time period and mapping relationship of the access time of the experience data in the experience data pool, and the sampling weight is calculated. According to the sampling weight, all positive samples in the experience data of the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain training data. The visitor's statement, the guiding statement, and the first prompt word constructed based on the corresponding basic information in the training data are respectively input into the Actor network and the Critic network to obtain the output of the Actor network and the output of the Critic network. The score in the reward function is calculated according to the feedback signal and sentiment tendency. The reward function is calculated according to the score in the reward function and the KL divergence between the output of the Actor network and the output of the Critic network. The TD error is calculated using the reward function. The advantage function is calculated using the TD error after multiple steps. The loss function is calculated using the advantage function. The next version of the open-mouth robot model is trained based on the loss function.

[0085] Test module 4 is configured to test the next version of the open-mouth robot model. If the test fails, it will roll back to the previous N steps and repeat the above steps for training until the open-mouth robot model passes the test. If the test passes, the parameters of the next version of the open-mouth robot model will be updated in the online inference engine's open-mouth robot model.

[0086] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 4 As shown, the electronic device in this embodiment includes a processor 401 and a memory 402; wherein the memory 402 is used to store computer execution instructions; and the processor 401 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0087] Alternatively, the memory 402 can be either standalone or integrated with the processor 401.

[0088] When the memory 402 is set up independently, the electronic device also includes a bus 403 for connecting the memory 402 and the processor 401.

[0089] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 401, implement the above method.

[0090] This invention also provides a computer program product, including a computer program that, when executed by a processor 401, implements the above-described method.

[0091] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0092] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0093] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0094] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 401 to execute some steps of the methods of the various embodiments of this application.

[0095] It should be understood that the processor 401 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 401 can be any conventional processor 401. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 401 executing the steps, or as a combination of hardware and software modules within the processor 401 executing the steps.

[0096] The memory 402 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0097] Bus 403 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 403 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 403 in the accompanying drawings of this application is not limited to only one bus 403 or one type of bus 403.

[0098] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0099] An exemplary storage medium is coupled to processor 401, enabling processor 401 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 401. Processor 401 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 401 and storage medium can exist as discrete components in an electronic device or host device.

[0100] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimization method for an open-mouth robot model based on online feedback, characterized in that, Includes the following steps: The interaction logs in the historical database are obtained and the historical average opening rate of visitors in each time period is calculated to construct a mapping relationship between time periods and historical average opening rates. The system acquires visitor statements and basic information collected online in real time. It inputs the visitor statements and a first prompt word constructed based on the basic information into the open robot model currently deployed in the online inference engine, outputs a guiding statement, determines the sentiment of the guiding statement, acquires the corresponding feedback signal after the visitor receives the guiding statement, and stores it in the experience data pool in combination with the visitor statement, basic information, sentiment, guiding statement and access time. The same speaking robot model as the one currently deployed in the online inference engine is used as the current version of the speaking robot model, and a modified PPO algorithm is used for reinforcement learning training. In the modified PPO algorithm, the current version of the speaking robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The historical average speaking rate is determined according to the time period and mapping relationship of the access time of the experience data in the experience data pool, and the sampling weight is calculated. The sampling weight is inversely proportional to the historical average speaking rate. According to the sampling weight, all positive samples in the experience data of the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain training data. The visitor statement, the guiding statement, and the first prompt word constructed based on the corresponding basic information in the training data are respectively input into the Actor network and the Critic network to obtain the output of the Actor network and the output of the Critic network. The score in the reward function is calculated according to the feedback signal and sentiment tendency, specifically including: If the feedback signal indicates that the visitor outputs a reply after receiving the guidance statement, the score corresponding to the feedback signal is 1; if the feedback signal indicates that the visitor does not output a reply after receiving the guidance statement, the score corresponding to the feedback signal is 0. If the sentiment of the guiding statement is positive, the score is 0.5; if the sentiment of the guiding statement is neutral, the score is 0; if the sentiment of the guiding statement is negative, the score is -0.

5. The score in the reward function is the sum of the score corresponding to the feedback signal and the score corresponding to the sentiment tendency; the reward function is calculated based on the score in the reward function and the KL divergence between the output of the Actor network and the output of the Critic network; the TD error is calculated using the reward function; the advantage function is calculated using the TD error after multiple steps; the loss function is calculated using the advantage function; and the next version of the open-mouth robot model is trained based on the loss function. The next version of the open-mouth robot model is tested. If the test fails, the model is rolled back to the previous N-step version and the above steps are repeated for training until the open-mouth robot model passes the test. If the test passes, the parameters of the next version of the open-mouth robot model are updated in the online inference engine's open-mouth robot model.

2. The optimization method for an open-mouth robot model based on online feedback according to claim 1, characterized in that, The next version of the open-mouth robot model will be tested, specifically including: The next version of the open-mouth robot model is compared with the current version of the open-mouth robot model to obtain the test results. If the test results show that the next version of the open-mouth robot model is better than the current version of the open-mouth robot model, then the test is determined to pass; otherwise, the test is determined to fail.

3. The optimization method for an open-mouth robot model based on online feedback according to claim 1, characterized in that, Determining the sentiment of the guiding statement specifically includes: Construct and train an emotion classification model to obtain a trained emotion classification model; A second prompt word is constructed to guide the judgment of the sentiment tendency of the guiding statement. The guiding statement and the second prompt word are input into the trained sentiment classification model to obtain the corresponding sentiment tendency.

4. The optimization method for an open-mouth robot model based on online feedback according to claim 1, characterized in that, The basic information includes the search source, search keywords, operating system, source province / city, and / or source operator.

5. An optimization device for an open-mouth robot model based on online feedback, characterized in that, include: The mapping relationship establishment module is configured to obtain interaction logs from the historical database and calculate the historical average opening rate of visitors in each time period in order to build a mapping relationship between the time period and the historical average opening rate. The data processing module is configured to acquire visitor statements and basic information collected online in real time, input the visitor statements and a first prompt word constructed based on the basic information into the open robot model currently deployed in the online inference engine, output a guiding statement, determine the sentiment tendency of the guiding statement, acquire the corresponding feedback signal after the visitor receives the guiding statement, and store it in the experience data pool in combination with the visitor statement, basic information, sentiment tendency, guiding statement and access time. The training module is configured to use the same speaking robot model as the currently deployed online inference engine as the current version of the speaking robot model and perform reinforcement learning training using an improved PPO algorithm. In the improved PPO algorithm, the current version of the speaking robot model is used as the Actor network, and the pre-trained large language model is used as the Critic network. The module determines the corresponding historical average speaking rate based on the time period and mapping relationship of the access time of the experience data in the experience data pool and calculates the sampling weights, which are inversely proportional to the historical average speaking rate. Based on the sampling weights, all positive samples from the experience data in the corresponding time period in the experience data pool are weighted and sampled uniformly to obtain training data. The visitor statements, guiding statements, and first prompt words constructed based on the corresponding basic information in the training data are respectively input into the Actor network and the Critic network to obtain the outputs of the Actor network and the Critic network. The module calculates the score in the reward function based on the feedback signal and sentiment tendency, specifically including: If the feedback signal indicates that the visitor outputs a reply after receiving the guidance statement, the score corresponding to the feedback signal is 1; if the feedback signal indicates that the visitor does not output a reply after receiving the guidance statement, the score corresponding to the feedback signal is 0. If the sentiment of the guiding statement is positive, the score is 0.5; if the sentiment of the guiding statement is neutral, the score is 0; if the sentiment of the guiding statement is negative, the score is -0.

5. The score in the reward function is the sum of the score corresponding to the feedback signal and the score corresponding to the sentiment tendency; the reward function is calculated based on the score in the reward function and the KL divergence between the output of the Actor network and the output of the Critic network; the TD error is calculated using the reward function; the advantage function is calculated using the TD error after multiple steps; the loss function is calculated using the advantage function; and the next version of the open-mouth robot model is trained based on the loss function. The testing module is configured to test the next version of the open-mouth robot model. If the test fails, it rolls back to the previous N-step version and repeats the above steps for training until the open-mouth robot model passes the test. If the test passes, the parameters of the next version of the open-mouth robot model are updated in the online inference engine's open-mouth robot model.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images