Intelligent agent assistant user feedback data filtering method and related device

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an agent assistant user feedback data filtering method and a related device, and the method comprises the steps: obtaining a user instruction, agent assistant execution data, agent assistant screenshot completion and user feedback data; according to the user instruction and the agent assistant, finishing screenshot to obtain a task completion degree value based on image-text multi-modality, and according to the user instruction and the agent assistant execution data, obtaining a process completion degree value based on action summarization; and performing weighted stacking on the task completion degree value and the process completion degree value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data according to the final evaluation score and the deviation of the user feedback data. The task completion degree of the agent assistant is evaluated from the two aspects of image-text multi-mode and action summary, the filtering accuracy of user feedback data is effectively improved, meanwhile, the robustness of the agent assistant to malicious or wrong user feedback data is enhanced, and therefore the performance of the agent assistant and the user experience are optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and relates to a method and related device for filtering user feedback data of an agent assistant. Background Art

[0002] In recent years, the breakthroughs in large models in terms of understanding and reasoning capabilities have prompted researchers to re-examine the issue of mobile task automation and shift it to a model-centric perspective. From this perspective, agent assistant systems based on multimodal large models have emerged, such as representative systems like BrowserUse and AutoGLM. These systems first parse the user's natural language instructions, and then integrate advanced visual perception tools to capture and analyze the visual and text elements of the mobile application interface in real time, comprehensively perceiving the operating environment. Based on the understanding of the instructions and scenarios, the agent assistant system conducts task planning and decomposition through a unified language model, breaking down complex operations into specific steps and instructions to ensure efficient execution.

[0003] Data poisoning attacks refer to malicious attackers injecting malicious or harmful data into the training data, affecting the learning process of the model, and thereby damaging its performance or manipulating its behavior. In the supply chain of agent assistants, attackers can use the user feedback mechanism to inject forged or manipulated data into the training dataset, causing biases in the internal model of the agent or making it perform incorrect operations. To ensure the quality of the training data, existing data cleaning techniques, including statistical methods, machine learning methods, and deep learning methods, etc., are widely applied in the data preprocessing process. These methods can effectively identify and correct errors, inconsistencies, and noises in the data, thereby improving the accuracy and reliability of the data.

[0004] However, in the application of user feedback data of agent assistants, existing data cleaning techniques still have many deficiencies. On the one hand, traditional single-modal behavior evaluation may ignore the mutual influence between different modal information, overlooking that multimodal evaluation of images and text can more deeply understand the comprehensive capabilities of the agent in a multimodal environment. On the other hand, traditional agent behavior evaluation methods are difficult to understand based on multiple steps taken by the agent assistant, making it difficult for data cleaning strategies to verify data according to specific application scenarios. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings of the existing technology and provide a method and related device for filtering user feedback data of an agent assistant.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] In the first aspect of the present invention, a method for filtering user feedback data of an agent assistant is provided, including: obtaining a user instruction, agent assistant execution data, an agent assistant completion screenshot, and user feedback data; obtaining a task completion value based on text and image multimodality according to the user instruction and the agent assistant completion screenshot, and obtaining a process completion value based on action summary according to the user instruction and the agent assistant execution data; weighted superposition of the task completion value and the process completion value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

[0008] Optionally, the obtaining a task completion value based on text and image multimodality according to the user instruction and the agent assistant completion screenshot includes: obtaining a feature representation of the user instruction as an instruction feature through the same multimodal model, and obtaining a feature representation of the agent assistant completion screenshot as an initial image feature; weighting the initial image feature based on the attention mechanism of the instruction feature to obtain a weighted image feature focused on the target area of the user instruction; obtaining and obtaining the similarity between the instruction feature and the weighted image feature to obtain the task completion value.

[0009] Optionally, the weighting the initial image feature based on the attention mechanism of the instruction feature to obtain a weighted image feature focused on the target area of the user instruction includes:

[0010] Obtaining a weighted image feature focused on the target area of the user instruction through the following formula

[0011]

[0012] where, v T is the instruction feature, v I is the initial image feature, is the dimension of v I and softmax(·) is a weighted coefficient conversion function.

[0013] Optionally, the obtaining and obtaining the similarity between the instruction feature and the weighted image feature to obtain the task completion value includes:

[0014] Obtaining the similarity between the instruction feature and the weighted image feature through the following formula

[0015]

[0016] where, v T is the instruction feature, is the weighted image feature.

[0017] Or obtaining the similarity between the instruction feature and the weighted image feature through the following formula

[0018]

[0019] Among them, is the k-th dimensional vector of v T , and is 's k-th dimensional vector. n is the dimension of v T and , and σ is an adjustable parameter.

[0020] The task completion value C multi (T, I) is obtained through the following formula:

[0021]

[0022] Among them, θ H is the preset first high threshold, and θ L is the preset first low threshold.

[0023] Optionally, the process completion value based on action summary obtained according to the user instruction and the data executed by the intelligent agent assistant includes: using a pre-trained text generation model to summarize the data executed by the intelligent agent assistant to obtain an action summary; vectorizing the user instruction and the action summary to obtain an instruction feature and an action summary feature; obtaining and obtaining the process completion value according to the similarity between the instruction feature and the action summary feature.

[0024] Optionally, obtaining and obtaining the process completion value according to the similarity between the instruction feature and the action summary feature includes:

[0025] The similarity S(v T , v Sum ) between the instruction feature and the action summary feature is obtained through the following formula:

[0026]

[0027] Among them, v T is the instruction feature, v Sum is the action summary feature, ||v T || is the norm of v T , and ||v Sum || is the norm of v Sum .

[0028] Or the similarity S(v T , v Sum ) between the instruction feature and the action summary feature is obtained through the following formula:

[0029]

[0030] Among them, is vT The k-th dimensional vector of is v sum The k-th dimensional vector of, where n is and The dimension of, and σ is an adjustable parameter.

[0031] The process completion value C is obtained through the following formula action (T, Sum):

[0032]

[0033] where γ H is the preset second high threshold, and γ L is the preset second low threshold.

[0034] Optionally, the weighted superposition task completion value and the process completion value are used to obtain the final evaluation score, and based on the deviation between the final evaluation score and the user feedback data, the filtering result of the user feedback data includes:

[0035] The final evaluation score C is obtained through the following formula:

[0036]

[0037] where α is the preset weight, C multi is the task completion value, and C action is the process completion value, is the first indicator function. When C multi ≠0 and C action ≠0 holds, the return value of the first indicator function is 1, otherwise the return value is 0;

[0038] The filtering result indicator value Filter of the user feedback data is obtained through the following formula:

[0039]

[0040] where is the second indicator function, C feedback is the user feedback data, and Λ is the preset threshold. When |C - C feedback | > Λ holds, the return value of the second indicator function is 1, otherwise the return value is 0;

[0041] When the filtering result indicator value Filter is 1, the filtering result of the user feedback data is filtered; when the filtering result indicator value Filter is 1, the filtering result of the user feedback data is not filtered.

[0042] In a second aspect of the present invention, there is provided an intelligent agent assistant user feedback data filtering system, including: a data acquisition module for acquiring user instructions, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data; a completion evaluation module for obtaining a task completion value based on graphic and text multi-modalities according to the user instructions and the intelligent agent assistant completion screenshots, and obtaining a process completion value based on action summary according to the user instructions and the intelligent agent assistant execution data; a decision-making module for weighted superposition of the task completion value and the process completion value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

[0043] In a third aspect of the present invention, there is provided a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the above-mentioned intelligent agent assistant user feedback data filtering method are implemented.

[0044] In a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned intelligent agent assistant user feedback data filtering method are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The intelligent agent assistant user feedback data filtering method of the present invention obtains a task completion value based on graphic and text multi-modalities according to the user instructions and the intelligent agent assistant completion screenshots, and obtains a process completion value based on action summary according to the user instructions and the intelligent agent assistant execution data, evaluates the task completion degree of the intelligent agent assistant from two aspects of graphic and text multi-modalities and action summary, then weighted superposes the task completion value and the process completion value to obtain a final evaluation score, and obtains a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data. Combining precise multi-modal and action analysis, it effectively improves the filtering accuracy of the user feedback data, and at the same time enhances the robustness of the intelligent agent assistant to malicious or incorrect user feedback data, thereby optimizing the performance and user experience of the intelligent agent assistant. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the intelligent agent assistant user feedback data filtering method according to an embodiment of the present invention.

[0048] Figure 2 It is a schematic diagram of the application environment of the intelligent agent assistant according to an embodiment of the present invention.

[0049] Figure 3 It is a flowchart of obtaining the task completion value according to an embodiment of the present invention.

[0050] Figure 4 It is a flowchart for obtaining the process completion value of an embodiment of the present invention.

[0051] Figure 5 It is a structural block diagram of an intelligent agent assistant user feedback data filtering system according to an embodiment of the present invention. Detailed implementation manners

[0052] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data used may be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings:

[0055] Refer to Figure 1 , in an embodiment of the present invention, a method for filtering intelligent agent assistant user feedback data is provided, which can accurately filter user feedback data to improve the retraining effect of the intelligent agent assistant.

[0056] Specifically, the method for filtering intelligent agent assistant user feedback data of the present invention includes the following steps:

[0057] S1: Obtain user instructions, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data.

[0058] S2: Obtain a task completion value based on text and graphics multimodality according to the user instructions and the intelligent agent assistant completion screenshots, and obtain a process completion value based on action summary according to the user instructions and the intelligent agent assistant execution data.

[0059] S3: Weightedly superimpose the task completion value and the process completion value to obtain the final evaluation score, and obtain the filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

[0060] The method for filtering user feedback data of the intelligent agent assistant of the present invention obtains a task completion value based on text and image multimodality by taking screenshots according to the user instruction and the intelligent agent assistant, and obtains a process completion value based on action summary according to the user instruction and the execution data of the intelligent agent assistant. The task completion of the intelligent agent assistant is evaluated from two aspects of text and image multimodality and action summary, and then the task completion value and the process completion value are weightedly superimposed to obtain the final evaluation score. According to the deviation between the final evaluation score and the user feedback data, the filtering result of the user feedback data is obtained. Combining precise multimodality and action analysis effectively improves the filtering accuracy of the user feedback data, and at the same time enhances the robustness of the intelligent agent assistant to malicious or incorrect user feedback data, thereby optimizing the performance and user experience of the intelligent agent assistant.

[0061] Explanatory, artificial intelligence endows computer systems with the ability of self-learning, reasoning, judgment and decision-making by simulating human intelligent behavior and using technologies such as machine learning and deep learning, and is widely used in fields such as autonomous driving, intelligent customer service, and financial risk control. Among them, the present invention mainly relates to the direction of training data cleaning. Training data cleaning refers to removing or correcting noise, errors and outliers in training data through a series of technical means, especially identifying and removing malicious data and invalid data, so as to ensure that the artificial intelligence model can learn correct and real information during the training process, and improve the robustness and generalization ability of the model.

[0062] Explanatory, see Figure 2 , which shows a feasible application environment of the intelligent agent assistant. In the shown application environment, the user interacts with the intelligent agent assistant and inputs a user instruction to the intelligent agent assistant. The intelligent agent assistant interacts with the terminal to complete task execution. Among them, the intelligent agent assistant understands the user instruction, and then uses its internal large model, visual component and tool to complete the perception, planning and decision-making of task execution, and finally completes the task. The terminal of the intelligent agent can be but is not limited to various desktop computers, laptop computers, smart phones, tablet computers and other devices with a user graphical interface. After the user instruction is completed, the user can give feedback on the execution situation of this task, that is, user feedback data. Finally, the intelligent agent developer sorts out the user feedback data for training or fine-tuning the model to optimize the execution efficiency of the intelligent agent assistant. Among them, representative intelligent agent assistants include MobileAgent, BrowserUse, etc. The user instruction can be a text instruction.

[0063] Taking the method for filtering user feedback data of the intelligent agent assistant of the present invention as an application toFigure 2 Take the terminal in [description] as an example for illustration. This terminal is an intelligent agent assistant for mobile devices. The intelligent agent assistant understands user instructions through an internal large model, combines the understanding of the mobile phone operation interface, reasons about the next operation to be performed, and finally determines the next operation to be executed, and then sends it to the mobile phone through the large model to execute the corresponding operation. Based on the behavior of this intelligent agent, the user provides feedback scores to the application provider, that is, user feedback data. This user feedback data and the behavior records of the intelligent agent assistant will be used as the dataset for the re-training of the internal large model of the intelligent agent assistant. Before re-training, it is necessary to filter out invalid user feedback data.

[0064] Explanatorily, user feedback data is the evaluation data that users provide to the intelligent agent assistant provider regarding the performance of the intelligent agent assistant when performing tasks, usually in the form of a scoring system. The execution data of the intelligent agent assistant is the detailed process data during the task execution of the intelligent agent assistant, generally including the operation records of each operation of the intelligent agent assistant, screenshots required for the intelligent agent assistant to perceive the operation interface before and after each operation, etc. User instructions refer to the instructions composed of natural language provided by users to the intelligent agent assistant, and these user instructions provide the specific behaviors or purposes of the operations performed by the intelligent agent assistant. The screenshot taken by the intelligent agent assistant refers to the last screen capture image retained before the operation of the intelligent agent assistant ends.

[0065] Specifically, through the integrated logging function and user interface interaction, feedback data can be captured in real time. These feedback data not only include user feedback data, but also cover the interaction details of the intelligent agent assistant during task execution, such as actions like clicks and swipes, as well as screenshots before and after task completion.

[0066] In a possible implementation manner, refer to Figure 3 , the obtaining of the task completion degree value based on text and image multi-modalities according to user instructions and the screenshot taken by the intelligent agent assistant includes:

[0067] S21: Through the same multi-modal model, obtain the feature representation of the user instruction as the instruction feature, and obtain the feature representation of the screenshot taken by the intelligent agent assistant as the initial image feature.

[0068] S22: Weight the initial image feature based on the attention mechanism of the instruction feature to obtain a weighted image feature focused on the target area of the user instruction.

[0069] S23: Obtain and calculate the similarity between the instruction feature and the weighted image feature to obtain the task completion degree value.

[0070] Explanatory, the same multimodal model refers to a model that can simultaneously process and understand multiple different types of data inputs (such as text, images, etc.). It extracts and fuses feature representations from different modalities through deep learning techniques. The key feature of this model lies in its ability to establish connections between different modalities and support cross-modal tasks, such as the correlation analysis between images and text; the joint feature representation of the text and image refers to the vector representation in a unified feature space obtained after processing the text data and image data through a multimodal model. This joint feature vector can capture and express the semantic information of the text and image, enabling originally heterogeneous data to be compared and calculated in the same space. In this way, the model can more accurately evaluate the completion degree of the agent's tasks and make effective decisions and inferences in a multimodal environment.

[0071] Exemplarily, the CLIP multimodal model is selected as the multimodal model. By separately inputting the user instruction T and the screenshot I completed by the agent assistant, the instruction feature v T and the initial image feature v I .

[0072] In the CLIP model, the image encoder is usually ResNet or Vision Transformer (ViT), while the text encoder is an architecture based on Transformer, such as BERT or GPT-2. Therefore, the feature vectors encoded by the image encoder and the text will be mapped through a linear layer to obtain feature vectors with the same dimension, and L2 normalization is performed. Therefore, the feature vector of the linear layer after projection is selected as the selected feature vector. After such operations, the instruction feature v T generated according to the user instruction T is obtained, which contains key information of the user instruction such as the target task and operation type; at the same time, the initial image feature v I is generated according to the screenshot completed by the agent assistant.

[0073] Exemplarily, other models can be selected as the selected multimodal model, such as CLI-VIT, etc.

[0074] Exemplarily, a pre-processed dataset can be used to pre-train or fine-tune the selected multimodal model to obtain a multimodal model more suitable for the execution of this task.

[0075] In a possible implementation manner, the attention mechanism based on the instruction feature weights the initial image feature to obtain a weighted image feature focused on the target area of the user instruction, including:

[0076] The weighted image feature focused on the target area of the user instruction is obtained through the following formula

[0077]

[0078] Among them, v T is the instruction feature, and v I is the initial image feature. is the dimension of v I , and softmax(·) is the weighted coefficient conversion function.

[0079] Explanatory, the attention mechanism utilizes the instruction feature v T and the initial image feature v I , and calculates the importance weight of each image region through the attention mechanism. This attention mechanism can be dot-product-based attention or a more complex Transformer layer. In this embodiment, the above cross-attention calculation formula is adopted to calculate the attention weight.

[0080] Explanatory, the weighted image feature focusing on the user instruction target region refers to weighting the initial image feature v I according to the calculated attention weight to highlight the region more relevant to the user instruction. After being weighted by the attention mechanism, the purpose is to ignore irrelevant elements and make the task evaluation more focused.

[0081] In a possible implementation manner, the obtaining the task completion degree value according to the similarity between the instruction feature and the weighted image feature includes:

[0082] Obtaining the similarity between the instruction feature and the weighted image feature through the following formula

[0083]

[0084] Among them, v T is the instruction feature, is the weighted image feature.

[0085] Or obtaining the similarity between the instruction feature and the weighted image feature through the following formula

[0086]

[0087] Among them, is the k-th dimensional vector of v T , is 's k-th dimensional vector, n is the dimension of v T and , and σ is an adjustable parameter used to control the distribution range of the similarity score and can be optimized according to the actual data set.

[0088] The task completion value C is obtained by the following formula multi (T, I):

[0089]

[0090] where θ H is the preset first high threshold, and θ L is the preset first low threshold.

[0091] Explanatorily, the similarity can be calculated by the cosine similarity method. Calculate the cosine similarity between the instruction feature v T and the weighted image feature , which is a similarity metric method for measuring the angle between two vectors. Optionally, another optional similarity metric method is to calculate the Euclidean distance between two feature vectors, which measures the actual distance of the vectors in the Euclidean space. The smaller the Euclidean distance, the more similar the two vectors are. Since the smaller the Euclidean distance indicates the more similar the two vectors are, the Euclidean distance can be converted into a similarity score.

[0092] Explanatorily, according to the calculated similarity the task completion degree can be judged. If the similarity reaches a certain threshold, the task is considered completed; if the similarity is low, the task is not completed. θ H and θ L respectively represent the high and low standards for task completion degree evaluation. If the similarity score is higher than θ H , it is considered that the task completion degree is very high; if the similarity score is between θ L and θ H , it is considered that the task completion degree is moderate, and at this time, it is adjusted according to the linear formula; if the similarity score is lower than θ L , it is considered that the task completion degree is low.

[0093] Exemplarily, determine one or more thresholds θ H , θ L for evaluating whether the similarity score indicates that the task completion degree meets the standard. These thresholds can be determined based on historical data, expert knowledge or experiments. Other similar evaluation formulas that can be used to quantitatively represent the multi-modal task scores are still applicable to the present invention.

[0094] In a possible implementation manner, referring to Figure 4 , the process completion value based on the action summary obtained from the user instruction and the intelligent agent assistant execution data includes:

[0095] S31: Use a pre-trained text generation model to summarize the intelligent agent assistant execution data to obtain an action summary.

[0096] S32: Vectorize the user instruction and action summary to obtain instruction features and action summary features.

[0097] S33: Obtain and determine the process completion value based on the similarity between the instruction features and the action summary features.

[0098] Explanatory, the execution data of the agent assistant includes the actions called by the agent assistant during the execution of the task. And for the purpose of adapting to subsequent operations, this action sequence should be in text form. Optionally, this text action sequence can be passed in the form of HTML, in the form of JSON, etc. Optionally, in addition to the action sequence, other text data related to task execution generated by the large model inside the agent assistant during the execution of the agent assistant, such as reflections, plans, and memories, can also be passed in.

[0099] Exemplary, the pre-trained text generation model refers to a text generation model suitable for sequence summarization, which can be used by calling the API or in the form of local deployment, such as DeepSeek series models and QWEN series models, etc. At the same time, when using the model for summarization, a system prompt can be predefined to explain the task purpose and task mode.

[0100] Exemplary, the collected action data can be cleaned and formatted to meet the input requirements of the selected model. A text model specifically for summarizing action sequences can also be trained using the preprocessed data.

[0101] Explanatory, the execution data of the agent assistant is a set A={a1, a2, a3, ……} of text descriptions of a series of actions performed by the agent assistant during the execution of the task. Among them, each action a i is in the text form of an operation step, recording each interaction of the agent assistant. Use a pre-trained text generation model to summarize a series of action data and convert it into a concise natural language description. The pre-trained text generation model can be a large language model or a pre-trained small model M such as BERT. This model can comprehensively analyze multiple actions in the action sequence A and compress them into a concise natural language summary Sum. This summary should be able to retain the core operation steps and task objectives during the execution process. That is, Sum = M(A).

[0102] Exemplarily, vectorize the user instruction and action summary, including selecting a vectorization model to perform vectorization processing on the user instruction and action summary; obtain instruction features and action summary features for the user instruction and action summary. Optionally, the selected vectorization model can be a model capable of processing text tasks such as the BERT model. Optionally, the generated instruction features and action summary features can also be dimensionally adjusted to match the requirements of subsequent processing steps.

[0103] In one possible implementation manner, the obtaining the process completion value according to the similarity between the instruction feature and the action summary feature includes:

[0104] Obtain the similarity S(v T ,v Sum ) of the instruction feature and the action summary feature through the following formula:

[0105]

[0106] where, v T is the instruction feature, v Sum is the action summary feature, ||v T || is the norm of v T , ||v Sum || is the norm of v Sum .

[0107] Or obtain the similarity S(v T ,v Sum ) of the instruction feature and the action summary feature through the following formula:

[0108]

[0109] where, v Tk is the k-th dimensional vector of v T , is the k-th dimensional vector of v sum , n is the dimension of and , and σ is an adjustable parameter used to control the distribution range of the similarity score and can be optimized according to the actual data set.

[0110] Obtain the process completion value C action (T,Sum) through the following formula:

[0111]

[0112] where, γ H is the preset second highest threshold, and γ L is the preset second lowest threshold.

[0113] Explanatorily, according to the calculated similarity S(vT , v Sum ) is compared with a set threshold. If the similarity score is higher than γ H , it is considered that the task completion degree is very high; if the similarity score is between γ L and γ H , it is considered that the task completion degree is moderate, and at this time, it is adjusted according to a linear formula; if the similarity score is lower than γ L , it is considered that the task completion degree is low; optionally, one or more thresholds γ H , γ L are determined to evaluate whether the similarity score indicates that the task completion meets the standard. These thresholds can be determined based on historical data, expert knowledge or experiments.

[0114] In a possible implementation manner, the weighted superposition of the task completion value and the process completion value is used to obtain a final evaluation score, and according to the deviation between the final evaluation score and the user feedback data, the filtering result of the user feedback data includes: The final evaluation score C is obtained by the following formula:

[0115]

[0116] where α is a preset weight, C multi is the task completion value, C action is the process completion value, is the first indicator function. When C multi ≠ 0 and C action ≠ 0 hold, the return value of the first indicator function is 1, otherwise the return value is 0.

[0117] The filtering result indicator value Filter of the user feedback data is obtained by the following formula:

[0118]

[0119] where, is the second indicator function, C feedback is the user feedback data, Λ is a preset threshold. When |C - C feedback | > A holds, the return value of the second indicator function is 1, otherwise the return value is 0.

[0120] When the filtering result indicator value Filter is 1, the filtering result of the user feedback data is filtered; when the filtering result indicator value Filter is 1, the filtering result of the user feedback data is not filtered.

[0121] Explanatory, based on the task completion value of the text-image multi-modal and the process completion value of the action summary, calculate the final evaluation score. If the difference between this final evaluation score and the user feedback data exceeds a preset threshold, it is considered that the user feedback data may be inaccurate or malicious, and these suspicious feedbacks will be filtered out. This process not only improves the accuracy of the user feedback data, but also enhances the robustness of the agent assistant to malicious or incorrect feedback, thus optimizing the performance of the agent assistant and the user experience.

[0122] Explanatory, the final evaluation score is based on the weighted superposition of the task completion value and the process completion value, and its weight should be related to the data itself and the required screening quality; the calculation of the final evaluation score involves the weighting of the task completion value based on the text-image multi-modal and the process completion value based on the action summary. The allocation of weights should be determined according to the characteristics of the data and the required screening quality. For example, if the text-image multi-modal evaluation can more accurately reflect the task completion, a higher weight may be given.

[0123] Explanatory, by comparing the difference between the final evaluation score and the user feedback data. If the difference exceeds the threshold, it is considered that the user feedback data is inaccurate or malicious, and these user feedback data will be filtered out.

[0124] Explanatory, the preset threshold Λ is related to the required screening quality, and the preset threshold Λ can be set according to the required screening quality. The higher the threshold, it means that only when the user feedback data is highly consistent with the final evaluation score, the user feedback data will be accepted, which helps to improve the overall data quality of the user feedback data, but may lead to a reduction in the amount of user feedback data.

[0125] Exemplary, by adjusting the preset threshold Λ, a balance can be found between data quality and quantity. A higher preset threshold Λ may lead to a reduction in the amount of data, but can ensure the high quality of the remaining data. On the contrary, a lower preset threshold Λ may increase the amount of data, but may contain more inaccurate or malicious feedback. In practical applications, the weights and the preset threshold Λ can be dynamically adjusted according to the performance of the agent assistant and the actual situation of the user feedback to achieve the best screening effect.

[0126] The method for filtering user feedback data of the agent assistant of the present invention realizes the efficient processing and accurate evaluation of the user feedback data in the agent assistant through an automated workflow, improves the efficiency and accuracy of data processing, reduces the need for manual intervention, and saves costs.

[0127] The following is the device embodiment of the present invention, which can be used to execute the method embodiment of the present invention. For the details not disclosed in the device embodiment, please refer to the method embodiment of the present invention.

[0128] SeeFigure 5 , in another embodiment of the present invention, there is provided an intelligent agent assistant user feedback data filtering system, which can be used to implement the above-mentioned intelligent agent assistant user feedback data filtering method. Specifically, the intelligent agent assistant user feedback data filtering system includes a data acquisition module, a completion evaluation module, and a decision-making module.

[0129] Among them, the data acquisition module is used to acquire user instructions, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data; the completion evaluation module is used to obtain a task completion value based on text and image multimodality according to the user instructions and the intelligent agent assistant completion screenshots, and obtain a process completion value based on action summary according to the user instructions and the intelligent agent assistant execution data; the decision-making module is used to weightedly superimpose the task completion value and the process completion value to obtain a final evaluation score, and obtain a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

[0130] All relevant contents of each step involved in the foregoing embodiments of the intelligent agent assistant user feedback data filtering method can be cited in the function descriptions of the corresponding functional modules of the intelligent agent assistant user feedback data filtering system in the embodiments of the present invention, and will not be elaborated herein.

[0131] The division of modules in the embodiments of the present invention is illustrative, only a logical function division. In actual implementation, there may be other division methods. In addition, in each embodiment of the present invention, each functional module can be integrated in one processor, or can exist alone physically, or two or more modules can be integrated in one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0132] In another embodiment of the present invention, a computer device is provided. The computer device includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the intelligent agent assistant user feedback data filtering method.

[0133] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the intelligent agent assistant user feedback data filtering method in the above embodiments.

[0134] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0135] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0136] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent substitutions, and any modification or equivalent substitution that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. An intelligent agent assistant user feedback data filtering method, characterized in that, Including: Obtain user instructions, agent assistant execution data, agent assistant completed screenshots, and user feedback data; Obtain a task completion value based on text and image multimodality according to the user instructions and the agent assistant completed screenshots, and obtain a process completion value based on action summary according to the user instructions and the agent assistant execution data; Weightedly superimpose the task completion value and the process completion value to obtain a final evaluation score, and obtain a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

2. The intelligent agent assistant user feedback data filtering method according to claim 1, wherein The obtaining a task completion value based on text and image multimodality according to the user instructions and the agent assistant completed screenshots includes: Through the same multimodal model, obtain the feature representation of the user instructions as instruction features, and obtain the feature representation of the agent assistant completed screenshots as initial image features; weight the initial image features by the attention mechanism based on the instruction features to obtain weighted image features focused on the target area of the user instructions; obtain and obtain the task completion value according to the similarity between the instruction features and the weighted image features.

3. The intelligent agent assistant user feedback data filtering method according to claim 2, wherein The weighting the initial image features by the attention mechanism based on the instruction features to obtain weighted image features focused on the target area of the user instructions includes: The weighted image features focused on the target area of the user instruction are obtained by the following formula Among them, v T is an instruction feature, v I is an initial image feature, is the dimension of v I , and softmax(·) is a weighted coefficient conversion function.

4. The intelligent agent assistant user feedback data filtering method according to claim 2, wherein The obtaining and obtaining the task completion value according to the similarity between the instruction features and the weighted image features includes: Obtain the similarity between the instruction feature and the weighted image feature through the following formula Among them, v T is an instruction feature, and is a weighted image feature; Or obtain the similarity between the instruction feature and the weighted image feature through the following formula Among them, is the k-th dimensional vector of v T , is the k-th dimensional vector of, n is the dimension of v T and , and σ is an adjustable parameter; The task completion value C is obtained by the following formula multi (T, I): where θ H is a preset first high threshold, and θ L is a preset first low threshold.

5. The method for filtering agent assistant user feedback data according to claim 1, wherein The obtaining a process completion value based on action summary according to the user instructions and the agent assistant execution data includes: using a pre-trained text generation model to summarize the agent assistant execution data to obtain an action summary; vectorize the user instructions and the action summary to obtain instruction features and action summary features; obtain and obtain the process completion value according to the similarity between the instruction features and the action summary features.

6. The intelligent agent assistant user feedback data filtering method according to claim 1, wherein The obtaining and obtaining the process completion value according to the similarity between the instruction features and the action summary features includes: The similarity S(v T , v Sum ) between the instruction feature and the action summary feature is obtained by the following formula: Among them, v T is an instruction feature, v Sum is an action summary feature, ||v T || is the norm of v T ||, ||v Sum || is the norm of v Sum ; Or obtain the similarity S(v T , v Sum ) between the instruction feature and the action summary feature through the following formula: Among them, is the k-th dimensional vector of v T , is the k-th dimensional vector of v sum , n is the dimension of and , and σ is an adjustable parameter; The process completion value C is obtained by the following formula action (T, Sum): where γ H is a preset second highest threshold, and γ L is a preset second lowest threshold.

7. The method for filtering intelligent agent assistant user feedback data according to claim 1, wherein The weighting and superimposing the task completion value and the process completion value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data includes: Obtain the final evaluation score C through the following formula: Among them, α is a preset weight, C multi is the task completion value, C action is the process completion value, is the first indicator function. When C multi ≠0 and C action ≠0 holds, the return value of the first indicator function is 1, otherwise the return value is 0; Obtain the filtering result indication value Filter of the user feedback data through the following formula: Among them, is the second indication function, C feedback is the user feedback data, Λ is the preset threshold. When |C - C feedback | > A holds, the return value of the second indication function is 1, otherwise the return value is 0; When the filtering result indication value Filter is 1, the filtering result of the user feedback data is filtered; when the filtering result indication value Filter is 1, the filtering result of the user feedback data is not filtered.

8. An intelligent agent assistant user feedback data filtering system, characterized in that, Including: A data acquisition module, configured to acquire user instructions, agent assistant execution data, agent assistant completed screenshots, and user feedback data; A completion evaluation module, configured to obtain a task completion value based on text and image multimodality according to the user instructions and the agent assistant completed screenshots, and obtain a process completion value based on action summary according to the user instructions and the agent assistant execution data; A decision module, configured to weightedly superimpose the task completion value and the process completion value to obtain a final evaluation score, and obtain a filtering result of the user feedback data according to the deviation between the final evaluation score and the user feedback data.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method for filtering user feedback data of the agent assistant according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the intelligent agent assistant user feedback data filtering method according to any one of claims 1 to 7.

Images