Intelligent agent assistant user feedback data filtering method and related apparatus

Abstract

The application belongs to the technical field of artificial intelligence, and discloses an agent assistant user feedback data filtering method and related device, which comprises obtaining user instructions, agent assistant execution data, agent assistant completed screenshots and user feedback data; obtaining a task completion degree value based on a graphic-text multimodal according to the user instructions and the agent assistant completed screenshots, and obtaining a process completion degree value based on an action summary according to the user instructions and the agent assistant execution data; weighting and superimposing the task completion degree value and the process completion degree value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data according to the final evaluation score and the deviation of the user feedback data. The task completion degree of the agent assistant is evaluated from the graphic-text multimodal and the action summary, which effectively improves the filtering accuracy of the user feedback data, enhances the robustness of the agent assistant to malicious or incorrect user feedback data, and optimizes the performance and user experience of the agent assistant.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method and related device for filtering user feedback data of an intelligent assistant. Background Technology

[0002] In recent years, breakthroughs in the understanding and reasoning capabilities of large-scale models have prompted researchers to re-examine the problem of automating mobile tasks, shifting their focus to a model-centric perspective. Under this perspective, intelligent agent assistant systems based on multimodal large-scale models have emerged, such as BrowserUse and AutoGLM. These systems first parse the user's natural language commands, then integrate advanced visual perception tools to capture and analyze the visual and textual elements of the mobile application interface in real time, comprehensively perceiving the operating environment. Based on the understanding of commands and scenarios, intelligent agent assistant systems use a unified language model for task planning and decomposition, breaking down complex operations into specific steps and commands to ensure efficient execution.

[0003] Data poisoning attacks refer to malicious attackers injecting malicious or harmful data into training data to influence the model's learning process, thereby damaging its performance or manipulating its behavior. In the supply chain of intelligent assistants, attackers can exploit user feedback mechanisms to inject forged or manipulated data into the training dataset, causing the agent's internal model to become biased or perform erroneous operations. To ensure the quality of training data, existing data cleaning techniques, including statistical methods, machine learning methods, and deep learning methods, are widely used in data preprocessing. These methods can effectively identify and correct errors, inconsistencies, and noise in the data, thereby improving the accuracy and reliability of the data.

[0004] However, existing data cleaning techniques still have many shortcomings in the application of user feedback data in intelligent agent assistants. On the one hand, traditional single-modal behavior evaluation may ignore the interaction between different modal information and overlook the fact that multimodal evaluation using images and text can provide a deeper understanding of the agent's comprehensive capabilities in a multimodal environment. On the other hand, traditional intelligent agent behavior evaluation methods struggle to understand the multiple steps performed by the intelligent agent assistant, making it difficult to validate data cleaning strategies according to specific application scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for filtering user feedback data in an intelligent assistant.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] In a first aspect, the present invention provides a method for filtering user feedback data of an intelligent agent assistant, comprising: acquiring user instructions, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data; obtaining a task completion value based on image and text multimodality based on the user instructions and intelligent agent assistant completion screenshots, and obtaining a process completion value based on action summary based on the user instructions and intelligent agent assistant execution data; weighting and superimposing the task completion value and the process completion value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data based on the deviation between the final evaluation score and the user feedback data.

[0008] Optionally, obtaining the task completion value based on the image and text multimodal model by taking a screenshot based on the user instruction and the intelligent assistant includes: obtaining the feature representation of the user instruction as the instruction feature through the same multimodal model, and obtaining the feature representation of the screenshot taken by the intelligent assistant as the initial image feature; weighting the initial image feature based on the attention mechanism of the instruction feature to obtain the weighted image feature focused on the target area of ​​the user instruction; and obtaining the task completion value based on the similarity between the instruction feature and the weighted image feature.

[0009] Optionally, the attention mechanism based on instruction features weights the initial image features to obtain weighted image features focused on the user instruction target region, including:

[0010] The weighted image features focused on the target region of the user command are obtained by the following formula.

[0011]

[0012] Among them, v T As a characteristic of instructions, v I As initial image features, For v I The dimension is softmax(·), which is the weighted coefficient transformation function.

[0013] Optionally, obtaining the task completion value based on the similarity between the instruction features and the weighted image features includes:

[0014] The similarity between instruction features and weighted image features is obtained using the following formula.

[0015]

[0016] Among them, v T As a characteristic of instructions, These are weighted image features.

[0017] Alternatively, the similarity between the instruction features and the weighted image features can be obtained using the following formula.

[0018]

[0019] in, For v T The k-th dimension vector, for The k-th dimension vector, where n is v T and The dimension is σ, which is an adjustable parameter.

[0020] The task completion value C is obtained using the following formula. multi (T,I):

[0021]

[0022] Where, θ H To preset the first high threshold, θ L The preset first low threshold is set.

[0023] Optionally, obtaining the process completion value based on the action summary according to the user instructions and the intelligent agent assistant execution data includes: summarizing the intelligent agent assistant execution data using a pre-trained text generation model to obtain an action summary; vectorizing the user instructions and action summary to obtain instruction features and action summary features; and obtaining the process completion value based on the similarity between the instruction features and the action summary features.

[0024] Optionally, obtaining the process completion value based on the similarity between instruction features and action summary features includes:

[0025] The similarity S(v) between instruction features and action summary features is obtained by the following formula. T ,v Sum ):

[0026]

[0027] Among them, v T As a characteristic of instructions, v Sum To summarize the characteristics of the action, ||v T ||For v T The norm of ||v Sum ||For v Sum The norm of .

[0028] Alternatively, the similarity S(v) between instruction features and action summary features can be obtained using the following formula. T ,v Sum ):

[0029]

[0030] in, For vT The k-th dimension vector, For v sum The k-th dimension vector, where n is and The dimension is σ, which is an adjustable parameter.

[0031] The process completion value C is obtained using the following formula. action (T,Sum):

[0032]

[0033] Where, γ H To preset the second highest threshold, γ L This is the preset second lowest threshold.

[0034] Optionally, the weighted sum of the task completion value and the process completion value yields the final evaluation score, and the filtering result of the user feedback data is obtained based on the deviation between the final evaluation score and the user feedback data, including:

[0035] The final evaluation score C is obtained using the following formula:

[0036]

[0037] Where α is the preset weight, C multi C represents the task completion rate. action This represents the process completion rate. As the first indicator function, when C multi ≠0andC action If ≠0 is true, the first indicator function returns 1; otherwise, it returns 0.

[0038] The filter result indicator value Filter of the user feedback data is obtained by the following formula:

[0039]

[0040] in, For the second indicator function, C feedback For user feedback data, Λ is a preset threshold, when |CC feedback When |>Λ is true, the second indicator function returns 1; otherwise, it returns 0.

[0041] When the filter result indicator value Filter is 1, the user feedback data is filtered; when the filter result indicator value Filter is 1, the user feedback data is not filtered.

[0042] In a second aspect, the present invention provides a user feedback data filtering system for an intelligent agent assistant, comprising: a data acquisition module for acquiring user instructions, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data; a completion evaluation module for obtaining a task completion value based on image and text multimodality according to user instructions and intelligent agent assistant completion screenshots, and obtaining a process completion value based on action summary according to user instructions and intelligent agent assistant execution data; and a decision module for weighted summarization of the task completion value and the process completion value to obtain a final evaluation score, and obtaining a filtering result of the user feedback data based on the deviation between the final evaluation score and the user feedback data.

[0043] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described intelligent assistant user feedback data filtering method.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described intelligent assistant user feedback data filtering method.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] This invention discloses a method for filtering user feedback data in an intelligent assistant. It obtains a task completion value based on a multimodal approach (text and image) by analyzing user commands and screenshots taken by the intelligent assistant, and a process completion value based on action summaries by analyzing user commands and execution data from the intelligent assistant. The method evaluates the intelligent assistant's task completion from both multimodal and action summary perspectives. A final evaluation score is then calculated by weighting and combining the task completion value and the process completion value. The deviation between the final evaluation score and the user feedback data is used to obtain the filtered user feedback data. By combining precise multimodal and action analysis, this method effectively improves the accuracy of filtering user feedback data and enhances the robustness of the intelligent assistant against malicious or erroneous user feedback data, thereby optimizing the performance and user experience of the intelligent assistant. Attached Figure Description

[0047] Figure 1 This is a flowchart of the user feedback data filtering method for an intelligent assistant according to an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the application environment of the intelligent assistant according to an embodiment of the present invention.

[0049] Figure 3 This is a flowchart illustrating the process of obtaining the task completion value according to an embodiment of the present invention.

[0050] Figure 4 This is a flowchart illustrating the process completion value acquisition process according to an embodiment of the present invention.

[0051] Figure 5 This is a block diagram of the user feedback data filtering system for the intelligent assistant according to an embodiment of the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings:

[0055] See Figure 1 In one embodiment of the present invention, a method for filtering user feedback data of an intelligent assistant is provided, which can accurately filter user feedback data to improve the retraining effect of the intelligent assistant.

[0056] Specifically, the intelligent assistant user feedback data filtering method of the present invention includes the following steps:

[0057] S1: Obtain user commands, intelligent assistant execution data, intelligent assistant completion screenshots, and user feedback data.

[0058] S2: Obtain the task completion value based on image and text multimodal processing by taking screenshots according to user instructions and intelligent assistant, and obtain the process completion value based on action summary according to user instructions and intelligent assistant execution data.

[0059] S3: The final evaluation score is obtained by weighting and combining the task completion value and the process completion value. Based on the deviation between the final evaluation score and the user feedback data, the filtered results of the user feedback data are obtained.

[0060] This invention discloses a method for filtering user feedback data in an intelligent assistant. It obtains a task completion value based on a multimodal approach (text and image) by analyzing user commands and screenshots taken by the intelligent assistant, and a process completion value based on action summaries by analyzing user commands and execution data from the intelligent assistant. The method evaluates the intelligent assistant's task completion from both multimodal and action summary perspectives. A final evaluation score is then calculated by weighting and combining the task completion value and the process completion value. The deviation between the final evaluation score and the user feedback data is used to obtain the filtered user feedback data. By combining precise multimodal and action analysis, this method effectively improves the accuracy of filtering user feedback data and enhances the robustness of the intelligent assistant against malicious or erroneous user feedback data, thereby optimizing the performance and user experience of the intelligent assistant.

[0061] In an explanatory sense, artificial intelligence (AI) is the ability to simulate human intelligent behavior and, through technologies such as machine learning and deep learning, endows computer systems with the capabilities of self-learning, reasoning, judgment, and decision-making. It is widely applied in fields such as autonomous driving, intelligent customer service, and financial risk control. This invention primarily focuses on training data cleaning. Training data cleaning refers to the process of removing or correcting noise, errors, and outliers in training data through a series of technical means, particularly identifying and removing malicious and invalid data. This ensures that the AI ​​model can learn correct and realistic information during training, thereby improving the model's robustness and generalization ability.

[0062] Explanatory, see Figure 2 This paper illustrates a feasible application environment for an intelligent agent assistant. In this environment, the user interacts with the intelligent agent assistant, inputting user commands. The intelligent agent assistant interacts with the terminal to complete task execution. Specifically, the intelligent agent assistant understands the user commands and uses its internal large model, vision components, and tools to perceive, plan, and make decisions regarding task execution, ultimately completing the task. The terminal for this intelligent agent can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, and other devices with a graphical user interface. After the user commands are completed, the user can provide feedback on the task execution status, i.e., user feedback data. Finally, the intelligent agent developer organizes the user feedback data for training or fine-tuning the model to optimize the execution efficiency of the intelligent agent assistant. Representative intelligent agent assistants include MobileAgent and BrowserUse. The user commands can be text commands.

[0063] The intelligent assistant user feedback data filtering method of the present invention is applied to Figure 2 Let's take a mobile terminal as an example. This terminal is a mobile-based intelligent assistant. The assistant uses an internal large model to understand user commands, combines this understanding with its knowledge of the mobile phone's interface, and infers the next action to be performed. Finally, it determines the action to be executed and sends it to the mobile phone for execution. Based on the assistant's behavior, the user provides a score to the application provider—this is the user feedback data. This user feedback data, along with the assistant's behavior records, will serve as the dataset for retraining the assistant's internal large model. Before retraining, invalid user feedback data needs to be filtered out.

[0064] Explanatory user feedback data refers to user evaluations of the intelligent assistant's performance during task execution, typically presented as a rating system. Intelligent assistant execution data comprises detailed process data, generally including records of each operation and screenshots taken before and after each operation to assess the user interface. User commands are instructions provided by the user to the intelligent assistant in natural language, specifying the concrete actions or objectives to be performed. The intelligent assistant completion screenshot is the last screenshot taken before the operation is completed.

[0065] Specifically, through integrated logging and user interface interaction, feedback data can be captured in real time. This feedback data includes not only user feedback but also details of the intelligent assistant's interactions during task execution, such as clicks and swipes, as well as screenshots before and after task completion.

[0066] In one possible implementation, see Figure 3 The step of obtaining a task completion value based on image and text multimodal processing by taking screenshots according to user instructions and intelligent assistants includes:

[0067] S21: Using the same multimodal model, obtain the feature representation of the user's instruction as the instruction feature, and obtain the feature representation of the screenshot completed by the intelligent assistant as the initial image feature.

[0068] S22: The initial image features are weighted by an attention mechanism based on the instruction features to obtain weighted image features focused on the target region of the user instruction.

[0069] S23: Obtain the task completion score based on the similarity between the instruction features and the weighted image features.

[0070] Interpretive, a single multimodal model refers to a model capable of simultaneously processing and understanding multiple different types of data inputs (such as text, images, etc.). It extracts and fuses feature representations from different modalities through deep learning techniques. A key characteristic of this model is its ability to establish connections between different modalities and support cross-modal tasks, such as the correlation analysis between images and text. The joint feature representation of text and images refers to the vector representation in a unified feature space obtained after processing text and image data through the multimodal model. This joint feature vector can capture and express the semantic information of text and images, enabling heterogeneous data to be compared and computed within the same space. In this way, the model can more accurately evaluate the agent's task completion and perform effective decision-making and reasoning in a multimodal environment.

[0071] For example, the CLIP multimodal model is selected as the multimodal model. By inputting the user command T and the screenshot I completed by the intelligent agent assistant respectively, the command feature v is obtained. T and initial image features v I .

[0072] In the CLIP model, the image encoder is typically ResNet or Vision Transformer (ViT), while the text encoder is based on a Transformer architecture, such as BERT or GPT-2. Therefore, the feature vectors from the image encoder and text encoder are mapped through a linear layer to obtain feature vectors with the same dimensionality, and then L2 normalized. Thus, the feature vector after projection through the linear layer is selected as the chosen feature vector. Through this process, the instruction feature v generated according to the user instruction T is obtained. T It contains key information about the user's instruction, such as the target task and operation type; simultaneously, it generates initial image features v based on the screenshot taken by the intelligent assistant. I .

[0073] For example, other models can be selected as the chosen multimodal model, such as CLI-VIT.

[0074] For example, a pre-processed dataset can be used to pre-train or fine-tune the selected multimodal model to obtain a multimodal model that is more suitable for the task.

[0075] In one possible implementation, the attention mechanism based on instruction features weights the initial image features to obtain weighted image features focused on the user instruction target region, including:

[0076] The weighted image features focused on the target region of the user command are obtained by the following formula.

[0077]

[0078] Among them, v T As a characteristic of instructions, v I As initial image features, For v I The dimension is softmax(·), which is the weighted coefficient transformation function.

[0079] Interpretive attention mechanisms utilize instruction features v T and initial image features v I The importance weight of each image region is calculated using an attention mechanism. This attention mechanism can be based on dot product attention or a more complex Transformer layer. In this embodiment, the cross-attention calculation formula described above is used to calculate the attention weights.

[0080] Interpretive, weighted image features focused on the target region of the user command refer to the initial image features v weighted according to the calculated attention weights. I Weighting is applied to highlight areas more relevant to user instructions. This attention-based weighting aims to ignore irrelevant elements, making task evaluation more focused.

[0081] In one possible implementation, obtaining the task completion value based on the similarity between the instruction features and the weighted image features includes:

[0082] The similarity between instruction features and weighted image features is obtained using the following formula.

[0083]

[0084] Among them, v T As a characteristic of instructions, These are weighted image features.

[0085] Alternatively, the similarity between the instruction features and the weighted image features can be obtained using the following formula.

[0086]

[0087] in, For v T The k-th dimension vector, for The k-th dimension vector, where n is v T and The dimension σ is an adjustable parameter used to control the distribution range of similarity scores, which can be optimized according to the actual dataset.

[0088] The task completion value C is obtained using the following formula. multi (T,I):

[0089]

[0090] Where, θ H To preset the first high threshold, θ L The preset first low threshold is set.

[0091] Interpretively, similarity can be calculated using cosine similarity. Calculate the instruction feature v. T and weighted image features Cosine similarity is a metric used to measure the similarity between two vectors at an angle. Alternatively, another alternative similarity metric is the Euclidean distance between two feature vectors, which measures the actual distance between the vectors in Euclidean space. A smaller Euclidean distance indicates greater similarity between the two vectors. Since a smaller Euclidean distance signifies greater similarity, it can be converted into a similarity score.

[0092] Interpretive, based on calculated similarity This allows us to determine the task's completion rate. If the similarity reaches a certain threshold, the task is considered complete; if the similarity is low, the task is considered incomplete. θ H and θ L These represent the high and low standards for task completion evaluation. If the similarity score is higher than θ, then... H If the similarity score is within θ, then the task completion rate is considered very high; if the similarity score is between θ and θ, then the task completion rate is considered very high. L and θ H If the similarity score is between θ and θ, the task completion rate is considered moderate, and adjustments are made according to the linear formula; if the similarity score is below θ... L If the completion rate is low, then the task completion rate is considered low.

[0093] For example, determine one or more thresholds θ H ,θ L The similarity score is used to assess whether the task completion rate is satisfactory. These thresholds can be determined based on historical data, expert knowledge, or experiments. Other similar evaluation formulas that can be used to quantify multimodal task scores are also applicable to this invention.

[0094] In one possible implementation, see Figure 4 The process completion value obtained based on action summary according to user instructions and intelligent assistant execution data includes:

[0095] S31: Use a pre-trained text generation model to summarize the action summary of the agent assistant's execution data.

[0096] S32: Vectorize the user commands and actions to obtain command features and action summary features.

[0097] S33: Obtain the process completion value based on the similarity between instruction features and action summary features.

[0098] Interpretive data from the intelligent agent assistant includes the actions invoked by the assistant during task execution. This action sequence should be in text format to adapt to subsequent operations. Optionally, this text action sequence can be passed in HTML, JSON, etc. Alternatively, in addition to the action sequence, other text data related to task execution generated by the intelligent agent assistant's internal large model, such as reflection, planning, and memory, can also be passed in.

[0099] For example, the pre-trained text generation model refers to a text generation model suitable for sequence summarization. It can be used by calling an API or deployed locally, such as the DeepSeek series models and the QWEN series models. Furthermore, when using the model for summarization, a predefined system prompt can be used to explain the task objective and task mode.

[0100] For example, the collected motion data can be cleaned and formatted to suit the input requirements of the selected model. Alternatively, a text model specifically designed to summarize motion sequences can be trained using the preprocessed data.

[0101] Interpretive, the intelligent agent assistant's execution data is a set of text descriptions A = {a1, a2, a3, ...} of a series of actions performed by the intelligent agent assistant while executing a task, where each action a i This is a textual record of the operational steps, documenting each interaction of the intelligent assistant. A pre-trained text generation model is used to summarize a series of action data, transforming it into a concise natural language description. This pre-trained text generation model can be a large language model or a pre-trained small model M such as BERT. This model can comprehensively analyze multiple actions in the action sequence A and compress it into a concise natural language summary, Sum. This summary should retain the core operational steps and task objectives during execution. That is, Sum = M(A).

[0102] For example, vectorizing user commands and action summaries includes selecting a vectorization model to process the user commands and action summaries; and obtaining command features and action summary features for the user commands and action summaries. Optionally, the selected vectorization model can be a model capable of handling text tasks, such as the BERT model. Optionally, the dimensions of the generated command features and action summary features can also be adjusted to match the requirements of subsequent processing steps.

[0103] In one possible implementation, obtaining the process completion value based on the similarity between instruction features and action summary features includes:

[0104] The similarity S(v) between instruction features and action summary features is obtained by the following formula. T ,v Sum ):

[0105]

[0106] Among them, v T As a characteristic of instructions, v Sum To summarize the characteristics of the action, ||v T ||For v T The norm of ||v Sum ||For v Sum The norm of .

[0107] Alternatively, the similarity S(v) between instruction features and action summary features can be obtained using the following formula. T ,v Sum ):

[0108]

[0109] Among them, v Tk For v T The k-th dimension vector, For v sum The k-th dimension vector, where n is and The dimension σ is an adjustable parameter used to control the distribution range of similarity scores, which can be optimized according to the actual dataset.

[0110] The process completion value C is obtained using the following formula. action (T,Sum):

[0111]

[0112] Where, γ H To preset the second highest threshold, γ L This is the preset second lowest threshold.

[0113] Interpretive, based on the calculated similarity S(v)T ,v Sum The similarity score is compared with a set threshold; if the similarity score is higher than γ, the similarity score is not higher. H If the similarity score is within γ, then the task completion rate is considered very high; if the similarity score is between γ and γ, then the task completion rate is considered very high. L and γ H If the similarity score is between γ and γ, the task completion rate is considered moderate, and adjustments are made according to the linear formula; if the similarity score is below γ... L If the threshold γ is not met, the task completion rate is considered low; alternatively, one or more thresholds γ can be determined. H ,γ L These similarity scores are used to assess whether a task completion rate is met. These thresholds can be determined based on historical data, expert knowledge, or experiments.

[0114] In one possible implementation, the weighted summation of task completion value and process completion value yields the final evaluation score. Based on the deviation between the final evaluation score and the user feedback data, the filtered result of the user feedback data is obtained, including: obtaining the final evaluation score C using the following formula:

[0115]

[0116] Where α is the preset weight, C multi C represents the task completion rate. action This represents the process completion rate. As the first indicator function, when C multi ≠0andC action If ≠0 is true, the first indicator function returns 1; otherwise, it returns 0.

[0117] The filter result indicator value Filter of the user feedback data is obtained by the following formula:

[0118]

[0119] in, For the second indicator function, C feedback For user feedback data, Λ is a preset threshold, when |CC feedback If A is true, the second indicator function returns 1; otherwise, it returns 0.

[0120] When the filter result indicator value Filter is 1, the user feedback data is filtered; when the filter result indicator value Filter is 1, the user feedback data is not filtered.

[0121] Interpretive evaluation calculates the final score based on task completion values ​​using a multimodal approach (text and images) and process completion values ​​based on action summaries. If the difference between this final score and the user feedback data exceeds a preset threshold, the user feedback data is considered potentially inaccurate or malicious, and such suspicious feedback is filtered out. This process not only improves the accuracy of user feedback data but also enhances the robustness of the intelligent assistant to malicious or erroneous feedback, thereby optimizing the performance of the intelligent assistant and the user experience.

[0122] Interpretive, the final evaluation score is based on a weighted sum of task completion and process completion values, with the weights determined by the data itself and the required screening quality. The calculation of the final evaluation score involves a weighted sum of task completion values ​​based on text-image multimodal assessment and process completion values ​​based on action summaries. The allocation of weights should be determined based on the characteristics of the data and the required screening quality. For example, if text-image multimodal assessment more accurately reflects task completion, it may be given a higher weight.

[0123] Interpretive analysis is performed by comparing the final rating score with the user feedback data. If the difference exceeds a threshold, the user feedback data is considered inaccurate or malicious, and this user feedback data will be filtered out.

[0124] Interpretively, the preset threshold Λ is related to the required screening quality, and can be set according to the desired screening quality. A higher threshold means that user feedback data will only be accepted when it is highly consistent with the final rating score. This helps improve the overall data quality of user feedback data, but may result in a reduction in the amount of user feedback data.

[0125] For example, by adjusting the preset threshold Λ, a balance can be found between data quality and quantity. A higher preset threshold Λ may result in a smaller amount of data, but it can ensure the high quality of the remaining data. Conversely, a lower preset threshold Λ may increase the amount of data, but it may include more inaccurate or malicious feedback. In practical applications, the weights and preset threshold Λ can be dynamically adjusted based on the performance of the intelligent assistant and the actual user feedback to achieve the best filtering effect.

[0126] The present invention provides a method for filtering user feedback data in an intelligent assistant. Through an automated workflow, it achieves efficient processing and accurate evaluation of user feedback data in the intelligent assistant, improves the efficiency and accuracy of data processing, reduces the need for manual intervention, and saves costs.

[0127] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0128] See Figure 5 In another embodiment of the present invention, a smart agent assistant user feedback data filtering system is provided, which can be used to implement the above-mentioned smart agent assistant user feedback data filtering method. Specifically, the smart agent assistant user feedback data filtering system includes a data acquisition module, a completion evaluation module, and a decision module.

[0129] The data acquisition module is used to acquire user commands, intelligent agent assistant execution data, intelligent agent assistant completion screenshots, and user feedback data. The completion evaluation module is used to obtain a task completion value based on image and text multimodal processing based on user commands and intelligent agent assistant completion screenshots, and to obtain a process completion value based on action summary based on user commands and intelligent agent assistant execution data. The decision module is used to weight and superimpose the task completion value and the process completion value to obtain the final evaluation score, and to obtain the filtering result of user feedback data based on the deviation between the final evaluation score and the user feedback data.

[0130] All relevant content of each step involved in the aforementioned embodiments of the intelligent assistant user feedback data filtering method can be referenced to the functional description of the corresponding functional module of the intelligent assistant user feedback data filtering system in the embodiments of the present invention, and will not be repeated here.

[0131] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0132] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a user feedback data filtering method in an intelligent assistant.

[0133] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the intelligent assistant user feedback data filtering method in the above embodiments.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for filtering user feedback data in an intelligent assistant, characterized in that, include: Acquire user commands, intelligent assistant execution data, screenshots of intelligent assistant completion, and user feedback data; The task completion value based on image and text multimodal processing is obtained by taking screenshots based on user instructions and intelligent assistant, and the process completion value based on action summary is obtained by taking screenshots based on user instructions and intelligent assistant execution data. The final evaluation score is obtained by weighting and combining the task completion value and the process completion value. Based on the deviation between the final evaluation score and the user feedback data, the filtered results of the user feedback data are obtained. The process of obtaining a multimodal task completion value based on images and text by taking screenshots according to user instructions and an intelligent assistant includes: Using the same multimodal model, the feature representation of the user's instruction is obtained as the instruction feature, and the feature representation of the screenshot completed by the intelligent assistant is obtained as the initial image feature. The initial image feature is weighted by an attention mechanism based on the instruction feature to obtain a weighted image feature focused on the target region of the user instruction. The task completion value is obtained and obtained based on the similarity between the instruction feature and the weighted image feature. The process of obtaining the task completion score based on the similarity between the instruction features and the weighted image features includes: The similarity between instruction features and weighted image features is obtained using the following formula. : in, As a characteristic of instructions, Weighted image features; Alternatively, the similarity between the instruction features and the weighted image features can be obtained using the following formula. : in, for The k A dimensional vector, for The k A dimensional vector, n for and The dimension is σ, which is an adjustable parameter; The task completion value is obtained using the following formula. : in, To preset the first high threshold, The first low threshold is preset; The process completion value based on action summary obtained from user instructions and intelligent assistant execution data includes: summarizing the intelligent assistant execution data using a pre-trained text generation model to obtain action summary; vectorizing user instructions and action summary to obtain instruction features and action summary features; and obtaining and obtaining the process completion value based on the similarity between instruction features and action summary features. The process completion value is obtained by acquiring and calculating the similarity between instruction features and action summary features, including: The similarity between instruction features and action summary features is obtained using the following formula. : in, As a characteristic of instructions, Summarize the characteristics of the action. for norm, for The norm; Alternatively, the similarity between instruction features and action summary features can be obtained using the following formula. : in, for The k A dimensional vector, for The k A dimensional vector, n for and The dimension is σ, which is an adjustable parameter; The process completion value is obtained using the following formula. : in, To preset the second highest threshold, This is the preset second lowest threshold.

2. The intelligent assistant user feedback data filtering method according to claim 1, characterized in that, The attention mechanism based on instruction features weights the initial image features to obtain weighted image features focused on the user instruction target region, including: The weighted image features focused on the target region of the user command are obtained by the following formula. : in, As a characteristic of instructions, As initial image features, for Dimensions This is the weighted coefficient transformation function.

3. The method for filtering user feedback data in an intelligent assistant according to claim 1, characterized in that, The weighted sum of task completion value and process completion value yields the final evaluation score. Based on the deviation between the final evaluation score and the user feedback data, the filtered results of the user feedback data are obtained, including: The final evaluation score is obtained using the following formula. : in, To preset weights, This represents the task completion rate. This represents the process completion rate. For the first indicator function, when If true, the first indicator function returns 1; otherwise, it returns 0. The filtering result indicator value of user feedback data is obtained by the following formula. : in, For the second indicator function, For user feedback data, For the preset threshold, when If true, the second indicator function returns 1; otherwise, it returns 0. When the filter result indicator value When the value is 1, the filtered result of the user feedback data is filtered; when the filter result indicator value is 1... When the value is 0, the filtering result for user feedback data is that it is not filtered.

4. A user feedback data filtering system for an intelligent assistant based on the user feedback data filtering method for an intelligent assistant as described in claim 1, characterized in that, include: The data acquisition module is used to acquire user commands, intelligent assistant execution data, screenshots taken by the intelligent assistant, and user feedback data. The completion evaluation module is used to obtain a task completion value based on image and text multimodality by taking screenshots based on user instructions and intelligent assistant, and to obtain a process completion value based on action summary by taking screenshots based on user instructions and intelligent assistant execution data. The decision module is used to weight and superimpose the task completion value and the process completion value to obtain the final evaluation score, and to obtain the filtered results of user feedback data based on the deviation between the final evaluation score and the user feedback data.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent assistant user feedback data filtering method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent assistant user feedback data filtering method as described in any one of claims 1 to 3.

Images