Discrimination method, model training method and device, discrimination system and related product

Abstract

The invention provides a discrimination method, a model training method and device, a discrimination system and a related product, and belongs to the technical field of artificial intelligence. The judgment method comprises the steps of calling a first judgment model to perform first judgment on a target agent based on target execution data under the condition that a judgment request for the target agent is received, and obtaining a first judgment result; under the condition that the first judgment result meets a preset screening condition, calling a second judgment model to perform second judgment on the target agent based on the target execution data to obtain a second judgment result; wherein the target execution data is execution data which is carried in the discrimination request and is used for representing that the target agent executes the target instruction, and the preset screening condition is used for determining whether to call a second discrimination model to discriminate the target agent again according to the confirmation degree of the first discrimination result. According to the embodiment of the invention, resource overhead and time delay can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a discrimination method, a model training method, a discrimination device, a model training device, a discrimination system, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of Large Language Models (LLMs) and Vision-Language Models (VLMs), Graphical User Interface (GUI) agents (such as Computer-Using Agents, CUAs) are widely used to automate tasks on mobile devices or computers. To reduce error accumulation in long-process tasks, discriminative models can be introduced to evaluate each step of the GUI agent's operation. In related technologies, large multimodal models with a large number of parameters (e.g., 7 bytes or more) are typically used as discriminative models, and a uniform discrimination method is applied to all operations. This can easily lead to wasted computing power and high latency, making it difficult to meet real-time interaction requirements. Summary of the Invention

[0003] This disclosure provides a discrimination method, a model training method, a discrimination device, a model training device, a discrimination system, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, this disclosure provides a discrimination method applied to a discrimination system. The discrimination system includes a first discrimination model and a second discrimination model for discriminating the instruction execution status of a user graphical interface agent, wherein the number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model. The discrimination method includes: upon receiving a discrimination request for a target agent, invoking the first discrimination model to perform a first discrimination on the target agent based on target execution data to obtain a first discrimination result; and if the first discrimination result satisfies a preset filtering condition, invoking the second discrimination model to perform a second discrimination on the target agent based on the target execution data to obtain a second discrimination result. The target execution data is the execution data carried in the discrimination request, used to characterize the target agent's execution of a target instruction, and the preset filtering condition is used to determine whether to invoke the second discrimination model to re-discriminate the target agent based on the confidence level of the first discrimination result.

[0005] Secondly, this disclosure provides a model training method, which includes: inputting training data into an initial discriminant model for second discrimination to obtain a predicted discriminant result; the training data including sample instructions from a sample agent, operation sequences corresponding to the sample instructions, and at least one screenshot of the sample agent executing the sample instructions on a user graphical interface; the predicted discriminant result including a predicted discriminant conclusion and predicted inference text; inputting the predicted discriminant result and at least one screenshot into a first discriminant model for semantic consistency analysis to obtain a consistency reward component; the consistency reward component being used to characterize the degree of consistency between the predicted inference text and the predicted discriminant result; adjusting some model parameters of the initial discriminant model according to the consistency reward component and a preset general reward component; and obtaining a second discriminant model based on the current initial discriminant model when a preset convergence condition is met; wherein the first discriminant model and the second discriminant model constitute a discriminant system, and the discriminant system is used to execute the above-described discriminant method.

[0006] Thirdly, this disclosure provides a discrimination device applied to a discrimination system. The discrimination system includes a first discrimination model and a second discrimination model for discriminating the instruction execution status of a user graphical interface agent. The number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model. The discrimination device includes: a first discrimination module, configured to, upon receiving a discrimination request for a target agent, invoke the first discrimination model to perform a first discrimination on the target agent based on target execution data, obtaining a first discrimination result; and a second discrimination module, configured to, if the first discrimination result satisfies preset filtering conditions, invoke the second discrimination model to perform a second discrimination on the target agent based on the target execution data, obtaining a second discrimination result; wherein the target execution data is the execution data carried in the discrimination request, used to characterize the target agent's execution of target instructions, and the preset filtering conditions are used to determine whether to invoke the second discrimination model to re-discriminate the target agent based on the degree of certainty of the first discrimination result.

[0007] Fourthly, this disclosure provides a model training apparatus, comprising: a prediction module, used to input training data into an initial discriminant model for second discrimination to obtain a prediction discriminant result, wherein the training data includes sample instructions of a sample agent, operation sequences corresponding to the sample instructions, and at least one screenshot of the sample agent executing the sample instructions on a user graphical interface, and the prediction discriminant result is determined to include a prediction discriminant conclusion and prediction inference text; an analysis module, used to input the prediction discriminant result and at least one screenshot of the interface into a first discriminant model for semantic consistency analysis to obtain a consistency reward component, wherein the consistency reward component is used to characterize the degree of consistency between the prediction inference text and the prediction discriminant result; an adjustment module, used to adjust some model parameters of the initial discriminant model according to the consistency reward component and a preset general reward component; and an acquisition module, used to obtain a second discriminant model based on the current initial discriminant model when a preset convergence condition is met; wherein the first discriminant model and the second discriminant model constitute a discriminant system, and the discriminant system is used to execute the above-described discriminant method.

[0008] Fifthly, this disclosure provides a discrimination system comprising: a cascaded first discrimination model and a second discrimination model; wherein the discrimination system is used to perform the discrimination method described above.

[0009] In a sixth aspect, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the aforementioned discrimination method or model training method.

[0010] In a seventh aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described discrimination method or model training method.

[0011] Eighthly, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described discrimination method or model training method.

[0012] In this embodiment of the disclosure, the discrimination system includes a first discrimination model and a second discrimination model. The former has fewer parameters than the latter. Correspondingly, the former requires less resource overhead and latency during runtime than the latter. Based on this system architecture, when discriminating against a target intelligent agent, the smaller first discrimination model is first invoked to perform a simple first discrimination, quickly and efficiently obtaining the first discrimination result. Only when the confidence level of the first discrimination result is low will the larger second discrimination model be invoked to perform a complex second discrimination against the target intelligent agent, thereby achieving a second discrimination of the target intelligent agent and obtaining the corresponding second discrimination result.

[0013] In other words, the embodiments of this disclosure can quickly intercept simple errors of the target agent using a small-scale first discrimination model, reducing resource overhead and latency. At the same time, when the first discrimination model cannot give a first discrimination result with a high degree of confidence, the complex logic of the second discrimination model can be introduced to perform a second discrimination on the target agent to obtain a second discrimination result with relatively higher accuracy, thereby effectively improving the accuracy of the discrimination.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of a discrimination method provided in an embodiment of this disclosure;

[0017] Figure 2 A schematic diagram illustrating a discrimination method provided in an embodiment of this disclosure;

[0018] Figure 3 A flowchart of a model training method provided in this embodiment of the disclosure;

[0019] Figure 4 A block diagram of a discrimination device provided in an embodiment of this disclosure;

[0020] Figure 5 A block diagram of a model training apparatus provided in an embodiment of this disclosure;

[0021] Figure 6A block diagram of a discrimination system provided in an embodiment of this disclosure;

[0022] Figure 7 A block diagram of an electronic device provided in an embodiment of this disclosure;

[0023] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0026] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0028] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example, appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely identifying specific individuals.

[0030] In related technologies, large single-model systems are typically used as the discrimination model for GUI agents to achieve high discrimination accuracy. For example, multimodal large models with a large number of parameters (such as 7B or more) can be used as the discrimination model. Furthermore, during the discrimination process, regardless of whether the GUI agent performs a simple click operation or an operation with complex logic, the large model is invoked to perform complete visual encoding and text generation inference. For simple operations, this incurs the same resource overhead and latency as complex operations, resulting in resource waste and reduced resource utilization. Additionally, current training of discrimination models typically employs rule-based consistency training methods. This means that during model training, to ensure consistency between the model's "reasoning process" and "discrimination conclusion (e.g., pass / fail (Yes / No))," a pre-set lexicon and corresponding rule matching methods are used. For example, reward signals are calculated by statistically analyzing the frequency of keywords such as "error" and "invalid," thereby updating the model parameters. This reward calculation method is too rigid and makes it difficult to understand complex semantic expressions (such as double negations and euphemisms), resulting in unstable training gradients and limiting the upper limit of model performance.

[0031] Furthermore, existing discrimination models typically employ a binary feedback approach, where the model only outputs a pass or fail conclusion, and may also output some textual reasons, without providing specific error correction location information. This is equivalent to only informing the agent that an error has occurred, but without pointing out where the correct error is. Therefore, the agent can only blindly retry, which easily leads to extremely low task repair efficiency.

[0032] In view of the above, embodiments of this disclosure provide a discrimination method, a model training method, a discrimination device, a model training device, a discrimination system, an electronic device, a computer-readable storage medium, and a computer program product.

[0033] In this embodiment of the disclosure, the discrimination of the target agent is performed by using a first discrimination model and a second discrimination model of different scales. For some simple discrimination requests, the first discrimination model can accurately and quickly make a discrimination and give a first discrimination result. For cases where the first discrimination model cannot give a certain discrimination result, the second discrimination model can be further called to perform a relatively complex discrimination, thereby giving a second discrimination result with higher accuracy and ensuring the accuracy of the discrimination.

[0034] Firstly, embodiments of this disclosure provide a discrimination method.

[0035] Figure 1 This is a flowchart illustrating a discrimination method provided in an embodiment of the present disclosure. This discrimination method can be applied to a discrimination system, which includes a first discrimination model and a second discrimination model for judging the instruction execution status of a user graphical interface agent. The number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model. (Refer to...) Figure 1 The discrimination method may include the following steps:

[0036] Step S11: Upon receiving a discrimination request for the target agent, the first discrimination model is invoked to perform a first discrimination on the target agent based on the target execution data, and a first discrimination result is obtained;

[0037] Step S12: If the first discrimination result meets the preset screening conditions, the second discrimination model is called to perform a second discrimination on the target agent based on the target execution data to obtain the second discrimination result;

[0038] Among them, the target execution data is the execution data carried in the discrimination request, which is used to characterize the target agent's execution of the target instruction. The preset filtering conditions are used to determine whether to call the second discrimination model to re-discriminate the target agent based on the degree of confidence of the first discrimination result.

[0039] For example, a graphical user interface (GUI) agent is an artificial intelligence proxy system that can visually understand a graphical interface and simulate human interactions such as clicking and inputting. A GUI agent can "see" graphical interface elements on the screen, such as buttons, text boxes, and icons, and understand how to click and fill in information, ultimately completing tasks such as online shopping, form filling, and information searching. Furthermore, unlike traditional automated scripts, a GUI agent does not execute data processing along a fixed path; instead, it can understand the purpose first, plan the path, and finally execute the corresponding operations, just like a human.

[0040] In some alternative implementations, the first and second discriminant models are used to discriminate the user interface agent. These models can be cascaded; input data is first processed by the first discriminant model, and then it is determined whether it needs to enter the second discriminant model for further processing. Therefore, the second discriminant model does not process all input data indiscriminately, but only processes a portion of it. This selective processing method reduces the amount of data processed, resource overhead, and latency.

[0041] Furthermore, as mentioned earlier, the first discriminant model has fewer parameters than the second discriminant model. This difference stems primarily from the fact that the first discriminant model is a lightweight model, typically used for convenient discrimination of target agents through methods such as visual scanning. In contrast, the second discriminant model can be an expert-level inference engine, such as a multimodal large model, capable of performing complex inference and suitable for discriminating target agents in complex scenarios and tasks. Therefore, for the same input data, the first discriminant model requires less resource overhead and has lower latency than the second discriminant model when performing data processing.

[0042] Based on the aforementioned model characteristics, this embodiment constructs a cascaded architecture consisting of a "lightweight gatekeeper-like" first discrimination system and an "expert-level inferencer-like" second discrimination model. Preset filtering conditions are set to determine whether the second discrimination model needs to be further invoked to re-discriminate the target agent based on the confidence level of the first discrimination result. In other words, the discrimination system of this embodiment can quickly intercept simple errors about the target agent using the first discrimination model. Only when the first discrimination model cannot provide a confident first discrimination result is it necessary to further utilize the second model to perform complex logical reasoning to obtain a more accurate and reliable second discrimination result. This approach breaks the limitation of uniformly using a large-scale second discrimination model for reasoning and discrimination in all situations, reduces the need to use the second model for reasoning and discrimination of simple errors, thereby reducing unnecessary resource overhead.

[0043] Therefore, in this embodiment, the discrimination system includes a cascaded first discrimination model and a second discrimination model. The former has fewer parameters than the latter, and correspondingly, the former requires less resource overhead and latency during runtime. Based on this system architecture, when discriminating against a target agent, the smaller first discrimination model is first invoked to perform simple logic for first discrimination, quickly and efficiently obtaining the first discrimination result. Only when the confidence level of the first discrimination result is low will the larger second discrimination model be invoked to perform complex logic for second discrimination against the target agent, achieving a second discrimination of the target agent and obtaining the corresponding second discrimination result. In other words, this embodiment can utilize a small-scale first discrimination model to quickly intercept simple errors of the target agent, reducing resource overhead and latency. Furthermore, when the first discrimination model cannot provide a highly confident first discrimination result, the complex logic of the second discrimination model can be introduced to perform a second discrimination of the target agent, obtaining a relatively more accurate second discrimination result, thereby effectively improving the accuracy of the discrimination.

[0044] The discrimination method of the present disclosure embodiments will be described in detail below.

[0045] In some alternative implementations, a discrimination request for a target agent refers to a request to perform a discrimination on the target agent, such as a request to determine whether the target agent can accurately execute the target instruction; wherein, the target agent includes a user graphical interface agent.

[0046] In some optional implementations, the discrimination request carries target execution data, which can characterize the execution data of the target agent executing the target instruction. It can include data that reflects the process of the target agent executing the target instruction, or data that reflects the execution result of the target agent on the target instruction. This disclosure does not limit this.

[0047] In some optional implementations, the target execution data may include target instructions and at least one screenshot, which is a screenshot of the target agent executing the target instructions on a preset user graphical interface.

[0048] In some alternative implementations, the target execution data includes target instructions and execution video, wherein the execution video includes one or more video slices of the target agent executing the target instructions on a preset user graphical interface, and each video slice includes multiple video frames.

[0049] For example, the target instruction is a shopping instruction, which instructs the target agent to purchase a specified product on a preset shopping website. The screenshot is a screenshot of the target agent performing the shopping process on the preset shopping website, and the number of screenshots can be one or more.

[0050] In some optional implementations, the target execution data includes a target instruction, a corresponding operation sequence, and at least one screenshot. The operation sequence comprises multiple operations, and there is a correspondence between the screenshot and at least one operation. The operation sequence refers to a sequence of multiple GUI operations generated by interpreting the processing instruction's intent and planning the task. In other words, the operation sequence is a set of atomic UI interaction actions arranged in execution order, used to instruct the target agent to act sequentially on the graphical user interface to complete the target task corresponding to the target instruction.

[0051] For example, the target instruction is a shopping instruction, which instructs the target intelligent agent to purchase a specified product on a preset shopping website. The operation sequence includes entering the preset shopping website, logging into the preset shopping website, searching and filtering products, adding products to the cart, submitting the order, and settling the order. The interface screenshot is a screenshot of the target intelligent agent performing the shopping process on the preset shopping website, and each interface screenshot corresponds to one or more operations. An operation may have one or more corresponding interface screenshots, or it may not have a corresponding interface screenshot. For example, the interface screenshot may include the interface screenshot corresponding to submitting the order.

[0052] Therefore, target execution data can include target instructions and related data (such as screenshots and operation sequences). The target instructions clarify which data processing the target agent needs to perform, and the related data reflects whether the target agent executes the target instructions accurately and orderly. Based on this, the first discriminant model (or the second discriminant model) can perform discriminative processing using the target execution data, thereby providing the corresponding first discriminant result (or second discriminant result). The first and second discriminant results reflect whether the target agent executes the target instructions accurately, completely, and efficiently.

[0053] It should be noted that in the discrimination system, the first discrimination model and the second discrimination model are connected in a cascaded manner. When a discrimination request is received, in order to reduce the waste of resources caused by directly calling the large-scale second discrimination model to perform discrimination for simple scenarios or simple tasks, the discrimination system will first call the smaller first discrimination model to perform the first discrimination and obtain the first discrimination result. Based on this, if the confidence level of the first discrimination result is low, the second discrimination model can be called to perform the second discrimination. Conversely, if the confidence level of the first discrimination result is high, there is no need to call the second discrimination model, and the discrimination process stops after obtaining the first discrimination result.

[0054] In some optional implementations, the confidence level of the first discrimination result can be characterized by the confidence level. Based on this, for cases with high confidence, the discrimination process can be stopped after obtaining the first discrimination result. For cases with low confidence, the second discrimination model is further called to perform a more complex discrimination to obtain a more accurate and reliable second discrimination result.

[0055] In some optional implementations, the first discrimination result includes a first discrimination conclusion and a confidence level corresponding to the first discrimination conclusion. The first discrimination conclusion is used to characterize whether the target agent passes the discrimination. After calling the first discrimination model to perform the first discrimination on the target agent based on the target execution data and obtaining the first discrimination result, the discrimination method may further include: if the confidence level is less than or equal to a preset confidence threshold, determining that the first discrimination result meets preset screening conditions, so as to obtain a second discrimination result based on the second discrimination model, and using the second discrimination result as the final discrimination result of the target agent; if the confidence level is greater than the preset confidence threshold, determining that the first discrimination result does not meet the preset screening conditions, so as to use the first discrimination result as the final discrimination result of the target agent.

[0056] For example, the first discrimination result includes the first discrimination conclusion S and the corresponding confidence level P. The first discrimination conclusion includes two cases: discrimination pass and discrimination fail. If P≤Thr, it is determined that the first discrimination result meets the preset screening conditions, and the second discrimination model needs to be called to perform the second discrimination. The obtained second discrimination result is used as the final discrimination result. If P>Thr, it is determined that the first discrimination result does not meet the preset screening conditions, and the first discrimination result can be directly used as the final discrimination result without calling the second discrimination model again, which saves resource consumption, reduces latency, and improves the real-time performance of interaction. Here, Thr is a preset confidence threshold.

[0057] Therefore, after obtaining a first discrimination result with a high degree of confidence, the discrimination request can be intercepted and sent to the second discrimination model to reduce unnecessary resource overhead. After obtaining a first discrimination result with a low degree of confidence, the second discrimination model can be called to perform further discrimination to improve the accuracy and reliability of the discrimination.

[0058] In some optional implementations, after obtaining the final judgment result, it can be sent to the target agent and / or its downstream agents. The downstream agent refers to a GUI agent that is located after the target agent and adjacent to or near the target agent in terms of data flow, task flow direction, etc. For example, the downstream agent needs to further execute the processing instructions received by the downstream agent based on the result of the target agent's instruction execution. For instance, the downstream agent can only execute its received processing instructions after the target agent has executed the target instruction.

[0059] In some alternative implementations, the discrimination method may further include: sending the final discrimination result to the target agent to instruct the target agent to correct and re-execute the target instruction based on the final discrimination result.

[0060] In some optional implementations, the discrimination method may further include: sending the final discrimination result to the downstream agent of the target agent to instruct the downstream agent to re-execute the target instruction based on the final discrimination result, and executing the instruction received by the downstream agent based on the execution result of the target instruction.

[0061] Therefore, once the final judgment result is obtained, it can be fed back to the target agent and / or downstream agents. If the target agent receives the final judgment result, it can correct itself based on the result to better execute the target instruction or similar processing instructions. Furthermore, the target agent can re-execute the target instruction based on the judgment result. If the downstream agent receives the final judgment result, it can execute the target instruction on behalf of the target agent based on the final judgment result and other information, and then execute the processing instructions it received. This eliminates the need for the target agent to repeatedly execute the target instruction, thus saving processing time and improving processing efficiency.

[0062] In some optional implementations, the first discrimination model is invoked to perform a first discrimination on the target agent based on the target execution data to obtain a first discrimination result. This includes: inputting the target execution data into the first discrimination model so that the first discrimination model can determine the target area and its sub-states in the interface screenshot through visual scanning, and determine the first discrimination result based on the target area and its sub-states; wherein, the interface screenshot is a screenshot of the target agent executing the target instruction on a preset user graphical interface.

[0063] Visual scanning refers to the first discrimination model performing global and / or directional visual analysis on the screenshot of the target agent's interface, determining the region of interest as the target region, and extracting the corresponding interface elements, text information, and other information as the sub-states of the target region. Based on this, the model completes the discrimination of the target agent's operation state, task progress, behavior correctness, etc., and obtains the corresponding discrimination results.

[0064] In some optional implementations, when the target execution data includes target instructions and at least one screenshot, the first discrimination model performs the first discrimination process including: for any screenshot, determining a first target region and its first sub-state in the screenshot according to the target instructions; visually scanning the screenshot to determine a second target region and its second sub-state in the screenshot; determining a first screenshot discrimination sub-result based on the first target region, the first sub-state, the second target region, and the second sub-state; and, if the first screenshot discrimination sub-results for all screenshots are obtained, determining a first discrimination result based on all the first screenshot discrimination sub-results, wherein the target region includes the first target region and the second target region, and the sub-state includes the first sub-state and the second sub-state.

[0065] In other words, when a target instruction and at least one screenshot are input into the first discrimination model, during the first discrimination process, for each screenshot, the first discrimination model can, on the one hand, determine the first target area in the screenshot that is highly related to the target instruction and the first sub-state that the first target area should possess, based on the target instruction; on the other hand, it can determine the second target area by visually scanning the screenshot, and determine the second sub-state based on the actual state of the second target area. By comparing the first target area, the first sub-state, the second target area, and the second sub-state, the similarities and differences are determined, and the corresponding first screenshot discrimination sub-result is obtained based on these similarities and differences. Furthermore, if there are multiple screenshots, the first discrimination result is determined jointly based on the multiple first screenshot discrimination sub-results. The first sub-state can characterize several attributes of the corresponding first target area, such as whether the first target area is configured with a certain interface element, the text content of the first target area, whether the first target area is selected, and whether the first target area can be clicked. The second sub-state is similar, and this embodiment does not limit this.

[0066] For example, the target instruction determines that the controls in the first target area of ​​the screenshot should be selected (i.e., the first sub-state is selected), while visual scanning determines that the controls in the corresponding second target area are unselected (i.e., the second sub-state is unselected). The first and second target areas correspond to the same locations in the screenshot. Based on this, the corresponding first screenshot discrimination sub-result can be determined as failing, thus determining the first discrimination result as failing. It should be noted that the confidence level of the first discrimination result as failing can be further determined to reflect its degree of certainty.

[0067] In some optional implementations, when the target execution data includes a target instruction, an operation sequence corresponding to the target instruction, and at least one screenshot, the first discrimination model performs the first discrimination process as follows: for any screenshot, determine a first target region and its first sub-state in the screenshot based on the target instruction and the operation corresponding to the screenshot; perform a visual scan of the screenshot based on the operation corresponding to the screenshot to determine a second target region and its second sub-state in the screenshot; determine a second screenshot discrimination sub-result based on the first target region, the first sub-state, the second target region, and the second sub-state; and, if the second screenshot discrimination sub-results for all screenshots are obtained, determine a first discrimination result based on all the second screenshot discrimination sub-results, wherein the target region includes the first target region and the second target region, and the sub-state includes the first sub-state and the second sub-state.

[0068] Therefore, compared to target execution data that only includes target instructions and interface screenshots, if the target execution data also includes operation sequences, it is equivalent to providing more reference information for the first discrimination model, thereby improving the discrimination accuracy of the first discrimination model.

[0069] In some optional implementations, the second discriminant model is invoked to make a second discrimination against the target agent based on the target execution data, and a second discrimination result is obtained. This includes: inputting the target execution data into the second discriminant model so that the second discriminant model determines the second discrimination result through thought chain reasoning.

[0070] Therefore, the second discriminant model can perform deep reasoning based on the chain of thought (CoT), thereby determining a more accurate and reliable second discriminant result. The chain of thought (CoT) refers to the ability to break down the reasoning process of the second discriminant model into explicit, ordered, and traceable logical steps. This reflects the model's reasoning process, helps the model better understand and solve problems, and significantly improves its ability to solve complex problems. Returning to the embodiments of this disclosure, the second discriminant model, through the chain of thought, can improve its ability to discriminate target agents in complex scenarios and tasks, thereby obtaining more accurate discriminant results.

[0071] In some optional implementations, the target execution data includes a target instruction and at least one screenshot, or the target execution data includes a target instruction, an operation sequence corresponding to the target instruction, and at least one screenshot, wherein the screenshot is a screenshot of the target agent executing the target instruction on a preset user graphical interface, the operation sequence includes multiple operations, and there is a correspondence between the screenshot and at least one operation; accordingly, the second discrimination model performs the second discrimination process including: performing thought chain reasoning based on the target execution data, determining the actual execution state of the target agent through thought chain reasoning, and determining the second discrimination result based on the actual execution state of the target agent.

[0072] The actual execution status includes task completion degree and logical sequence status. Task completion degree can reflect the degree to which the target agent completes the target task corresponding to the target instruction. Logical sequence status can characterize whether the logical sequence adopted by the target agent in the process of executing the target instruction is accurate and reasonable.

[0073] It should be noted that the above description of the actual execution state is merely an example, and the embodiments disclosed herein do not impose any limitations on it.

[0074] Therefore, it can be seen that the second discriminant model is larger in scale than the first discriminant model, and thus can support more complex reasoning, including reasoning based on thought chains, judging the actual execution state of the target agent through reasoning, and obtaining a more accurate second discriminant result based on the actual execution state.

[0075] In some optional implementations, if the discrimination result indicates that the target agent has failed, the generated first discrimination result and / or second discrimination result can carry regression position information, which is the information of the correct position. This can instruct the target agent and / or downstream agents to perform corresponding operations in the corresponding correct area based on the regression position information, thereby effectively alleviating the problem of low processing efficiency caused by the agent blindly retrying when it fails the discrimination.

[0076] In some optional implementations, if the first discrimination result indicates that the target agent fails the discrimination, the first discrimination result also includes first regression position information, which is the information of the correct position in the user graphical interface corresponding to the error position of the target agent, determined by the first discrimination model based on the target execution data; or, if the second discrimination result indicates that the target agent fails the discrimination, the second discrimination result also includes second regression position information and / or inference text, whereby the second regression position information is the information of the correct position in the user graphical interface corresponding to the error position of the target agent, determined by the second discrimination model based on the target execution data, and the inference text is used to represent the thought chain-based inference process and inference result of the second discrimination model.

[0077] For example, if the first discrimination model determines that the target agent fails the discrimination and determines that the reason for the failure is that the mouse did not click on the target button, based on this, the first discrimination model can determine the first regression position information according to the position information of the target button in the user's graphical interface, thereby guiding the target agent and / or downstream agent to accurately click on the target button when executing the target instruction again, so as to accurately execute the target instruction.

[0078] In some alternative implementations, the first discriminant model can determine the first regression location information through a visual regression network structure.

[0079] For example, during the training of the first discriminative model, a visual regression head can be introduced to learn the ability to predict the bounding box, center position, and other information representing the correct position of the operation, thereby enabling the output of the first regression position information during the inference stage.

[0080] For example, if the confidence level of the first discrimination result obtained by the first discrimination model is less than a preset confidence threshold, a second discrimination model is introduced based on thought chain reasoning to perform discrimination on the target agent again. Furthermore, if the second discrimination model determines that the target agent failed the discrimination, and determines that the reason for the failure was that the mouse did not click on the target button, then the second discrimination model can, on the one hand, output corresponding reasoning text based on the thought chain to instruct the target agent to replan the execution method of the target instruction; on the other hand, it can determine the second regression position information based on the position information of the target button in the user's graphical interface, thereby guiding the target agent and / or downstream agents to accurately click the target button when executing the target instruction again, so as to accurately execute the target instruction.

[0081] In some alternative implementations, for scenarios where a target agent makes purchases on a shopping platform, an evaluation system is used to assess the target agent to determine whether the target agent has accurately executed the target instruction for purchasing the specified goods on the shopping platform.

[0082] First, a discrimination request for the target agent, carrying target execution data, is received. This request is parsed to obtain the target execution data, which is then input into the first discrimination model to perform a first discrimination, yielding a first discrimination result. The target execution data includes a target instruction, a screenshot of the interface, and an operation sequence, including selecting a product and submitting an order. Correspondingly, the first discrimination process includes: after processing the target execution data through visual scanning, the first discrimination model determines that the pixels of the selected product in the shopping cart area are blank, and the mouse clicks on the submit button. Since the first discrimination model is a lightweight model, it focuses more on physical-level verification, such as determining whether the mouse position is reasonable and accurate. Based on this, the first discrimination model gives a first discrimination conclusion that the discrimination has passed. However, considering that the pixels of the selected product are blank, the corresponding confidence level P is relatively low.

[0083] Furthermore, the confidence level P is compared with the preset confidence threshold Thr, and it is determined that P < Thr. Therefore, the second discriminant model is invoked to further discriminate the target agent.

[0084] The target execution data is input into the second discriminant model, which performs a thought chain reasoning based on the input data: "The user wants to purchase a specific product, but the current shopping cart is empty. Clicking the submit button directly will cause an error. The user should add the specific product to be purchased first." Based on this, the second discriminant result is output: the discriminant fails because "the shopping cart is empty".

[0085] Furthermore, the second judgment result can be fed back to the target agent and / or downstream agents, allowing the target agent to correct itself or re-execute the target instruction based on the second judgment result. Since the reason given for the "shopping cart is empty" is provided, the target agent can adjust its execution method to ensure that the specified item is selected before clicking the submit button. Similarly, the downstream agent can accurately execute the target instruction based on the second judgment result, and then execute its own received processing instructions, thus speeding up the processing progress and improving the processing efficiency of the GUI agent.

[0086] In some alternative implementations, for scenarios where a target agent performs a setting update on a preset client, a discrimination system is used to discriminate the target agent to determine whether the target agent accurately executed the target instruction for changing settings on the preset client.

[0087] First, the system receives a discrimination request for the target agent, carrying target execution data. It parses the request to obtain the target execution data and inputs it into the first discrimination model to perform a first discrimination, resulting in a first discrimination result. The target execution data includes a target instruction and a screenshot of the interface. Correspondingly, the first discrimination model performs the following process: after visual scanning and other processing based on the target execution data, the first discrimination model determines that the setting button is located in the target area of ​​the user's graphical interface, while the mouse icon representing the click behavior is outside the target area. Based on this, it determines that the click behavior cannot act on a valid control (i.e., the setting button). Therefore, the first discrimination model outputs a first discrimination conclusion that it has failed the discrimination, with a high confidence level. In addition, the first discrimination model generates first regression position information based on the position of the setting button, using this information to guide the target agent or downstream agent to execute the target instruction. The first regression position information can be the center coordinates of the setting button [x0, y0], or a range of positions for the entire area where the setting button is located [x1, x2; y1, y2].

[0088] Figure 2 This is a schematic diagram of a discrimination method provided in an embodiment of this disclosure. (Refer to...) Figure 2 A discrimination system is used to perform discrimination on a target intelligent agent. Its input data may include the target intelligent agent's target instructions, operation sequence and interface screenshot. The discrimination system includes a cascaded first discrimination model, a second discrimination model and a decision node. The decision node is a processing node used to determine whether to call the second discrimination model.

[0089] Upon receiving the aforementioned input data, the first discrimination model is invoked to perform the first discrimination, such as through visual scanning, to determine and output the first discrimination result. The specific processing may include Task A, Task B, and Task C. Task A is a binary classification task, and its processing result corresponds to the first discrimination conclusion, used to characterize whether the target agent passes the discrimination. The result value is either pass or fail, or it can be yes or no, etc., and this embodiment does not impose any limitations on this. Task B is a confidence calculation task, used to calculate the confidence level of the task processing result of Task A to characterize its degree of certainty. Task C is a visual regression task, corresponding to the first regression position information, used to output the correct position information when the target agent fails the discrimination, to guide the target agent and / or downstream agents to execute the target instruction.

[0090] Furthermore, such as Figure 2As shown, after obtaining the first discrimination result, the decision node needs to determine whether to perform an interception operation. The interception operation refers to intercepting the discrimination request, and therefore not calling the second discrimination system to perform a second discrimination. Not performing the interception operation means calling the second discrimination system to perform a second discrimination on the target agent in order to obtain a more accurate discrimination result.

[0091] In some optional implementations, if the confidence level in the first discrimination result is high, an interception operation is performed; if the confidence level in the first discrimination result is low, no interception operation is performed. The determination of the confidence level can be based on a preset confidence threshold, or on processing requirements, experience, statistical data, simulation data, etc. This disclosure does not impose any limitations on this method.

[0092] Furthermore, if an interception operation is performed, the first discrimination result can be used as the final discrimination result and quickly fed back to the target agent and / or downstream agent; if an interception operation is not performed, the second discrimination model performs thought chain reasoning based on the aforementioned input data to determine several actual execution states of the target agent, and based on this, determines and outputs the second discrimination result, which is fed back to the target agent and / or downstream agent as the final discrimination result.

[0093] Secondly, this disclosure provides a model training method.

[0094] Figure 3 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 3 The model training method may include the following steps:

[0095] Step S31: Input the training data into the initial discrimination model for the second discrimination to obtain the prediction discrimination result. The training data includes the sample instructions of the sample agent, the operation sequence corresponding to the sample instructions, and at least one screenshot of the interface during the process of the sample agent executing the sample instructions on the user's graphical interface. The prediction discrimination result includes the prediction discrimination conclusion and the prediction inference text.

[0096] Step S32: Input the prediction and discrimination results and at least one screenshot of the interface into the first discrimination model to perform semantic consistency analysis and obtain the consistency reward component. The consistency reward component is used to characterize the degree of consistency between the prediction inference text and the prediction and discrimination results.

[0097] Step S33: Adjust some model parameters of the initial discrimination model based on the consistency reward component and the preset general reward component;

[0098] Step S34: Under the condition of satisfying the preset convergence condition, obtain the second discriminant model based on the current initial discriminant model;

[0099] The first discrimination model and the second discrimination model constitute a discrimination system, which is used to execute the discrimination method described in any one of the embodiments of this disclosure.

[0100] In some optional implementations, the preset convergence condition may be that the number of training iterations reaches a preset threshold, or that the model performance reaches a preset performance requirement, etc., and the embodiments disclosed herein do not limit this.

[0101] In some alternative implementations, the initial discriminant model is essentially a second discriminant model that has not yet been fully trained. Therefore, the process of the initial discriminant model performing the second discriminant process is similar to the process of the second discriminant model performing forward inference in the evaluation system. Please refer to the relevant content for details, which will not be described in detail here.

[0102] Furthermore, in this embodiment of the disclosure, the first discriminant model is used as the reward model for the second discriminant model to determine the consistency reward component for the second discriminant model. Combined with other general reward components, some model parameters of the initial discriminant model are adjusted until a preset convergence condition is met. At this point, the initial discriminant model is used as the final second discriminant model.

[0103] In some optional implementations, for any screenshot of an interface, the first discriminant model is used to perform semantic understanding of the interface based on the screenshot, obtain the semantic understanding result, determine the degree of consistency between the predicted inference text and the predicted discriminant result based on the semantic understanding result, and map the degree of consistency to a preset reward interval to obtain the consistency reward component. The preset reward interval is a continuous scalar interval.

[0104] For example, in the reinforcement learning phase (such as the GRPO phase) for the initial discriminant model, a first discriminant model is used as the reward model. The model parameters of the first discriminant model are frozen, and the first discriminant model can be a model that has already been fine-tuned. After the initial discriminant model outputs the predicted discriminant conclusion and the predicted inference text, both these and a screenshot are input into the first discriminant model. The first discriminant model performs GUI semantic understanding on the screenshot and, based on the semantic understanding results, determines whether the inference logic of the predicted inference text supports the predicted discriminant conclusion. That is, it determines whether the inference logic of the initial discriminant model is semantically consistent with its given discriminant conclusion and determines the degree of consistency. Then, according to a preset mapping relationship, this degree of consistency is mapped to a preset reward range, and the specific reward value of the mapping is used as the consistency reward component.

[0105] It should be noted that the first discriminant model is introduced when determining the consistency reward component. This is equivalent to using the processing power of the first discriminant model to evaluate the discrimination result of the initial discriminant model. This involves processing methods such as GUI semantic understanding, which can still give accurate and reasonable evaluations for complex semantic expressions. This allows us to determine the accurate value of the consistency reward component, which in turn helps to improve the training effect of the initial discriminant model.

[0106] In some optional implementations, in addition to the consistency reward component, a general reward component is also set. The general reward component may include the accuracy reward component, the format reward component, etc. The accuracy reward component is determined based on the degree of matching and accuracy between the output content and the standard answer. The format reward component is determined based on whether the output meets the preset format specifications (such as data structure, constraints, etc.).

[0107] It should be noted that the above examples of general reward components are merely illustrative and are not intended to limit the scope of this disclosure.

[0108] Furthermore, in some optional implementations, different weight coefficients can be assigned to different reward components, and a weighted reward value can be obtained by weighting the weight coefficients and each reward component. Then, some model parameters can be adjusted based on the weighted reward value.

[0109] For example, the weighted reward value R can be calculated using the following formula:

[0110] R = w1 × R1 + w2 × R2 + w3 × R3

[0111] Where w1, w2, and w3 are weighting coefficients, R1 represents the accuracy reward component, R2 represents the format reward component, and R3 represents the consistency reward component.

[0112] In summary, in this embodiment, the reward value is no longer calculated rigidly based on dictionary and rule matching methods. Instead, the first discriminant model is used as the reward model to calculate the consistency reward component. The discriminative ability of the first discriminant model is applied to the evaluation of the discriminative result of the second discriminant model. This allows for accurate and reasonable reward components to be given even in cases of complex semantic expressions, thereby making the training gradient more stable and not affecting the upper limit of model performance.

[0113] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0114] In addition, this disclosure also provides a discrimination device, a model training device, a discrimination system, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the discrimination methods or model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section on methods and will not be repeated here.

[0115] Thirdly, embodiments of this disclosure provide a discrimination device.

[0116] Figure 4 This is a block diagram of a discrimination device provided in an embodiment of the present disclosure.

[0117] Reference Figure 4 This disclosure provides a discrimination device 400 disposed in a discrimination system. The discrimination system includes a first discrimination model and a second discrimination model for judging the instruction execution status of a user graphical interface agent, wherein the number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model.

[0118] The first discrimination module 401 is used to, upon receiving a discrimination request for the target intelligent agent, invoke the first discrimination model to perform a first discrimination on the target intelligent agent based on the target execution data, and obtain a first discrimination result.

[0119] The second discrimination module 402 is used to call the second discrimination model to perform a second discrimination on the target intelligent agent based on the target execution data when the first discrimination result meets the preset screening conditions, so as to obtain the second discrimination result;

[0120] Among them, the target execution data is the execution data carried in the discrimination request, which is used to characterize the target agent's execution of the target instruction. The preset filtering conditions are used to determine whether to call the second discrimination model to re-discriminate the target agent based on the degree of confidence of the first discrimination result.

[0121] In this embodiment, the discrimination system includes a first discrimination model and a second discrimination model. The former has fewer parameters than the latter, and correspondingly, the former requires less resource overhead and latency during runtime. Based on this system architecture, when discriminating against a target agent, the smaller first discrimination model is first invoked to perform simple logic for first discrimination, quickly and efficiently obtaining the first discrimination result. Only when the confidence level of the first discrimination result is low will the larger second discrimination model be invoked to perform complex logic for second discrimination against the target agent, achieving a second discrimination of the target agent and obtaining the corresponding second discrimination result. In other words, this embodiment can utilize a small-scale first discrimination model to quickly intercept simple errors of the target agent, reducing resource overhead and latency. Furthermore, when the first discrimination model cannot provide a highly confident first discrimination result, the complex logic of the second discrimination model can be introduced to perform a second discrimination of the target agent, obtaining a relatively more accurate second discrimination result, thereby effectively improving the accuracy of the discrimination.

[0122] Figure 5 This is a block diagram of a model training apparatus provided in an embodiment of the present disclosure.

[0123] Reference Figure 5 This disclosure provides a model training apparatus 500, which includes:

[0124] The prediction module 501 is used to input training data into the initial discrimination model for second discrimination to obtain the prediction discrimination result. The training data includes sample instructions of the sample agent, operation sequence corresponding to the sample instructions, and at least one screenshot of the sample agent executing the sample instructions on the user graphical interface. The prediction discrimination result includes the prediction discrimination conclusion and the prediction inference text.

[0125] Analysis module 502 is used to input the prediction and discrimination results and at least one screenshot of the interface into the first discrimination model to perform semantic consistency analysis and obtain consistency reward components. The consistency reward components are used to characterize the degree of consistency between the prediction inference text and the prediction and discrimination results.

[0126] The adjustment module 503 is used to adjust some model parameters of the initial discrimination model based on the consistency reward component and the preset general reward component.

[0127] The module 504 is used to obtain a second discrimination model based on the current initial discrimination model, provided that a preset convergence condition is met.

[0128] The first discrimination model and the second discrimination model constitute a discrimination system, which is used to execute the discrimination method described in any one of the embodiments of this disclosure.

[0129] In this embodiment, the reward value is no longer calculated rigidly based on the dictionary and rule matching method. Instead, the first discriminant model is used as the reward model to calculate the consistency reward component. The discriminative ability of the first discriminant model is applied to the evaluation of the discriminative result of the second discriminant model. This allows for accurate and reasonable reward components to be given even in cases of complex semantic expressions, thereby making the training gradient more stable and not affecting the upper limit of model performance.

[0130] Figure 6 This is a block diagram of a discrimination system provided in an embodiment of the present disclosure.

[0131] Reference Figure 6 This disclosure provides a discrimination system 600, which includes: a cascaded first discrimination model 610 and a second discrimination model 620;

[0132] The discrimination system 600 is used to execute the discrimination method described in any one of the embodiments of this disclosure.

[0133] In this embodiment, the discrimination system includes a cascaded first discrimination model and a second discrimination model. The former has fewer parameters than the latter, and correspondingly, the former requires less resource overhead and latency during runtime. Based on this system architecture, when discriminating against a target agent, the smaller first discrimination model is first invoked to perform simple logic for first discrimination, quickly and efficiently obtaining the first discrimination result. Only when the confidence level of the first discrimination result is low will the larger second discrimination model be invoked to perform complex logic for second discrimination against the target agent, achieving a second discrimination of the target agent and obtaining the corresponding second discrimination result. In other words, this embodiment can utilize a small-scale first discrimination model to quickly intercept simple errors of the target agent, reducing resource overhead and latency. Furthermore, when the first discrimination model cannot provide a highly confident first discrimination result, the complex logic of the second discrimination model can be introduced to perform a second discrimination of the target agent, obtaining a relatively more accurate second discrimination result, thereby effectively improving the accuracy of the discrimination.

[0134] Each module in the aforementioned discrimination device and model training method can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0135] Figure 7 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0136] Reference Figure 7This disclosure provides an electronic device, which includes: at least one processor 701, at least one memory 702, and one or more I / O interfaces 703; wherein, the memory 702 stores one or more computer programs that can be executed by at least one processor 701, and the one or more computer programs are executed by at least one processor 701 to enable at least one processor 701 to perform the above-described discrimination method or model training method.

[0137] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0138] Reference Figure 8 This disclosure provides an electronic device that includes multiple processing cores 801 and an on-chip network 802. The multiple processing cores 801 are all connected to the on-chip network 802, which is used to exchange data between the multiple processing cores and external data.

[0139] One or more processing cores 801 store one or more instructions, and the one or more instructions are executed by one or more processing cores 801 to enable one or more processing cores 801 to execute the above-mentioned discrimination method or model training method.

[0140] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0141] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned discrimination method or model training method. The computer-readable storage medium may be volatile or non-volatile.

[0142] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described discrimination method or model training method.

[0143] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0144] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0145] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0146] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0147] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0148] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0152] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A discrimination method, characterized in that, The method is applied to a discrimination system, which includes a first discrimination model and a second discrimination model for judging the instruction execution status of a user graphical interface agent, wherein the number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model. Upon receiving a discrimination request for a target intelligent agent, the first discrimination model is invoked to perform a first discrimination on the target intelligent agent based on the target execution data, and a first discrimination result is obtained. If the first discrimination result meets the preset screening conditions, the second discrimination model is invoked to perform a second discrimination on the target agent based on the target execution data, and a second discrimination result is obtained; The target execution data is the execution data carried in the discrimination request, which is used to characterize the target agent's execution of the target instruction. The preset filtering conditions are used to determine whether to call the second discrimination model to re-discriminate the target agent based on the degree of confidence of the first discrimination result.

2. The method according to claim 1, characterized in that, The first discrimination result includes a first discrimination conclusion and a confidence level corresponding to the first discrimination conclusion. The first discrimination conclusion is used to characterize whether the target agent passes the discrimination. After the method calls the first discrimination model to perform a first discrimination on the target agent based on the target execution data and obtains the first discrimination result, the method further includes: If the confidence level is less than or equal to a preset confidence threshold, the first discrimination result is determined to satisfy the preset screening conditions, so as to obtain the second discrimination result based on the second discrimination model, and the second discrimination result is used as the final discrimination result of the target agent; If the confidence level is greater than the preset confidence threshold, it is determined that the first discrimination result does not meet the preset screening conditions, so that the first discrimination result is taken as the final discrimination result of the target agent.

3. The method according to claim 2, characterized in that, The method further includes: The final judgment result is sent to the target agent to instruct the target agent to correct and re-execute the target instruction based on the final judgment result.

4. The method according to claim 2, characterized in that, The method further includes: The final judgment result is sent to the downstream agent of the target agent to instruct the downstream agent to re-execute the target instruction based on the final judgment result, and to execute the instruction received by the downstream agent based on the execution result of the target instruction.

5. The method according to any one of claims 1-4, characterized in that, The step of calling the first discrimination model to perform a first discrimination on the target agent based on the target execution data and obtaining a first discrimination result includes: The target execution data is input into the first discrimination model so that the first discrimination model can determine the target area and its sub-state in the interface screenshot by visual scanning, and determine the first discrimination result based on the target area and its sub-state; The screenshot is a screenshot taken during the process of the target intelligent agent executing the target instruction on a preset user graphical interface.

6. The method according to claim 5, characterized in that, The target execution data includes the target instruction and at least one of the interface screenshots; The first discrimination model performs the first discrimination process, which includes: For any of the aforementioned screenshots, determine the first target region and its first sub-state in the screenshot according to the target instruction; Visually scan the screenshot to determine the second target area and its second sub-state in the screenshot; The first screenshot discrimination sub-result is determined based on the first target area, the first sub-state, the second target area, and the second sub-state; Given the first screenshot discrimination sub-results of all the interface screenshots, the first discrimination result is determined based on all the first screenshot discrimination sub-results; The target region includes the first target region and the second target region, and the sub-state includes the first sub-state and the second sub-state.

7. The method according to claim 5, characterized in that, The target execution data includes the target instruction, the operation sequence corresponding to the target instruction, and at least one of the interface screenshots. The operation sequence includes multiple operations, and there is a correspondence between the interface screenshot and at least one operation. The first discrimination model performs the first discrimination process, which includes: For any of the aforementioned screenshots, a first target region and its first sub-state in the screenshot are determined based on the target instruction and the operation corresponding to the screenshot. Visually scan the screenshot based on the operation corresponding to the screenshot to determine the second target area and its second sub-state in the screenshot. The second screenshot discrimination sub-result is determined based on the first target area, the first sub-state, the second target area, and the second sub-state; Given the second screenshot discrimination sub-results of all interface screenshots, the first discrimination result is determined based on all the second screenshot discrimination sub-results; The target region includes the first target region and the second target region, and the sub-state includes the first sub-state and the second sub-state.

8. The method according to any one of claims 1-4, characterized in that, The step of calling the second discrimination model to perform a second discrimination on the target agent based on the target execution data, and obtaining a second discrimination result, includes: The target execution data is input into the second discrimination model so that the second discrimination model determines the second discrimination result through a thought chain reasoning method.

9. The method according to claim 8, characterized in that, The target execution data includes the target instruction and at least one interface screenshot, or the target execution data includes the target instruction, the operation sequence corresponding to the target instruction, and at least one interface screenshot, wherein the interface screenshot is a screenshot of the target agent executing the target instruction on a preset user graphical interface, the operation sequence includes multiple operations, and there is a correspondence between the interface screenshot and at least one operation; The second discrimination model performs the second discrimination process, which includes: Based on the target execution data, perform thought chain reasoning to determine the actual execution state of the target agent; The second discrimination result is determined based on the actual execution state of the target intelligent agent.

10. The method according to any one of claims 1-4, characterized in that, If the first discrimination result indicates that the target agent has failed the discrimination, the first discrimination result also includes first regression position information; Wherein, the first regression position information is the information of the correct position in the user graphical interface corresponding to the error position of the target agent, determined by the first discrimination model based on the target execution data; or... If the second discrimination result indicates that the target agent has failed the discrimination, the second discrimination result also includes second regression location information and / or inference text; Wherein, the second regression position information is the information of the correct position in the user graphical interface corresponding to the error position of the target agent, which is determined by the second discriminant model based on the target execution data, and the reasoning text is used to characterize the reasoning process and reasoning result of the second discriminant model based on the thought chain.

11. A model training method, characterized in that, The method includes: The training data is input into the initial discrimination model for a second discrimination to obtain the prediction discrimination result. The training data includes sample instructions of the sample agent, the operation sequence corresponding to the sample instructions, and at least one screenshot of the sample agent executing the sample instructions on the user's graphical interface. The prediction discrimination result is determined to include the prediction discrimination conclusion and the prediction inference text. The prediction and discrimination results and at least one of the interface screenshots are input into the first discrimination model for semantic consistency analysis to obtain a consistency reward component. The consistency reward component is used to characterize the degree of consistency between the predicted inference text and the prediction and discrimination results. Based on the consistency reward component and the preset general reward component, adjust some model parameters of the initial discrimination model; Under the condition that the preset convergence condition is met, a second discriminant model is obtained based on the current initial discriminant model; The first discrimination model and the second discrimination model constitute a discrimination system, which is used to execute the discrimination method as described in any one of claims 1-10.

12. The method according to claim 11, characterized in that, For any of the aforementioned screenshots, the first discriminant model is used to perform interface semantic understanding based on the screenshots to obtain semantic understanding results; The degree of consistency between the predicted inference text and the predicted discrimination result is determined based on the semantic understanding result; The consistency level is mapped to a preset reward interval to obtain the consistency reward component, wherein the preset reward interval is a continuous scalar interval.

13. A discrimination device, characterized in that, An apparatus is applied to a discrimination system, the discrimination system comprising a first discrimination model and a second discrimination model for judging the instruction execution status of a user graphical interface intelligent agent, wherein the number of parameters in the first discrimination model is less than the number of parameters in the second discrimination model, the apparatus comprising: The first discrimination module is used to, upon receiving a discrimination request for a target intelligent agent, invoke the first discrimination model to perform a first discrimination on the target intelligent agent based on the target execution data, and obtain a first discrimination result; The second discrimination module is used to call the second discrimination model to perform a second discrimination on the target agent based on the target execution data when the first discrimination result meets the preset screening conditions, so as to obtain the second discrimination result; The target execution data is the execution data carried in the discrimination request, which is used to characterize the target agent's execution of the target instruction. The preset filtering conditions are used to determine whether to call the second discrimination model to re-discriminate the target agent based on the degree of confidence of the first discrimination result.

14. A model training device, characterized in that, include: The prediction module is used to input training data into the initial discrimination model for a second discrimination to obtain a prediction discrimination result. The training data includes sample instructions of the sample agent, the operation sequence corresponding to the sample instructions, and at least one screenshot of the sample agent executing the sample instructions on the user's graphical interface. The prediction discrimination result is determined to include a prediction discrimination conclusion and a prediction inference text. The analysis module is used to input the prediction and discrimination results and at least one of the interface screenshots into the first discrimination model to perform semantic consistency analysis and obtain a consistency reward component. The consistency reward component is used to characterize the degree of consistency between the prediction inference text and the prediction and discrimination results. The adjustment module is used to adjust some model parameters of the initial discrimination model based on the consistency reward component and the preset general reward component. The acquisition module is used to obtain a second discriminant model based on the current initial discriminant model, provided that a preset convergence condition is met. The first discrimination model and the second discrimination model constitute a discrimination system, which is used to execute the discrimination method as described in any one of claims 1-10.

15. A discrimination system, characterized in that, include: Cascaded first and second discriminant models; The discrimination system is used to perform the discrimination method as described in any one of claims 1-10.

16. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the discrimination method as described in any one of claims 1-10, or the model training method as described in any one of claims 11-12.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the discrimination method as described in any one of claims 1-10, or the model training method as described in any one of claims 11-12.

18. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the discrimination method as described in any one of claims 1-10, or the model training method as described in any one of claims 11-12.

Images