Prediction method of graphical user interface and related device

Abstract

A method and related apparatus for predicting graphical user interfaces (GUIs). The method includes: processing text information of a first interface using symbols to obtain a first symbolized interface; inputting the first symbolized interface and an operation to be executed into a diffusion model to predict a second symbolized interface; and predicting the text corresponding to each symbol in the second symbolized interface to obtain the second interface. This application achieves decoupling of graphical content prediction and textual content prediction, solving the problem of mixed processing of graphical and textual information; it also solves the problem of uncertainty in the interaction logic when generating the second interface solely based on the operation to be executed, ensuring the consistency between adjacent first and second interfaces. This improves the accuracy of the prediction results and solves the problem of missing visual information in the interface prediction process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to prediction methods and related apparatus for graphical user interfaces. Background Technology

[0002] In the use of electronic devices such as smartphones and web browsers, intelligent agents based on large language models (APPs) are widely used to automate APP operations. To complete user tasks, these agents typically need to execute long sequences of operations and handle complex state transitions within a graphical user interface (GUI). Therefore, a method for predicting the GUI is needed to predict the GUI after each operation in the sequence; the generated GUI is then used to evaluate and optimize the operations, resulting in more reasonable target operations. This avoids problems such as execution failures or task termination caused by inappropriate interactions when directly executing potentially erroneous operations on the actual GUI. Summary of the Invention

[0003] To address the aforementioned issues, this application provides a prediction method and related apparatus for graphical user interfaces, which improves the prediction accuracy of graphical user interfaces, ensures the accuracy of target operations performed by intelligent agents on real GUIs, and avoids problems such as execution failure or task termination caused by unreasonable interactions when performing potentially erroneous operations directly on real GUIs.

[0004] Therefore, the following technical solution is adopted in this application:

[0005] In a first aspect, this application provides a method for predicting a graphical user interface (GUI), mainly comprising the following steps: processing text information of a first interface using symbols to obtain a first symbolized interface. The first interface represents a GUI to be operated, and includes text information; the first symbolized interface represents a GUI where the text information is covered by symbols. The first symbolized interface and the operation to be performed are input into a diffusion model to predict a second symbolized interface. The second symbolized interface represents the GUI after performing the operation on the first symbolized interface. The text corresponding to each symbol in the second symbolized interface is predicted to obtain a second interface. The second interface represents the predicted GUI after performing the operation on the first interface.

[0006] In this application, the text in the first interface is masked using symbols to obtain a first symbolized interface. A diffusion model predicts a second symbolized interface based on the image information and the operation to be performed in the first symbolized interface. The text corresponding to each symbol in the second symbolized interface is then predicted to obtain the second interface. This decouples the prediction of graphic content and text content, solving the problem of mixed processing of graphic and text information. Mixed processing of graphic and text information can lead to problems such as blurred and improperly arranged text content in the generated interface, affecting readability, especially for smaller text, which is not clearly displayed in pixel form. Generating the second interface based on the first interface before the operation and the operation to be performed solves the problem of uncertainty in the interaction logic when generating the second interface solely based on the operation to be performed. In interface interaction tasks, it is necessary to ensure the consistency between adjacent first and second interfaces. This application not only generates corresponding images based on the operation to be performed but also considers the state of the first interface, thereby constraining the layout of the second interface and ensuring the consistency between adjacent first and second interfaces. In this application, the first symbolic interface and the first interface maintain a high degree of visual consistency. The first symbolic interface can provide accurate visual information about the interface, which improves the accuracy of the prediction results of the second symbolic interface and solves the problem of missing visual information in the interface prediction process.

[0007] In one possible implementation, the above-mentioned use of symbols to process the text information of the first interface to obtain the first symbolized interface includes: detecting the text information and its position information of the first interface; and using symbols to mask the text information according to the position information to obtain the first symbolized interface.

[0008] In this implementation, the text information and its location information of the first interface are detected. Based on the location information, the text information is masked using symbols to obtain the first symbolized interface. Thus, the first symbolized interface no longer contains text information. The diffusion model no longer performs text prediction during the prediction of the second symbolized interface, achieving decoupling between graphic content prediction and text content prediction. This improves the accuracy of the second symbolized interface, and consequently, the accuracy of the second interface itself.

[0009] In one possible implementation, the text information in the first interface includes static text, which represents non-editable text in the first interface. The above-described method of processing the text information of the first interface using symbols to obtain a first symbolized interface includes: processing text information other than static text in the first interface using symbols to obtain the first symbolized interface.

[0010] In this implementation, the text information within the first interface is divided into dynamic text and static text. Static text remains unchanged across different interfaces, thus preserving it within the first symbolic interface. Static text represents non-editable text within the first interface, resulting in a low correlation between static text and the operation to be performed. Consequently, predicting static text is more difficult, less accurate, and more prone to errors. Therefore, this implementation avoids the prediction of static text, improving the accuracy of text prediction and reducing the possibility of prediction errors.

[0011] In one possible implementation, the above-mentioned use of symbols to process text information other than static text in the first interface to obtain the first symbolized interface includes: inputting the first interface into a large language model to identify static text in the first interface; and using symbols to process text information other than static text in the first interface based on the identified static text to obtain the first symbolized interface.

[0012] In this implementation, the large language model has strong text analysis capabilities, which can identify static text in the first interface and thus retain the static text in the first symbolic interface.

[0013] In one possible implementation, the above-mentioned inputting the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface includes: determining a noisy image; inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, wherein the diffusion model uses the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; and performing denoising processing on the noisy image based on the predicted noise to obtain the second symbolic interface.

[0014] In this implementation, the diffusion model uses the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; the noisy image is then denoised to obtain the second symbolic interface. Furthermore, the denoising stage of the diffusion model is used to predict the second symbolic interface.

[0015] In one possible implementation, the aforementioned noisy image is obtained by adding noise to the first symbolic interface.

[0016] In this implementation, noise is added to the first symbolized interface to obtain a noisy image. This allows the diffusion model to learn and generate the second symbolized interface from the distribution of the noisy image corresponding to the first symbolized interface, rather than learning from the distribution of the input dataset. This improves the robustness of the diffusion model.

[0017] In one possible implementation, the above-mentioned inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model includes: performing feature concatenation on the noisy image and the first symbolic interface to obtain fused features; and inputting the fused features and the operation to be performed into the diffusion model.

[0018] In this implementation, both the noisy image and the first symbolized interface are image-type data. Features from the noisy image and the first symbolized interface can be concatenated using the same convolution kernel to obtain fused features. These fused features and the operation to be performed are then input into the diffusion model. This results in a simpler model structure and higher computational efficiency.

[0019] In one possible implementation, the above-mentioned prediction of the text corresponding to each symbol in the second symbolized interface to obtain the second interface includes: inputting the second symbolized interface, the first interface and the operation to be performed into a large language model to predict the text corresponding to each symbol in the second symbolized interface; replacing each symbol in the second symbolized interface with the corresponding text to obtain the second interface.

[0020] In this implementation, the large language model has strong text understanding and analysis capabilities. Based on its understanding of the first interface and the operation to be performed, the large language model can accurately predict the text corresponding to each symbol in the second symbolized interface.

[0021] In one possible implementation, the above-mentioned inputting the second symbolized interface, the first interface, and the operation to be performed into a large language model to predict the text corresponding to each symbol in the second symbolized interface includes: determining prompt words based on the second symbolized interface, the first interface, and the operation to be performed, wherein each symbol in the second symbolized interface is the analysis object, and the first interface and the operation to be performed are the context information; and inputting the prompt words into the large language model to predict the text corresponding to each symbol in the second symbolized interface.

[0022] In this implementation, the prompt words use the symbols in the second symbolized interface as the analysis objects, and the first interface and the operation to be performed as contextual information. In this way, the prompt words can effectively guide the large language model to predict the text corresponding to each symbol in the second symbolized interface.

[0023] In one possible implementation, the aforementioned prompts include at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface.

[0024] In this implementation, the prompt includes at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface. In this way, the large language model can distinguish each symbol in the second symbolized interface based on its identification information (such as number), size information, and position information, thereby predicting the text corresponding to each symbol.

[0025] In one possible implementation, the above-mentioned acquisition of the first interface and the operation to be executed includes: acquiring user input information, the input information representing the user's task to be executed; decomposing the input information into an operation sequence, the operation sequence including at least one operation; acquiring the operation to be executed from the operation sequence, and using the graphical user interface after the execution of the previous operation as the first interface; wherein, when the operation to be executed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be executed.

[0026] This implementation provides a method for obtaining a first interface and operations to be performed. When processing user input, the input is decomposed into an operation sequence; each operation in the sequence is taken as an operation to be performed; then, the method of this application is used to simulate each operation in the operation sequence to verify its feasibility and effectiveness.

[0027] In one possible implementation, the method further includes: evaluating the second interface to obtain the evaluation result of the operation to be executed; and adjusting the operation to be executed based on the evaluation result to obtain the target operation.

[0028] In this implementation, the second interface is evaluated, and the operations to be executed are adjusted based on the evaluation results to obtain the target operation. Then, the target operation is executed to obtain the final execution result. This automates the process of processing user input. This application makes each operation in the sequence more reasonable, thereby avoiding potentially erroneous operations performed directly on the real interface and reducing problems such as execution failure or task termination caused by unreasonable interactions.

[0029] In one possible implementation, the aforementioned operation to be executed is obtained by decomposing the input information into tasks, where the input information represents the user's task to be executed. The aforementioned evaluation of the second interface to obtain the evaluation result of the operation to be executed includes: inputting the input information and the second interface into a large language model to output the evaluation result.

[0030] In this implementation, the large language model has strong text understanding and analysis capabilities. The large language model is a type of large language model that can process multimodal data. It combines visual input (second interface) and language input (user input information) to perform multimodal understanding, thereby determining whether the current feedback second interface is on the correct path, that is, the degree of consistency between the predicted second interface and the user input information. It generates confidence scores to evaluate the rationality and accuracy of the current operation to be evaluated. Based on the evaluation results, it adjusts the current operation to be executed to obtain the optimized target operation.

[0031] In one possible implementation, the above-mentioned operation to be executed is provided with at least one alternative operation. The step of adjusting the operation to be executed based on the evaluation results to obtain the target operation includes: determining the target operation from the operation to be executed and the at least one alternative operation based on the evaluation results of the operation to be executed and the evaluation results of the at least one alternative operation.

[0032] This implementation provides a method for determining the target operation. By setting multiple operations (the operation to be executed and at least one alternative operation) for each operation step, the target operation can be determined based on the evaluation results (e.g., confidence levels) of the multiple operations. This ensures the reasonableness of the target operation; for example, the target operation is the one with the highest confidence level among the multiple operations.

[0033] In one possible implementation, the training method of the above diffusion model includes: constructing a training dataset; the training dataset includes one or more training sample pairs, each training sample pair including a training sample and a label for the training sample, the training sample including a symbolic first interface sample and an operation sample, the symbolic first interface sample being obtained by symbolically processing the text information of the first interface sample, the label of the training sample representing the ground truth of the symbolic second interface, the symbolic second interface representing the symbolic graphical user interface after performing the operation sample on the symbolic first interface sample; inputting the symbolic first interface sample and the operation sample into the diffusion model to generate a predicted value for the symbolic second interface; and adjusting the parameters of the diffusion model based on the loss value between the ground truth and the predicted value of the symbolic second interface.

[0034] In this implementation, during the training of the diffusion model, symbolized first interface samples and operational samples are input into the diffusion model to generate predicted values ​​for symbolized second interfaces. The parameters of the diffusion model are adjusted based on the loss value between the ground truth and predicted values ​​of the symbolized second interfaces. Thus, the trained diffusion model functions as a denoising autoencoder for symbolized images, predicting noise in noisy image samples to achieve noise removal and obtain the symbolized second interface.

[0035] Secondly, this application provides a graphical user interface (GUI) prediction device, mainly comprising: a symbolization processing module, used to process text information of a first interface using symbols to obtain a first symbolized interface; wherein the first interface represents a GUI to be operated, the first interface includes text information, and the first symbolized interface represents a GUI where the text information is covered by symbols. A symbolized interface prediction module, used to input the first symbolized interface and the operation to be performed into a diffusion model to predict a second symbolized interface. The second symbolized interface represents the GUI after performing the operation on the first symbolized interface. A text prediction module, used to predict the text corresponding to each symbol in the second symbolized interface to obtain a second interface. The second interface represents the predicted GUI after performing the operation on the first interface.

[0036] In one possible implementation, when processing the text information of the first interface using symbols to obtain the first symbolized interface, the aforementioned symbolization processing module is specifically used to: detect the text information and its position information of the first interface; and, based on the position information, use symbols to mask the text information to obtain the first symbolized interface.

[0037] In one possible implementation, the text information in the first interface includes static text, which represents non-editable text in the first interface. When processing the text information of the first interface using symbols to obtain the first symbolized interface, the symbolization processing module described above is specifically used to: process text information other than static text in the first interface using symbols to obtain the first symbolized interface.

[0038] In one possible implementation, the above-mentioned use of symbols to process text information other than static text in the first interface to obtain the first symbolized interface includes: inputting the first interface into a large language model to identify static text in the first interface; and using symbols to process text information other than static text in the first interface based on the identified static text to obtain the first symbolized interface.

[0039] In one possible implementation, when the first symbolic interface and the operation to be performed are input into the diffusion model to predict the second symbolic interface, the aforementioned symbolic interface prediction module is specifically used to: determine the noisy image; input the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, the diffusion model using the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; and perform denoising processing on the noisy image based on the predicted noise to obtain the second symbolic interface.

[0040] In one possible implementation, the aforementioned noisy image is obtained by adding noise to the first symbolic interface.

[0041] In one possible implementation, when inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, the aforementioned symbolic interface prediction module is specifically used to: concatenate the features of the noisy image and the first symbolic interface to obtain fused features; and input the fused features and the operation to be performed into the diffusion model.

[0042] In one possible implementation, when predicting the text corresponding to each symbol in the second symbolized interface to obtain the second interface, the aforementioned text prediction module is specifically used to: input the second symbolized interface, the first interface, and the operation to be performed into the large language model to predict the text corresponding to each symbol in the second symbolized interface; and replace each symbol in the second symbolized interface with the corresponding text to obtain the second interface.

[0043] In one possible implementation, when the second symbolic interface, the first interface, and the operation to be performed are input into a large language model to predict the text corresponding to each symbol in the second symbolic interface, the aforementioned text prediction module is specifically used to: determine prompt words based on the second symbolic interface, the first interface, and the operation to be performed, wherein each symbol in the second symbolic interface is the analysis object and the first interface and the operation to be performed are the context information; and input the prompt words into the large language model to predict the text corresponding to each symbol in the second symbolic interface.

[0044] In one possible implementation, the aforementioned prompts include at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface.

[0045] In one possible implementation, the above-described apparatus further includes: a preprocessing module for acquiring user input information, the input information representing the user's task to be executed; decomposing the input information into an operation sequence, the operation sequence including at least one operation; acquiring the operation to be executed from the operation sequence, and using the graphical user interface after the execution of the previous operation as the first interface; wherein, when the operation to be executed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be executed.

[0046] In one possible implementation, the above-described apparatus further includes: an evaluation module for evaluating the second interface to obtain an evaluation result of the operation to be executed; and adjusting the operation to be executed based on the evaluation result to obtain the target operation.

[0047] In one possible implementation, the aforementioned operation to be executed is obtained by decomposing the input information into tasks, where the input information represents the user's task to be executed. When evaluating the second interface to obtain the evaluation result of the operation to be executed, the aforementioned evaluation module is specifically used for:

[0048] Input information and the second interface are fed into a large language model to output evaluation results.

[0049] In one possible implementation, the above-mentioned operation to be executed is provided with at least one alternative operation. When adjusting the operation to be executed based on the evaluation results to obtain the target operation, the evaluation module is specifically used to: determine the target operation from the operation to be executed and the at least one alternative operation based on the evaluation results of the operation to be executed and the evaluation results of the at least one alternative operation.

[0050] In one possible implementation, the above-described apparatus further includes a model training module for training the diffusion model. Specifically, this model training module is used to: construct a training dataset; the training dataset includes one or more training sample pairs, each training sample pair including a training sample and a label for the training sample; the training sample includes a symbolic first interface sample and an operation sample; the symbolic first interface sample is obtained by symbolically processing the text information of the first interface sample; the label of the training sample represents the ground truth of a symbolic second interface; the symbolic second interface represents a symbolic graphical user interface after performing the operation sample on the symbolic first interface sample; input the symbolic first interface sample and the operation sample into the diffusion model to generate a predicted value for the symbolic second interface; and adjust the parameters of the diffusion model based on the loss value between the ground truth and the predicted value of the symbolic second interface.

[0051] Thirdly, this application provides a world model for predicting a second interface when a first interface and an operation to be performed are input into the world model. The first interface represents a graphical user interface to be operated, and the first interface includes text information. The second interface represents a predicted graphical user interface after the operation to be performed on the first interface.

[0052] Fourthly, this application provides an intelligent agent, including the world model of the third aspect; wherein the intelligent agent is used to acquire user input information, the input information representing the user's task to be performed; and to decompose the input information into a task sequence, the operation sequence including at least one operation; and to acquire the operation to be performed from the operation sequence, and to use the graphical user interface after the execution of the previous operation of the operation to be performed as a first interface; wherein, when the operation to be performed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be performed; and to invoke the world model so that the world model predicts a second interface based on the first interface and the operation to be performed, the second interface representing the predicted graphical user interface after the execution of the operation to be performed on the first interface.

[0053] Fifthly, this application provides an electronic device, including: the intelligent agent of the fourth aspect. The electronic device is used to acquire user input information and invoke the intelligent agent to cause the intelligent agent to perform a task corresponding to the input information.

[0054] In a sixth aspect, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in the first aspect and any possible implementation thereof.

[0055] In a seventh aspect, this application provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform the methods described in the first aspect and any possible implementation thereof.

[0056] Eighthly, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in the first aspect and any possible implementation thereof.

[0057] Ninthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect and any possible implementation thereof, wherein the cluster of computing devices includes at least one computing device.

[0058] In a tenth aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect and any possible implementation thereof, wherein the cluster of computing devices includes at least one computing device.

[0059] It is understood that the beneficial effects of aspects two through ten above can be found in the relevant descriptions in aspect one above, and will not be repeated here. Attached Figure Description

[0060] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0061] Figure 1a A schematic diagram illustrating an application scenario of an electronic device provided in an embodiment of this application;

[0062] Figure 1b A schematic diagram illustrating the working process of an intelligent agent provided in an embodiment of this application;

[0063] Figure 2 A schematic diagram illustrating the working process of the world model provided in the embodiments of this application;

[0064] Figure 3 A schematic diagram of the training phase of a diffusion model provided in an embodiment of this application;

[0065] Figure 4 A schematic diagram of the inference stage of a diffusion model provided in an embodiment of this application;

[0066] Figure 5 A schematic diagram illustrating an example of a prediction method for a graphical user interface provided in an embodiment of this application;

[0067] Figure 6a A schematic diagram illustrating the first example of an application scenario provided in this application embodiment;

[0068] Figure 6b A schematic diagram illustrating the working process of the intelligent agent in the first example of the application scenario provided in this application embodiment;

[0069] Figure 6c A schematic diagram illustrating the working process of the world model in the first example of the application scenario provided in this application embodiment;

[0070] Figure 7a A schematic diagram illustrating a second example of an application scenario provided in the embodiments of this application;

[0071] Figure 7b A schematic diagram illustrating the working process of the intelligent agent in the second example of the application scenario provided in this application embodiment;

[0072] Figure 7c A schematic diagram illustrating the working process of the world model in the second example of the application scenario provided in this application embodiment;

[0073] Figure 8 A flowchart illustrating a prediction method using a graphical user interface provided in an embodiment of this application;

[0074] Figure 9 This is a schematic diagram illustrating the composition of a prediction device with a graphical user interface provided in an embodiment of this application;

[0075] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0076] Figure 11 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0077] Figure 12 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0079] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0080] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0081] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0082] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc. "Several" means one or more.

[0083] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0084] Large Language Model (LLM): A large language model is a natural language processing (NLP) model trained using deep learning techniques. It is capable of understanding, generating, and manipulating human language, text data, and more. Large language models are typically built from deep neural networks and trained on large-scale text data to learn the syntax, semantics, contextual relationships, and reasoning abilities of a language.

[0085] A Visual Language Model (VLM) is a deep learning model that combines visual input (such as images and videos) with language input (such as text and speech). VLMs are capable of multimodal understanding and perform well in tasks such as image recognition, text generation, and question-answering systems. In this embodiment, VLM is used to generate tasks and decompose complex, long-term sequential tasks through semantic reasoning and task context. A VLM can be understood as a language model or large language model capable of processing multimodal data.

[0086] Diffusion models, originating from non-equilibrium thermodynamics, are a type of generative model based on probability likelihood. In other words, diffusion models are deep learning-based generative models. Their basic architecture includes an image encoder and an image decoder, and they involve both a forward noise addition process and a reverse denoising process. The forward noise addition process, or diffusion, progressively adds noise to a real image to obtain a noisy image that follows a standard normal distribution. The reverse denoising process can be used for image generation tasks. Using the noisy image as the denoising target, the diffusion model predicts the noise in the noisy image based on the input conditional information, then performs denoising processing on the noisy image to generate an image that removes the noise and meets the conditional information.

[0087] Symbolic Text Representation (STR): A special form of graphical user interface (GUI) representation where the text content is masked or hidden by symbols, retaining only the interface structure, interactive controls, and symbols. For example, optical character recognition (OCR) technology is used to detect text regions on the interface, and a uniform symbol is used to cover the text content, ensuring that GUI prediction is unaffected by text changes. This method eliminates the interference of text information on interface state prediction, making interface modeling more stable and providing a clear structural reference for subsequent text recovery.

[0088] World models are an emerging concept in the field of artificial intelligence (AI), aiming to endow AI with the ability to understand and predict the real world by simulating the dynamic interactions and spatial consistency of the physical world. A world model is a digital representation of the physical world, capable of simulating physical phenomena, object motion, and interactive behaviors in the real world.

[0089] In the use of electronic devices such as smartphones and web browsers, intelligent agents based on large language models (APPs) are widely used to automate APP operations. However, current technological development faces challenges. For example, in real-world environments, the control tasks of electronic devices (such as smartphones) are complex; in scenarios like smartphone operation and web browsing, intelligent agents need to navigate through complex and ever-changing environments, which include numerous applications, diverse user interfaces, and constantly updated content. To effectively perform tasks, intelligent agents must be able to predict and adapt to these changes. Intelligent agents often need to execute long sequences of actions and cope with complex state transitions. Therefore, intelligent agents require world models to help predict feasible outcomes after action execution, enabling them to determine more accurate actions and ultimately achieve user goals.

[0090] Understandably, the purpose of a world model is to simulate the process of performing operations in a graphical user interface (GUI) and obtain the operation results (i.e., the predicted GUI after the operation). The agent evaluates the operation results to determine whether the operation is reasonable, thus identifying the target operation. The agent executes the target operation to obtain the final execution result. This avoids directly executing unoptimized operations in the real environment, reducing the possibility of execution failure or task termination due to unreasonable interactions.

[0091] In related technologies, a world model based on text descriptions is provided to predict behavioral outcomes. For example, it predicts web pages or GUIs based on text, and evaluates and adjusts operations based on the predicted web pages or GUIs. However, text descriptions contain limited information, and the world model's evaluation results for operations are inaccurate. This leads to the agent being unable to obtain accurate operations, resulting in execution failures or results that do not meet user expectations.

[0092] In view of this, embodiments of this application provide a world model for predicting graphical user interfaces (GUIs). This application constructs a vision-based world model and uses symbols to mask the text in the GUI, enabling the world model to predict the next GUI based on image information and the current operation, resulting in more accurate predictions. Specifically, masking the text in the GUI decouples graphical content generation from text content generation, improving the visual consistency of the generated GUI. This efficient visual GUI provides accurate visual information, improving the accuracy of GUI predictions. It also improves the accuracy of evaluating the current operation, thereby significantly enhancing the accuracy of the target operation performed by the agent. Furthermore, it enhances the agent's perception capabilities in GUI interaction tasks, providing a more efficient and robust solution for long-term interactions.

[0093] This application addresses the problem of missing visual information in related technologies. GUI interaction methods in related technologies primarily rely on text descriptions, neglecting the visual information of the interface. This makes it difficult for the agent to accurately understand the spatial relationships, sizes, color contrasts, and other visual features of interface elements. This deficiency prevents the agent from utilizing the overall layout information of the interface to optimize interaction decisions when performing operations. For example, when the agent needs to locate a button, relying solely on text information may not accurately determine the button's actual position, especially when the interface is dynamically changing. Furthermore, color contrast plays a crucial role in interface readability; a lack of visual information may cause the agent to ignore prompts or misclassify important components, thus affecting the evaluation of the current state and ultimately impacting the completion of the overall task.

[0094] This application addresses the problem of mixed processing of graphic and textual information in related technologies. GUI tasks typically involve the co-presentation of text and graphic elements, but existing methods often fail to effectively distinguish between these two types of information when generating and recognizing interface content. This can lead to problems such as blurry text content and improper layout, affecting the readability of the GUI interface, especially for text with small fonts, which is not clearly displayed in pixel form. Related technologies, when generating GUI interfaces using world models that generate text from images, lack control over the text content. For example, when changing GUI information, although the changes do not involve other image-level GUI modifications, the text generation still needs to be highly consistent with user instructions. Furthermore, the methods in related technologies cannot control the pixels in the image, thus affecting the generation of text content and ultimately functionality. In addition, adaptation issues with different devices and resolutions exacerbate the distortion of text and graphic information, causing the user interface to fail to render and interact correctly in certain scenarios.

[0095] This application addresses the problem of uncertainty in interaction logic in related technologies. The interaction logic in related technologies involves generating a corresponding image based on the user's text instructions. In GUI interaction tasks, the agent needs to ensure consistency between adjacent GUIs. However, due to the diversity of interface designs, related technologies do not consider the current GUI interface, making it difficult to guarantee the stability of the interaction logic before and after GUI operations. In different applications, the same operation may trigger different results. For example, in setting an alarm clock, although the task is the same, different UI layouts lead to inconsistent final results. Related technologies only consider the user's text instructions, without considering the current GUI state, making inconsistencies or significant differences between the GUIs before and after the current state possible. This application provides an accurate world model that not only generates corresponding images based on user instructions but also considers the current GUI state, thereby constraining the GUI layout for the next state. In this application, the text in the GUI is masked using symbols, decoupling the generation of graphic content from the generation of text content, improving the visual consistency of the generated GUI, enhancing the consistency and coherence of the GUI before and after operations, and ensuring the stability of the GUI's interaction logic.

[0096] To better understand the graphical user interface prediction method of the present application embodiment, the application scenarios of the graphical user interface prediction method of the present application embodiment are described below, but the description of the application scenarios does not constitute a limitation on the present application embodiment.

[0097] Figure 1a This is a schematic diagram illustrating an application scenario of an electronic device provided in an embodiment of this application. For example... Figure 1a As shown in this embodiment, the user sends input information to the electronic device, which represents the user's task to be performed on the electronic device. The electronic device is equipped with an intelligent agent that can invoke a world model. Based on the input information, the intelligent agent predicts one or more possible user behaviors. The intelligent agent evaluates the behavioral outcome of each possible user behavior by invoking the world model. Based on the evaluation results of the one or more possible user behaviors, the intelligent agent selects a target behavior. The intelligent agent invokes a tool to execute the target behavior and obtains the execution result. The electronic device displays the execution result to the user. The modality of the input information includes voice, image, video, text, etc., and this embodiment does not limit the modality of the input information.

[0098] In this embodiment, an electronic device refers to any device with a GUI; the terminal 110 in this embodiment can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this embodiment does not impose any restrictions on it.

[0099] Figure 1b This is a schematic diagram illustrating the working process of an intelligent agent provided in an embodiment of this application. Figure 1b As shown, the agent acquires user input information. This input is typically a complex task. Based on the input, the agent decomposes the task into one or more operations. The agent inputs an operation and a GUI (GUI object) to be executed into the world model, outputting the resulting GUI. The agent evaluates the generated GUI and adjusts the current operation based on the evaluation results to obtain a more reasonable target operation. The agent then executes the target operation, yielding the final execution result.

[0100] For example, GUI0 and operation 1 are input into the world model, the world model predicts the next GUI1 after operation 1 is performed on GUI0, the agent evaluates the accuracy of GUI1 and adjusts operation 1 according to the evaluation result; GUI1 and operation 2 are input into the world model, the world model predicts the next GUI2 after operation 2 is performed on GUI1, and so on.

[0101] In this embodiment of the application, the working process of the intelligent agent may include a task decomposition stage, a GUI prediction stage, a GUI evaluation and operation adjustment stage, and an execution stage.

[0102] Furthermore, in the task decomposition stage, this embodiment utilizes a Vision Language Model (VLM) to reason about and parse the user's input information. Through its multimodal understanding capabilities, the VLM decomposes the high-level task input by the user into one or more short-sequential, operable operations. An operation can be understood as an action performed on the GUI. In this stage, the VLM generates one or more candidate interactive operations by analyzing the user's input information, the current visual information of the GUI, and existing user operation data. The task decomposition stage ensures that the agent has preliminary policy planning before actual interaction. This policy planning represents one or more sequentially executed operations obtained to decompose the user task. After subsequent optimization, this policy planning yields the final one or more operations.

[0103] Furthermore, in the GUI prediction phase, the agent simulates the interactive operations of the current GUI using a world model to verify its feasibility and effectiveness. The agent invokes the world model to simulate clicking, inputting, scrolling, and other operations on the current GUI in a virtual environment, generating the next GUI after the operation. For example, if GUI0 and operation 1 are input into the world model, the world model predicts the next GUI1 after performing operation 1 on GUI0; if GUI1 and operation 2 are input into the world model, the world model predicts the next GUI2 after performing operation 2 on GUI1, and so on.

[0104] Furthermore, in the GUI evaluation and operation adjustment phase, the agent simulates the interactive operations of the current GUI using a world model, generating the next GUI after the operation. The next GUI is then evaluated to assess the rationality and accuracy of the current operation. Based on the evaluation results, the current operation is adjusted to obtain the optimized target operation.

[0105] For example, the agent evaluates the predicted GUI using a Vision Language Model (VLM). A VLM is a large language model capable of handling multimodal data. It combines visual input (the predicted GUI) and linguistic input (the user's input) to perform multimodal understanding, determining whether the currently presented GUI is on the correct path—that is, the degree of consistency between the predicted GUI and the user's input—and generating a confidence score. When the confidence score in the evaluation result is lower than a threshold, the user's input is re-decomposed into tasks based on the evaluation results, thus adjusting the current operation. This results in a higher confidence score and greater accuracy for the final target operation. Alternatively, one or more alternative operations can be set for each operation. Based on the evaluation results of the current operation and the alternatives, the operation with the highest confidence score is selected as the final target operation.

[0106] Furthermore, during the execution phase, the agent operates in a real environment, interacts with the GUI, and executes one or more of the target operations. Throughout this process, the agent continuously adapts to the dynamic changes in the GUI (such as asynchronous content loading, pop-up interference, CAPTCHA verification, etc.) and provides real-time feedback and adjusts its interaction strategies.

[0107] Furthermore, the method of this application embodiment improves the accuracy of GUI prediction, enabling the agent to complete complex interactive tasks more stably and enhancing the user experience.

[0108] Figure 2 This is a schematic diagram illustrating the working process of the world model provided in the embodiments of this application. Based on Figure 1b The working process of the intelligent agent is further explained, focusing on the GUI prediction stage of the world model. For example... Figure 2 As shown, the world model takes GUI(k) as input and operation (k+1) as output, and outputs GUI(k+1). GUI(k) represents the k-th GUI, is the current GUI without operation (k+1) executed, and is the object to be operated on (k+1). Operation (k+1) refers to the (k+1)-th operation after decomposing the user's input information. GUI(k+1) is the next GUI predicted by the world model after executing operation (k+1) on GUI(k). k is an integer greater than or equal to zero.

[0109] Optionally, the world model includes a processing module 1. Processing module 1 performs Symbolic Text Representation (STR) processing on the GUI(k) to obtain STR(k). For example, the STR processing procedure of processing module 1 generally includes: acquiring the GUI(k); detecting the text content in the GUI(k); and converting the text content into certain symbols to obtain STR(k). STR(k) can be understood as the GUI that has undergone STR processing; STR(k) represents the text-symbolized GUI corresponding to the GUI(k).

[0110] In this embodiment, the symbol can be any graphic structure other than text. For example, the graphic structure can be a graphic structure with or without fill in the middle. A graphic structure without fill in the middle can be a polygon, circle, or other graphic structure enclosed by one or more borders. A graphic structure with fill in the middle can be a solid color frame, a gradient color frame, a frame with a specific texture, or other graphic structures. A graphic structure refers to a graphic structure with a certain area, the size of which is greater than or equal to the size of the text being replaced, and the shape of which can be any shape. In this process of symbolizing text to represent STR, the text information in the GUI interface is structured into independent symbols, avoiding prediction errors caused by the mixing of graphics and text. For example, the symbol can also be a certain area near the original position of the text after processing it using methods such as blurring or mosaic processing.

[0111] In this embodiment, Symbolic Text Representation (STR) is introduced to decouple graphic and text content, providing a clear and flexible representation for the prediction of both graphics and text. In the GUI generation process of related technologies, text information is often mixed with graphic content, leading to problems such as disordered interface information, blurred text, or improper layout in the generated GUI. STR stores text information in an independent symbolic representation, separating the prediction of text from the prediction of graphic elements, making the text and graphic generation processes more independent, thereby improving the accuracy of GUI interface prediction for the world model.

[0112] Optionally, the world model includes a diffusion model. The diffusion model takes STR(k) and operation (k+1) as input and outputs STR(k+1). STR(k+1) represents the symbolized GUI interface of the text after performing operation (k+1) on STR(k). In predicting the next STR(k+1) based on STR(k) and operation (k+1), the diffusion model treats the symbols transformed from the text as a graphical structure in the STR(k) image, generating STR(k+1) that contains this graphical structure.

[0113] Furthermore, both the input STR(k) and the generated STR(k+1) of the diffusion model are GUI interfaces where the text has been symbolically processed. A diffusion model is used to predict the new symbolic representation of the text in STR(k+1) to ensure accurate interface layout and maintain the integrity of graphical elements. During the denoising process, the diffusion model progressively optimizes the interface structure, making the graphical information of the generated STR(k+1) more consistent with STR(k), the current operation, user input, or historical data. This method can generate high-quality user interface (UI) components in STR(k+1), including buttons, icons, and input boxes, adapting them to different screen sizes and interface layouts.

[0114] Optionally, the world model also includes a processing module 2. The processing module 2 takes STR(k+1) as input, performs text prediction based on the symbols in STR(k+1), and adds the predicted text to STR(k+1) to obtain GUI(k+1). Optionally, to improve the accuracy of the predicted text, operation (k+1) and GUI(k) can be used as contextual information for STR(k+1) to predict the text corresponding to the symbols in STR(k+1), making the input information of the prediction process richer and the accuracy of the predicted text higher.

[0115] For example, processing module 2 includes a Large Language Model (LLM). During the text generation stage, the LLM can be used to generate specific text content for each symbol, and the generated text content is precisely overlaid onto the symbolized STR(k+1) image to ensure high clarity and readability of the text in the GUI(k+1) interface. This method ensures that the text information in the GUI(k+1) interface not only conforms to natural language expression but also adapts to UI interfaces of different resolutions, improving the accuracy and consistency of GUI interaction.

[0116] This application proposes a GUI generation method based on Symbolic Text Representation (STR), which solves problems such as inaccurate interface understanding, ambiguous text, and unreasonable layout caused by the mixing of graphical and textual information in GUI interaction tasks. This application achieves efficient separation of graphics and text during GUI generation and optimizes the accuracy of interface prediction by combining diffusion models and large language models. This application not only improves the quality of GUI generation but also enhances the adaptability of intelligent agents in complex interaction tasks, providing a new technical path for automated GUI design.

[0117] The deployment endpoint (such as a smartphone, intelligent vehicle, or other electronic device) in this application embodiment primarily serves the task execution process in complex graphical user interface (GUI) environments. It determines the credibility of an operation by predicting its subsequent state. During execution, the agent first generates a list of operations to be performed based on user input (i.e., the target task) and the current GUI state. Then, the first operation in the list and the current GUI state are used as input to the world model. The world model outputs a prediction of the next GUI state, combined with Symbolic Text Representation (STR) to improve prediction accuracy.

[0118] Figure 3 This diagram illustrates the training phase of a diffusion model according to an embodiment of this application. Based on the aforementioned diffusion model, the training phase of the diffusion model will be further described exemplarily. Figure 3 As shown in one example, during the training phase of the diffusion model, the training dataset includes one or more training sample pairs; each training sample pair includes a training sample and its label. For example, the training sample includes a symbolized text representation STR(k) sample, an operation (k+1) sample, and the label STR(k+1)′ of the training sample. The STR(k) sample is obtained by performing symbolized text representation (STR) processing on the GUI(k). STR(k+1)′ is obtained by performing symbolized text representation (STR) processing on the GUI(k+1)′. The GUI(k) sample represents the k-th GUI sample in the training dataset. The GUI(k) sample is the current GUI that has not yet performed the operation (k+1) sample, and the GUI(k) sample is the operation object that needs to perform the operation (k+1) sample. The operation (k+1) sample refers to the (k+1)-th operation sample in the training dataset. The GUI(k+1)′ sample is the next GUI sample after performing the operation (k+1) sample on the GUI(k) sample. A GUI(k+1)′ sample can be understood as the ground truth value of a GUI(k) sample after performing operations on (k+1) samples. For example, if the training dataset is a dataset where operations are performed on the GUI continuously, then GUI(k+1)′ is the (k+1)th GUI sample in the training dataset. In other words, the STR(k+1)′ label is the ground truth value corresponding to the STR(k) sample.

[0119] Furthermore, during the training phase, the diffusion model takes STR(k) samples, operation (k+1) samples, and noisy image samples as input, and outputs the predicted value of STR(k+1). The loss value is calculated based on the ground truth value of STR(k+1)′ and the predicted value of STR(k+1). The parameters of the diffusion model are then adjusted based on the calculated loss value.

[0120] In this embodiment, the diffusion model includes a noise prediction module and a denoising module; adjusting the parameters of the diffusion model means adjusting the parameters of the noise prediction module and the denoising module. The diffusion model may also include more neural network modules, and these additional neural network modules can also be the objects of parameter adjustment.

[0121] In this application embodiment, the methods for calculating the loss value include mean squared error (MSE), cross-entropy loss, etc., and this application embodiment does not limit the methods.

[0122] In this embodiment of the application, during the training phase, in the GUI prediction process, the diffusion model uses noisy image samples as initial noise, i.e., the denoising object; and uses the STR(k) representation of the current GUI(k) sample (i.e., the STR(k) sample) and the operation (k+1) sample as conditional inputs. The diffusion model is equivalent to a denoising autoencoder, which predicts the noise (such as Gaussian noise) in the noisy image samples under the guidance of the STR(k) sample and the operation (k+1) sample; and then removes the noise in the noisy image samples to obtain the STR(k+1) predicted value.

[0123] For example, the diffusion model includes a noise prediction module for predicting noise in a noisy image sample under the guidance of STR(k) samples and operation (k+1) samples; the diffusion model includes a denoising module for removing noise from the noisy image sample to obtain the STR(k+1) predicted value.

[0124] Optionally, the noisy image sample can be a noisy image of an STR(k) sample with added noise. The noisy image sample can also be a random noisy image.

[0125] In this embodiment of the application, during the GUI prediction process, an STR predictor is proposed based on the powerful generation capability of the diffusion model. The trained diffusion model is obtained by training the diffusion model. The trained diffusion model can be understood as a denoising autoencoder within the STR predictor. This STR predictor can understand a given STR(k) sample and operation(k) sample, and generate the corresponding next-step STR(k+1) prediction value. In other words, this embodiment of the application fine-tunes a pre-trained stable diffusion model. The fine-tuned diffusion model is equivalent to a denoising autoencoder to predict Gaussian noise in noisy image samples, thereby achieving noise removal and obtaining the STR(k+1) prediction value.

[0126] For example, STR(k) samples can be encoded into feature representations, and then noisy features can be added to the features of the STR(k) samples to obtain the features of the noisy image samples. The features of the noisy image samples are then input into a diffusion model, which acts as a denoising autoencoder. This model can predict Gaussian noise in the noisy image samples, thereby achieving noise removal and obtaining the predicted value of STR(k+1). For instance, STR(k) samples can be encoded into a latent representation z, and Gaussian noise can be added to the latent representation z to obtain a noisy latent representation z′. This noisy latent representation z′ is then input into a diffusion model, which acts as a denoising autoencoder. This model can predict Gaussian noise in the noisy latent representation z′, thereby achieving noise removal and obtaining the predicted value of STR(k+1).

[0127] Optionally, in order to support the image of STR(k) samples as conditional input, an additional input channel is added to the first convolutional layer of the neural network model, and the representation of STR(k) samples and the representation of the noisy image after adding noise are concatenated and input into the noise prediction model.

[0128] Figure 4 This diagram illustrates the inference stage of a diffusion model provided in an embodiment of this application. The diffusion model is a key component of the world model, and the inference stage of the diffusion model is further illustrated by example. Figure 4 As shown, in one example, during the inference phase of the diffusion model, the input to the diffusion model is STR(k) and operation (k+1), and the output is STR(k+1).

[0129] In this embodiment of the application, during the GUI prediction process, the diffusion model uses the noisy image as the initial noise, i.e., the denoising object; and uses the STR(k) representation and operation (k+1) of the current GUI(k) as conditional inputs. The diffusion model is equivalent to a denoising autoencoder, which predicts the noise (such as Gaussian noise) in the noisy image under the instruction of STR(k) and operation (k+1); and then removes the noise in the noisy image to obtain STR(k+1).

[0130] For example, the diffusion model includes a noise prediction module for predicting noise in a noisy image under the instruction of STR(k) and operation (k+1); the diffusion model includes a denoising module for removing noise from the noisy image to obtain STR(k+1).

[0131] Optionally, a noisy image can be obtained by adding noise to STR(k). Therefore, the input to the diffusion model is STR(k) and operation (k+1), and the output is STR(k+1).

[0132] For example, the diffusion model also includes a noise-adding module. This module adds arbitrary noise to STR(k) to obtain a noisy image. For instance, adding Gaussian noise to STR(k) yields a noisy Gaussian image. The main purpose of adding noise is to transform the input STR(k) into pure Gaussian noise, allowing the diffusion model to learn to generate STR(k+1) from the distribution of the Gaussian noise corresponding to STR(k), rather than learning from the distribution of the input dataset. This improves the robustness of the diffusion model.

[0133] Optionally, STR(k) can be encoded into a feature representation, and then noise features can be added to the features of STR(k) to obtain the features of the noisy image. For example, STR(k) can be encoded into a latent representation z, and Gaussian noise can be added to the latent representation z to obtain the noisy latent representation z′. Alternatively, noise can be added to STR(k) to obtain a noisy image, and then the noisy image can be encoded into a feature representation to obtain the feature representation of the noisy image. For example, Gaussian noise can be added to STR(k) to obtain a Gaussian noise image; the Gaussian noise image can then be encoded into a latent representation z to obtain the noisy latent representation z′.

[0134] Optionally, the noisy image can also be a random noisy image. Therefore, the input to the diffusion model is STR(k), the operation (k+1), and the noisy image, and the output is STR(k+1). For example, the diffusion model can also omit the noise-adding module; instead, it can directly acquire an initial noisy image (e.g., a Gaussian noise image) and perform a denoising process to obtain STR(k+1).

[0135] Optionally, the diffusion model includes an encoding module. This encoding module encodes the input into feature representations for the model to analyze. For example, it encodes STR(k) into an STR representation; STR(k) encoding is implemented by adding convolutional layers to the model network or adding additional input channels to the convolutional layers. For example, it encodes operation (k+1) into an operation representation; operation (k+1) encoding is implemented by adding fully connected layers to the model network. For example, it encodes noisy images into feature representations to generate features of the noisy image; noisy image encoding is implemented by adding convolutional layers to the model network.

[0136] Optionally, the STR representation of STR(k) and the features of the noisy image can be concatenated and used as input to the noise prediction module. This concatenation method preserves the original information of different feature maps while providing richer feature representations for subsequent layers. Furthermore, by increasing feature dimensionality and diversity, it helps improve the model's expressive power and performance.

[0137] Optionally, the diffusion model also includes a decoding module. This decoding module is used to decode the features in the diffusion model. For example, it converts the feature representation of the output STR(k+1) into an image format.

[0138] This application provides a vision-based GUI world model capable of directly generating future GUI images. This world model can more accurately capture details such as the color, position, and style of elements in the GUI, thereby effectively improving the realism and accuracy of the images. This method solves the problem of inaccurately representing image details when generating GUIs based on text, and provides clearer visual guidance for agents in long-term planning.

[0139] This application provides a GUI image prediction and LLM text prediction based on STR (Stroke Order Language). GUI image prediction based on text symbolized by STR eliminates the reliance on precise pixel-level rendering of the text during image generation, thus improving the quality and efficiency of GUI generation. Based on a large language model, the text corresponding to each symbol in the STR is predicted accurately, ensuring the semantic and positional correctness of the generated text.

[0140] To further illustrate the technical solutions of the embodiments of this application, an example is provided to further explain the prediction method of the graphical user interface described above. This example is intended to illustrate the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions of the embodiments of this application.

[0141] Figure 5 This is a schematic diagram illustrating an example of a prediction method for a graphical user interface provided in an embodiment of this application. Figure 5 As shown in one example, the user's input is "Open Settings and set an alarm for 7 a.m.

[0142] Furthermore, the agent parses the user's input information based on the deployed VLM model. VLM, through its multimodal understanding capabilities, decomposes the high-level tasks input by the user into one or more short-sequence, executable operations. These executable operations may include steps such as: "opening the alarm clock application," "clicking the 'Add Alarm' button to add a new alarm," "setting the alarm parameters to 7 AM," and "clicking the 'Save' button to save the alarm."

[0143] Furthermore, assuming in the first example, the current GUI (x) k This is the GUI after the operation "Set alarm parameters to 7 AM" is executed; the current operation (a k The next GUI element predicted is "Click the save button to save the alarm".k+1 ) is the current GUI(x) k ) Execute the current operation (a k ) GUI for prediction.

[0144] Furthermore, in the current GUI(x) k ) of STR(x k In the feature generation phase, the agent first obtains the current GUI interface and gets the GUI(x) k Agent detection GUI (x k The text content in ) is converted into a symbolic text representation STR(x) k In this process, GUI(x) k The text information in the interface is structured into independent symbols to avoid prediction errors caused by mixing graphics and text input into the diffusion model.

[0145] In this example, to avoid generating specific text content during the prediction of the GUI, this application proposes a method for symbolic text representation (STR). Wherein, GUI(x k All text content (i.e. actual words) in the image is symbolized (overwritten) as uniform text symbols (placeholders), thus converting the text into symbols.

[0146] In this example, GUI(x) k The text information within the `<textarea>` tag includes text in pop-up windows, text on the alarm clock interface, and text in the navigation box. For example, text in pop-up windows might include "07:00", "AM", "PM", "Delete", or "Save". Text on the alarm clock interface might include "Alarm", "5:00", "AM", "8:00", "PM", "9:25", or "AM". Text in the navigation box might include "8:16". The text "Alarm" on the alarm clock interface is static text. To obtain the symbolic STR(x)... k This function processes text in pop-up windows (e.g., "07:00", "AM", "PM", "Delete", "Save"), alarm clock interfaces (e.g., "Alarm", "5:00", "AM", "8:00", "PM", "9:25", "AM"), and navigation boxes (e.g., "8:16") into rectangular prisms. These prisms are white-filled rectangles with black borders. The position and size of the prisms are determined by the size of the text they cover.

[0147] In this example, the processing module 1 in the world model is a text processing module. This embodiment of the application uses the text processing module to detect the GUI (x). kThe text information within the GUI image is extracted and symbols are overlaid at the locations where text exists, thus converting the given GUI image into an STR representation. First, an Optical Character Recognition (OCR) model is used to detect the GUI (x...). k The text information within the GUI is first identified, and its position is located. Secondly, the detected text is filled with white and covered with a black border to mask the actual text content, making the text location a placeholder. Finally, LLM filtering is used for GUI(x)... k Static text on elements, and preserves this static text in STR(x) k )middle.

[0148] Furthermore, GUI(x) k The text information inside can be divided into two types: dynamic text and static text.

[0149] Static text refers to text that does not undergo semantic evolution or dynamic change, such as static UI components like keyboards, alarm clock interfaces, and application icons. Static UI components have inherent text that is typically not editable by the user; examples include the text on an application icon representing the application name, numbers on an alarm clock wheel, and letters on a keyboard. Static text does not undergo semantic evolution or dynamic change; the text in these static components remains unchanged across different GUI interfaces.

[0150] Dynamic text refers to text that involves semantic evolution or dynamic changes, such as UI interfaces for message editing and data statistics. For example, the number of new messages a user sends may change in different GUI interfaces. This application's embodiments have implemented related methods, and experimental results show that predicting these static texts with complex spatial patterns is a significant challenge for LLM (Local Management Model).

[0151] This application embodiment will use GUI(x) k The text information within STR(x) is divided into dynamic text and static text. Dynamic text is the part that LLM needs to predict, while static text does not require text prediction. In other words, static text remains unchanged across different GUIs, and thus can be preserved in STR(x). k In this way, the predicted next STR(x) k+1 This static text is included in the next STR(x). Alternatively, the static text is also symbolized and included in the text prediction phase, in the next STR(x) prediction. k+1 Then replace the symbols at the corresponding positions with static text.

[0152] In this embodiment of the application, through the above process, GUI(x)k The image is converted to STR representation STR(x) k In this model, text content is abstracted into symbols, thus simplifying the task of generating semantic text content in the world model to predicting the position and size of the symbols.

[0153] Furthermore, in the STR prediction phase, a diffusion model is adopted based on the current GUI(x) k The text of STR(x) is symbolically represented. k ) and user operations (a k Predict the new GUI (x) k+1 The STR of ) represents STR(x) k+1 The diffusion model is used to predict changes in interface layout, enabling the new interface to accurately map the results of user actions.

[0154] In this example, during the GUI prediction process, this application embodiment proposes a specially trained STR predictor based on the powerful generative capabilities of the diffusion model. This predictor is capable of understanding a given STR(x) k ) and user operations (a k ), and generate the corresponding next step STR(x) k+1 For example, embodiments of this application fine-tune a pre-trained stable diffusion model to predict the next STR(x). k+1 This fine-tuning process is based on the current GUI (x k The STR of ) represents STR(x) k ) and user operations (a k ) as conditional input. When GUI(x) is received k ) of STR(x k When ), first STR(x) k The latent representation z is encoded as a potential representation z. Gaussian noise is then added to this latent representation z. Next, a denoising autoencoder is trained to predict the Gaussian noise in the latent representation z, thereby achieving noise removal. The loss value during training is STR(x). k+1 The loss value between the predicted and true values ​​of STR(x). k+1 The predicted value of STR(x) is obtained by decoding after removing noise from the latent representation z. k+1 ) represents predicting the new GUI(x) k+1 The STR of ) represents STR(x) k+1 ). STR(x) k+1 The truth value of ) is GUI(x) k+1 The truth value is represented by STR.

[0155] In this example, operation (a kFor example, "Click the save button to save the alarm clock at 7 AM" is the generation STR(x) of the diffusion model. k+1 The conditional information of the process, therefore, STR(x) k+1 The alarm clock for 7 a.m. is set to "on".

[0156] Furthermore, through the trained diffusion model, the STR predictor in this embodiment of the application can predict based on a given STR(x) k ) and user operations (a k ), synthesize the next step STR(x) k+1 ).

[0157] Furthermore, in the text content recovery phase, the text predictor utilizes LLM to parse the new STR(x) k+1 The text symbols in the interface, and based on the new STR(x) k+1 The process generates the corresponding text content. This process ensures that the position, size, and clarity of the text in the interface conform to natural interface standards, and finally overlays the text onto the symbolic image to obtain the complete GUI interface for the next step.

[0158] In this example, the world model includes a text predictor. During text prediction, the symbolized STR(x) is... k+1 As input to the text predictor, the symbol loading module in the text predictor can recognize STR(x). k+1 The symbol loading module outputs information such as the position and size of each symbol in the STR (Stroke Order Array). For example, the output might be {"ID":(x,y,w,h), ...}. Here, "ID" represents the identity document (ID) of each symbol, such as a number, sequence number, or name code; "x,y" represent the position coordinates of each symbol; and "w,h" represent the size of each symbol. For instance, firstly, symbols in the STR are located using color matching and boundary detection methods, and their unique identifiers (IDs) and their position coordinates and sizes are extracted. Then, leveraging the image processing and task understanding capabilities of the LLM (Low-Level Model), the STR(x,y,w,h) is predicted based on the context information of the STR. k+1 The text content in the GUI STR. This design ensures flexible text generation while enabling the predicted GUI STR to provide accurate text content prediction as contextual information.

[0159] Furthermore, the context information of STR is based on the current GUI (x k ) and operations (a k Determine based on the current GUI (x). For example, based on the current GUI (x) k ), Current GUI (x) k The STR of ) represents STR(x) k ) and operations (ak Determine the prompt word. Enter the prompt word into the LLM, and the LLM will use the current GUI (x k ) and operations (a k ) is STR(x k Analyze the context information of STR(x) k The text corresponding to the symbols in ).

[0160] In this example, based on the current GUI (x) k ), STR(x) k ) and operations (a k The specific prompt word could be something like "There are currently two GUI interface images [GUI(x)]". k )] and [STR(x k+1 The second GUI image is an operation performed on the first GUI image. k The next GUI interface after []. The text in the second GUI image has been symbolized; please predict the text corresponding to each symbol in the second GUI image. We provide information about one or more symbols in the predicted STR image, for example, ID is [ID1]. k+1 The size of the symbol is [w1] k+1 h1 k+1 ], located at [x1 k+1 y1 k+1 ]; ID is [ID2] k+1 The size of the symbol is [w2] k+1 h2 k+1 ], located at [x2 k+1 y2 k+1 ]wait.".

[0161] In this example, based on the current GUI (x) k ), STR(x) k ) and operations (a k A specific prompt could be something like, "There are currently two GUI interface images [GUI(x)]". k We retrieved the text from this GUI interface and used symbols to overlay the text, resulting in an STR image of the text after symbol processing [STR(x]]. k Specifically, the position is [x1] k y1 k The text of [text1] k The image was replaced with an ID of [ID1]. k ], size is [w1 k h1 k The symbol ] is located at [x2] k y2 k The text of [text2]k The image was replaced with an ID of [ID2]. k ], Dimensions are

[0162] [w2 k h2 k The symbols, etc., are used to predict the operation [a] performed on the STR image. k The text in the next GUI interface after ] is symbolically processed by the STR image [STR(x k+1 We provide information about one or more symbols in the predicted STR image, for example, with ID [ID1]. k+1 The size of the symbol is [w1] k+1 h1 k+1 ], located at [x1 k+1 y1 k+1 ]; ID is [ID2] k+1 The size of the symbol is [w2] k+1 h2 k+1 ], located at [x2 k+1 y2 k+1 Please use the above information as context to predict the STR image [STR(x...]. k+1 The text corresponding to each of the one or more symbols in the symbol )]".

[0163] In this example, based on the current GUI (x) k ), STR(x) k ) and operations (a k A specific prompt could be something like, "There is currently an image of a GUI interface [GUI(x)]". k We retrieved the text from this GUI interface and used symbols to overlay the text, resulting in an STR image of the text after symbol processing [STR(x]]. k Specifically, the position is [x1] k y1 k The text of [text1] k The image was replaced with an ID of [ID1]. k ], size is [w1 k h1 k The symbol ] is located at [x2] k y2 k The text of [text2] k The image was replaced with an ID of [ID2]. k ], Dimensions are

[0164] [w2 k h2 k The symbols, etc., are used to predict the operation [a] performed on the STR image. kThe text in the next GUI interface after ] is symbolically processed by the STR image [STR(x k+1 We provide information about one or more symbols in the predicted STR image, for example, with ID [ID1]. k+1 The size of the symbol is [w1] k+1 h1 k+1 ], located at [x1 k+1 y1 k+1 ]; ID is [ID2] k+1 The size of the symbol is [w2] k+1 h2 k+1 ], located at [x2 k+1 y2 k+1 Please use the above information as context to predict the STR image [STR(x...]. k+1 The text corresponding to each of the one or more symbols in the symbol )]".

[0165] In this example, the world model includes a text overlay module. The text overlay module is used to overlay the text content of each symbol onto STR(x). k+1 Reconstruct the complete GUI image GUI(x) based on the corresponding symbols in ) k+1 Specifically, according to STR(x) k+1 The coordinate information of each symbol in the code is used to replace each symbol with the corresponding text. The text style is dynamically adjusted according to the size of the symbol and the surrounding color to ensure the readability and aesthetics of the final interface.

[0166] In this example, GUI(x) k+1 The text information within STR(x) includes the text on the alarm clock interface and the text in the navigation box. For example, the text on the alarm clock interface might be "Alarm Clock", "5:00", "AM", "6:30", "AM", "7:00", "AM", "1:05", "PM", "4:10", "PM", "8:00", "PM", "9:25", "AM". The text in the navigation box might be something like "8:16". For STR(x) k+1After predicting the text corresponding to each symbol in the code, the predicted text is overlaid on the corresponding symbol. The text style is dynamically adjusted based on the symbol's size and surrounding color to ensure the final interface's readability and aesthetics. For example, the symbols on the alarm clock interface are replaced with text such as "Alarm Clock," "5:00," "AM," "6:30," "AM," "7:00," "AM," "1:05," "PM," "4:10," "PM," "8:00," "PM," "9:25," and "AM"; the symbols in the navigation box are replaced with text such as "8:16." "7:00 AM" is a newly added alarm clock, and the alarm clock switch control on the right is in the on state.

[0167] In this example, the text processing module can be understood as an example of processing module 1, and the text predictor and text overlay module can be understood as examples of processing module 2.

[0168] To further illustrate the technical solutions of the embodiments of this application, two application scenario examples are also provided in the embodiments of this application.

[0169] The technical solutions of this application have wide application scenarios in complex graphical user interface (GUI) tasks, and can significantly improve task execution efficiency and the adaptability of intelligent agents. During deployment, symbolic text representation (STR) is combined with a diffusion model to achieve interface state prediction, and the text content is recovered through a large language model (LLM) to ensure the accuracy and consistency of interface generation.

[0170] Figure 6a This is a schematic diagram illustrating the first example of an application scenario provided in the embodiments of this application. For example... Figure 6a As shown in the first example of the application scenario, the application of this embodiment in the intelligent assistant of a mobile device aims to address the challenges of complex, long-sequential user commands, providing an efficient and reliable GUI interface prediction and interaction solution. This embodiment mainly includes two steps: GUI state prediction and dynamic interface updating. By introducing Symbolic Text Representation (STR) and a Diffusion Model for GUI change prediction, and text recovery based on a Large Language Model (LLM), this embodiment achieves accurate modeling and real-time adaptation of complex interface interactions.

[0171] For example, when a user inputs a complex task objective via voice or text, such as "open settings and enable Bluetooth," the world model predicts the future state of the GUI. Based on the evaluation of the predicted GUI, the agent optimizes the operations derived from the task decomposition, executes the optimized target operation, and obtains the execution result. The user receives the "open settings and enable Bluetooth" execution result in an end-to-end manner. The STR method of this application improves the accuracy of predicting the future GUI, enhances the accuracy of the agent's operation optimization and target operation generation, ensures the accuracy of the execution result, and improves the user experience.

[0172] In this example, within a mobile device, this embodiment of the application can be used to enhance the interface generation capabilities of intelligent assistants, particularly in tasks such as application automation testing, interface layout optimization, and cross-device adaptation. Modern smartphones present highly dynamic application scenarios with complex arrangement of interface elements and interaction patterns, making it difficult for related technologies to adapt to real-time interface changes. This embodiment of the application predicts the interface structure using STR (Stroke Order Prediction) and combines it with LLM (Low-Level Text Generation) to generate text content, ensuring that the interface state remains stable during task execution. For example, in automated testing tasks, the agent can predict changes in the application interface under different operation steps and generate corresponding test cases. In adaptation tasks, this technology can be used for interface optimization across different screen sizes and resolutions, ensuring cross-device interface consistency.

[0173] In this example, compared to intelligent assistants in related technologies, this embodiment provides prediction of the future state of the GUI during task execution. Based on the evaluation results of the predicted GUI, it optimizes the operations obtained by the agent through task decomposition, providing stronger guidance for the agent's strategy. This strategy optimization mechanism significantly improves the agent's performance in determining the correct path, enabling the agent to efficiently complete complex tasks in a dynamic and ever-changing environment.

[0174] The applications of this application are wide-ranging, including but not limited to multi-step tasks such as installing applications, sending messages, and configuring devices. By providing visual state predictions of future states, the intelligent assistant can quickly adapt to different task scenarios, significantly improve task completion rates, and demonstrate excellent robustness and flexibility in dynamic environments, providing strong technical support for the functional implementation of intelligent assistants on mobile devices.

[0175] Figure 6b This is a schematic diagram illustrating the working process of the intelligent agent in the first example of the application scenario provided in this application embodiment. For example... Figure 6bAs illustrated, for example, during the inference process, when a user inputs a complex task objective via voice or text, such as "turn on settings and enable Bluetooth," the system first relies on the deployed VLM model to parse the task. The VLM, through its multimodal understanding capabilities, decomposes the high-level task input by the user into multiple short-sequence, executable operations. These operations may include: Operation 1, Operation 2, and Operation 3.

[0176] In this example, action 1 is "Open the Settings app". The VLM analyzes the semantics of action 1 and translates it into a screen interaction action. The agent calls the environment awareness module to detect the current screen content and locate the icon of the "Settings" app.

[0177] In this example, operation 2 is "turn on Bluetooth option". In the opened settings page, the smart agent locates the "Bluetooth" option using the UI element recognition function and navigates to the relevant interface.

[0178] In this example, operation 3 is "Enable Bluetooth switch". On the Bluetooth settings page, the smart agent detects the status of the Bluetooth switch button and performs the enable operation.

[0179] In this example, the world model predicts the next GUI state based on the current GUI for each operation. The predicted GUI state is used to evaluate the credibility or confidence level of each corresponding operation. Finally, based on the evaluation result of the predicted GUI state in this embodiment, the agent selects the target operation to be executed; the target operation is executed to obtain the execution result on the current GUI, thereby updating the GUI state.

[0180] In this example, the world model predicts the state of GUI1 for the next step based on action 1 "Open Settings app" and the current GUI0. The agent evaluates the credibility or confidence level of action 1 based on the predicted state of GUI1. Based on the evaluation result of the predicted GUI1 state, the agent optimizes action 1 "Open Settings app". These steps are repeated until the credibility or confidence level of action 1 meets the requirements, at which point the target action corresponding to the final action 1 is selected for execution.

[0181] In this example, the world model predicts the state of the next step, GUI2, based on action 2 "turn on Bluetooth option" and the current GUI1. The agent evaluates the credibility or confidence level of action 2 based on the predicted state of GUI2. The agent optimizes action 2 "turn on Bluetooth option" based on the evaluation result of the predicted state of GUI2. These steps are repeated until the credibility or confidence level of action 2 meets the requirements, at which point the target action corresponding to the final action 2 is selected for execution.

[0182] In this example, the world model predicts the state of the next step, GUI3, based on action 3 "Enable Bluetooth switch" and the current GUI2. The agent evaluates the credibility or confidence level of action 3 based on the predicted state of GUI3. The agent optimizes action 3, "Enable Bluetooth switch," based on the evaluation result of the predicted state of GUI3. These steps are repeated until the credibility or confidence level of action 3 meets the requirements, at which point the target action corresponding to the final action 3 is selected for execution.

[0183] Optionally, in the future GUI state evaluation phase, embodiments of this application rely on a pre-deployed VLM model to determine whether the predicted GUI is on the correct path. For example, the user input information "open settings and enable Bluetooth" and the predicted GUI state are input into the VLM model. The VLM, through its multimodal understanding capabilities, determines the degree of matching between the predicted GUI state and the user input information "open settings and enable Bluetooth". Based on the degree of matching, the confidence level of the GUI state is determined, and the evaluation result is obtained.

[0184] Figure 6c This is a schematic diagram illustrating the working process of the world model in the first example of the application scenario provided in this application embodiment. For example... Figure 6c As shown in the embodiments of this application, examples of various GUI interfaces and corresponding operations for world model input and output in the first example application scenario are provided.

[0185] This example further illustrates the prediction process of GUI1. GUI0 is the current GUI interface on the mobile device. GUI0 includes a navigation bar, a main interface, and a shortcut bar. The navigation bar includes weather information, the time information "11:10", and battery level information. The main interface includes main controls and application (APP) controls. The main controls include the time information "11:10, February 15th, Thursday". APP controls include "Pictures", "Music", "Settings", "Mail", "Clock", "Navigation", etc. The shortcut bar includes shortcuts for "Phone", "SMS", "Camera", etc.

[0186] Operation 1 is "Open Settings app", which means opening the "Settings" app in GUI0. Operation 1 "Open Settings app" and the current GUI0 are input into the world model. The world model predicts the GUI interface after opening the "Settings" app, i.e., GUI1. Then, based on the predicted GUI1 of opening the "Settings" app, the agent evaluates the credibility or confidence level of operation 1 "Open Settings app" and further adjusts operation 1 accordingly.

[0187] GUI1 is the interface within the "Settings" app. The "Settings" app interface includes controls for options such as "Account Login," "System Information," "Network," "Bluetooth," and "Volume."

[0188] This example further illustrates the prediction process for GUI2. Operation 2 is "turn on Bluetooth option," which means turning on the control corresponding to the "Bluetooth" option. Operation 2, "turn on Bluetooth option," and the current GUI1 are input into the world model. The world model predicts the GUI interface after turning on the control corresponding to "Bluetooth," i.e., GUI2. Based on the predicted state of GUI2, the agent evaluates the credibility or confidence level of operation 2 and then adjusts operation 2 accordingly.

[0189] GUI2 is the GUI interface for the "Bluetooth" option. GUI2 includes a "Bluetooth" toggle control, which is set to "OFF".

[0190] This example further illustrates the prediction process of GUI3. Operation 3 is "Enable Bluetooth switch," which means updating the state of the "Bluetooth" switch control in GUI2 to "ON." Operation 3, "Enable Bluetooth switch," and the current GUI2 are input into the world model. The world model predicts a GUI interface where the "Bluetooth" switch control is "ON," i.e., GUI3. Then, based on GUI3 with the "Bluetooth" switch control in an "ON" state, the agent evaluates the credibility or confidence level of operation 3, "Enable Bluetooth switch," and adjusts operation 3 accordingly.

[0191] GUI3 is a GUI interface where the Bluetooth switch control is set to "ON". GUI3 includes the Bluetooth switch control and controls for available devices. The Bluetooth switch control in GUI3 is set to "ON". Available devices represent electronic devices that can currently be connected via Bluetooth, such as electronic devices with names like "ABC", "QWE", and "Zhang San".

[0192] Figure 7a This is a schematic diagram illustrating a second example of an application scenario provided in the embodiments of this application. For example... Figure 7aAs shown, in the second example of the application scenario, within an in-vehicle system, the GUI prediction technology of this application embodiment can improve the intelligence level of driving interaction. The interface design of in-vehicle systems is limited by screen size, interaction methods, and driving safety requirements, thus necessitating a more intelligent interface adaptation solution. This application embodiment can predict the dynamic changes of the in-vehicle interface during driving, such as interface switching in the navigation system and operational adjustments in the entertainment system. Simultaneously, by combining STR prediction with LLM-generated text content, this technology can be used to optimize the interface feedback of the voice assistant, improving the driver's operational efficiency and interactive experience. For example, when the driver issues the navigation command "Set navigation to the nearest gas station and play music," the system can predict the next interface layout based on the current interface state and dynamically generate adapted text instructions, making interface guidance more intuitive.

[0193] In this example, the application of this application embodiment in the human-machine interaction task of an in-vehicle system aims to address the driver's need for complex, multi-step operations on the in-vehicle system during driving. This application embodiment generates a predicted GUI interface through a diffusion model and optimizes operations based on the GUI interface evaluation method; based on operation judgment and optimization techniques, this application embodiment can provide an efficient and reliable solution for in-vehicle human-machine interaction. In an in-vehicle scenario, this application embodiment can be used to solve the problem of predicting future states without relying on actual interaction with the environment, providing operation-driven task execution and policy updates based on operation feedback.

[0194] Figure 7b This is a schematic diagram illustrating the working process of the intelligent agent in a second example of an application scenario provided in this application embodiment. Figure 7a In the second example of the application scenario, such as Figure 7b As illustrated, for example, when a user inputs information via voice (e.g., "set navigation to the nearest gas station and play music"), the VLM parses the user's input and generates an operation sequence. These operations may include operation 1 "open navigation application", operation 2 "search for the nearest gas station", operation 3 "confirm navigation route", and operation 4 "start music playback". This embodiment utilizes the current GUI interface of the in-vehicle system and the decomposed operations to predict the GUI state after the operation; and based on the predicted GUI, determines the accuracy of the corresponding operation. During execution, the agent can dynamically respond to interface changes, such as route recommendations being updated or control buttons being moved, adjusting the confidence level of the operation in real time to ensure the continuity and accuracy of task execution.

[0195] For example, the world model predicts the state of GUI1 for the next step based on operation 1 "open the navigation application" and the current GUI0. The agent evaluates the credibility or confidence level of operation 1 based on the predicted state of GUI1. Based on the evaluation result of the predicted state of GUI1, the agent optimizes operation 1 "open the navigation application". The above steps are repeated until the credibility or confidence level of operation 1 meets the requirements, and then the target operation corresponding to operation 1 is selected for final execution.

[0196] For example, the world model predicts the state of the next step, GUI2, based on operation 2 "search for the nearest gas station" and the current GUI1. The agent evaluates the credibility or confidence level of operation 2 based on the predicted state of GUI2. The agent optimizes operation 2, "search for the nearest gas station," based on the evaluation result of the predicted state of GUI2. The above steps are repeated until the credibility or confidence level of operation 2 meets the requirements, and then the target operation corresponding to operation 2 is selected for final execution.

[0197] For example, the world model predicts the state of the next GUI3 based on operation 3 "confirm navigation path" and the current GUI2. The agent evaluates the credibility or confidence level of operation 3 based on the predicted state of GUI3. Based on the evaluation result of the predicted state of GUI3, the agent optimizes operation 3 "confirm navigation path". The above steps are repeated until the credibility or confidence level of operation 3 meets the requirements, and then the target operation corresponding to the final operation 3 is selected for execution.

[0198] For example, the world model predicts the state of GUI4 in the next step based on operation 4 "start music playback" and the current GUI3. The agent evaluates the credibility or confidence level of operation 4 based on the predicted state of GUI4. Based on the evaluation result of the predicted state of GUI4, the agent optimizes operation 4 "start music playback". The above steps are repeated until the credibility or confidence level of operation 3 meets the requirements, and then the target operation corresponding to operation 3 is selected for final execution.

[0199] Optionally, during the evaluation phase of future GUI states, a pre-deployed VLM module is used to assess whether the operation's result is on the correct path, generating evaluation results for each operation, such as confidence, credibility, and reward signals. This enhances the agent's understanding of the current operation, thereby improving execution efficiency and stability. This example is applicable to tasks such as navigation settings, music control, and in-vehicle system configuration, providing users with a safe and efficient interactive experience through operation future state generation and policy optimization.

[0200] It is understood that the embodiments of this application, by combining the multimodal processing capabilities of symbolic text representation (STR), diffusion models, and large language models, provide technical support for the efficient execution of complex GUI tasks. Whether in mobile devices, web page interactions, or in-vehicle systems, intelligent agents can rely on the interface state prediction and text generation mechanisms of the embodiments of this application to improve the accuracy and stability of interface interactions. Based on the modular design and powerful scalability of electronic devices, the embodiments of this application can adapt to various application scenarios, providing users with more intelligent interface generation and optimization solutions.

[0201] Furthermore, the technical solutions of the embodiments of this application will be further described to illustrate possible application scenarios. The technical solutions of the embodiments of this application, especially the vision-based GUI world model, are not only applicable to the task execution domain of current GUI intelligent agents, but also have broad cross-domain application potential.

[0202] For example, the application field of the technical solutions in this application embodiment includes the field of robotics. The visual world model in this application embodiment can be applied to robot control systems. Robots need to perform tasks in complex and dynamic environments, such as home automation, manufacturing assembly lines, and autonomous driving. In these tasks, robots predict the world state after performing the task in advance, thereby improving learning efficiency and task success rate.

[0203] For example, the application areas of the technical solutions in this application include the fields of intelligent assistants and personal assistants. Besides applications in mobile device and web page interactions, the technical solutions in this application can be widely applied to smart home assistants or virtual assistants. For instance, when a user inputs complex commands via voice or text, this application can ensure efficient execution of user commands by predicting the future state of simple commands (such as controlling multiple home appliances, adjusting environmental settings, etc.). This cross-device, cross-platform operation generation and dynamic adjustment capability will enhance the actual usability of the virtual assistant.

[0204] For example, the application areas of the technical solutions in this application include the fields of industrial automation and logistics. In automated factory or warehouse management systems, the method based on future state prediction can be used to efficiently schedule production line operations and warehouse management tasks, automatically generate the future states of sub-tasks, reduce the possibility of errors in advance, reduce human intervention, and improve production efficiency.

[0205] For example, the application areas of the technical solutions in this application embodiment include the medical and health field. In medical auxiliary decision-making systems, the technical solutions in this application embodiment can help doctors break down and optimize the diagnosis and treatment process step by step. Through algorithms based on task outcome prediction, intelligent systems can help predict the execution consequences for each stage of the medical process in advance, thereby improving efficiency. For example, developing personalized treatment plans or managing complex patient data processes.

[0206] For example, the application areas of the technical solutions in this application include the field of education and training. In adaptive learning systems, simulations based on real-world states can help the system dynamically adjust teaching content according to students' learning progress and interests. Through student interaction and visual stimulation, it can improve students' understanding of the consequences of their current actions and stimulate their learning interest.

[0207] In this embodiment, the visual information generation process is illustrated by way of example. Visual information in the GUI can be predicted using STR-based graphical states, or other visual states, such as a complete UI image, can be used for graphical state prediction. For example, when the GUI does not contain text, the next GUI element after an operation can be predicted based on a complete UI image. By introducing different types of visual information, a complete GUI interface can be generated based solely on visual input without relying on a large oracle model. Furthermore, the STR form is merely illustrative; other forms can be designed to represent textual information in the GUI, such as blurring the text.

[0208] This application provides an exemplary description of the text symbol processing process in the GUI interface. This application uses LLM to analyze the predicted STR and predict the specific text content within each symbol. Other neural network models or fine-tuned language models can also be used to predict text, thereby achieving text prediction. The accuracy of text prediction can be improved by enhancing the stability of other algorithms and by collecting more GUI data.

[0209] Figure 7c This is a schematic diagram illustrating the working process of the world model in a second example of an application scenario provided in this application embodiment. For example... Figure 7c As shown in the embodiments of this application, examples of various GUI interfaces and corresponding operations for world model input and output in the second example application scenario are provided.

[0210] This example further illustrates the prediction process of GUI1. GUI0 is the current GUI interface in the vehicle system. GUI0 includes a navigation bar, a main interface, and a shortcut bar. The navigation bar includes weather information, time information "11:10, February 15th", network signal strength information, and battery level information. The main interface includes multiple application (APP) controls; APP controls include "Pictures", "Music", "Settings", "Email", "Clock", "Navigation", "Camera", etc. The shortcut bar includes controls for shortcut keys such as "Phone", "SMS", and "Camera".

[0211] Operation 1 is "Open the navigation application," which means opening the "Navigation" app in GUI0. Operation 1, "Open the navigation application," and the current GUI0 are input into the world model. The world model predicts the GUI interface after opening the "Navigation" app, i.e., GUI1. Furthermore, the agent evaluates the credibility or confidence level of operation 1, "Open the navigation application," based on the predicted GUI1, and adjusts operation 1 accordingly.

[0212] GUI1 is the interface of the "Navigation" app. The "Navigation" app interface includes a map display area and an operation area. The map display area includes icons for the user's current location, roads, and buildings. The road icon is a crossroads. Building icons include icons for office buildings, gas stations, restrooms, parking lots, etc. The operation area includes controls for "Delete," "Search," "Back," "Forward," and files.

[0213] This example further illustrates the prediction process for GUI2. Operation 2 is "Search for the nearest gas station," which involves using the "Search" control to search for gas stations on the map and displaying the one closest to the user's location. Operation 2, "Search for the nearest gas station," and the current GUI1 are input into the world model. The world model predicts the GUI interface that will appear on the map after "Search for the nearest gas station," i.e., GUI2. Based on the predicted state of GUI2, the agent evaluates the credibility or confidence level of Operation 2 and adjusts Operation 2 accordingly.

[0214] GUI2 is the GUI interface displayed after "Searching for the nearest gas station" on the map. GUI2 includes gas station icons and a navigation prompt pop-up. The gas station icon is the icon of the nearest gas station after the search, which is located in the upper left corner of the map and is highlighted. The navigation prompt pop-up is a pop-up asking "Do you need navigation?" and includes "Yes" and "No" option controls.

[0215] This example further illustrates the prediction process of GUI3. Operation 3 is "Confirm Navigation Path," which is clicking the "Yes" option control in the navigation prompt pop-up window asking "Do you need navigation?". Operation 3, "Confirm Navigation Path," and the current GUI2 are input into the world model. The world model predicts the GUI interface that appears after clicking the "Yes" option control in the navigation prompt pop-up window, i.e., GUI3. Then, based on GUI3, the agent evaluates the credibility or confidence level of operation 3, "Confirm Navigation Path," and adjusts operation 3 accordingly.

[0216] GUI3 refers to the GUI interface displayed after clicking the "Yes" option control in the navigation prompt pop-up. GUI3 includes a highlighted gas station icon, an icon of the user's current location, and a navigation path. This navigation path refers to the route from the user's current location to the nearest gas station.

[0217] This example further illustrates the prediction process for GUI4. Operation 4 is "start music playback," which means clicking the music app control while navigating and playing music in the background. Operation 4, "start music playback," and the current GUI3 are input into the world model. The world model predicts the GUI interface where the music app control is clicked and music plays in the background, i.e., GUI4. Then, based on GUI4, the agent evaluates the credibility or confidence level of operation 4, "start music playback," and adjusts operation 3 accordingly.

[0218] GUI4 is the GUI interface for background music playback. GUI4 includes a navigation interface and controls for background music playback. The navigation interface is described in the relevant section of GUI3 and will not be repeated here. Controls for background music playback include, for example, the "***" control for background playback.

[0219] Figure 8 This is a flowchart illustrating a prediction method using a graphical user interface provided in an embodiment of this application. Figure 8 As shown in the figure, this application provides a prediction method for a graphical user interface, which mainly includes the following steps:

[0220] Step S810: Process the text information of the first interface using symbols to obtain a first symbolized interface. Here, the first interface represents the graphical user interface to be operated, and includes text information; the first symbolized interface represents the graphical user interface where the text information is covered by symbols.

[0221] Step S820: Input the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface. The second symbolic interface represents the graphical user interface after the operation to be performed on the first symbolic interface.

[0222] Step S830: Predict the text corresponding to each symbol in the second symbolized interface to obtain the second interface. The second interface represents the predicted graphical user interface after the operation to be performed on the first interface.

[0223] In this embodiment, the text in the first interface is masked using symbols to obtain a first symbolized interface. A diffusion model predicts a second symbolized interface based on the image information and the operation to be performed in the first symbolized interface. The text corresponding to each symbol in the second symbolized interface is predicted to obtain the second interface. This decouples the graphic content prediction (step S820) and text content prediction (step S830), solving the problem of mixed processing of graphic and text information. Mixed processing of graphic and text information can lead to problems such as blurred and improperly arranged text content in the generated interface, affecting readability, especially for smaller text, which is not clearly displayed in pixel form. Generating the second interface based on the first interface before the operation and the operation to be performed solves the problem of uncertainty in the interaction logic when generating the second interface solely based on the operation to be performed. In interface interaction tasks, it is necessary to ensure the consistency between adjacent first and second interfaces. This embodiment not only generates corresponding images based on the operation to be performed but also considers the state of the first interface, thereby constraining the layout of the second interface and ensuring the consistency between adjacent first and second interfaces. In this embodiment, the first symbolic interface and the first interface maintain a high degree of visual consistency. The first symbolic interface can provide accurate visual information about the interface, which improves the accuracy of the prediction results of the second symbolic interface and solves the problem of missing visual information in the interface prediction process.

[0224] In one possible implementation, the above-mentioned use of symbols to process the text information of the first interface to obtain the first symbolized interface includes: detecting the text information and its position information of the first interface; and using symbols to mask the text information according to the position information to obtain the first symbolized interface.

[0225] In this implementation, the text information and its location information of the first interface are detected. Based on the location information, the text information is masked using symbols to obtain the first symbolized interface. Thus, the first symbolized interface no longer contains text information. The diffusion model no longer performs text prediction during the prediction of the second symbolized interface, achieving decoupling between graphic content prediction and text content prediction. This improves the accuracy of the second symbolized interface, and consequently, the accuracy of the second interface itself.

[0226] In one possible implementation, the text information in the first interface includes static text, which represents non-editable text in the first interface. The above-described method of processing the text information of the first interface using symbols to obtain a first symbolized interface includes: processing text information other than static text in the first interface using symbols to obtain the first symbolized interface.

[0227] In this implementation, the text information within the first interface is divided into dynamic text and static text. Static text remains unchanged across different interfaces, thus preserving it within the first symbolic interface. Static text represents non-editable text within the first interface, resulting in a low correlation between static text and the operation to be performed. Consequently, predicting static text is more difficult, less accurate, and more prone to errors. Therefore, this implementation avoids the prediction of static text, improving the accuracy of text prediction and reducing the possibility of prediction errors.

[0228] In one possible implementation, the above-mentioned use of symbols to process text information other than static text in the first interface to obtain the first symbolized interface includes: inputting the first interface into a large language model to identify static text in the first interface; and using symbols to process text information other than static text in the first interface based on the identified static text to obtain the first symbolized interface.

[0229] In this implementation, the large language model has strong text analysis capabilities, which can identify static text in the first interface and thus retain the static text in the first symbolic interface.

[0230] In one possible implementation, the above-mentioned inputting the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface includes: determining a noisy image; inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, wherein the diffusion model uses the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; and performing denoising processing on the noisy image based on the predicted noise to obtain the second symbolic interface.

[0231] In this implementation, the diffusion model uses the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; the noisy image is then denoised to obtain the second symbolic interface. Furthermore, the denoising stage of the diffusion model is used to predict the second symbolic interface.

[0232] In one possible implementation, the aforementioned noisy image is obtained by adding noise to the first symbolic interface.

[0233] In this implementation, noise is added to the first symbolized interface to obtain a noisy image. This allows the diffusion model to learn and generate the second symbolized interface from the distribution of the noisy image corresponding to the first symbolized interface, rather than learning from the distribution of the input dataset. This improves the robustness of the diffusion model.

[0234] In one possible implementation, the above-mentioned inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model includes: performing feature concatenation on the noisy image and the first symbolic interface to obtain fused features; and inputting the fused features and the operation to be performed into the diffusion model.

[0235] In this implementation, both the noisy image and the first symbolized interface are image-type data. Features from the noisy image and the first symbolized interface can be concatenated using the same convolution kernel to obtain fused features. These fused features and the operation to be performed are then input into the diffusion model. This results in a simpler model structure and higher computational efficiency.

[0236] In one possible implementation, the above-mentioned prediction of the text corresponding to each symbol in the second symbolized interface to obtain the second interface includes: inputting the second symbolized interface, the first interface and the operation to be performed into a large language model to predict the text corresponding to each symbol in the second symbolized interface; replacing each symbol in the second symbolized interface with the corresponding text to obtain the second interface.

[0237] In this implementation, the large language model has strong text understanding and analysis capabilities. Based on its understanding of the first interface and the operation to be performed, the large language model can accurately predict the text corresponding to each symbol in the second symbolized interface.

[0238] In one possible implementation, the above-mentioned inputting the second symbolized interface, the first interface, and the operation to be performed into a large language model to predict the text corresponding to each symbol in the second symbolized interface includes: determining prompt words based on the second symbolized interface, the first interface, and the operation to be performed, wherein each symbol in the second symbolized interface is the analysis object, and the first interface and the operation to be performed are the context information; and inputting the prompt words into the large language model to predict the text corresponding to each symbol in the second symbolized interface.

[0239] In this implementation, the prompt words use the symbols in the second symbolized interface as the analysis objects, and the first interface and the operation to be performed as contextual information. In this way, the prompt words can effectively guide the large language model to predict the text corresponding to each symbol in the second symbolized interface.

[0240] In one possible implementation, the aforementioned prompts include at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface.

[0241] In this implementation, the prompt includes at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface. In this way, the large language model can distinguish each symbol in the second symbolized interface based on its identification information (such as number), size information, and position information, thereby predicting the text corresponding to each symbol.

[0242] In one possible implementation, the above-mentioned acquisition of the first interface and the operation to be executed includes: acquiring user input information, the input information representing the user's task to be executed; decomposing the input information into an operation sequence, the operation sequence including at least one operation; acquiring the operation to be executed from the operation sequence, and using the graphical user interface after the execution of the previous operation as the first interface; wherein, when the operation to be executed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be executed.

[0243] This implementation provides a method for obtaining a first interface and operations to be performed. When processing user input information, the input information is decomposed into an operation sequence; each operation in the operation sequence is taken as an operation to be performed; then, the method of this embodiment is used to simulate each operation in the operation sequence to verify its feasibility and effectiveness.

[0244] In one possible implementation, the method further includes: evaluating the second interface to obtain an evaluation result of the operation to be executed; and adjusting the operation to be executed based on the evaluation result to obtain the target operation.

[0245] In this implementation, the second interface is evaluated, and the operation to be executed is adjusted based on the evaluation result to obtain the target operation. Then, the target operation is executed to obtain the final execution result. This automates the process of completing user input. The embodiments of this application make each operation in the operation sequence more reasonable, thereby avoiding potentially erroneous operations performed directly on the real interface and reducing problems such as execution failure or task termination caused by unreasonable interaction.

[0246] In one possible implementation, the aforementioned operation to be executed is obtained by decomposing the input information into tasks, where the input information represents the user's task to be executed. The aforementioned evaluation of the second interface to obtain the evaluation result of the operation to be executed includes: inputting the input information and the second interface into a large language model to output the evaluation result.

[0247] In this implementation, the large language model has strong text understanding and analysis capabilities. The large language model is a type of large language model that can process multimodal data. It combines visual input (second interface) and language input (user input information) to perform multimodal understanding, thereby determining whether the current feedback second interface is on the correct path, that is, the degree of consistency between the predicted second interface and the user input information. It generates confidence scores to evaluate the rationality and accuracy of the current operation to be evaluated. Based on the evaluation results, it adjusts the current operation to be executed to obtain the optimized target operation.

[0248] In one possible implementation, the above-mentioned operation to be executed is provided with at least one alternative operation. The step of adjusting the operation to be executed based on the evaluation results to obtain the target operation includes: determining the target operation from the operation to be executed and the at least one alternative operation based on the evaluation results of the operation to be executed and the evaluation results of the at least one alternative operation.

[0249] This implementation provides a method for determining the target operation. By setting multiple operations (the operation to be executed and at least one alternative operation) for each operation step, the target operation can be determined based on the evaluation results (e.g., confidence levels) of the multiple operations. This ensures the reasonableness of the target operation; for example, the target operation is the one with the highest confidence level among the multiple operations.

[0250] In one possible implementation, the training method of the above diffusion model includes: constructing a training dataset; the training dataset includes one or more training sample pairs, each training sample pair including a training sample and a label for the training sample, the training sample including a symbolic first interface sample and an operation sample, the symbolic first interface sample being obtained by symbolically processing the text information of the first interface sample, the label of the training sample representing the ground truth of the symbolic second interface, the symbolic second interface representing the symbolic graphical user interface after performing the operation sample on the symbolic first interface sample; inputting the symbolic first interface sample and the operation sample into the diffusion model to generate a predicted value for the symbolic second interface; and adjusting the parameters of the diffusion model based on the loss value between the ground truth and the predicted value of the symbolic second interface.

[0251] In this implementation, during the training of the diffusion model, symbolized first interface samples and operational samples are input into the diffusion model to generate predicted values ​​for symbolized second interfaces. The parameters of the diffusion model are adjusted based on the loss value between the ground truth and predicted values ​​of the symbolized second interfaces. Thus, the trained diffusion model functions as a denoising autoencoder for symbolized images, predicting noise in noisy image samples to achieve noise removal and obtain the symbolized second interface.

[0252] Based on the same concept as the foregoing embodiments, this application also provides a prediction device with a graphical user interface.

[0253] Figure 9 This is a schematic diagram illustrating the composition of a prediction device with a graphical user interface provided in an embodiment of this application. Figure 9 As shown, in a second aspect, this application provides a prediction device 900 with a graphical user interface, mainly comprising:

[0254] The symbolization processing module 910 is used to process the text information of the first interface using symbols to obtain the first symbolized interface; wherein, the first interface represents the graphical user interface to be operated, the first interface includes text information, and the first symbolized interface represents the graphical user interface in which the text information is covered by symbols.

[0255] The symbolic interface prediction module 920 is used to input the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface. The second symbolic interface represents the graphical user interface after the operation to be performed on the first symbolic interface.

[0256] The text prediction module 930 is used to predict the text corresponding to each symbol in the second symbolized interface to obtain the second interface. The second interface represents the predicted graphical user interface after the operation to be performed on the first interface.

[0257] In one possible implementation, when processing the text information of the first interface using symbols to obtain the first symbolized interface, the aforementioned symbolization processing module 910 is specifically used to: detect the text information and its position information of the first interface; and, based on the position information, use symbols to mask the text information to obtain the first symbolized interface.

[0258] In one possible implementation, the text information in the first interface includes static text, which represents non-editable text in the first interface. When processing the text information of the first interface using symbols to obtain the first symbolized interface, the symbolization processing module 910 is specifically used to: process text information other than static text in the first interface using symbols to obtain the first symbolized interface.

[0259] In one possible implementation, the above-mentioned use of symbols to process text information other than static text in the first interface to obtain the first symbolized interface includes: inputting the first interface into a large language model to identify static text in the first interface; and using symbols to process text information other than static text in the first interface based on the identified static text to obtain the first symbolized interface.

[0260] In one possible implementation, when the first symbolic interface and the operation to be performed are input into the diffusion model to predict the second symbolic interface, the symbolic interface prediction module 920 described above is specifically used to: determine the noisy image; input the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, wherein the diffusion model uses the noisy image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noisy image; and perform denoising processing on the noisy image based on the predicted noise to obtain the second symbolic interface.

[0261] In one possible implementation, the aforementioned noisy image is obtained by adding noise to the first symbolic interface.

[0262] In one possible implementation, when inputting the noisy image, the first symbolic interface, and the operation to be performed into the diffusion model, the symbolic interface prediction module 920 described above is specifically used to: perform feature concatenation on the noisy image and the first symbolic interface to obtain fused features; and input the fused features and the operation to be performed into the diffusion model.

[0263] In one possible implementation, when predicting the text corresponding to each symbol in the second symbolized interface to obtain the second interface, the text prediction module 930 described above is specifically used to: input the second symbolized interface, the first interface, and the operation to be performed into the large language model to predict the text corresponding to each symbol in the second symbolized interface; and replace each symbol in the second symbolized interface with the corresponding text to obtain the second interface.

[0264] In one possible implementation, when the second symbolic interface, the first interface, and the operation to be performed are input into a large language model to predict the text corresponding to each symbol in the second symbolic interface, the aforementioned text prediction module 930 is specifically used to: determine prompt words based on the second symbolic interface, the first interface, and the operation to be performed, wherein each symbol in the second symbolic interface is the analysis object, and the first interface and the operation to be performed are the context information; and input the prompt words into the large language model to predict the text corresponding to each symbol in the second symbolic interface.

[0265] In one possible implementation, the aforementioned prompts include at least one of the following: identification information, size information, and position information of each symbol in the second symbolized interface.

[0266] In one possible implementation, the graphical user interface prediction device 900 further includes: a preprocessing module for acquiring user input information, the input information representing the user's task to be executed; decomposing the input information into an operation sequence, the operation sequence including at least one operation; acquiring the operation to be executed from the operation sequence, and using the graphical user interface after the execution of the previous operation as the first interface; wherein, when the operation to be executed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be executed.

[0267] In one possible implementation, the graphical user interface prediction device 900 further includes: an evaluation module for evaluating the second interface to obtain an evaluation result of the operation to be executed; and adjusting the operation to be executed based on the evaluation result to obtain the target operation.

[0268] In one possible implementation, the aforementioned operation to be executed is obtained by decomposing the input information into tasks, where the input information represents the user's task to be executed. When evaluating the second interface to obtain the evaluation result of the operation to be executed, the aforementioned evaluation module is specifically used for:

[0269] Input information and the second interface are fed into a large language model to output evaluation results.

[0270] In one possible implementation, the above-mentioned operation to be executed is provided with at least one alternative operation. When adjusting the operation to be executed based on the evaluation results to obtain the target operation, the evaluation module is specifically used to: determine the target operation from the operation to be executed and the at least one alternative operation based on the evaluation results of the operation to be executed and the evaluation results of the at least one alternative operation.

[0271] In one possible implementation, the aforementioned graphical user interface prediction device 900 further includes a model training module for training a diffusion model. Specifically, this model training module is used to: construct a training dataset; the training dataset includes one or more training sample pairs, each training sample pair including a training sample and a label for the training sample; the training sample includes a symbolic first interface sample and an operation sample; the symbolic first interface sample is obtained by symbolically processing the text information of the first interface sample; the label of the training sample represents the ground truth value of the symbolic second interface; the symbolic second interface represents the symbolic graphical user interface after performing the operation sample on the symbolic first interface sample; input the symbolic first interface sample and the operation sample into the diffusion model to generate a predicted value for the symbolic second interface; and adjust the parameters of the diffusion model based on the loss value between the ground truth value and the predicted value of the symbolic second interface.

[0272] The following is about... Figure 9The software and hardware implementation of the graphical user interface prediction device 900 shown (hereinafter referred to as: the device of the present application embodiment) needs further explanation.

[0273] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0274] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0275] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0276] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0277] It should be noted that, in other embodiments, the apparatus of this application embodiment additionally provides one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the apparatus of this application embodiment can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the apparatus of this application embodiment.

[0278] This application also provides a computing device 1000. For example... Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, memory 1006, and communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, laptop computer, or smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0279] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10The bus 1004 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0280] The processor 1004 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0281] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, the memory 1006 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0282] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1006, or two or more types of storage media can be configured to realize the function of memory 1006. This application does not limit this.

[0283] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 1006 stores instructions for executing the methods described in the above embodiments.

[0284] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 1006 stores instructions for executing the methods described in the above embodiments.

[0285] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0286] As one possible implementation, the computing device 1000 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0287] As one possible implementation, the computing device 1000 may include various types of processors 1004, that is, the computing device 1000 is a heterogeneous device. For example, the computing device 1000 includes a CPU and a GPU, and at least one of the processors 1004 can execute the operation steps corresponding to the method of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0288] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0289] like Figure 11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0290] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the methods described in the above embodiments.

[0291] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the apparatus in this application embodiment. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules of the apparatus in this application embodiment.

[0292] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of one or more modules of the apparatus of this application embodiment. Meanwhile, the memory 1006 in computing device 1000B stores instructions for the functions of another one or more modules of the apparatus of this application embodiment.

[0293] It should be understood that Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0294] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 11 and Figure 12 The connection method of the computing device cluster is different in that the memory 1006 of one or more computing devices 1000 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0295] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for performing the aforementioned method.

[0296] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0297] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0298] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0299] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0300] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0301] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0302] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A prediction method for a graphical user interface, characterized in that, include: The text information of the first interface is processed using symbols to obtain the first symbolized interface; The first interface represents a graphical user interface to be operated, and the first interface includes text information. The first symbolic interface represents a graphical user interface in which the text information is covered by the symbol. The first symbolic interface and the operation to be performed are input into the diffusion model to predict the second symbolic interface; the second symbolic interface represents the graphical user interface after the operation to be performed is executed on the first symbolic interface; Predict the text corresponding to each symbol in the second symbolized interface to obtain the second interface, which represents the predicted graphical user interface after the operation to be performed on the first interface.

2. The method according to claim 1, characterized in that, The process of using symbols to process the text information of the first interface to obtain the first symbolized interface includes: Detect the text information and its location information on the first interface; Based on the location information, the text information is masked using symbols to obtain the first symbolized interface.

3. The method according to claim 1 or 2, characterized in that, The text information includes static text, which represents non-editable text in the first interface; The process of using symbols to process the text information of the first interface to obtain the first symbolized interface includes: The first symbolic interface is obtained by processing text information other than the static text in the first interface using symbols.

4. The method according to claim 3, characterized in that, The process of using symbols to process text information other than the static text in the first interface to obtain the first symbolized interface includes: The first interface is input into a large language model to recognize the static text in the first interface; Based on the identified static text, text information other than the static text in the first interface is processed using symbols to obtain a first symbolized interface.

5. The method according to any one of claims 1-4, characterized in that, The step of inputting the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface includes: Identify the noisy image; The noise image, the first symbolic interface, and the operation to be performed are input into the diffusion model. The diffusion model uses the noise image as the denoising object and the first symbolic interface and the operation to be performed as conditional information to predict the noise in the noise image. Based on the predicted noise, the noisy image is denoised to obtain the second symbolic interface.

6. The method according to claim 5, characterized in that, The noisy image is obtained by adding noise to the first symbolic interface.

7. The method according to claim 5 or 6, characterized in that, The step of inputting the noise image, the first symbolic interface, and the operation to be performed into the diffusion model includes: The noisy image and the first symbolized interface are concatenated to obtain fused features; The fusion features and the operation to be performed are input into the diffusion model.

8. The method according to any one of claims 1-7, characterized in that, The step of predicting the text corresponding to each symbol in the second symbolized interface to obtain the second interface includes: The second symbolized interface, the first interface, and the operation to be performed are input into a large language model to predict the text corresponding to each symbol in the second symbolized interface. Replace each symbol in the second symbolized interface with its corresponding text to obtain the second interface.

9. The method according to claim 8, characterized in that, The step of inputting the second symbolized interface, the first interface, and the operation to be performed into a large language model to predict the text corresponding to each symbol in the second symbolized interface includes: Based on the second symbolic interface, the first interface, and the operation to be executed, a prompt word is determined, wherein each symbol in the second symbolic interface is the analysis object, and the first interface and the operation to be executed are the context information; The prompt words are input into a large language model to predict the text corresponding to each symbol in the second symbolized interface.

10. The method according to claim 9, characterized in that, The prompt includes at least one of the following: the identification information, size information, and position information of each symbol in the second symbolized interface.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Obtain user input information, wherein the input information represents the user's task to be performed; The input information is decomposed into a task to obtain an operation sequence, the operation sequence including at least one operation; The operation to be executed is obtained from the operation sequence, and the graphical user interface after the execution of the operation preceding the operation to be executed is used as the first interface; wherein, when the operation to be executed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be executed.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: The second interface is evaluated to obtain the evaluation result of the operation to be performed; Based on the evaluation results, the operation to be executed is adjusted to obtain the target operation.

13. The method according to claim 12, characterized in that, The operation to be performed is obtained by decomposing the input information into tasks, where the input information represents the user's task to be performed. The evaluation of the second interface to obtain the evaluation result of the operation to be performed includes: The input information and the second interface are input into the large language model to output the evaluation result.

14. The method according to claim 12 or 13, characterized in that, The operation to be performed has at least one alternative operation; The step of adjusting the operation to be executed based on the evaluation result to obtain the target operation includes: Based on the evaluation results of the operation to be executed and the evaluation results of the at least one alternative operation, the target operation is determined from the operation to be executed and the at least one alternative operation.

15. The method according to any one of claims 1-14, characterized in that, The training method for the diffusion model includes: Construct a training dataset; the training dataset includes one or more training sample pairs, each training sample pair includes a training sample and a label for the training sample, the training sample includes a symbolic first interface sample and an operation sample, the symbolic first interface sample is obtained by processing the text information of the first interface sample using symbolic processing, the label of the training sample represents the truth value of the symbolic second interface, and the symbolic second interface represents the symbolic graphical user interface after performing the operation sample on the symbolic first interface sample; The symbolized first interface sample and the operation sample are input into the diffusion model to generate the predicted value of the symbolized second interface; The parameters of the diffusion model are adjusted based on the loss value between the true value and the predicted value of the symbolized second interface.

16. A prediction device with a graphical user interface, characterized in that, include: The symbolization processing module is used to process the text information of the first interface using symbols to obtain the first symbolized interface; The first interface represents a graphical user interface to be operated, and the first interface includes text information. The first symbolic interface represents a graphical user interface in which the text information is covered by the symbol. The symbolic interface prediction module is used to input the first symbolic interface and the operation to be performed into the diffusion model to predict the second symbolic interface; the second symbolic interface represents the graphical user interface after the operation to be performed is executed on the first symbolic interface; The text prediction module is used to obtain the text corresponding to each symbol in the second symbolized interface to obtain the second interface, which represents the graphical user interface after the predicted operation is performed on the first interface.

17. A world model, characterized in that, When a first interface and an operation to be performed are input into the world model, the method described in any one of claims 1-15 is executed to predict a second interface; the first interface represents a graphical user interface to be operated, the first interface includes text information, and the second interface represents a predicted graphical user interface after the operation to be performed is executed on the first interface.

18. An intelligent agent, characterized in that, The system includes the world model as described in claim 17; wherein the agent is configured to acquire user input information, the input information representing a user's task to be performed; and to decompose the input information into a task sequence, the operation sequence including at least one operation; and to acquire the operation to be performed from the operation sequence, and to use the graphical user interface after the execution of the previous operation of the operation to be performed as the first interface; wherein when the operation to be performed is the first operation in the operation sequence, the first interface is obtained by detecting the graphical user interface before the execution of the operation to be performed; and to invoke the world model so that the world model predicts a second interface based on the first interface and the operation to be performed, the second interface representing the predicted graphical user interface after the execution of the operation to be performed on the first interface.

19. An electronic device, characterized in that, include: The intelligent agent as described in claim 18; the electronic device is used to acquire user input information and invoke the intelligent agent to enable the intelligent agent to perform the task corresponding to the input information.

20. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the method as described in any one of claims 1-15.

21. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-15.

22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-15.

23. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-15, wherein the cluster of computing devices includes at least one computing device.

24. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-15, wherein the computing device cluster includes at least one computing device.

Images