Information determination method, device, and computer-readable storage medium

Abstract

The application provides an information determination method, device and computer program product; the method comprises: determining a target hypertext markup language text representation corresponding to a multi-modal user interface of screen information of a virtual mobile device; wherein the screen information is generated based on user input on the virtual mobile device; based on the text representation and a large language model, generating a thinking chain prompt corresponding to the text representation and a to-be-executed task; wherein the thinking chain prompt is used to lead out a logical reasoning process in the large language model, so that the large language model clarifies its thinking process before reaching a final answer; based on the thinking chain prompt, the to-be-executed task is executed by an executor to obtain an operation result corresponding to the to-be-executed task.

Description

Technical Field

[0001] This application relates to task automation technology, and more particularly to an information determination method, device, and computer-readable storage medium. Background Technology

[0002] Currently, with the deep integration of cloud-native architecture and generative artificial intelligence, the cloudification of mobile applications based on cloud phones has become an important development trend. In the field of task automation, existing technologies mainly fall into two categories: one relies on mobile applications to demonstrate programming of IoT devices, using manual operation of smartphone applications to demonstrate task flows. However, this method suffers from poor scalability, difficulty in handling large-scale or complex tasks, and insufficient understanding of the task, easily leading to instruction errors or inefficient execution. The other approach connects a basic model to an application programming interface (API) to complete tasks, utilizing API calls to implement operations. However, this approach suffers from incomplete API descriptions, difficulty in fully covering executor capabilities, and a disconnect between task understanding and execution, resulting in generated instructions that do not match actual needs and poor execution reliability. In summary, related technologies generally face technical problems such as poor scalability, insufficient reliability, and a lack of flexibility in the execution process during cloud phone task automation, urgently requiring a solution that can achieve efficient, reliable, and intelligent task automation. Summary of the Invention

[0003] This application provides an information determination method, device, and computer-readable storage medium, which provide highly reliable and highly interpretable task automation execution capabilities for virtualized mobile devices.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an information determination method, the method comprising: The text representation of the target hypertext markup language corresponding to the multimodal user interface that determines the screen information of the virtualized mobile device; wherein the screen information is generated based on user input on the virtualized mobile device; Based on the text representation and the large language model, thought chain prompts and tasks to be performed are generated corresponding to the text representation; wherein, the thought chain prompts are used to elicit the logical reasoning process in the large language model, so that the large language model can clarify its thought process before arriving at the final answer; The executor executes the task to be executed based on the thought chain prompts, and obtains the operation result corresponding to the task to be executed.

[0005] In the above scheme, the text representation of the target Hypertext Markup Language corresponding to the multimodal user interface that determines the screen information of the virtualized mobile device includes: Obtain the view hierarchy in the multimodal user interface and represent the view hierarchy as a view hierarchy tree; Based on a depth-first search, the user interface elements corresponding to each node in the view hierarchy tree are traversed to determine the text representation of the Hypertext Markup Language corresponding to each user interface element. For each text representation of the Hypertext Markup Language to be processed, an index corresponding to the text representation of the Hypertext Markup Language to be processed is determined, and the index is inserted into the text representation of the Hypertext Markup Language to be processed to obtain the text representation of the target Hypertext Markup Language; wherein, the index is generated according to the depth-first search order in the view hierarchy tree; Based on class mapping rules, the Android user interface component classes of the virtualized mobile device are mapped to the text representations of the target Hypertext Markup Language with corresponding functions; wherein, the class mapping rules are used to indicate the mapping between Android user interface component classes and Hypertext Markup Language.

[0006] In the above scheme, the step of traversing the user interface elements corresponding to each node in the view hierarchy tree based on depth-first search to determine the text representation of the Hypertext Markup Language corresponding to each user interface element includes: The view hierarchy tree is traversed by depth-first search, and the user interface element corresponding to each node in the view hierarchy tree is converted into a text representation of the initial hypertext markup language. For each of the initial hypertext markup language text representations, a target attribute set for the user interface element corresponding to the text representation of the initial hypertext markup language is determined; wherein, the target attribute set includes class attributes, text attributes, resource identifier attributes, and content description attributes; The target attribute set is written into the text representation of the initial hypertext markup language to obtain the text representation of the hypertext markup language to be processed.

[0007] In the above scheme, the step of mapping the Android user interface component classes of the virtualized mobile device to the text representations of the target Hypertext Markup Language with corresponding functions based on class mapping rules includes: Based on the class mapping rules, text view classes are mapped to paragraph tags, button-related classes are mapped to button tags, image-related classes are mapped to image tags, and text input classes are mapped to input tags. All other classes in the Android user interface component classes, except for the text view class, the button-related class, the image-related class, and the text input class, are mapped to partition tags. The Android user interface component classes include the text view class, the button-related class, the image-related class, the text input class, and the other classes. The text representation in the target hypertext markup language includes the paragraph tags, the button tags, the image tags, the input tags, and the partition tags.

[0008] In the above scheme, generating thought chain prompts and tasks to be executed corresponding to the text representation based on the text representation and the large language model includes: The text representation is processed using the large language model to obtain intermediate inference results; Based on multiple intermediate reasoning results, the thought chain prompt is determined; Based on the task prompt structure and the thought chain prompt, the task to be executed is generated; wherein, the task prompt structure includes a preface, multiple examples and a test screen; the preface is used to explain the purpose of the prompt, each example includes an input, a thought chain and an output, and the test screen is the screen input to be processed.

[0009] An information processing method, the method comprising: Receive the target task input by the user, and use the target large language model to parse the target task to obtain one or more first subtasks; For each of the first subtasks, a feasibility predictor processes the first subtask and the first current screenshot of the virtualized mobile device to obtain a feasibility result for the first subtask; wherein, the feasibility predictor is used to predict the feasibility of the first subtask. If the feasibility result indicates that the feasibility is successful, the first subtask is executed by the executor to obtain a screenshot after the first execution. The integrity verifier processes the expected screenshot of the first subtask and the screenshot after execution to obtain an integrity result for the first subtask; wherein the integrity verifier is used to evaluate the integrity of the first subtask.

[0010] In the above scheme, the step of processing the current screenshot of the first subtask and the virtualized mobile device through a feasibility predictor to obtain a feasibility result for the first subtask includes: The current screenshot is used to extract features by the visual encoder in the feasibility predictor to obtain the image features of the current screenshot; The semantic information of the first subtask is obtained by performing semantic parsing on the first subtask through the language decoder in the feasibility predictor. Based on the image features and the semantic information, the feasibility result is determined.

[0011] In the above scheme, the process of using an integrity verifier to process the expected screenshot of the first subtask and the screenshot after execution to obtain an integrity result for the first subtask includes: The integrity verifier determines the degree of matching between the expected screenshot and the screenshot after the first execution. The integrity result is determined based on the degree of matching.

[0012] In the above scheme, after processing the expected screenshot of the first subtask and the first executed screenshot using the integrity verifier to obtain the integrity result for the first subtask, the scheme further includes: If there is a discrepancy between the expected screenshot and the first executed screenshot, the target large language model is used to parse the discrepancy and the target task to obtain one or more second subtasks; For each of the second subtasks, the feasibility predictor processes the second subtask and the second current screenshot of the virtualized mobile device to obtain the feasibility result for the second subtask. If the feasibility result indicates that the feasibility is successful, the second subtask is executed by the executor to obtain a screenshot of the second execution.

[0013] In the above scheme, the information processing method further includes: Sensitive information in the target task is identified using a target recognition model, and placeholders corresponding to the sensitive information are determined. The placeholder and the sensitive information are stored as a storage pair in offline memory; In the above scheme, the step of executing the first subtask through an executor to obtain a screenshot after the first execution includes: Based on the storage pair, the target placeholder in the first subtask is restored to the target sensitive information to obtain the restored subtask; The executor executes the restored subtask to obtain the first screenshot after execution.

[0014] This application provides an information determination device, including: A conversion unit is used to determine the target hypertext markup language text representation corresponding to the multimodal user interface of the screen information of the virtualized mobile device; wherein the screen information is generated based on the user's input on the virtualized mobile device; The processing unit is used to generate thought chain prompts and tasks to be executed corresponding to the text representation based on the text representation and the large language model; wherein, the thought chain prompts are used to elicit the logical reasoning process in the large language model, so that the large language model can clarify its thought process before arriving at the final answer; The generation unit is used to execute the task to be executed based on the thought chain prompts through the executor, and obtain the operation result corresponding to the task to be executed.

[0015] This application provides an information processing apparatus, including: The parsing unit is used to receive the target task input by the user and parse the target task using the target large language model to obtain one or more first subtasks; A feasibility analysis unit is used to process the first sub-task and the first current screenshot of the virtualized mobile device through a feasibility predictor for each first sub-task to obtain a feasibility result for the first sub-task; wherein, the feasibility predictor is used to predict the feasibility of the first sub-task. An execution unit is used to execute the first subtask through an executor when the feasibility result characterizes the feasibility as passed, and to obtain a screenshot after the first execution. The integrity analysis unit is used to process the expected screenshot of the first subtask and the screenshot after execution using an integrity verifier to obtain an integrity result for the first subtask; wherein the integrity verifier is used to evaluate the integrity of the first subtask.

[0016] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0017] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0018] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the method provided in this application.

[0019] The embodiments of this application have the following beneficial effects: By converting the multimodal user interface of the virtualized mobile device into a text representation of Hypertext Markup Language, the large language model can overcome the perception barrier of multimodal data, accurately understand the interface structure and element semantics, and effectively solve the technical problem that the large language model cannot directly process mobile interface data; furthermore, by generating thought chain prompts, the large language model is guided to output a clear logical reasoning process before arriving at the final answer, making the decision path of complex tasks explicit, significantly improving the accuracy and interpretability of task execution; finally, based on the thought chain prompts, the executor executes the task to be executed, realizing a natural and efficient conversational interaction between the user and the virtualized mobile device. Therefore, this method greatly simplifies user operation, improves the intelligence level of task execution, and has good scalability, which can flexibly adapt to various task scenarios, providing virtualized mobile devices with highly reliable and highly interpretable automated task execution capabilities. Attached Figure Description

[0020] Figure 1 This is a first flowchart illustrating the information determination method provided in the embodiments of this application; Figure 2 These are schematic diagrams illustrating different dialogue scenarios in the information determination method provided in the embodiments of this application; Figure 3 This is a second flowchart illustrating the information processing method provided in the embodiments of this application; Figure 4 This is a third flowchart illustrating the information determination method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the fourth process in the information determination method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the fifth process of the information processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the feasibility predictor processing procedure in the information processing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the sixth process in the information processing method provided in the embodiments of this application; Figure 9 This is a schematic diagram of module interactions in the information determination method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the information determination device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the information processing device provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of the device provided in the embodiments of this application; Figure 13 This is a schematic diagram of the structure of the information processing device provided in the embodiments of this application.

[0021] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It's important to note that current digital infrastructure is rapidly migrating to cloud-native architectures, forming deep technological synergy with generative artificial intelligence. In the cloud phone field, virtualization technology based on distributed ARM architecture has achieved commercial-grade deployment, supporting the core capabilities of mobile application cloudification through containerized orchestration and lightweight sandbox isolation. Simultaneously, the intelligent interaction paradigm driven by Large Language Models (LLM) is propelling task automation into a new cognitive enhancement stage. It reconstructs the human-machine collaboration interface through natural language understanding, forming a closed-loop link from perception to decision-making. The convergence of these two technological tracks marks a crucial leap for intelligent cloud services from resource virtualization to cognitive intelligence. In terms of technological ecosystem infrastructure, mainstream cloud platforms have completed the full-stack reconstruction of mobile application runtime environments, establishing technical benchmarks in fundamental capabilities such as real-time streaming, low-latency interaction, and multi-instance concurrency, providing a stable platform for the deployment of cloud-based intelligent agents. Breakthroughs in technologies such as visual-language joint representation learning and tool-invoking enhanced reasoning enable LLM to possess the potential for cross-modal environment understanding and operational planning, injecting a cognitive dimension into cloud-based task automation. The combination of a cloud-edge-device collaborative architecture and dedicated AI acceleration hardware provides computing power support for the collaborative execution of complex model inference and real-time control commands. Cutting-edge research focuses on two main dimensions: first, constructing a cloud-edge-device collaborative intelligent execution framework to optimize real-time response capabilities through dynamic resource scheduling; second, exploring a fusion architecture of Mixture of Experts (MoE) and reinforcement learning to enhance complex task planning capabilities. However, existing solutions have not effectively addressed core issues such as multimodal representation alignment and domain knowledge transfer in open environments, which is precisely the breakthrough direction of this technical solution. Existing solutions are as follows: Mobile application demonstrations are used to program IoT devices. This solution uses mobile applications to demonstrate programming IoT devices, overcoming this limitation by leveraging smartphones as the hub for IoT automation. It directly manipulates the corresponding smartphone application for each IoT device to demonstrate the desired behavior, enabling the creation of automations for most consumer-grade IoT devices on smartphones. The system also supports using smartphone applications with context and external web services as triggers and data for automation, enabling the creation of highly context-aware IoT applications. However, this solution has the following drawbacks: 1) Lack of scalability: This solution relies on manual task setup and demonstration, making it difficult to effectively scale to complex or large-scale IoT systems. Each IoT device requires a separate smartphone application to demonstrate the task, making it difficult to automate a large number of devices or diverse tasks. 2) Insufficient task understanding: Because the system is based on simple task demonstrations, the LLM may not have a sufficient understanding of the task's complexity and environmental conditions, failing to accurately issue appropriate instructions to the IoT devices. This lack of deep understanding may lead to incorrect or inefficient operations by the IoT devices. In summary, the relevant solutions suffer from insufficient task scalability, insufficient reliability, and a lack of flexibility in the execution process during cloud phone task automation.

[0024] Based on this, embodiments of this application provide an information determination method, referring to... Figure 1 As shown, an information determination device applied to electronic devices may specifically include the following steps: Step 101: Determine the text representation of the multimodal user interface of the virtualized mobile device's screen information in the corresponding target Hypertext Markup Language.

[0025] The screen information is generated based on the user's input on the virtualized mobile device.

[0026] In this embodiment, a virtualized mobile device refers to a mobile device instance virtualized through cloud technology, which can be remotely operated by the user via the network, and its operating environment is built based on containerization or virtualization technology; screen information refers to the data of the currently displayed interface of the virtualized mobile device, including the view hierarchy (UI tree) and screenshots; multimodal user interface refers to the display interface that simultaneously contains multiple information forms such as text, images, and layout structure, and belongs to multimodal data; the target hypertext markup language text representation refers to the complete HTML text form in which attribute information and indexes are added to the text description of the initial Hypertext Markup Language (HTML) format user interface obtained after directly converting the multimodal user interface.

[0027] In this embodiment, since existing large language models can only accept text input, while user interfaces are multimodal, their view hierarchy and screenshots contain text, images, and structural information, etc. Furthermore, directly inputting the view hierarchy into the large language model is not feasible because it contains too much information, such as the detailed attributes of each user interface element. This information can easily exceed the input length limit of the large language model. Therefore, it is necessary to first convert the multimodal user interface into a text representation. Moreover, since HTML is already a markup language for representing web user interfaces (Web UI), HTML is suitable for representing multimodal user interface elements. Specifically, the view hierarchy tree of the virtualized mobile device system (such as the Android system) can be obtained first. Each node in the view hierarchy tree (i.e., each node corresponds to a user interface element) can be traversed through a depth-first search, and each node can be converted into an HTML text representation with corresponding functions. Then, attribute information and indexes are added to each user interface element, and finally a structured and complete hypertext markup language text representation is output for the LLM to understand the current interface content. In this way, the problem that large language models cannot directly process multimodal user interface data is solved. In one feasible implementation, the virtualized mobile device can specifically be a cloud phone.

[0028] Step 102: Based on text representation and large language model, generate thought chain prompts and tasks to be performed corresponding to the text representation.

[0029] Among them, the thought chain prompt is used to elicit the logical reasoning process in the large language model, so that the large language model can explain its thought process before arriving at the final answer.

[0030] In this embodiment, a large language model refers to a pre-trained model with natural language understanding and generation capabilities, such as a Generative Pre-trained Transformer (GPT) or a Pathways Language Model (PaLM), used to understand user interface (UI) text and generate task reasoning processes. Thought chain hints are a hinting engineering technique that guides the LLM to output a progressive logical reasoning process before generating the final answer by inserting intermediate reasoning steps into the input, thereby enhancing the model's reasoning ability and interpretability. Before processing the text representation using a large language model, the LLM needs to be pre-trained. Specifically, this can be done by predicting subsequent labels with the maximum probability based on training data. The pre-training data comes from natural language and code on the internet, including 24 common programming languages ​​such as Java, HTML, and Python. Then, the large language model is used to process the text representation, and intermediate logical results are obtained during the processing. Thought chain hints are determined based on multiple intermediate logical results. In this way, by generating thought chain hints corresponding to the text representation, the accuracy and interpretability of executing the task can be improved, making it easier for users to understand the model's behavior.

[0031] In this embodiment of the application, the HTML text representation first needs to be encapsulated according to the predefined "task prompt structure" and then input into the large language model. After receiving the above prompt structure, the large language model will follow the pattern in the example in the task prompt structure, first output the thought chain prompt, that is, before the final answer is obtained, a text describing the intermediate reasoning process is generated. After generating the thought chain prompt, the large language model outputs the specific task to be executed according to the thought chain prompt.

[0032] Step 103: Using the executor, execute the task to be executed based on the thought chain prompts, and obtain the operation result corresponding to the task to be executed.

[0033] In this embodiment, the executor can refer to a model in a specific domain. The executor can input thought chain prompts and tasks to be executed into the executor. The executor executes the tasks to be executed according to the thought chain prompts, thereby obtaining the operation results corresponding to the tasks to be executed. In this way, end-to-end conversational task automation is realized, and users can control virtualized mobile devices through natural language.

[0034] It should be noted that user interactions with virtualized mobile devices (such as cloud phones) typically involve human users exchanging information with conversational intelligent assistants. (See reference...) Figure 2As shown, this paper categorizes different dialogue scenarios when users and conversational intelligent assistants interact on the user interface (UI) of a cloud phone to complete tasks. Specifically, the categorization has two axes: initiative and purpose. Initiative refers to the agent (i.e., large language model + executor) actively soliciting new information from the user or providing information to the user, and vice versa. This application provides an example task for each category, where the agent's operations represent the target output of the LLM. In this process, by converting UI elements into parsable text, the LLM can better understand and manipulate complex UI structures, thereby improving the accuracy and efficiency of task execution. Through flexible application in different dialogue scenarios, this method can automatically identify the user's intent and dynamically generate execution strategies adapted to the current task environment, thus achieving more intelligent and efficient task automation.

[0035] The information determination method provided in this application converts the multimodal user interface of a virtualized mobile device into a text representation using Hypertext Markup Language (HTML). This enables a large language model to overcome the perceptual barriers of multimodal data, accurately understand the interface structure and element semantics, and effectively solves the technical problem that large language models cannot directly process mobile interface data. Furthermore, by generating thought chain prompts, the method guides the large language model to output a clear logical reasoning process before arriving at the final answer, making the decision-making path of complex tasks explicit and significantly improving the accuracy and interpretability of task execution. Finally, based on the thought chain prompts, the executor executes the task to be performed, realizing a natural and efficient conversational interaction between the user and the virtualized mobile device. Therefore, this method greatly simplifies user operations, improves the intelligence level of task execution, and has good scalability, flexibly adapting to various task scenarios, providing virtualized mobile devices with highly reliable and highly interpretable automated task execution capabilities.

[0036] It should be noted that current task automation technologies based on large language models generally face a disconnect between task understanding and execution when handling user instructions in complex and open environments. Existing solutions fall into two categories: one relies on manual demonstrations or pre-set scripts, which struggle to dynamically adapt to interface changes and lack the ability to predict operational feasibility, easily generating invalid instructions; the other connects to application programming interfaces (APIs) through a basic model to execute tasks, but is limited by incomplete interface descriptions and insufficient environmental awareness, often resulting in mismatches between instructions and the actual interface state, and deviations from expected goals during execution, leading to low task execution success rates and poor reliability. In virtualized mobile device scenarios such as cloud phones, these problems are particularly pronounced due to the diverse task types and dynamic interface changes, necessitating an automated method that can verify feasibility before task execution and validate results after execution to improve the accuracy and reliability of task execution.

[0037] Based on this, embodiments of this application provide an information processing method, referring to... Figure 3 As shown, an information processing device applied in electronic devices may specifically include the following steps: Step 201: Receive the target task input by the user and parse the target task using the target large language model to obtain one or more first subtasks.

[0038] In this embodiment, the target task refers to a vague or complex high-level command issued by the user through natural language or other interactive methods. The target large language model is a trained large language model obtained by training an initial large language model with sample data. It is used to decompose the user-input target task into a machine-executable, logically coherent sequence of operation steps (i.e., the first subtask). The first subtask refers to the operation steps generated by the target large language model after parsing the target task. After receiving the user-input target task, the target large language model is invoked for semantic understanding and task decomposition. Then, the target large language model is used to decompose the complex target task into a series of sequentially executed subtasks (i.e., the first subtask). In this way, 1) complex tasks can be automatically decomposed, reducing the user's operational threshold; 2) the powerful semantic understanding capability of the target large language model is utilized to improve the accuracy and flexibility of task parsing, adapting to diverse user inputs.

[0039] In one feasible implementation, when the target task is "to open WeChat on a virtualized mobile device (such as a cloud phone) and send a message to Zhang San", the multiple first subtasks can specifically be a sequence of operation steps such as "clicking the WeChat icon", "finding Zhang San in the contacts", and "entering the message content". When the target task is "to open application A, search for phone cases and add them to the shopping cart", the multiple first subtasks can specifically be a sequence of operation steps such as "clicking the icon of application A", "entering phone case in the search box", "clicking the search button", "clicking the first item", and "clicking to add to the shopping cart".

[0040] Step 202: For each first subtask, the feasibility predictor processes the first subtask and the first current screenshot of the virtualized mobile device to obtain the feasibility result for the first subtask.

[0041] The feasibility predictor is used to predict the feasibility of the first subtask.

[0042] In this embodiment, the feasibility predictor is a multimodal framework component based on a domain-specific model, consisting of a visual encoder and a language decoder. Its specific function is to predict whether a subtask is feasible in the current interface before execution (e.g., whether a button is clickable or whether an element exists). The first current screenshot refers to the real-time interface image of the virtualized mobile device at the current moment, which is used to reflect the current environmental state of the virtualized mobile device. The feasibility result refers to the judgment result output by the feasibility predictor, which is used to indicate whether the first subtask can be successfully executed in the current interface, and the feasibility result specifically includes feasible and infeasible.

[0043] In this embodiment, before executing each first subtask, execution is not done blindly. Instead, a feasibility predictor is invoked for pre-execution checks. Specifically, the first subtask and the current screenshot are input into the feasibility predictor. The visual encoder in the feasibility predictor processes the current screenshot, and the language decoder in the feasibility predictor processes the first subtask. The processing results are then used to comprehensively determine whether the subtask is feasible in the current interface. If it is not feasible, the result is "failure," triggering replanning; if it is feasible, the execution of the first subtask begins. This not only avoids the target large language model generating invalid instructions, improving the success rate of task execution, but also allows for pre-execution intervention, reducing the impact of erroneous operations on the system state.

[0044] Step 203: If the feasibility result characterization is successful, execute the first subtask through the executor to obtain a screenshot after the first execution.

[0045] In this embodiment, the executor is a component responsible for actually performing specific operations in the virtualized mobile device environment, such as simulating clicks, swipes, and text input; the first screenshot after execution refers to the screenshot of the new state of the virtualized mobile device interface after the first subtask is completed. When the result output by the feasibility predictor indicates that the current first subtask can be executed, the executor simulates user operations in the virtualized mobile device instance, and after the execution is completed, captures the screenshot after execution (i.e., the first screenshot after execution).

[0046] Step 204: Process the expected screenshot of the first subtask and the screenshot after the first execution using the integrity verifier to obtain the integrity result for the first subtask.

[0047] The integrity verifier is used to evaluate the integrity of the first subtask.

[0048] In this embodiment, the integrity verifier is a component used to evaluate whether the execution result achieves the expected goal. It determines whether the operation is complete and correct by comparing the expected screenshot with the actual screenshot after execution (i.e., the first screenshot after execution). The expected screenshot is the interface state after execution estimated according to the task logic (e.g., the "message has been sent" prompt expected to appear after clicking the "send" button). The integrity result refers to the judgment result output by the integrity verifier, which is used to indicate whether the first subtask has been executed correctly and completely, and the integrity result includes completeness and incompleteness.

[0049] In this embodiment, after executing a first subtask in step 203, success is not assumed. Instead, a completeness verifier is invoked for post-execution checks. Specifically, for each first subtask, the expected screenshot and the actual screenshot after execution can be input into the completeness verifier. The completeness verifier then compares the expected screenshot with the actual screenshot after execution to determine the degree of matching between the two, thereby evaluating whether the first subtask has achieved the expected effect during planning. If the degree of matching is high, i.e., there is no deviation, the completeness result will indicate completeness. If the degree of matching is low, i.e., there is a deviation, the completeness result will indicate incompleteness. At this time, a self-correction mechanism is triggered, feeding the deviation back to the target large language model. The target large language model then performs replanning or compensation operations based on the target task and the deviation. In this way, the reliability of task execution can be improved, ensuring that each step of the operation meets expectations.

[0050] The information processing method provided in this application uses a target large language model to parse the user-input target task into executable subtasks. Before execution, a feasibility predictor, as an end-to-end multimodal model, uses the current screenshot and the subtask to be executed as input to accurately predict the feasibility of the operation, effectively filter invalid instructions, and avoid negative impacts on the system state caused by erroneous operations. After execution, an integrity verifier compares the expected screenshot with the actual execution result to accurately evaluate the task completion status. Therefore, this application significantly improves the success rate and robustness of automated complex tasks. Through the dual guarantee mechanism of feasibility prediction before execution and integrity verification after execution, it effectively reduces the risk of execution failure caused by dynamic changes in the interface or instruction deviations, and improves the reliability and security of task processing.

[0051] Based on the foregoing embodiments, this application provides yet another information determination method, referring to... Figure 4 As shown, an information determination device applied to electronic devices may specifically include the following steps: Step 301: Obtain the view hierarchy in the multimodal user interface and represent the view hierarchy as a view hierarchy tree.

[0052] In this embodiment, the view hierarchy refers to the tree-like structure representation of user interface elements (UI) in a virtualized mobile device system (such as Android). Each node corresponds to a user interface element, including various attribute information of that user interface element, such as class, visibility to the user, and the boundaries of the user interface element. The view hierarchy tree is an abstract representation of the view hierarchy in the form of a tree. The root node represents the entire screen of the virtualized mobile device, and the child nodes represent nested user interface components, facilitating subsequent traversal and transformation. The view hierarchy can be accessed from the current user interface of the virtualized mobile device (see reference 1) through Android's View system or Accessibility service. Figure 5 The view hierarchy is extracted from the diagram and organized into a tree structure (i.e., a view hierarchy tree). This tree fully reflects the hierarchical relationships and attribute information of all user interface components on the current screen, providing a structured data foundation for subsequent traversal and transformation. In this way, the complex, unstructured view hierarchy is transformed into a clearly structured tree representation, facilitating algorithmic processing.

[0053] Step 302: Based on depth-first search, traverse the user interface elements corresponding to each node in the view hierarchy tree to determine the text representation of the Hypertext Markup Language to be processed for each user interface element.

[0054] In this embodiment, Depth-First Search (DFS) is a traversal algorithm that prioritizes exploring child nodes in the view hierarchy tree to ensure that the nesting relationship of user interface elements is reflected in the converted text. A user interface element refers to each node in the view hierarchy tree, representing a specific user interface component, such as a button, text box, or image. The text representation of the Hypertext Markup Language (HTML) to be processed refers to the HTML tags generated for each user interface element after conversion, with added attribute information to describe the user interface element in the text representation. The initial HTML representation corresponding to each user interface element can be determined first, and then attribute information can be added to the initial HTML tags to ultimately obtain the HTML representation of the user interface element corresponding to the HTML to be processed.

[0055] It should be noted that step 302 can be achieved in the following way: Step 302A1: Traverse the view hierarchy tree using a depth-first search and convert the user interface element corresponding to each node in the view hierarchy tree into a text representation of the initial hypertext markup language.

[0056] In this embodiment of the application, the initial hypertext markup language text representation refers to the basic HTML text representation obtained by directly converting the user interface element type before adding attribute information; during the process of traversing the view hierarchy tree using depth-first search, each user interface element can be converted into an initial HTML text representation with corresponding functions according to the Android class type of each user interface element through class mapping rules.

[0057] Step 302A2: For each initial hypertext markup language text representation, determine the target attribute set of the user interface element corresponding to the initial hypertext markup language text representation.

[0058] The target attribute set includes class attributes, text attributes, resource identifier attributes, and content description attributes.

[0059] In this embodiment, the target attribute set refers to the key attributes extracted from user interface elements, used to describe the characteristics of the user interface element (i.e., a subset of attributes related to the text description of the user interface element). This target attribute set includes: class: Android object type; text: user-visible element text content; resource_id: text identifier referencing the resource; content_desc: content description (alternative text, used for accessibility). To avoid HTML text exceeding the LLM input length limit (e.g., 2048 tokens for GPT-3; 1920 tokens for PaLM), the attribute information of each user interface element in the view hierarchy can be filtered, retaining only the subset of attributes related to the text description of the user interface element (i.e., the target attribute set), instead of all attributes. This achieves two goals: 1) streamlines input information, avoiding exceeding the LLM token limit; 2) retains the most critical semantic attribute information, ensuring that the LLM can accurately understand the meaning and function of each user interface element.

[0060] Step 302A3: Write the target attribute set into the initial hypertext markup language text representation to obtain the hypertext markup language text representation to be processed.

[0061] In this embodiment of the application, the text representation of the Hypertext Markup Language to be processed refers to the final generated HTML tags containing complete attribute information, such as... <button text=""发送”class="Button”resource_id="btn_send”">The target attribute set can be written into the initial Hypertext Markup Language (HTML) tag as HTML attributes, forming a text representation of HTML with attribute information; in this way, by generating a structurally complete and semantically rich HTML text representation, LLM can accurately identify and manipulate user interface elements.

[0062] Step 303: For each text representation of the Hypertext Markup Language to be processed, determine the index corresponding to the text representation of the Hypertext Markup Language to be processed, and insert the index into the text representation of the Hypertext Markup Language to be processed to obtain the text representation of the target Hypertext Markup Language.

[0063] The index is generated according to the depth-first search order in the view hierarchy tree.

[0064] In this embodiment, the index is a unique identifier assigned to each user interface element, specifically a numeric identifier, and is generated according to the depth-first search (DFS) order in the view hierarchy tree. After completing the class mapping and generating HTML tags with attributes, an index can be assigned to each tag, and the generation of the index follows the depth-first search order in the view hierarchy tree: starting from the root node, for each node visited, the index is incremented in the order of visit, and this number is written as an identifier attribute (i.e., the id attribute) into the corresponding HTML tag, for example, 、 、 < / button> <button id="3">, obtain reference Figure 5 The complete target Hypertext Markup Language text representation is shown; thus, 1) in subsequent task execution, LLM can directly reference a user interface element by index (e.g., clicking the button with index 3) without outputting lengthy HTML tag descriptions, reducing the complexity of instruction generation; 2) index referencing avoids element location errors caused by duplicate or similar tag attributes, improving operational accuracy. Step 304: Based on class mapping rules, map the Android user interface component classes of the virtualized mobile device to the text representations of the target Hypertext Markup Language with corresponding functions.

[0065] Among them, class mapping rules are used to indicate the mapping between Android user interface component classes and Hypertext Markup Language.

[0066] In this embodiment, the class mapping rule refers to the conversion rule that maps native user interface component classes (such as TextView, Button, etc.) in the Android system to HTML tags with similar functional semantics; Android user interface component classes include text view classes (TextView), button-related classes (Button, ImageButton), image-related classes (ImageView), text input classes (EditText), and other classes (such as container classes like LinearLayout); the text representation of the target hypertext markup language includes paragraph tags ( ), button labels ( < / button> <button>Image tags ), Input tags ( <input> ) and partition labels ( Each class in the Android user interface component class can be mapped to a text representation of a target Hypertext Markup Language (HTML) with similar functionality to that class, according to class mapping rules. In this way, native user interface types in the Android system can be mapped to HTML tags already learned in the LLM pre-training, enabling the LLM to leverage its existing web page understanding capabilities.

[0067] It should be noted that step 304 can be achieved in the following way: Step 304B1: Based on the class mapping rules, map the text view class to the paragraph label, the button-related class to the button label, the image-related class to the image label, and the text input class to the input label. Also, map all other classes in the Android user interface component classes except for the text view class, the button-related class, the image-related class, and the text input class to the partition label.

[0068] The Android user interface component classes include text view classes, button-related classes, image-related classes, text input classes, and other classes; the target hypertext markup language text representations include paragraph tags, button tags, image tags, input tags, and section tags.

[0069] In this embodiment of the application, since the TextView class is used to display text content, and paragraph labels ( It is also used to display text content, i.e., TextView and If it has similar functionality, then you can map the TextView to... (i.e., TextView→) ); because the classes associated with buttons (Button, ImageButton) represent clickable button elements, and the button label ( < / button> <button> () also represents clickable button elements, namely Button, ImageButton, and...< / button> <button> If they have similar functionality, then the classes associated with the buttons can be mapped to...< / button> <button> (i.e., Button, ImageButton →)< / button> <button>); because the ImageView class is used to display images, and image tags ( It is also used to display images, i.e., ImageView and If it has similar functionality, then you can map an ImageView to... (i.e., ImageView → ); because the text input class (EditText) is used to receive user text input, and the input label ( <input> It is also used to receive user text input, i.e., EditText and <input> If it has similar functionality, then EditText can be mapped to... <input> (i.e., EditText→) <input> For Android user interface component classes other than those mentioned above, such as container classes (LinearLayout, RelativeLayout), they are all mapped to partition labels. (i.e., other classes →) ).

[0070] Step 305: Process the text representation using a large language model to obtain intermediate inference results.

[0071] In this embodiment, intermediate inference results refer to the staged inference outputs generated by the LLM when processing input, such as "The current interface contains a send button (id=3), the user requests to send a message, therefore the button needs to be clicked first." These intermediate results are not the final answer, but rather components of the complete inference chain. (See also...) Figure 5 As shown, the target Hypertext Markup Language text representation can be used as input and passed to the LLM for processing. Based on its HTML understanding and common sense knowledge acquired during the pre-training phase, the LLM analyzes the current interface and outputs a series of intermediate inference results. These intermediate inference results typically include: interface element recognition, matching analysis between the task target and the interface state, and logical judgment of the next operation. In this way, the black-box output of the LLM can be transformed into traceable inference steps, enhancing the transparency of the task execution process.

[0072] Step 306: Based on multiple intermediate reasoning results, determine the thought chain prompts.

[0073] In this embodiment, multiple intermediate reasoning results can be organized and integrated to form a structured thought chain prompt. The thought chain prompt typically adopts the form of "intermediate step 1 → intermediate step 2 → … → final conclusion", clearly showing the complete reasoning path from the initial state to the final decision. This improves reasoning accuracy: the thought chain prompt forces LLM to reason step by step, reducing jump errors and increasing the success rate of complex tasks.

[0074] It should be noted that generating and linking intermediate results to obtain the final answer, i.e., thought chain hints, can improve the reasoning ability of LLM. This is because by attaching a thought chain describing the intermediate result before the answer in the hint, the model will then follow the pattern to generate thought chains during the reasoning process, and the result is also more interpretable because the model explains its thought process before arriving at the answer.

[0075] Step 307: Generate tasks to be executed based on the task prompt structure and mind chain prompts.

[0076] The task prompt structure includes a preface, multiple examples, and a test screen. The preface explains the purpose of the prompt, each example includes input, thought chain, and output, and the test screen is the screen input to be processed.

[0077] In this embodiment, the HTML text representation can be encapsulated according to a predefined "task prompt structure," which specifically includes the following three parts: 1) Introduction: A text is added at the beginning of the prompt to explain the purpose and task background of this interaction to the LLM; 2) Example (few-shot learning): After the introduction, one or more complete "input-thought chain-output" examples are provided, specifically including: Input: A mobile screen example represented in HTML syntax; Thought chain: Demonstrates how to deduce the intermediate reasoning steps of the execution logic from the screen information; Output: The target result of the corresponding example task (such as clicking a button, inputting text, etc.); 3) Current input screen: After the example, the screen information that is currently to be processed and has been converted to HTML syntax is appended as the actual input for this prediction; Then, the large language model outputs the specific task to be executed based on the encapsulated HTML text representation according to the thought chain prompt.

[0078] Step 308: Execute the task to be executed through the executor based on the thought chain prompts, and obtain the operation result corresponding to the task to be executed.

[0079] It should be noted that this application uses depth-first search to traverse the view hierarchy, converting user interface elements into HTML syntax to achieve textual representation of these elements. Then, a series of conversational tasks are executed using a large language model (LLM), enabling natural interaction between human users and a conversational intelligent assistant. This method combines machine learning and natural language processing techniques, allowing users to interact with virtualized mobile devices (such as cloud phones) through simple dialogue, automating complex tasks. Its core lies in using HTML syntax to represent user interface elements, enabling the LLM to accurately identify and manipulate these elements, thus efficiently executing tasks. The benefits of this method are significantly simplified user-device interaction, improved task execution intelligence, and enhanced accuracy and interpretability of task reasoning through thought chain prompts.

[0080] It should be noted that the descriptions of the same steps and contents as in other embodiments can be found in other embodiments and will not be repeated here.

[0081] The information determination method provided in this application converts the multimodal user interface of a virtualized mobile device into a text representation using Hypertext Markup Language (HTML). This enables a large language model to overcome the perceptual barriers of multimodal data, accurately understand the interface structure and element semantics, and effectively solves the technical problem that large language models cannot directly process mobile interface data. Furthermore, by generating thought chain prompts, the method guides the large language model to output a clear logical reasoning process before arriving at the final answer, making the decision-making path of complex tasks explicit and significantly improving the accuracy and interpretability of task execution. Finally, based on the thought chain prompts, the executor executes the task to be performed, realizing a natural and efficient conversational interaction between the user and the virtualized mobile device. Therefore, this method greatly simplifies user operations, improves the intelligence level of task execution, and has good scalability, flexibly adapting to various task scenarios, providing virtualized mobile devices with highly reliable and highly interpretable automated task execution capabilities.

[0082] Based on the foregoing embodiments, this application provides yet another information processing method, referring to... Figure 6 As shown, an information processing device applied in electronic devices may specifically include the following steps: Step 401: Receive the target task input by the user and parse the target task using the target large language model to obtain one or more first subtasks.

[0083] Step 402: For each first subtask, the visual encoder in the feasibility predictor extracts features from the first current screenshot to obtain the image features of the current screenshot.

[0084] The feasibility predictor is used to predict the feasibility of the first subtask.

[0085] In this embodiment, the feasibility predictor includes two sub-modules: a visual encoder and a language decoder. The visual encoder is responsible for extracting visual features from the screenshot. Image features refer to the abstract feature representation extracted by the visual encoder from the current screenshot, encoding the visual information (such as position, size, type, etc.) of user interface elements in the interface, and specifically can be image feature vectors. (See reference...) Figure 7 As shown, before executing each first subtask, a first current screenshot of the virtualized mobile device is first acquired and input into the visual encoder of the feasibility predictor. The visual encoder extracts features from the first current screenshot and outputs a high-dimensional image feature vector. This feature vector captures the visual distribution and layout information of the user interface elements in the interface, providing an environmental perception basis for subsequent feasibility judgment. In this way, by transforming visual information into structured features, the target large language model can perceive the current interface state.

[0086] Step 403: Semantically parse the first subtask using the language decoder in the feasibility predictor to obtain the semantic information of the first subtask.

[0087] In this embodiment, the language decoder is responsible for semantic parsing of the first subtask and converting the first subtask into a semantic representation that can be fused with other modal information. The semantic information refers to the semantic features extracted by the language decoder from the instructions of the first subtask, which encodes information such as the type of operation (such as click, input, swipe) and the description of the target element.

[0088] In the embodiments of this application, for each first subtask, refer to Figure 7 As shown, the current first subtask (such as "Log in to this application using account {account_name} and password {password}") can be input into the language decoder of the feasibility predictor. The language decoder performs semantic parsing on the first subtask, extracts the operation intention and target information, and outputs a semantic feature vector. This feature vector captures the core semantics of the instruction. In this way, not only is the natural language instruction transformed into structured semantic features, which is convenient for joint reasoning with visual information, but the operation intention (click, input, swipe, etc.) can also be accurately captured, providing instruction-side basis for feasibility judgment.

[0089] Step 404: Determine the feasibility result based on image features and semantic information.

[0090] In the embodiments of this application, reference is made to Figure 7 As shown, the obtained image features and semantic information can be fused (usually through splicing, attention mechanisms, or multimodal interaction), and input into a classifier or discriminator network to output a feasibility result, with feasibility represented by 1 and infeasibility by 0. For example, if the first subtask is "click the send button", but the "send button" does not exist in the current screenshot, the feasibility predictor outputs 0; if it exists and is clickable, it outputs 1. In this way, 1) invalid commands can be filtered in advance: infeasible operations are identified before execution to avoid negative impacts on the system state caused by incorrect execution; 2) the execution failure rate is reduced: through feasibility judgment, only commands that are confirmed to be feasible are executed, significantly improving the task execution success rate.

[0091] Step 405: If the feasibility result characterization is successful, identify sensitive information in the target task using the target identification model and determine the placeholders corresponding to the sensitive information.

[0092] In this embodiment, the target recognition model refers to a Named Entity Recognition (NER) model used to identify sensitive information, which can automatically identify sensitive information (i.e., privacy data) from the target task input by the user; sensitive information refers to privacy data that needs to be protected, such as the user's credit card number, account number, password, etc., which should not be uploaded to the target large language model in the cloud; placeholders are identifiers used to replace sensitive information during transmission, such as "{credit_card}", "{password}", etc.

[0093] In this embodiment of the application, before executing each first subtask by the executor, the target task input by the user is first scanned by the NER model to identify sensitive information. For each piece of sensitive information, a placeholder corresponding to that sensitive information is generated. For example, the placeholder corresponding to "1234-5678-9012-3456" is "{credit_card}". In this way, sensitive information can be identified at the source, preventing privacy data from being uploaded to the cloud.

[0094] Step 406: Store the placeholders and sensitive information as storage pairs in offline memory.

[0095] In this embodiment, the storage pair refers to a key-value pair structure, where the key is a placeholder and the value is sensitive information; offline memory refers to storage space deployed at the edge or on the user's local machine, isolated from the cloud. After determining the placeholder for each piece of sensitive information, each piece of sensitive information and its corresponding placeholder can be stored in offline memory in the form of a key-value pair; in this way, sensitive information is only stored locally and does not interact with the cloud, significantly reducing the risk of privacy leakage.

[0096] Step 407: Based on the storage pair, restore the target placeholder in the first subtask to the target sensitive information to obtain the restored subtask.

[0097] In this embodiment, the target placeholder refers to the placeholder appearing in the first subtask, such as "{credit_card} in 'enter {credit_card} on the payment page'". The restored subtask refers to the subtask after replacing the placeholder with sensitive information. For example, if the sensitive information corresponding to {credit_card} is "1234-5678-9012-3456", then the restored subtask could specifically be "enter 1234-5678-9012-3456 on the payment page". After the target large language model generates the first subtask containing the placeholder, the executor queries the offline storage pair before execution to restore the target placeholder in the first subtask to the corresponding target sensitive information. This conversion is only performed locally, and the cloud can never access the sensitive information. In this way, the sensitive information remains encrypted / placeholder during transmission and processing, and is only restored during local execution.

[0098] Step 408: Execute the restored subtask through the executor to obtain the first screenshot after execution.

[0099] In this embodiment of the application, the restored subtask (containing sensitive information) is handed over to the executor to be executed in the virtualized mobile device. Then the executor simulates user operations (such as inputting text, clicking buttons, etc.) to complete the task steps, and captures a screenshot of the screen after execution.

[0100] Step 409: Determine the degree of matching between the expected screenshot and the screenshot after the first execution using the integrity validator.

[0101] In this embodiment, the matching degree refers to the similarity measure between the expected screenshot and the screenshot after the first execution, which can be calculated based on pixel-level comparison, feature matching, semantic similarity, etc. After the first subtask is executed, the integrity verifier can be called to compare the first screenshot after the first execution captured after executing the first subtask with the expected screenshot to calculate the matching degree between the two.

[0102] Step 410: Determine the integrity result based on the degree of matching.

[0103] In this embodiment, the system determines the result as incomplete when the matching degree is less than a set threshold, and as complete when the matching degree is greater than or equal to the set threshold. The integrity verifier outputs the final integrity result based on the calculated matching degree. If the matching degree is high (e.g., greater than the preset threshold (90%)), it is determined as complete, and the next first subtask is executed. If the matching degree is low (i.e., it is considered that there is a deviation between the executed screenshot and the expected screenshot, and the expected result is not achieved), it is determined as incomplete, triggering a self-correction mechanism in the following steps.

[0104] It should be noted that, based on this embodiment, the following steps may also be included: Step 411: If there is a discrepancy between the expected screenshot and the screenshot after the first execution, the target large language model is used to analyze the discrepancy and the target task to obtain one or more second subtasks.

[0105] In this embodiment, deviation refers to the difference between the expected screenshot and the actual screenshot found by the integrity verifier during post-execution comparison. This deviation may stem from various reasons, such as interface loading delay, element position changes, network fluctuations, and application state anomalies. The second subtask refers to the subtask obtained by the target large language model re-decomposing the target task based on the deviation. (Refer to...) Figure 8 As shown, when the integrity verifier determines that there is a deviation between the expected screenshot and the screenshot after the second execution (i.e., the matching degree is less than the set threshold), a self-correction mechanism is triggered. At this time, the deviation information can be input into the target large language model along with the original target task for re-parsing. Based on the understanding of the current state and the deviation analysis, the target large language model generates a new sequence of operation steps (i.e., the second subtask). Then, the second subtask is processed through the feasibility prediction period, and if the feasibility is passed, the second subtask is executed by the executor. Then, the integrity verifier verifies the screenshot after the second execution and the expected screenshot corresponding to the second subtask to determine whether there is a deviation, thus forming a loop, enabling the coordinator in the target large language model to have the ability to self-correct. In this way, 1) dynamic adaptive capability: it can perceive the execution deviation and actively respond, rather than blindly continuing or directly failing; 2) intelligent error correction: it uses the reasoning ability of the target large language model to analyze the cause of the deviation and generate targeted correction operations, rather than simply retrying; 3) improved task success rate: through the error correction mechanism, even if unexpected situations are encountered, the task still has a high probability of being completed in the end.

[0106] Step 412: For each second subtask, the feasibility predictor processes the second subtask and the second current screenshot of the virtualized mobile device to obtain the feasibility result for the second subtask.

[0107] In this embodiment, the second current screenshot refers to a real-time screenshot of the virtualized mobile device before the correction operation is performed, and this second current screenshot is different from the first current screenshot when the first subtask was initially executed. After generating one or more second subtasks, they are not executed immediately, but the feasibility predictor is called again to perform pre-execution verification for each second subtask. Specifically, the second subtask and the second current screenshot can be input into the feasibility predictor to determine whether the second subtask is feasible under the current interface; in this way, the high reliability principle of pre-execution verification is maintained in the error correction stage.

[0108] Step 413: If the feasibility result characterization is successful, execute the second subtask through the executor to obtain a screenshot after the second execution.

[0109] In this embodiment, the second post-execution screenshot refers to a screenshot of the new state of the virtualized mobile device's interface after the second subtask is executed. This second post-execution screenshot can re-enter the integrity verification loop, forming a closed-loop continuous error correction. When the feasibility predictor confirms the feasibility of the second subtask, the executor executes the second subtask and captures the post-execution screenshot (i.e., the second post-execution screenshot). Then, the second post-execution screenshot is evaluated by the integrity verifier to determine whether the correction operation has successfully corrected the deviation. If there is still a deviation, replanning can be triggered again (step 408), forming a multi-round error correction loop until the task is completed or the preset retry limit is reached, thereby forming a closed-loop continuous error correction of "execution → verification → deviation → replanning → execution → reverification". In this way, multi-round error correction can be supported, which can cope with uncertainties and abnormal situations in complex environments.

[0110] It should be noted that this application employs a multimodal framework to train a domain-specific model (proximal policy optimization algorithm) to assist the target large language model in optimizing the feasibility, completeness, and security of task execution. Specifically, the method first evaluates the generated sub-tasks using a feasibility predictor to ensure their practical feasibility. Next, an integrity verifier progressively checks the results of sub-task execution and provides timely feedback to the coordinator of the target large language model to ensure correct task execution. Furthermore, a security protector utilizes local memory mechanisms to protect users' sensitive information and prevent privacy leaks. This provides an efficient, secure, and self-correcting solution for the automated execution of complex tasks, ensuring highly reliable task execution across various task scenarios.

[0111] It should be noted that the information processing methods and information determination methods provided in this application both include references to... Figure 9 The system is structured into four layers: the basic business layer, the core business layer, the presentation layer, and the user layer. The basic business layer provides the underlying infrastructure and services for the entire system, offering resource, network, and computing power support to the core business layer and upper-layer applications; it forms the foundation of the entire system. The core business layer is the central hub of the system, responsible for resource management and scheduling to ensure efficient business operations. This primarily includes resource management, resource operation, and unified monitoring. The presentation layer provides users with a front-end interactive interface and supports Software Development Kits (SDKs) for mobile devices and personal computers (PCs), ensuring users can interact with the system through various terminal devices. The user layer provides targeted, scenario-based solutions for different user groups (such as mobile office users and game studio users) to meet their actual needs.

[0112] It should be noted that the descriptions of the same steps and contents as in other embodiments can be found in other embodiments and will not be repeated here.

[0113] The information processing method provided in this application uses a target large language model to parse the user-input target task into executable subtasks. Before execution, a feasibility predictor, as an end-to-end multimodal model, uses the current screenshot and the subtask to be executed as input to accurately predict the feasibility of the operation, effectively filter invalid instructions, and avoid negative impacts on the system state caused by erroneous operations. After execution, an integrity verifier compares the expected screenshot with the actual execution result to accurately evaluate the task completion status. Therefore, this application significantly improves the success rate and robustness of automated complex tasks. Through the dual guarantee mechanism of feasibility prediction before execution and integrity verification after execution, it effectively reduces the risk of execution failure caused by dynamic changes in the interface or instruction deviations, and improves the reliability and security of task processing.

[0114] Based on the foregoing embodiments, this application provides an information determining device, which can be applied to... Figure 1 and Figure 4 In the information determination method provided in the corresponding embodiment, refer to Figure 10 As shown, the information determining device 5 may include: a conversion unit 51, a processing unit 52, and a generation unit 53, wherein: The conversion unit 51 is used to determine the target hypertext markup language text representation corresponding to the multimodal user interface of the screen information of the virtualized mobile device; wherein the screen information is generated based on the user's input on the virtualized mobile device; Processing unit 52 is used to generate thought chain prompts and tasks to be executed corresponding to the text representation based on the text representation and the large language model; wherein, the thought chain prompts are used to elicit the logical reasoning process in the large language model, so that the large language model can clarify its thought process before arriving at the final answer. The generation unit 53 is used to execute the task to be executed based on the thought chain prompts through the executor and obtain the operation result corresponding to the task to be executed.

[0115] In other embodiments of this application, the conversion unit 51 is further configured to perform the following steps: Obtain the view hierarchy in the multimodal user interface and represent the view hierarchy as a view hierarchy tree; Based on a depth-first search, the user interface elements corresponding to each node in the view hierarchy tree are traversed to determine the text representation of the Hypertext Markup Language to be processed for each user interface element. For each text representation of the Hypertext Markup Language to be processed, the index corresponding to the text representation of the Hypertext Markup Language to be processed is determined, and the index is inserted into the text representation of the Hypertext Markup Language to be processed to obtain the text representation of the target Hypertext Markup Language; wherein, the index is generated according to the depth-first search order in the view hierarchy tree; Based on class mapping rules, Android user interface component classes of virtualized mobile devices are mapped to text representations of target Hypertext Markup Language with corresponding functions; wherein, class mapping rules are used to indicate the mapping between Android user interface component classes and Hypertext Markup Language.

[0116] In other embodiments of this application, the conversion unit 51 is further configured to perform the following steps: By traversing the view hierarchy tree using a depth-first search, the user interface element corresponding to each node in the view hierarchy tree is converted into a text representation of the initial Hypertext Markup Language. For each initial hypertext markup language text representation, determine the target attribute set of the user interface element corresponding to the initial hypertext markup language text representation; wherein, the target attribute set includes class attributes, text attributes, resource identifier attributes, and content description attributes; Write the target attribute set into the initial hypertext markup language text representation to obtain the hypertext markup language text representation to be processed.

[0117] In other embodiments of this application, the conversion unit 51 is further configured to perform the following steps: Based on class mapping rules, text view classes are mapped to paragraph tags, button-related classes are mapped to button tags, image-related classes are mapped to image tags, and text input classes are mapped to input tags. All other classes in the Android user interface component classes, except for text view classes, button-related classes, image-related classes, and text input classes, are mapped to partition tags. The Android user interface component classes include text view classes, button-related classes, image-related classes, text input classes, and other classes. The text representation in the target hypertext markup language includes paragraph tags, button tags, image tags, input tags, and partition tags.

[0118] In other embodiments of this application, the processing unit 52 is further configured to perform the following steps: A large language model is used to process the text representation to obtain intermediate inference results; Based on multiple intermediate reasoning results, determine the thought chain prompts; Based on the task prompt structure and thought chain prompts, tasks to be executed are generated. The task prompt structure includes a preface, multiple examples, and a test screen. The preface explains the purpose of the prompts, each example includes input, thought chain, and output, and the test screen is the screen input to be processed.

[0119] It should be noted that the specific implementation process of the steps performed by each module in the embodiments of this application can be referred to Figure 1 and Figure 4 The implementation process of the information determination method provided in the corresponding embodiments will not be described in detail here.

[0120] The information determination device provided in this application converts the multimodal user interface of a virtualized mobile device into a text representation using Hypertext Markup Language (HTML), enabling a large language model to overcome the perceptual barriers of multimodal data and accurately understand the interface structure and element semantics. This effectively solves the technical problem that large language models cannot directly process mobile interface data. Furthermore, by generating thought chain prompts, the large language model is guided to output a clear logical reasoning process before arriving at the final answer, making the decision-making path of complex tasks explicit and significantly improving the accuracy and interpretability of task execution. Finally, based on the thought chain prompts, the executor executes the task to be performed, realizing a natural and efficient conversational interaction between the user and the virtualized mobile device. Therefore, this method greatly simplifies user operations, improves the intelligence level of task execution, and has good scalability, flexibly adapting to various task scenarios, providing virtualized mobile devices with highly reliable and highly interpretable automated task execution capabilities.

[0121] Based on the foregoing embodiments, this application provides an information processing apparatus that can be applied to... Figure 3 and Figure 6 In the information processing method provided in the corresponding embodiment, refer to Figure 11 As shown, the information processing device 6 may include: a parsing unit 61, a feasibility analysis unit 62, an execution unit 63, and an integrity analysis unit 64, wherein: Parsing unit 61 is used to receive the target task input by the user and parse the target task using the target large language model to obtain one or more first subtasks; The feasibility analysis unit 62 is used to process the first sub-task and the first current screenshot of the virtualized mobile device through a feasibility predictor for each first sub-task to obtain the feasibility result for the first sub-task; wherein, the feasibility predictor is used to predict the feasibility of the first sub-task. Execution unit 63 is used to execute the first subtask through the executor and obtain a screenshot after the first execution if the feasibility result characterization is successful. The integrity analysis unit 64 is used to process the expected screenshot and the screenshot after the first execution of the first subtask through the integrity verifier to obtain the integrity result for the first subtask; wherein, the integrity verifier is used to evaluate the integrity of the first subtask.

[0122] In other embodiments of this application, the feasibility analysis unit 62 is further configured to perform the following steps: The visual encoder in the feasibility predictor extracts features from the current screenshot to obtain the image features of the current screenshot. The semantic information of the first subtask is obtained by semantic parsing of the language decoder in the feasibility predictor. Feasibility results are determined based on image features and semantic information.

[0123] In other embodiments of this application, the integrity analysis unit 64 is further configured to perform the following steps: The integrity validator determines the degree of matching between the expected screenshot and the screenshot after the first execution. The completeness result is determined based on the degree of matching.

[0124] In other embodiments of this application, the execution unit 63 is further configured to perform the following steps: If there is a discrepancy between the expected screenshot and the screenshot after the first execution, the target large language model is used to analyze the discrepancy and the target task to obtain one or more second subtasks; For each second subtask, the feasibility predictor processes the second subtask and the second current screenshot of the virtualized mobile device to obtain the feasibility result for the second subtask. If the feasibility result indicates that the feasibility is successful, the second subtask is executed by the executor to obtain a screenshot of the second execution.

[0125] In other embodiments of this application, the execution unit 63 is further configured to perform the following steps: The target recognition model identifies sensitive information in the target task and determines the placeholders corresponding to the sensitive information. Placeholders and sensitive information are stored in storage pairs in offline memory; In other embodiments of this application, the execution unit 63 is further configured to perform the following steps: Based on the storage pair, the target placeholder in the first subtask is restored to the target sensitive information to obtain the restored subtask; The executor executes the restored subtask to obtain the first screenshot after execution.

[0126] It should be noted that the specific implementation process of the steps performed by each module in the embodiments of this application can be referred to Figure 3 and Figure 6 The implementation process of the information processing method provided in the corresponding embodiments will not be described in detail here.

[0127] The information processing device provided in this application uses a target large language model to parse the user-input target task into executable subtasks. Before execution, a feasibility predictor, as an end-to-end multimodal model, uses the current screenshot and the subtask to be executed as input to accurately predict the feasibility of the operation, effectively filter invalid instructions, and avoid negative impacts on the system state caused by erroneous operations. After execution, an integrity verifier compares the expected screenshot with the actual execution result to accurately evaluate the task completion status. Therefore, this application significantly improves the success rate and robustness of complex task automation. Through the dual guarantee mechanism of feasibility prediction before execution and integrity verification after execution, it effectively reduces the risk of execution failure caused by dynamic changes in the interface or instruction deviations, and improves the reliability and security of task processing.

[0128] Based on the foregoing embodiments, embodiments of this application provide an electronic device, specifically including an information determining device, which can be applied to... Figure 1 and Figure 4 In the information determination method provided in the corresponding embodiment, refer to Figure 12 As shown, the information determining device 7 may include: a processor 71, a memory 72, and a communication bus 73, wherein: Communication bus 73 is used to realize the communication connection between processor 71 and memory 72; The processor 71 is used to execute the information determination program in the memory 72 to perform the following steps: The text representation of the target hypertext markup language corresponding to the multimodal user interface that determines the screen information of the virtualized mobile device; wherein the screen information is generated based on the user's input on the virtualized mobile device; Based on text representation and large language model, thought chain prompts and tasks to be performed are generated corresponding to the text representation; among them, thought chain prompts are used to elicit the logical reasoning process in the large language model, so that the large language model can explain its thought process before arriving at the final answer; The executor executes the task to be executed based on the thought chain prompts, and obtains the operation result corresponding to the task to be executed.

[0129] In other embodiments of this application, processor 71 is used to execute the information determination program in memory 72 to determine the text representation of the target Hypertext Markup Language corresponding to the multimodal user interface of the virtualized mobile device's screen information, in order to implement the following steps: Obtain the view hierarchy in the multimodal user interface and represent the view hierarchy as a view hierarchy tree; Based on a depth-first search, the user interface elements corresponding to each node in the view hierarchy tree are traversed to determine the text representation of the Hypertext Markup Language to be processed for each user interface element. For each text representation of the Hypertext Markup Language to be processed, the index corresponding to the text representation of the Hypertext Markup Language to be processed is determined, and the index is inserted into the text representation of the Hypertext Markup Language to be processed to obtain the text representation of the target Hypertext Markup Language; wherein, the index is generated according to the depth-first search order in the view hierarchy tree; Based on class mapping rules, Android user interface component classes of virtualized mobile devices are mapped to text representations of target Hypertext Markup Language with corresponding functions; wherein, class mapping rules are used to indicate the mapping between Android user interface component classes and Hypertext Markup Language.

[0130] In other embodiments of this application, processor 71 is used to execute a depth-first search traversal of the user interface elements corresponding to each node in the view hierarchy tree of the information determination program in memory 72, and to determine the text representation of the Hypertext Markup Language corresponding to each user interface element, so as to implement the following steps: By traversing the view hierarchy tree using a depth-first search, the user interface element corresponding to each node in the view hierarchy tree is converted into a text representation of the initial Hypertext Markup Language. For each initial hypertext markup language text representation, determine the target attribute set of the user interface element corresponding to the initial hypertext markup language text representation; wherein, the target attribute set includes class attributes, text attributes, resource identifier attributes, and content description attributes; Write the target attribute set into the initial hypertext markup language text representation to obtain the hypertext markup language text representation to be processed.

[0131] In other embodiments of this application, processor 71 is used to execute the class mapping rule of the information determination program in memory 72, mapping the Android user interface component classes of the virtualized mobile device to text representations of the target Hypertext Markup Language with corresponding functions, to achieve the following steps: Based on class mapping rules, text view classes are mapped to paragraph tags, button-related classes are mapped to button tags, image-related classes are mapped to image tags, and text input classes are mapped to input tags. All other classes in the Android user interface component classes, except for text view classes, button-related classes, image-related classes, and text input classes, are mapped to partition tags. The Android user interface component classes include text view classes, button-related classes, image-related classes, text input classes, and other classes. The text representation in the target hypertext markup language includes paragraph tags, button tags, image tags, input tags, and partition tags.

[0132] In other embodiments of this application, processor 71 is used to execute the information determination program in memory 72 based on text representation and a large language model to generate thought chain prompts and tasks to be performed corresponding to the text representations, in order to achieve the following steps: A large language model is used to process the text representation to obtain intermediate inference results; Based on multiple intermediate reasoning results, determine the thought chain prompts; Based on the task prompt structure and thought chain prompts, tasks to be executed are generated. The task prompt structure includes a preface, multiple examples, and a test screen. The preface explains the purpose of the prompts, each example includes input, thought chain, and output, and the test screen is the screen input to be processed.

[0133] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 4 The implementation process of the information determination method provided in the corresponding embodiments will not be described in detail here.

[0134] The information determination device provided in this application converts the multimodal user interface of a virtualized mobile device into a text representation using Hypertext Markup Language (HTML), enabling a large language model to overcome the perceptual barriers of multimodal data and accurately understand the interface structure and element semantics. This effectively solves the technical problem that large language models cannot directly process mobile interface data. Furthermore, by generating thought chain prompts, the large language model is guided to output a clear logical reasoning process before arriving at the final answer, making the decision-making path of complex tasks explicit and significantly improving the accuracy and interpretability of task execution. Finally, based on the thought chain prompts, the executor executes the task to be performed, realizing a natural and efficient conversational interaction between the user and the virtualized mobile device. Therefore, this method greatly simplifies user operations, improves the intelligence level of task execution, and has good scalability, flexibly adapting to various task scenarios, providing virtualized mobile devices with highly reliable and highly interpretable automated task execution capabilities.

[0135] Based on the foregoing embodiments, embodiments of this application provide an electronic device, specifically including an information processing device, which can be applied to... Figure 3 and Figure 6 In the information processing method provided in the corresponding embodiment, refer to Figure 13 As shown, the information processing device 8 may include: a processor 81, a memory 82, and a communication bus 83, wherein: Communication bus 83 is used to realize the communication connection between processor 81 and memory 82; The processor 81 is used to execute the information processing program in the memory 82 to perform the following steps: Receive the target task input by the user, and use the target large language model to parse the target task to obtain one or more first subtasks; For each first subtask, a feasibility predictor processes the first subtask and the first current screenshot of the virtualized mobile device to obtain a feasibility result for the first subtask; wherein, the feasibility predictor is used to predict the feasibility of the first subtask. If the feasibility result indicates that the feasibility is successful, the first subtask is executed by the executor to obtain a screenshot after the first execution. The integrity verifier processes the expected screenshot and the screenshot after the first execution of the first subtask to obtain the integrity result for the first subtask; the integrity verifier is used to evaluate the integrity of the first subtask.

[0136] In other embodiments of this application, processor 81 is used to execute the information processing program in memory 82 to process the first subtask and the current screenshot of the virtualized mobile device through a feasibility predictor to obtain a feasibility result for the first subtask, in order to implement the following steps: The visual encoder in the feasibility predictor extracts features from the current screenshot to obtain the image features of the current screenshot. The semantic information of the first subtask is obtained by semantic parsing of the language decoder in the feasibility predictor. Feasibility results are determined based on image features and semantic information.

[0137] In other embodiments of this application, the processor 81 is used to execute the information processing program in the memory 82 to process the expected screenshot and the first executed screenshot of the first subtask through an integrity verifier to obtain an integrity result for the first subtask, in order to implement the following steps: The integrity validator determines the degree of matching between the expected screenshot and the screenshot after the first execution. The completeness result is determined based on the degree of matching.

[0138] In other embodiments of this application, the processor 81 is used to execute an information processing method of an information processing program in the memory 82 to implement the following steps: If there is a discrepancy between the expected screenshot and the screenshot after the first execution, the target large language model is used to analyze the discrepancy and the target task to obtain one or more second subtasks; For each second subtask, the feasibility predictor processes the second subtask and the second current screenshot of the virtualized mobile device to obtain the feasibility result for the second subtask. If the feasibility result indicates that the feasibility is successful, the second subtask is executed by the executor to obtain a screenshot of the second execution.

[0139] In other embodiments of this application, the processor 81 is used to execute an information processing method of an information processing program in the memory 82 to implement the following steps: The target recognition model identifies sensitive information in the target task and determines the placeholders corresponding to the sensitive information. Placeholders and sensitive information are stored in storage pairs in offline memory.

[0140] In other embodiments of this application, the processor 81 is used to execute the information processing program in the memory 82 by executing a first subtask through an executor to obtain a screenshot after the first execution, in order to achieve the following steps: Based on the storage pair, the target placeholder in the first subtask is restored to the target sensitive information to obtain the restored subtask; The executor executes the restored subtask to obtain the first screenshot after execution.

[0141] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 3 and Figure 6 The implementation process of the information processing method provided in the corresponding embodiments will not be described in detail here.

[0142] The information processing device provided in this application uses a target large language model to parse the user-input target task into executable subtasks. Before execution, a feasibility predictor, as an end-to-end multimodal model, uses the current screenshot and the subtask to be executed as input to accurately predict the feasibility of the operation, effectively filter invalid instructions, and avoid negative impacts on the system state caused by erroneous operations. After execution, an integrity verifier compares the expected screenshot with the actual execution result to accurately evaluate the task completion status. Therefore, this application significantly improves the success rate and robustness of complex task automation. Through the dual guarantee mechanism of feasibility prediction before execution and integrity verification after execution, it effectively reduces the risk of execution failure caused by dynamic changes in the interface or instruction deviations, and improves the reliability and security of task processing.

[0143] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application. < / button>

Claims

1. A method for determining information, characterized in that, The method includes: The text representation of the target hypertext markup language corresponding to the multimodal user interface that determines the screen information of the virtualized mobile device; wherein the screen information is generated based on user input on the virtualized mobile device; Based on the text representation and the large language model, thought chain prompts and tasks to be performed are generated corresponding to the text representation; wherein, the thought chain prompts are used to elicit the logical reasoning process in the large language model, so that the large language model can clarify its thought process before arriving at the final answer; The executor executes the task to be executed based on the thought chain prompts, and obtains the operation result corresponding to the task to be executed. The text representation of the target Hypertext Markup Language corresponding to the multimodal user interface that determines the screen information of the virtualized mobile device includes: Obtain the view hierarchy in the multimodal user interface and represent the view hierarchy as a view hierarchy tree; Based on a depth-first search, the user interface elements corresponding to each node in the view hierarchy tree are traversed to determine the text representation of the Hypertext Markup Language corresponding to each user interface element. For each text representation of the Hypertext Markup Language to be processed, an index corresponding to the text representation of the Hypertext Markup Language to be processed is determined, and the index is inserted into the text representation of the Hypertext Markup Language to be processed to obtain the text representation of the target Hypertext Markup Language; wherein, the index is generated according to the depth-first search order in the view hierarchy tree; Based on class mapping rules, the Android user interface component classes of the virtualized mobile device are mapped to the text representations of the target Hypertext Markup Language with corresponding functions; wherein, the class mapping rules are used to indicate the mapping between Android user interface component classes and Hypertext Markup Language.

2. The method according to claim 1, characterized in that, The process of traversing the user interface elements corresponding to each node in the view hierarchy tree based on depth-first search to determine the text representation of the Hypertext Markup Language (HTML) for each user interface element includes: The view hierarchy tree is traversed by the depth-first search, and the user interface element corresponding to each node in the view hierarchy tree is converted into a text representation of the initial hypertext markup language. For each of the initial hypertext markup language text representations, a target attribute set for the user interface element corresponding to the text representation of the initial hypertext markup language is determined; wherein, the target attribute set includes class attributes, text attributes, resource identifier attributes, and content description attributes; The target attribute set is written into the text representation of the initial hypertext markup language to obtain the text representation of the hypertext markup language to be processed.

3. The method according to claim 1, characterized in that, The method of mapping Android user interface component classes of the virtualized mobile device to text representations of the target Hypertext Markup Language with corresponding functions based on class mapping rules includes: Based on the class mapping rules, text view classes are mapped to paragraph tags, button-related classes are mapped to button tags, image-related classes are mapped to image tags, and text input classes are mapped to input tags. All other classes in the Android user interface component classes, except for the text view class, the button-related class, the image-related class, and the text input class, are mapped to partition tags. The Android user interface component classes include the text view class, the button-related class, the image-related class, the text input class, and the other classes. The text representation in the target hypertext markup language includes the paragraph tags, the button tags, the image tags, the input tags, and the partition tags.

4. The method according to claim 1, characterized in that, The process of generating thought chain prompts and tasks to be performed corresponding to the text representation based on the text representation and the large language model includes: The text representation is processed using the large language model to obtain intermediate inference results; Based on multiple intermediate reasoning results, the thought chain prompt is determined; Based on the task prompt structure and the thought chain prompt, the task to be executed is generated; wherein, the task prompt structure includes a preface, multiple examples and a test screen; the preface is used to explain the purpose of the prompt, each example includes an input, a thought chain and an output, and the test screen is the screen input to be processed.

5. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that can be executed by one or more processors to implement the method of claims 1 to 4 or any one of them.

Images