An Automatic Operation Method for Mobile Terminal Graphical User Interface Based on Multimodal Model

Abstract

This invention discloses an automated operation method for mobile terminal GUI based on a multimodal model. The method includes: initializing the environment through a debugging interface; acquiring terminal interface images in real time and intelligently scaling them according to model visual token constraints; constructing prompts integrating historical operation sequences, task instructions, and tool definitions, and inputting them into a multimodal model to obtain structured decision suggestions; parsing the decision suggestions containing action intentions and function calls and executing them on the terminal, where text input is accurately injected by switching the debugging input method; and finally, storing the action intentions in the historical operation sequence and looping until the task terminates. This invention combines visual understanding and historical perception mechanisms, solving the problems of poor generalization ability and difficulty in recognizing non-standard controls in traditional solutions on complex interfaces, significantly improving the efficiency of automated execution on mobile terminals.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, human-computer interaction, and automatic control technology for mobile terminals, and specifically to a method for operating a graphical user interface on a real mobile terminal based on a multimodal large model and through a terminal debugging interface. Background Technology

[0002] With the rapid development of artificial intelligence technology, automated operation technology for mobile terminal graphical user interfaces is playing an increasingly important role in software automation testing, process automation, and intelligent decision-making assistance. However, traditional mobile terminal automation solutions mainly rely on rule-based or static element positioning technologies. These solutions often exhibit poor generalization ability, high maintenance costs, and a lack of deep logical understanding when faced with frequently updated interfaces, non-standard controls, or complex visual semantic environments, severely restricting the flexibility and success rate of automation tasks. In recent years, the rise of multimodal large models has provided new possibilities for intelligent operation, but it still faces many challenges in practical applications and struggles to support high-intensity industrial-grade automation requirements. This invention proposes an automated operation method for mobile terminal graphical user interfaces based on multimodal models. By introducing an intelligent image scaling mechanism to adapt to visual image size limitations and constructing prompt words based on historical perception decisions, combined with an enhanced debugging interface input method, this method achieves efficient collaboration, accurate execution, and flexible adaptation to heterogeneous interfaces for mobile terminal user graphical interface operations, thereby promoting the intelligent level of terminal control in mobile office, application development, and automated operation and maintenance scenarios. Summary of the Invention

[0003] This invention provides an automatic operation method for a mobile terminal graphical user interface based on a multimodal model. The method involves a mobile terminal, a terminal debugging interface, a large multimodal model, and a debugging input method. The method includes:

[0004] Step 1: The system initializes the environment by connecting to the mobile terminal through the debugging interface and obtaining the screen resolution parameters. Reset the terminal state and initialize the historical operation sequence. ;

[0005] Step 2: Capture the original image of the current graphical user interface of the mobile terminal through the debugging interface. , as the visual input for the current step;

[0006] Step 3: The system determines the maximum number of visual image tokens based on the preset limit. To address the limitations, the original image undergoes intelligent scaling, adjusting its size while maintaining the aspect ratio as much as possible, and adapting to the image patch size of the multimodal model. ;

[0007] Step 4: The system processes the historical operation sequence of the current task. The formatting process is performed and embedded into a preset task instruction template, while also incorporating other information, including the task objective. Description of tools that can be called by the model Text prompts for constructing multimodal large models;

[0008] Step 5: Simultaneously input the constructed prompts and the scaled graphical interface image into the multimodal large model to obtain the action descriptions generated by the model. and structured decision recommendations for function calls ;

[0009] Step 6: The system analyzes the decision suggestions output by the model and extracts the types of operations to be performed. and its corresponding parameters Such as coordinates, text, or duration;

[0010] Step 7: If the parsing result indicates that the task is completed or failed, the operation loop is terminated and the result is saved; otherwise, the system transmits the parsing result from step 6 to the action execution module.

[0011] Step 8: Based on the parsed operation type and parameters, perform simulated operations on the terminal. When performing actions involving text input, the system automatically detects and switches the mobile terminal's input method to the debug input method. Encoded transmission method completes non Precise character injection, and switching to the default input method after the input action is completed;

[0012] Step 9: After executing the action, wait for the preset waiting time. To ensure the interface refreshes completely, the current step is then recorded. Update to historical sequence Return to step 2 to proceed to the next iteration.

[0013] Specifically, in step 1, the system performs environment initialization, which involves the system acquiring relevant device information through the information acquisition module. For example, the system can acquire the original screen width of the mobile terminal through the screen size acquisition module. and height This provides a baseline for subsequent coordinate transformations; simultaneously, the system calls a restart function to ensure the target application is in its initial state and creates an empty list. Used to maintain the context operation chain during task execution.

[0014] Specifically, in step 2, capturing the original image through the debugging interface involves using an image capture module to capture pixel data of the mobile terminal screen in real time. And save it as an image file or binary stream in the specified path.

[0015] Specifically, in step 3, the intelligent scaling process for the original image involves: the system calculating the maximum total number of pixels allowed by the model. If the original image's pixel count exceeds this limit, the image will be scaled down to the closest possible size while maintaining the original aspect ratio as much as possible. And both width and height are The size is adjusted to a multiple of the model size to retain key visual features while reducing computational overhead.

[0016] Specifically, in step 4, the prompt word for constructing the multimodal model is: system formatting. The historical operation steps are represented as strings, which are then used to replace the historical information placeholders in the preset prompt template; the prompt template also includes a description of the task objective. Tool calling specifications And output format constraints.

[0017] Specifically, in step 5, obtaining structured decision suggestions involves the system matching text prompts with... The encoded image data is encapsulated into a message format that conforms to the requirements of the multimodal model interface. This is then used to call a local or remote multimodal large model service, receiving the returned formatted string, which contains a semantic description of the action. and specific tool names and parameters .

[0018] Specifically, in step 6, the decision suggestion output by the parsing model is as follows: the system uses a parser to process the formatted string output by the model and extracts the name of the target action. and parameter dictionary If click coordinates are involved, the system will restore the normalized coordinates output by the model to the pixel coordinates of the corresponding screen resolution.

[0019] Specifically, in step 7, the specific operation is performed as follows: if the decision is that the task is completed or the task fails, then the task is exited; otherwise, the parsed action type and parameters are passed to the action execution module.

[0020] Specifically, in step 8, the debugging input method-assisted text input involves: the system detecting the current default input method through the debugging terminal; if it is not the debugging input method, switching it via the input method switching module; and when inputting text, the system processes the string to be input. The code is encoded, sent to the debug input method, and input is completed. After input is completed, the system default input method is switched. If the action includes visualization requirements, the system draws a marker dot at the operation point and saves the image.

[0021] Specifically, in step 9, updating the system state involves: after the system completes a single physical operation, it calls a waiting module to wait for the graphical interface to complete the response update; subsequently, it updates the semantic description of the currently executed action. Add to In the process, it simultaneously enters the next image acquisition and decision-making loop until the task counter reaches the maximum number of steps or the termination condition is triggered. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 A flowchart of an automatic operation method for a mobile terminal graphical user interface based on a multimodal model; Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Current traditional image interface automation solutions based on rule-based or static element localization suffer from shortcomings when faced with complex and ever-changing terminal interfaces, including poor generalization ability, difficulty in handling non-standard controls, and lack of visual semantic understanding. This invention proposes an automated operation method for mobile terminal graphical user interfaces based on a multimodal large model. By combining the visual understanding capabilities of the multimodal large model, intelligent image preprocessing mechanisms, and terminal debugging interface communication, it achieves precise automatic control of the mobile interface, effectively addressing automation needs in complex business scenarios. The specific steps are as follows:

[0026] S101: System environment initialization and parameter configuration.

[0027] Specifically, the system uses a terminal debugging interface (e.g.) system The system can obtain the mobile terminal's status parameters via an interface. Specifically, it can perform operations such as obtaining the screen size (e.g., using...). (Instruction) Get the original screen width of the current mobile terminal and height This serves as the benchmark for subsequent visual coordinate reconstruction. Simultaneously, the system invokes the environment reset module to ensure the system is in a preset initial state and initializes the historical operation sequence. This provides a contextual basis for multi-round continuous decision-making.

[0028] S102: Real-time acquisition of visual data from the current graphical interface.

[0029] Specifically, the system utilizes the screenshot function of the debugging interface (such as executing...). Real-time capture of pixel data from the mobile terminal screen to obtain the raw image. Saved as a binary stream or an image file in a specified format (such as PNG), it serves as visual input for the agent to perceive the environment.

[0030] S103: Image scaling and visual image size optimization.

[0031] Specifically, to adapt to the input constraints of multimodal large models and optimize computational efficiency, the system processes the original image... Intelligent scaling is performed. First, based on the model's preset maximum number of visual tokens... And the parameter block size of a large multimodal model Calculate the maximum allowed total number of pixels. Next, while maintaining the original aspect ratio as much as possible, the scaling factor is calculated, and the image size is adjusted so that the scaled width and height are both equal. The total number of pixels is an integer multiple of the total number of pixels and is close to or less than or equal to the total number of pixels. This process effectively reduces the inference overhead of the model while preserving the key visual features of the interface.

[0032] S104: Dynamically construct multimodal reasoning prompts.

[0033] Specifically, the system will The recorded historical operation steps are formatted as text strings and dynamically injected into a designated area of ​​a preset prompt template; the prompt also integrates the user-inputted task objective instructions. List of callable atomization tools (Including tool function descriptions and parameter specifications) and output format constraints, to guide the multimodal large model to generate decision content that conforms to the expected structure.

[0034] S105: Call the multimodal model to perform task inference.

[0035] Specifically, the system will convert the preprocessed image data (such as into...) The encoded string and the text prompt words constructed in step S103 are encapsulated together into a message body conforming to the model interface protocol and submitted to the multimodal large model. The model performs logical reasoning based on visual information, historical records, and task objectives, and outputs structured decision suggestions containing subsequent operations.

[0036] S106: Analysis of reasoning results and extraction of action intent.

[0037] Specifically, the system receives the structured response returned by the model, parses it, and extracts the semantic description of the actions. and specific function calls . Includes the target tool name and execution parameters. If the parameters involve position coordinates, the system will use the normalized coordinates of the model output based on the coordinates obtained in step S101. The parameters are restored to the pixel coordinates of the physical screen.

[0038] S107: Task termination determination.

[0039] Specifically, the system first checks the parsed action type. If the model determines that the task has been completed, the task has failed, or the preset maximum number of attempts has been reached, the operation loop is terminated and the execution log is saved. Otherwise, the system transmits the decision data to the execution module, triggering a physical interaction operation.

[0040] S108: Atomic operation execution and input method enhancement based on the debugging interface.

[0041] Specifically, the system executes atomic operations on the mobile terminal through a debugging interface. For routine interactions such as clicking, swiping, and long-pressing, the system directly sends the corresponding underlying physical simulation commands (e.g., ...). ); for those involving non When a character is input, the system first executes the input method switching logic to switch the terminal input method to a debugging-specific input method that supports encoded transmission (such as...). ), and the text to be input will be processed After encoding, the data is sent to the terminal, where the debug input method decodes it and accurately fills the target position, thus circumventing the native debug protocol's limitations on non-native input methods. Limitations due to poor character support.

[0042] S109: Interface feedback waiting and state sequence update.

[0043] Specifically, after the physical operation is completed, the system waits for a preset waiting time. (such as calling) This ensures that the terminal's graphical interface completes refresh and response updates. Subsequently, the semantic description of the action performed is... Add to Once the state synchronization is completed, the process loops back to step S102 to begin a new round of perception and decision-making until the task termination condition is met.

[0044] This invention proposes an automatic operation method for a mobile terminal graphical user interface based on a multimodal model. It achieves high efficiency in visual understanding through intelligent image scaling, improves the coherence of decision-making by combining historical perception prompts, and solves the text input problem in mobile automation by using an enhanced input method scheme. Thus, it realizes safe, efficient, and closed-loop automatic operation in a mobile environment.

Claims

1. An automatic operation method for a mobile terminal graphical user interface based on a multimodal model, characterized in that, The method involves a mobile terminal, a multimodal large model, a terminal debugging interface, and a debugging input method. The method includes the following steps: Step 1: The system initializes the environment by connecting to the mobile terminal through the terminal communication interface and obtaining the screen resolution parameters. Reset the mobile terminal state and initialize the historical operation sequence. Empty; Step 2: Capture the original image of the current user graphical interface on the mobile terminal through the debugging interface. , as the visual input for the current step; Step 3: The system determines the maximum number of visual image tokens based on a preset limit. The original screenshot image is intelligently scaled, and the image size is adjusted to fit the multimodal model while maintaining the aspect ratio as much as possible. Step 4: The system processes the historical operation sequence of the current task. Perform formatting and match it with the task target. Tool Description Embedded in preset task instruction templates, prompts are used to construct multimodal large-scale models; Step 5: Input the constructed text prompts and the scaled screenshot image into the multimodal large model to obtain the model-generated action descriptions. and function calls Structured decision-making recommendations; Step 6: The system parses the function calls output by the model. Extract the operation type to be executed and its corresponding parameters Such as coordinates, text, or duration; Step 7: Determine if the task has terminated. If the parsing result indicates task completion or failure, terminate the operation loop; otherwise, the system will output the parsed operation type. and its corresponding parameters Send to the instruction execution module. Step 8: Execute the action. The system will determine the action based on the operation type. The system invokes the corresponding atomic tool commands to perform specific operations on the mobile terminal via the debugging interface. When performing actions involving text input, the system automatically detects and switches the mobile terminal's input method to the debugging input method. Text injection is accomplished using encoded transmission methods; Step 9: Update system status, execute the action, and wait for the preset waiting time. To ensure the interface is updated, the intent of the current step will then be displayed. Update to historical sequence Return to step 2 to proceed to the next iteration.

2. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 1, the system performs environment initialization, specifically: the system obtains the original screen width of the mobile terminal through a debugging interface. and height Simultaneously, the system calls the restart function to reset the system to its initial state and creates an empty list. Used to maintain operation records during task execution.

3. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 2, the system captures the original image of the graphical interface. Specifically, the system uses the screenshot command of the debugging interface to capture pixel data of the mobile terminal screen in real time and saves it as a binary stream or an image file in a specified path.

4. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 3, the intelligent scaling process specifically involves: given the maximum allowed number of image tokens. The maximum total number of pixels allowed by the system's calculation model. If the number of pixels in the original image exceeds Then, while maintaining the original aspect ratio as much as possible, the image is scaled to a width and height of 1000x1000x1000x2 ... Multiples and the total number of pixels is close to The size.

5. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 5, obtaining structured decision suggestions specifically involves: the system encapsulating the prompt words and scaled image data into a multimodal message format and sending it to the multimodal large model, and receiving the returned message containing... fields and The response of the field.

6. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 6, the analytical decision suggestion specifically involves the system extracting the target action type from the model response. and parameter dictionary If the parameters involve coordinates, the normalized coordinates output by the model will be restored to the pixel coordinates of the corresponding physical screen resolution.

7. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 7, determining whether the task has terminated specifically involves: if the action type is extracted... If the task is completed or fails, the task is terminated; otherwise, the action type and its parameters are sent to the action execution module.

8. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 8, the execution of the action specifically involves: executing the corresponding terminal instruction based on the action type, including but not limited to click, long press, double click, swipe, text input, wait, and key events. When inputting text, the system detects the current default input method; if it is not the preset debug input method, it executes the input method switching instruction, and then processes the text to be input. The code is encoded and sent to the debugging input method on the mobile terminal for decoding and filling. After the input is completed, the input method is switched to the system default input method.

9. The automatic operation method for a mobile terminal graphical user interface based on a multimodal model as described in claim 1, characterized in that, In step 9, updating the system state specifically means that the system pauses for a preset time interval after a single physical operation is completed. Then, the semantic description of this action will be given. Add to This triggers the next round of perception and decision-making.

Images