Interface operation instruction generation method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an interface operation instruction generation method, electronic equipment, a storage medium and a program product. The method relates to the field of man-machine interaction and the technical field of computer vision, and comprises the following steps: obtaining fusion features based on visual features of an operation interface and current semantic features of a user instruction; according to the fusion features, a region frame regression module is adopted, a target region frame corresponding to the current interface element is generated, the current interface element is an interactive element in the operation interface, and the target region frame is used for representing an interactive region of the current interface element; the region frame regression module is obtained by training based on reference interface elements marked with reference region frames; and generating an operation instruction for executing operation based on the target area frame. In the technical scheme of the embodiment of the invention, the technical problem that the accuracy of automatic operation execution is not ideal due to inaccurate interface element recognition in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the fields of human-computer interaction and computer vision technology, and in particular to a method for generating interface operation instructions, an electronic device, a storage medium, and a program product. Background Art

[0002] In current technological practice, computer use agents (CUAs) and browser use agents (BUAs) are two mainstream types of interface interaction agents. These agents must accurately perform actions such as clicking, typing, and scrolling on the GUI (graphical user interface) based on screen content and user instructions. The key to achieving these actions lies in precisely locating interface elements so that the agent can interact with them accurately.

[0003] Current related technologies directly output a point coordinate as the operation location. However, during the training process, the same interface element (such as a button) may be labeled with different coordinate points. For example, it may be labeled as the center point, the upper left corner, or any internal position in different training samples. Although they are all located in the same clickable area, the coordinate values ​​may differ greatly, resulting in the problem of ground truth jittering during training. However, in actual GUI operations, any point within the entire clickable area of ​​an interactive element (such as a button) should be considered the correct operation point. Ground truth jittering causes the model to receive confusing signals during training, resulting in optimization difficulties and slower model convergence. In addition, related technologies do not take into account the semantic ambiguity of single-point coordinates, making it difficult for the model to accurately understand and locate the operation area during training and prediction, ultimately affecting the accuracy and generalization ability of the model. Summary of the Invention

[0004] The embodiments of the present application provide an interface operation instruction generation method, an electronic device, a storage medium, and a program product to alleviate or solve the technical problem in the related art of inaccurate interface element recognition, resulting in unsatisfactory accuracy in automatic operation execution.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an interface operation instruction, comprising: Based on the visual features of the operation interface and the current semantic features of the user instructions, a fusion feature is obtained; Based on the fused features, a region frame regression module is used to generate a target region frame corresponding to the current interface element. The current interface element is an interactive element in the operation interface. The target region frame is used to represent the interactive area of ​​the current interface element. The region frame regression module is trained based on reference interface elements annotated with reference region frames. Generate an operation instruction to perform an operation based on the target area frame.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any method of the embodiment of the present application when executing the computer program.

[0007] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method of the embodiments of the present application when executed by a processor.

[0009] Based on the interface operation instruction generation method of the first aspect above, the present application has at least the following beneficial effects or advantages: through multimodal feature fusion, it is possible to accurately understand the correspondence between user intentions and interface elements, and avoid erroneous operations caused by unclear semantics in the coordinate point regression method used in related technologies. The area box regression module effectively solves the problem of annotation jitter by predicting the complete interactive area boundary box, so that the model maintains spatial consistency in the positioning results of the same element. Directly outputting the interactive area instead of the single point coordinates effectively improves the recognition ability of the interface elements, makes the generated operation instructions more fault-tolerant during actual execution, and improves the success rate of operations under complex interfaces.

[0010] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0012] Figure 1 A flow chart showing a method for generating interface operation instructions according to an embodiment of the present application is shown; Figure 2 A comparison diagram showing a method for generating interface operation instructions according to an embodiment of the present application is shown; Figure 3 A schematic diagram of an interface operation instruction generating device according to an embodiment of the present application is shown; Figure 4A block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0014] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0015] The following terms will be used in the following text: A graphical user interface (GUI) is an interface design that uses graphical elements (such as windows, icons, menus, buttons, and text boxes) to interact with the user. It allows users to interact with computer systems or software applications through mouse clicks, keyboard input, and touch operations, without relying on complex command line input.

[0016] Interface elements are the components of a graphical user interface used to achieve interaction and display information. Through reasonable layout and design, users can interact with software or devices intuitively.

[0017] In RPA (Robotic Process Automation) technology, agents automatically complete work tasks by simulating the interaction between human users and software systems.

[0018] The Vision-Language-Action (VLA) model is a multimodal artificial intelligence model that integrates visual perception, natural language understanding, and action execution.

[0019] A computer usage agent (CUA) is an AI agent capable of autonomously or semi-autonomously performing tasks within a computer operating system or application. Combining the visual capabilities of multimodal models with the reasoning power of reinforcement learning, a CUA can understand on-screen elements such as buttons, menus, and text boxes and perform complex tasks. For example, a CUA can open applications, browse web pages, and fill out forms without relying on a specific application interface.

[0020] The Browser Usage Agent (BUA) is an AI agent that specializes in performing tasks within a browser environment. It uses the structure of a web page (such as the DOM) and visual information to identify interactive elements and perform actions such as clicks and inputs. It leverages the web page's structural information to improve the accuracy of operations, for example, by identifying clickable areas on a web page to execute clicks.

[0021] Existing technologies such as CUA and BUA primarily use the point regression method to predict the user interface interaction location. This method directly outputs a precise coordinate point (x, y) as the operation location. However, the operable area of ​​a GUI element is an area, not a point. This point-based output does not meet actual operation requirements. In the training data, the same interface element may be annotated with different coordinate points, such as the center point, the upper left corner, or any internal location. Although these annotated points are all located within the same clickable area, their coordinate values ​​may vary significantly, resulting in the problem of true value jitter during training.

[0022] Due to the jittering ground truth, the model receives confusing signals during training, making optimization difficult and slowing convergence. Even for the same target element, the model must learn to map different visual features to different coordinate points, which not only increases the learning difficulty but also reduces the accuracy of the final model.

[0023] In real-world applications, even if the model's predicted point deviates slightly from the true optimal point, the action will still succeed as long as it remains within the clickable area. However, due to the requirement for point-level accuracy during training, the model may overfit to certain points instead of learning the more meaningful concept of "region," thus reducing the model's robustness in real-world scenarios.

[0024] It should be noted that the above-mentioned application scenarios or application examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solution. In addition, the user information (including but not limited to user device information, user personal information, user operation information, interface layout information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, user click data, interface data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse.

[0025] The following describes in detail the technical solution of this application and how it solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of this application in detail with reference to the accompanying drawings.

[0026] Figure 1 A flow chart of a method for generating interface operation instructions according to an embodiment of the present application is shown. Figure 1 As shown, the method may include step S101, step S102 and step S103.

[0027] Step S101: obtaining fusion features based on the visual features of the operation interface and the current semantic features of the user command; Step S102: Based on the fused features, a region frame regression module is used to generate a target region frame corresponding to the current interface element. The current interface element is an interactive element in the operation interface. The target region frame is used to represent the interactive area of ​​the current interface element. The region frame regression module is trained based on reference interface elements annotated with reference region frames. Step S103: generating an operation instruction for performing an operation based on the target area frame.

[0028] Exemplarily, the above-mentioned execution subject is in the form of an intelligent agent, and the optional application scenarios can be, for example: software robots (RPA robots), intelligent interactive systems, browser plug-ins, application extensions, Internet of Things (IoT) device controllers, human-machine collaborative robots, etc.

[0029] In the case of software robots, interactive agents for end users use natural language processing to understand user needs and operate the interface to complete services, such as querying orders and modifying account information. These agents can perform cross-system operations by switching between systems in the interface.

[0030] The above-mentioned intelligent interactive system can be a smart cockpit, smart home central control, etc., which is integrated into the interaction center of the hardware device (such as the car central control screen, smart home panel), can receive voice / gesture commands, and perform operations based on interface visual analysis. For example, in the car system, the user uses the voice command "open the search box of the navigation interface", and the system automatically locates the interface elements and triggers the interaction.

[0031] When interacting as a browser plug-in or application extension, a functional module is embedded in the browser or third-party application to enhance the automated interaction capabilities of specific scenarios (such as automatic filling of web forms and batch data extraction).

[0032] When interacting as an IoT device controller, the middleware connects physical devices with digital interfaces, such as the monitoring interface operation of industrial equipment and the remote control interface interaction of smart home appliances. For example: in the factory central control system, through the command "adjust the temperature slider of a production line parameter panel", the controller automatically locates the interface elements and sends adjustment instructions.

[0033] In the case of a human-machine collaborative robot, a collaborative robot that combines a physical robotic arm with an interface for interaction. For example, in a customer service scenario, the robot uses visual recognition of interface buttons and cooperates with the robotic arm to click operations while handling user inquiries.

[0034] According to the embodiment provided by the present application, the visual features of the operation interface are extracted to reflect the appearance and layout of the operation interface and the interface elements included therein. The semantic features of the user instructions are extracted to understand the operations that the user wants to perform. The visual features and semantic features are fused to obtain a comprehensive feature representation, namely the fusion feature. The area box regression module is obtained by training the reference interface elements marked with the reference area box, and can learn how to predict the correct interactive area from the visual and semantic features. Using the area box regression module, a target area box is generated for the interface element based on the fusion feature, and the interface element is selected as the data basis for the execution of subsequent operation instructions. Based on the generated target area box, the corresponding operation instruction is generated to guide the subsequent interactive operations (such as clicking, inputting, etc.).

[0035] By generating a target area box to represent the interactive area of ​​the interface element, it is more consistent with the actual operation requirements and avoids the semantic ambiguity problem of the regression point method provided by related technologies. The area box regression module can learn the overall interactive area of ​​the interface element instead of relying on a single labeled point. Even if there is jitter in the labeled points in the training data, the model can reduce the impact of this jitter by learning the area box. Even if there is a certain deviation between the predicted area box and the real area box, as long as the interactive area is still covered, the operation can still be successful, avoiding the overfitting of precise points in the regression point method, and has good adaptability to the identification requirements of interface elements with smaller icons in the operation interface, thereby improving the robustness of the model.

[0036] For example, screenshots of the operating interface are taken at predetermined intervals, and feature extraction is performed based on the screenshots to determine visual features. Optionally, the screenshots are taken within a predetermined range. For example, for a particular application, a partial screenshot is taken of the application interface. If the operating interface includes sub-interfaces of multiple virtual machines, the screenshot method can be used to select the interfaces of a portion of the virtual machines.

[0037] For example, before extracting visual features, the input screen image can be preprocessed to ensure image quality and consistency. For example, the screenshot can be scaled to a uniform size or pixel values ​​can be normalized to a specific range (such as 0 to 1 or -1 to 1). Noise can also be removed from the image, for example, using Gaussian filtering to reduce interference. Contrast enhancement, sharpening, and other operations can be used to highlight important features in the image.

[0038] According to the embodiment provided in this application, the output layer of the region box regression module includes multiple regression heads, and the corner points of the target region box correspond to different regression heads. In step S102: based on the fusion features, the region box regression module is used to generate the target region box corresponding to the current interface element, which may include the following specific steps: Based on the fused features, multiple regression heads are used to predict the initial coordinates of the corresponding corner points respectively; The initial coordinates are mapped to the corresponding corner coordinates through the fully connected layer set in the region bounding box regression module, so that the coordinates of multiple corner points are normalized to a predetermined size range; According to the coordinates of multiple corner points, the target area frame is obtained.

[0039] In the embodiment provided in the present application, the output layer of the region box regression module includes multiple regression heads, and each two regression heads are specifically responsible for predicting the initial coordinates of a corner point of the target region box. Since a coordinate includes the horizontal coordinate x and the vertical coordinate y, for a rectangular region box, there are two corner points (upper left and lower right). Therefore, there will be two regression heads for predicting the horizontal and vertical coordinates of the upper left corner point. Similarly, there will be two regression heads for predicting the lower right corner point, allowing the model to learn the features of each corner point separately, thereby improving the accuracy of the prediction. Each regression head predicts the initial coordinates of the corresponding corner point based on the input fusion features. The above initial coordinates can be regarded as the original prediction values ​​without normalization. A fully connected layer is set in the region box regression module to further map the initial coordinates of each corner point to the final corner point coordinates. During the mapping process, the fully connected layer will normalize the corner point coordinates to a predetermined size range. The normalization process makes the output of the model independent of the specific size of the input image, thereby improving the generalization ability of the model. Based on the normalized coordinates of multiple corner points, a target area box is constructed. The corner point coordinates define the shape and position of the area box, thereby selecting the interactive area of ​​the current interface element.

[0040] For example, there are two corner points, schematically represented as (x_min, y_min, x_max, y_max), representing the coordinates of the top-left and bottom-right corners of the bounding box, respectively. Two independent regression heads and a fully connected layer are set for each corner point. Compared to a single rectangle prediction method, this corner point prediction method can minimize invalid areas in the bounding box. It is more robust when handling deformations of the target bounding box (such as rotation and scaling).

[0041] Figure 2 A comparison chart shows the interface operation instruction generation methods of the embodiments of this application. Point regression is a method provided in related technologies, which outputs single-point coordinates to represent the position of the corresponding interface element. The box regression method, a solution provided in the embodiments of this application, calibrates the region box using corner point coordinates.

[0042] According to the embodiments provided in this application, before step S102, the method may include the following specific steps: Determine the region loss according to the degree of overlap between the reference region frame and the true region frame; Determine the coordinate loss according to the degree of deviation between the reference coordinates of the reference area frame and the real coordinates of the real area frame; Determine the comprehensive loss based on the regional loss and coordinate loss; The region bounding box regression module is trained until the comprehensive loss is less than a predetermined loss threshold, thereby obtaining a trained region bounding box regression module.

[0043] In the embodiment provided in the present application, the area loss is an indicator that measures the degree of overlap between the predicted reference area frame and the true area frame. The coordinate loss is an indicator that measures the degree of deviation between the coordinates of the predicted reference area frame and the coordinates of the true area frame. The comprehensive loss is the weighted sum of the area loss and the coordinate loss, which is used to comprehensively evaluate the performance of the model. By considering the area loss and the coordinate loss at the same time, the model can more accurately predict the position and shape of the target area frame. The area loss ensures that the degree of overlap between the predicted frame and the true frame is high, while the coordinate loss ensures that the corner coordinates of the predicted frame are close to the true coordinates. In each training iteration, the comprehensive loss is calculated and the model parameters are adjusted according to the loss value. By optimizing the comprehensive loss, the area frame regression module is trained until the comprehensive loss is less than the predetermined loss threshold, and the model training is considered to be completed, and a trained area frame regression module is obtained. The introduction of the comprehensive loss enables the model to optimize the shape and position of the area frame simultaneously during the training process, avoiding the problem of insufficient optimization or overfitting that may be caused by a single loss function.

[0044] For example, the region box regression module belongs to the VLA model for Web GUI regression operation area. The overall architecture of the VLA model includes five core modules, which realize the complete process from interface images and user instructions to operation instructions, including visual encoder, language encoder, multimodal fusion module, region box regression module, and operation execution module.

[0045] The visual encoder processes the input GUI screen image and uses visual models such as convolutional neural networks (CNNs) to extract visual features from the image, such as icon shape, color, and layout position. The speech encoder parses user command text and uses natural language processing (NLP) techniques to extract semantic features, converting natural language commands into semantic vectors to clarify the user's interaction intent. The multimodal fusion module fuses the visual features output by the visual encoder with the semantic features output by the speech encoder, generating a unified fused feature through concatenation or an attention mechanism. This can be used to bridge the semantic gap between visual and language modalities, enabling the model to associate the semantic correspondence between a "button in the image" and the user command. The bounding box regression module uses the multimodal fusion features to predict the bounding box coordinates of interactive elements in the interface, replacing traditional single-point coordinate predictions. The bounding box covers the entire interactive area (e.g., the clickable area of ​​a button). The operation execution module generates specific operation instructions based on the predicted bounding box coordinates, combined with the element type (e.g., button, slider) and operation type (e.g., click, drag). These instructions can directly drive automated tools such as RPA robots and browser plug-ins to perform interactive operations on the interface.

[0046] For example, the region loss can be calculated using the Intersection over Union (IoU), which is the ratio of the intersection area of ​​the predicted region box to the union area of ​​the ground-truth region box. Generalized IoU (GIoU) ​​and Distance-IoU (DIoU) can also be used for loss calculation. GIoU aims to introduce an additional penalty term so that the predicted region box is not only closer to the ground-truth box in area, but also closer in shape and size. DIoU aims to make the region box closer to the ground-truth box in position by penalizing the distance between the center points of the region box and the ground-truth box.

[0047] For example, coordinate loss is calculated using Mean Squared Error (MSE) or SmoothL1 Loss. MSE measures the difference between reference coordinates and true coordinates by taking the mean squared difference between the two. SmoothL1 Loss is an improved version of L1 Loss. It uses a piecewise function to approximate MSE when the error is small and L1 when the error is large, thus avoiding the exploding gradient problem.

[0048] For example, the parameters of the bounding box regression module are updated using a backpropagation algorithm and optimizers, such as Adaptive Moment Estimation (Adam) or Stochastic Gradient Descent (SGD). Adam dynamically adjusts the learning rate based on the historical gradients of the parameters, eliminating the need to manually adjust the learning of each parameter and performing well on sparse data. SGD calculates the gradient of each sample and updates the model parameters of the bounding box regression module in the opposite direction of the gradient. This allows for effective convergence to a near-global optimal solution when the data volume is large.

[0049] For example, the visual and language encoders are first pre-trained on a large-scale general dataset, followed by fine-tuning on a Web GUI dataset, and finally fine-tuning specifically for the region bounding box regression task. Using general datasets containing large amounts of images and text, such as image-text pairs, the model's visual encoder for extracting image features and its language encoder for understanding text semantics are initially trained. This allows the model to acquire basic visual feature extraction capabilities, enabling it to discern objects, colors, and shapes in user interface images, as well as basic language comprehension capabilities to understand text meaning and perform keyword extraction.

[0050] Further training was conducted using a dataset specifically tailored to web graphical user interfaces (GUIs), such as buttons, input boxes, menus, and other interface elements found on web pages. By adapting this dataset to specific interfaces, the model adapted to the visual characteristics of web interfaces, gaining better recognition of interface layouts and icon styles. Combined with interactive language commands like "click the search button" and "slide the scroll bar," the model was able to better understand and process elements and commands within web interfaces.

[0051] After completing the first two training steps, the model is further optimized for bounding box regression, specifically predicting the location of interactive elements in the interface and marking the clickable range of buttons with bounding boxes. Using a large amount of data labeled with bounding boxes for interface elements, the model is refined to accurately output the coordinate range of the elements, improving positioning accuracy and ultimately meeting the requirements of tasks such as automated interaction and interface analysis. Through a step-by-step training process of "general pre-training → domain fine-tuning → task-specific fine-tuning," the model gradually develops from basic capabilities to specialized capabilities for locating web interface elements, ensuring accurate understanding of commands and locating interactive areas in real-world applications.

[0052] For example, to improve training efficiency, mixed-precision training is used for VLA models, reducing memory usage and computation time while maintaining model accuracy. Mixed-precision training overcomes the limitations of a single data type by combining 16-bit floating-point numbers (float16) and 32-bit floating-point numbers (float32) during model training. During forward propagation, most operations, such as convolution and matrix multiplication, can be performed using the float16 data type. Float16 takes up only half the memory of float32 and offers faster computation speed, significantly reducing computation time and accelerating training. Because float16 has a smaller representation range and is prone to overflow and underflow, the float32 data type is still used in pre-defined high-precision layers, such as gradient calculations and normalization layers, to ensure model training stability and maintain final accuracy. This combination of data types enables efficient training while maintaining model accuracy.

[0053] For ease of understanding, let's use examples. For the visual encoder, it's preferred to use float16 precision. Convolutional neural networks are less sensitive to precision, and float16 can accelerate matrix operations. The last fully connected layer of the visual encoder is preferably set to float32 precision to avoid loss of detail in the feature maps of webpage screenshots. For the speech encoder, it's preferred to use float16 precision. For user instruction text, the text embedding layer should be kept in float32 to avoid semantic distortion. The Transformer layer used for natural language processing can use float16 to accelerate attention calculations. For the multimodal fusion module, it's preferred to use float32 precision because cross-modal feature alignment is sensitive to small changes, and attention score calculations require high precision to prevent overflow. For the region regression module, it's preferred to use float32 precision because coordinate prediction requires sub-pixel precision, and loss calculations require float32 precision to ensure numerical stability. The operation execution module is preferably set to float16 precision. This module determines the operation strategy based on element type and operation type, without requiring high-precision calculations.

[0054] The performance of the target region box model depends on the richness of the training set. To improve the performance of the region box regression module, the embodiment provided in this application expands the training set through a data augmentation method, thereby improving the generalization ability and robustness of the model. According to the embodiment provided in this application, the method may include the following steps: Construct an initial training set based on the reference interface elements marked with reference area boxes; Adjust the reference region frame to obtain multiple data enhancement frames, where the multiple data enhancement frames use the reference region frame as the maximum range boundary; Based on multiple data augmentation boxes, the initial training set is expanded to obtain the target training set for training the region box regression module.

[0055] In the embodiment provided in the present application, a reference interface element marked with a reference area box is used as the initial training set as the basis for model training. By adjusting the reference area box, multiple data enhancement boxes are generated. Optional adjustments include translation, scaling, rotation and other operations. All generated data enhancement boxes use the reference area box as the maximum range boundary, and the maximum range boundary is regarded as the boundary of the interactive range. The content of multiple data enhancement boxes within the interactive range ensures that the intelligent body can normally interact within the range selected by the interface element box. The generated multiple data enhancement boxes are added to the initial training set as an expansion means to form a target training set.

[0056] By generating diverse training samples through data augmentation, the model can learn more diverse region box shapes and positions, thereby improving its adaptability to different interface elements. The introduction of data augmentation boxes makes the model more robust to noise and changes, and can better handle complex scenarios in real applications.

[0057] For example, for the various adjustment methods described above, the translation method randomly selects a point within the reference region as the new center point to generate a new region. The scaling method randomly selects a scale factor within the reference region to scale the reference region. The rotation method randomly selects an angle within the reference region to rotate the reference region, which can achieve good data enhancement effects for non-center-symmetric reference region frames.

[0058] For example, augmenting training data is a key technical approach to improving model generalization. By simulating the diversity and variability of interfaces in real scenarios, the model can adapt to interface elements of different sizes, layouts, and visual styles. Web GUIs can present different visual appearances depending on factors such as device screen size (e.g., PC, tablet, mobile phone), browser zoom ratio, and theme style (e.g., night mode, day mode, eye protection mode). Simulating these variations through data augmentation prevents the model from over-relying on fixed button sizes, single color schemes, and other factors, which can lead to the training data only including specific features.

[0059] Therefore, other data augmentation techniques can also be applied to the training data, such as random scaling, cropping, color dithering, and blurring. Random scaling randomly scales the interface image by a certain ratio (e.g., 0.8x to 1.2x) to simulate screen resolutions or browser zoom effects on different devices. This allows the model to adapt to changes in element size and prevents the model from memorizing buttons with fixed pixel sizes. Random cropping randomly extracts local areas from the interface image as new samples to simulate scrolling or changes in user perspective. This allows the model to locate elements based on local visual features, improving its ability to retrieve target elements in complex interfaces. Color dithering can be used to adjust brightness / contrast by randomly adjusting the brightness and contrast of the image by a predetermined ratio to simulate different lighting conditions or themes. This makes the model less sensitive to color variations and allows it to focus on the shape and structural features of the elements. Gaussian blur can also be added to add Gaussian noise or blur to the image to simulate the visual blur of low-resolution screens or interface elements, such as the background blur caused by pop-up overlays, enhancing the model's robustness to low-quality images or visual distractions.

[0060] In the embodiment provided in this application, an initial training set is constructed based on reference interface elements marked with reference area frames, including: Extract historical semantic features based on historical user instructions; According to the element type and operation type of the reference interface element, a reference interface element marked with a type is obtained; An initial training set is generated according to reference interface elements with marked types and reference area boxes, as well as historical semantic features.

[0061] In the embodiments provided herein, in order to improve the performance of the region frame regression module, it is necessary not only to annotate the reference region frames, but also to combine the semantic features of historical user instructions, as well as the types and operation types of interface elements, to construct a richer and more representative initial training set. User instructions are extracted from historical interaction records. These instructions can be operations described in natural language (such as "click the login button", "enter your username", etc.). User instructions are converted into semantic feature vectors. These feature vectors can capture the semantic information of user instructions. Reference interface elements are classified and their types are annotated (such as buttons, text boxes, sliders, etc.). The operation types related to the interface elements are annotated (such as click, enter text, drag, etc.). The interface elements with annotated types are combined with the corresponding operation types to form annotated data. The interface elements with annotated types and reference region frames are combined with historical semantic features to form an initial training set. Each training sample can include the visual features of the interface element, the annotated type and operation type, the coordinates of the reference region frame, and historical semantic features.

[0062] By combining historical semantic features with the types of interface elements and their operation types, the model can learn richer contextual information, thereby improving its adaptability to different scenarios. Rich annotation information enables the model to better understand the semantics and operational requirements of interface elements, making it more robust in the face of noise and changes.

[0063] For example, natural language processing (NLP) techniques, such as word embedding and BERT (Bidirectional Encoder Representations from Transformers), are used to train language encoders to obtain semantic features. This can convert text information into numerical features that machines can understand and process, thereby supporting subsequent tasks (such as training the bounding box regression module). Word embedding is a technique that maps words into a high-dimensional vector space. These vectors can capture the semantic relationships between words. Common word embedding methods include Word2Vec and GloVe. Word2Vec learns word vector representations based on contextual information. GloVe (Global Vectors for Word Representation) learns word vector representations based on global word frequency statistics. By combining local contextual information with global word frequency statistics, it can better capture the semantic meaning of words.

[0064] The BERT network's language encoder can also be pre-trained using large amounts of text data, enabling it to capture deep semantic information about language. BERT uses a bidirectional Transformer encoder that simultaneously considers both forward and backward contextual information. It learns language representations by randomly masking words in the input sentence and predicting these masked words. This allows the language encoder to adapt to a variety of natural language tasks and demonstrates strong generalization capabilities.

[0065] According to the embodiment provided by this application, the method further includes: Based on the element type of the reference interface element, determine the visual boundaries and interactive area of ​​the reference interface element; Mark reference area boxes for reference interface elements based on visual boundaries and interactive areas.

[0066] In the embodiments provided in this application, the display range of an element in the interface is determined based on its visual features (such as the icon range of a button and the border of an input box). For example, the visual boundary of a button is the icon pixel range, and the visual boundary of a slider includes the track and the handle. Based on the overlap of the visual boundary and the interactive area, a rectangular frame is used to mark the interactive range of the element. The interactive characteristics of the element are judged by its type, so that the marked reference area frame not only covers the visual display range but also accurately reflects the actual operable area, providing accurate labels for the training of the subsequent area frame regression module.

[0067] According to the embodiment provided by the present application, in step S103: generating an operation instruction to perform an operation based on the target area frame may include the following specific steps: Determining an operation strategy for the target area frame based on the element type and operation type of the interface element, the operation strategy including an operation position and / or operation direction for the target area frame; According to the operation strategy, an operation instruction based on the target area frame is generated.

[0068] In the embodiments provided herein, a specific operation strategy can be determined based on the element type of the interface element (e.g., button, text box, slider, etc.) and the user-specified operation type (e.g., click, text input, drag, etc.). It will be appreciated that, with respect to the operation location, the center position is typically selected for a button click, while the upper left corner may be selected for text input in a text box. With respect to the operation direction, if the operation requires directionality, such as dragging a slider, the operation direction is determined to include left, right, up, down, and from one specified direction to another. Based on the determined operation strategy, specific operation instructions are generated to guide subsequent automated operations. Different interface elements may require different operation strategies. By dynamically adjusting the operation strategy based on the element type and operation type, it is possible to better adapt to various complex scenarios and improve robustness across different interfaces and operation tasks.

[0069] By determining the operation strategy, the most appropriate operation position and direction can be selected based on the type of interface element and the type of operation. This avoids the possibility of incorrect operation caused by blindly selecting the operation point, making the automated operation more consistent with the interaction habits of human users, such as avoiding clicking on the edge of a button, which may cause the operation to fail.

[0070] Exemplarily, the above-mentioned operation strategy generally selects the center point of the area box as the operation point, that is, the operation strategy of the area box is set to the default form and is implemented through global settings. Targetedly, generating an operation strategy based on the element type and operation type may include the following forms, for example: for a click operation, the center point of the area box is used to perform click processing. For a drag operation, the center points of the starting area box and the target area box are used to determine the starting and end points of the drag. For text input, select a predetermined position within the input box area for input, such as the default cursor position, or the upper left position. For scrolling operations, the starting and ending points of the scroll are determined according to the scrolling direction (such as vertical or horizontal) and the area box range.

[0071] According to the above embodiments and optional embodiments, the present application also provides an optional implementation method, which adopts the region box regression method to replace the traditional point regression method for the VLA model of Web GUI regression operation area. The overall architecture of the VLA model includes the following main parts, including a visual encoder (Visual Encoder): responsible for processing the input GUI screen image and extracting visual features; a language encoder (Language Encoder): processing user instructions and extracting semantic understanding; a multimodal fusion module (Multimodal Fusion Module): fusing visual and language features; a region box regression module (Bounding Box Regression Module): predicting the bounding box of the operation area instead of single point prediction; an action execution module (Action Execution Module): generating actual operation instructions based on the predicted region box.

[0072] Unlike the traditional regression point method, this optional implementation requires the model to output a region box, with the coordinates of two corner points represented by four coordinate values, schematically represented as (x_min, y_min, x_max, y_max), representing the coordinates of the upper left corner and lower right corner of the region box respectively.

[0073] The data annotation strategy used by the region box regression module no longer annotates individual points for GUI elements in the training data, but instead annotates the bounding boxes of the entire operational area. This annotation method better aligns with the semantics of actual operations and is easier to maintain consistency. For different types of interface elements, such as buttons, links, and input boxes, the corresponding bounding boxes are determined based on their visual boundaries and interactive areas.

[0074] The output layer of the region bounding box regression module is designed as four regression heads, which predict the four coordinate values ​​of the region bounding box (x_min, y_min, x_max, y_max). These regression heads can be implemented using fully connected layers or convolutional layers, and ensure that the output values ​​are normalized to the image size range.

[0075] The loss function for bounding box regression uses the Intersection over Union (IoU) loss or its variants (such as GIoU and DIoU). These loss functions directly optimize the overlap between the predicted and ground-truth bounding boxes, which is more consistent with the region's semantics. They can also be combined with the Smooth L1 loss to optimize coordinate accuracy.

[0076] This optional implementation optimizes model performance by augmenting the training data with random scaling, cropping, and color transformations to improve model generalization. Specifically, by randomly adjusting the size and position of bounding boxes (while remaining within the valid region), the model's understanding of regions rather than specific points is enhanced.

[0077] A multi-stage training strategy is adopted: the vision and language encoders are first pre-trained on a large-scale general dataset, then fine-tuned on a Web GUI dataset, and finally fine-tuned specifically for the region bounding box regression task.

[0078] To improve training efficiency, mixed precision training technology is used to reduce memory usage and computing time while maintaining model accuracy.

[0079] In actual applications, the predicted area box obtained after model inference needs to be converted into specific operation instructions. The general strategy is to select the center point of the area box as the operation point, but more detailed adjustments can also be made according to the specific interface element type and operation type. For click operations, the center point of the area box is used to perform click processing. For drag operations, the center points of the starting area box and the target area box are used to determine the starting and end points of the drag. For text input, select a predetermined position in the input box area for input, such as the default cursor position, or the upper left position. For scrolling operations, the starting and ending points of the scroll are determined according to the scrolling direction (such as vertical or horizontal) and the area box range.

[0080] Experimental results show that the average accuracy of the model using the region box regression method is improved by 15%-20% compared with point regression, especially in the positioning of complex interfaces and small elements.

[0081] Compared with the traditional regression point method, the VLA model based on region box regression proposed in this application has the following significant advantages: the region box regression method directly corresponds to the operable area of ​​GUI elements, has clear semantic meaning, is more in line with actual operation logic, and eliminates the semantic ambiguity problem in the point regression method; by annotating consistent region boxes, the jitter problem of the annotation points of the same element in different samples is avoided, providing the model with a more stable and consistent learning target, reducing training noise, and improving convergence speed and stability.

[0082] Region box prediction provides more spatial information. Even if the predicted area deviates from the true area, the operation can still succeed as long as there is sufficient overlap, greatly enhancing the model's robustness in practical applications. The region box method can better adapt to interface elements of varying sizes and shapes, effectively handling everything from small icons to large regions. It also provides a foundation for subsequent expansion to more complex interactive operations (such as dragging and region selection). This method is not only applicable to Web GUI interactions but can also be easily extended to other GUI environments such as mobile applications and desktop software, showing broad application prospects.

[0083] By innovatively adopting the region box regression method instead of the traditional point regression method, the problems of unclear semantics and true value jitter in the existing technology are effectively solved, and the training efficiency, prediction accuracy and operational robustness of the model are significantly improved. This technology has important application value and industrialization prospects in the fields of computer vision, natural language processing and human-computer interaction.

[0084] Figure 3 A schematic diagram of the interface operation instruction generating device according to an embodiment of the present application is shown. Figure 3 As shown, corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides an interface operation instruction generating device, including: A feature generation module 301 is used to obtain fusion features based on the visual features of the operation interface and the current semantic features of the user command; A frame regression module 302 is used to generate a target region frame corresponding to the current interface element based on the fusion features using a region frame regression module. The current interface element is an interactive element in the operation interface. The target region frame is used to represent the interactive area of ​​the current interface element. The region frame regression module is trained based on reference interface elements annotated with reference region frames. The instruction generation module 303 is used to generate an operation instruction for performing an operation based on the target area frame.

[0085] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0086] Figure 4 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402. The memory 401 stores a computer program that can be executed on the processor 402. When the processor 402 executes the computer program, the method of the above embodiment is implemented. The number of memory 401 and processor 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and exchanging data.

[0087] In a specific implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, the memory 401, processor 402, and communication interface 403 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0088] Optionally, in a specific implementation, if the memory 401 , the processor 402 , and the communication interface 403 are integrated on a chip, the memory 401 , the processor 402 , and the communication interface 403 may communicate with each other through an internal interface.

[0089] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0090] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0091] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0092] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0093] It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0094] Furthermore, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache memory. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0095] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0096] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0098] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0099] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus or device), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0100] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0101] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0102] The above are merely exemplary embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for generating an interface operation instruction, characterized in that: include: Based on the visual features of the operation interface and the current semantic features of the user instructions, a fusion feature is obtained; Based on the fusion features, a region frame regression module is used to generate a target region frame corresponding to the current interface element, where the current interface element is an interactive element in the operation interface, and the target region frame is used to represent the interactive area of ​​the current interface element. The region frame regression module is trained based on reference interface elements annotated with reference region frames. An operation instruction for performing an operation based on the target area frame is generated.

2. The method according to claim 1, characterized in that The output layer of the region box regression module includes multiple regression heads, and the corner points of the target region box correspond to different regression heads. Based on the fusion features, the region box regression module is used to generate the target region box corresponding to the current interface element, including: Based on the fusion features, the multiple regression heads are used to respectively predict the initial coordinates of the corresponding corner points; Mapping the initial coordinates to corresponding corner point coordinates through a fully connected layer provided in the region bounding box regression module, so that the coordinates of the multiple corner points are normalized to a predetermined size range; The target area frame is obtained according to the coordinates of the multiple corner points.

3. The method according to claim 1, characterized in that The method further comprises: Determining the region loss according to the degree of overlap between the reference region frame and the true region frame; determining a coordinate loss according to a degree of deviation between the reference coordinates of the reference area frame and the real coordinates of the real area frame; determining a comprehensive loss based on the area loss and the coordinate loss; The region box regression module is trained until the comprehensive loss is less than a predetermined loss threshold, thereby obtaining the trained region box regression module.

4. The method according to claim 1, wherein The generating of an operation instruction for performing an operation based on the target area frame includes: determining an operation strategy for the target area frame based on an element type and an operation type of the interface element, the operation strategy including an operation position and / or an operation direction for the target area frame; According to the operation strategy, the operation instruction based on the target area frame is generated.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: constructing an initial training set based on the reference interface elements marked with the reference area frame; Adjusting the reference area frame to obtain a plurality of data enhancement frames, wherein the plurality of data enhancement frames have the reference area frame as a maximum range boundary; Based on the multiple data enhancement boxes, the initial training set is expanded to obtain a target training set for training the region box regression module.

6. The method according to claim 5, characterized in that The constructing an initial training set based on the reference interface elements marked with the reference area frame includes: Extract historical semantic features based on historical user instructions; Obtaining the reference interface element marked with a type according to the element type and operation type of the reference interface element; The initial training set is generated according to the reference interface elements marked with types and the reference area boxes, and the historical semantic features.

7. The method according to claim 5, characterized in that The method further comprises: Determining the visual boundary and interactive area of ​​the reference interface element based on the element type of the reference interface element; The reference interface element is marked with the reference area frame according to the visual boundary and the interactive area.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images