Interface operation instruction generation method, electronic device, storage medium and program product

Abstract

Embodiments of the present application provide an interface operation instruction generation method, electronic equipment, storage medium and program product. It is related to the field of human-computer interaction and the field of computer vision technology. The method comprises: obtaining fusion features based on visual features of an operation interface and current semantic features of a user instruction; generating a target region frame corresponding to a current interface element according to the fusion features by using a region frame regression module, the current interface element being an interactive element in the operation interface, the target region frame being used to represent an interactive region of the current interface element, and the region frame regression module being trained based on reference interface elements with labeled reference region frames; and generating an operation instruction for executing an operation based on the target region frame. In the technical solution of the embodiments of the present application, the present application solves the technical problem of inaccurate interface element recognition in related technologies, which leads to unsatisfactory accuracy of automatic operation execution.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer interaction and the field of computer vision, and in particular, relates to an interface operation instruction generation method, an electronic device, a storage medium, and a program product. BACKGROUND

[0002] In current technical practice, Computer Use Agent (CUA) and Browser Use Agent (BUA) are two mainstream types of interface interaction agents. An agent needs to accurately perform clicking, inputting, scrolling, and other operations on a GUI (Graphical User Interface) according to screen content and user instructions. The key to achieving these operations is to accurately locate the position of an interface element so that the agent can accurately interact with it.

[0003] In the current related technology, a point coordinate is directly output as an operation position, but in the training process, the same interface element (such as a button) may be labeled as different coordinate points, such as being labeled as a center point, a top-left corner, or an arbitrary internal position in different training samples. Although they are all located within the same clickable area, they may differ greatly in coordinate values, resulting in the problem of Ground Truth Jittering during training. However, in actual GUI operations, any point within the entire clickable area of an interactive element (such as a button) should be considered as a correct operation point. Ground Truth Jittering causes the model to receive confusing signals during the training process, leading to optimization difficulties and slower model convergence speed. Moreover, the related technology does not consider the semantic ambiguity of single-point coordinates, making it difficult for the model to accurately understand and locate the operation area during training and prediction, ultimately affecting the precision and generalization ability of the model. SUMMARY

[0004] The embodiments of the present application provide an interface operation instruction generation method, an electronic device, a storage medium, and a program product to alleviate or solve the technical problem that the interface element recognition is not accurate in the related technology, resulting in unsatisfactory accuracy of automatic operation execution.

[0005] In a first aspect, the embodiments of the present application provide an interface operation instruction generation method, comprising:

[0006] obtaining a fused feature based on the visual features of the operation interface and the current semantic features of the user instruction;

[0007] According to the fused features, a region box regression module is used to generate a target region box corresponding to the current interface element, the current interface element is an interactive element in the operation interface, the target region box is used to represent the interactive region of the current interface element, and the region box regression module is trained based on a reference interface element with a labeled reference region box;

[0008] An operation instruction based on the target region box is generated.

[0009] In a second aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor implements the method of any of the embodiments of the present application when executing the computer program.

[0010] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0011] In a fourth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0012] Based on the interface operation instruction generation method of the first aspect, the present application has at least the following advantages: through multi-modal feature fusion, the correspondence between the user's intention and the interface element can be accurately understood, and the misoperation caused by the ambiguity of the semantic in the coordinate point regression method in the related art can be avoided. The region box regression module predicts a complete interactive region bounding box, effectively solves the labeling jitter problem, and makes the positioning results of the same element by the model keep spatial consistency. Instead of outputting a single point coordinate, the interactive region is directly output, which effectively improves the recognition ability of the interface element, makes the generated operation instruction have stronger fault tolerance when actually executed, and improves the operation success rate under a complex interface.

[0013] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the description can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the drawings, the same reference numbers in the drawings represent the same or similar elements or components. The drawings are not necessarily drawn to scale. It should be understood that the drawings only depict some embodiments according to the present application, and should not be considered as limiting the scope of the present application.

[0015] Figure 1A flow chart of the interface operation instruction generation method of the embodiment of the present application is shown.

[0016] Figure 2 A comparison chart of the interface operation instruction generation method of the embodiment of the present application is shown.

[0017] Figure 3 A schematic diagram of the interface operation instruction generation device of the embodiment of the present application is shown.

[0018] Figure 4 A block diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0020] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0021] In the following, the following terms will be used:

[0022] Graphical User Interface (GUI), is a kind of interface design through graphical elements (such as windows, icons, menus, buttons, text boxes, etc.) to interact with users. Allow users to interact with computer systems or software applications through mouse clicks, keyboard input, touch operation, etc. without relying on complex command line input.

[0023] Interface elements are components of graphical user interfaces that enable interaction and display information. Through reasonable layout and design, users can intuitively interact with software or devices.

[0024] Agent, in RPA (Robotic Process Automation, Robotic Process Automation) technology, simulates human user interaction with software systems to automatically complete work tasks.

[0025] Vision-Language-Action (VLA) model is a multi-modal artificial intelligence model that integrates visual perception, natural language understanding and action execution.

[0026] A computer usage agent (CUA) is an artificial intelligence agent that can autonomously or semi-autonomously perform tasks in a computer operating system or application. Combining the visual capabilities of multi-modal models with the reasoning capabilities of reinforcement learning, it can understand elements such as buttons, menus, and text boxes on the screen and perform complex tasks. For example, a CUA can open an application, browse a webpage, fill out a form, and so on without relying on specific application interfaces.

[0027] A browser usage agent (BUA) is an artificial intelligence agent that focuses on performing tasks in a browser environment. It can identify interactive elements through the structure of a webpage (such as the DOM) and visual information, and perform operations such as clicking and inputting. It uses structural information of the webpage to improve the accuracy of operations, such as identifying clickable areas on the webpage to perform clicking operations.

[0028] Existing technologies such as CUA and BUA mainly use the Point Regression method to predict the position of user interface interactions. Point Regression directly outputs an accurate coordinate point (x, y) as the operation position, but the operable area of a GUI element is a region, not a point. This point-like output does not match the actual operation requirements. In the training data, the same interface element may be labeled as different coordinate points, such as the center point, the top-left corner, or any internal position. These labeled points may differ significantly in coordinate values, causing the true value to fluctuate during training.

[0029] Due to the true value fluctuation problem, the model receives confusing signals during training, causing optimization difficulties and slower convergence speed. Even for the same target element, the model needs to learn to map different visual features to different coordinate points, which not only increases the learning difficulty but also reduces the accuracy of the final model.

[0030] In practical applications, even if the model's predicted point has some deviation from the true optimal point, as long as it is still within the clickable area, the operation can still be successful. However, due to the requirement for accuracy to the point during training, the model may overfit to certain specific points rather than learning the more meaningful "region" concept, thereby reducing the model's robustness in real-world scenarios.

[0031] It should be noted that the above application scenarios or application examples provided in the embodiments of the present application are for the convenience of understanding, and the application of the technical solutions in the embodiments of the present application is not specifically limited. In addition, the user information (including but not limited to user device information, user personal information, user operation information, interface layout information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, user click data, interface data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0032] The technical solutions of the present application and how the technical solutions of the present application solve the foregoing technical problems will be described in detail below with specific embodiments. Several specific embodiments listed can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 The flowchart of the interface operation instruction generation method of the embodiments of the present application is shown as shown in Figure 1 The method can include steps S101, S102 and S103.

[0034] Step S101: obtaining fusion features based on visual features of the operation interface and current semantic features of the user instruction;

[0035] Step S102: generating a target region frame corresponding to a current interface element in the operation interface by using a region frame regression module according to the fusion features, the current interface element being an interactive element in the operation interface, the target region frame being used to represent an interactive region of the current interface element, and the region frame regression module being trained based on reference interface elements with labeled reference region frames;

[0036] Step S103: generating an operation instruction for executing an operation based on the target region frame.

[0037] Exemplarily, the above execution subject is in the form of an intelligent agent, and optional application scenarios can be, for example: software robots (RPA robots), intelligent interaction systems, browser plug-ins, application program extensions, Internet of Things (IoT) device controllers, human-robot collaborative robots, etc.

[0038] In the case of a software robot, an interactive intelligent agent facing end users understands user needs through natural language processing and operates the interface to complete services such as querying orders and modifying account information. The above intelligent agent can perform cross-system operation processing through system switching in the interface.

[0039] The intelligent interaction system described above can be used in intelligent cockpits, intelligent home central controls, and interaction hubs integrated in hardware devices (such as vehicle-mounted central control screens and intelligent home panels), which can receive voice / gesture instructions, perform operations in combination with interface visual analysis, for example, in a vehicle-mounted system, a user gives a voice instruction “open the search box of the navigation interface”, and the system automatically locates the interface element and triggers the interaction.

[0040] When interacting as a browser plug-in or application extension, the functional module embedded in the browser or third-party application is used to enhance the automatic interaction capability in specific scenarios (such as automatic filling of web forms and batch data extraction).

[0041] When interacting as an Internet of Things device controller, the middleware connecting the physical device and the digital interface is used for, for example, monitoring interface operation of industrial equipment and remote control interface interaction of smart home appliances, for example, in a factory central control system, through an instruction “adjust the temperature slider of the parameter panel of a certain production line”, the controller automatically locates the interface element and sends an adjustment instruction.

[0042] In the case of a human-machine collaborative robot, a collaborative robot combining a physical mechanical arm and interface interaction, for example, in a customer service scenario, the robot identifies the interface button through visual recognition, cooperates with the mechanical arm to click the operation, and simultaneously processes user inquiries.

[0043] According to the embodiments provided in the present application, the visual features of the operation interface are extracted, which reflect the appearance and layout of the operation interface and the interface elements included therein. The semantic features of the user instruction are extracted to understand the operation that the user wants to perform. The visual features and the semantic features are fused to obtain a comprehensive feature representation, i.e., the fused features. The region box regression module is trained by using reference interface elements with labeled reference region boxes, and can learn how to predict the correct interactive region from the visual and semantic features. Using the region box regression module, the target region box of the interface element is generated based on the fused features, and the interface element is framed out as the data basis for the subsequent operation instruction execution. According to the generated target region box, the corresponding operation instruction is generated to guide the subsequent interaction operation (such as clicking, inputting, etc.).

[0044] The interface element is represented by generating a target region frame, which is more consistent with actual operation requirements, and avoids the problem of unclear semantics in the regression point method provided by the related art. The region frame regression module can learn the overall interactive region of the interface element, rather than relying on a single labeled point. Even if the labeled points in the training data have jitter, the model can reduce the influence of such jitter by learning the region frame. Even if there is a certain deviation between the predicted region frame and the true region frame, as long as the interactive region is still covered, the operation can still be successful, avoiding the overfitting of the regression point method to the precise point. The identification requirement for interface elements with small icons in the operation interface has good adaptability, and the robustness of the model is improved.

[0045] Illustratively, the operation interface is screenshot at a predetermined interval, and feature extraction is performed based on the screenshots to determine the visual features. Optionally, the above-mentioned screenshot is a screenshot taken within a predetermined range, for example, for a certain application, a local screenshot is used to capture the interface of the application. Assuming that the operation interface includes sub-interfaces of multiple virtual machines, the above-mentioned screenshot method can be used to capture the interface in selected virtual machines.

[0046] Illustratively, before extracting the visual features, the input screen image can be preprocessed to ensure the quality and consistency of the image. For example, the screenshot can be scaled to a uniform size, or the pixel values can be normalized to a certain range (such as 0 to 1 or -1 to 1). Noise in the image can also be removed, for example, by Gaussian filtering or other methods to reduce interference. Important features in the image can be highlighted through contrast enhancement, sharpening, and other operations.

[0047] According to the embodiments provided in the present application, the output layer of the region frame regression module includes multiple regression heads, and the corner points of the target region frame correspond to different regression heads. In step S102, the target region frame corresponding to the current interface element is generated by using the region frame regression module based on the fusion features, which can include the following specific steps:

[0048] According to the fusion features, multiple regression heads are used to predict the initial coordinates of the corresponding corner points.

[0049] The initial coordinates are mapped to the corresponding corner point coordinates by the fully connected layer in the region frame regression module, so that the multiple corner point coordinates are normalized to a predetermined size range.

[0050] The target region frame is obtained according to the multiple corner point coordinates.

[0051] In the embodiments provided in the present application, the output layer of the region frame regression module includes multiple regression heads, and each two regression heads are responsible for predicting the initial coordinates of one corner point of the target region frame. Since one coordinate includes the horizontal coordinate x and the vertical coordinate y, for a rectangular region frame, there are two corner points (top left and bottom right). Therefore, two regression heads are used to predict the horizontal and vertical coordinates of the top left corner point, and similarly, two regression heads are used to predict the bottom right corner point, allowing the model to learn the features of each corner point respectively, thereby improving the prediction accuracy. Each regression head predicts the initial coordinates of the corresponding corner point based on the input fusion features. The initial coordinates can be regarded as the original prediction values without normalization processing. The region frame regression module is provided with a full connection layer for further mapping the initial coordinates of each corner point to the final corner point coordinates. In the mapping process, the full connection layer normalizes the corner point coordinates to a predetermined size range. The normalization processing makes the output of the model independent of the specific size of the input image, thereby improving the generalization ability of the model. According to the normalized multiple corner point coordinates, the target region frame is constructed, and the corner point coordinates define the shape and position of the region frame, thereby framing the interactive region of the current interface element.

[0052] Exemplarily, the number of corner points is 2, which is represented as (x_min, y_min, x_max, y_max), representing the top left and bottom right coordinates of the region frame. Two independent regression heads and a full connection layer are provided for each corner point. Compared with the single rectangular prediction method, this corner point prediction method can minimize the framed invalid region. It has stronger robustness in processing the deformation (such as rotation and scaling) of the target region frame.

[0053] Figure 2 A comparison diagram of the interface operation instruction generation method of the embodiments of the present application is shown. The point regression is the method provided in the related art, which represents the position of the corresponding interface element by outputting a single point coordinate. The frame regression is the scheme provided in the embodiments of the present application, which marks the region frame by corner point coordinates.

[0054] According to the embodiments provided in the present application, before step S102, the method can include the following specific steps:

[0055] Determine the region loss according to the degree of overlap between the reference region frame and the real region frame;

[0056] Determine the coordinate loss according to the deviation degree between the reference coordinates of the reference region frame and the real coordinates of the real region frame;

[0057] Determine the comprehensive loss based on the region loss and the coordinate loss;

[0058] Train the region frame regression module until the comprehensive loss is less than a predetermined loss threshold, and obtain the trained region frame regression module.

[0059] In the embodiments provided in the present application, the region loss is an index for measuring the degree of overlap between the predicted reference region frame and the real region frame. The coordinate loss is an index for measuring the degree of deviation between the coordinates of the predicted reference region frame and the coordinates of the real region frame. The comprehensive loss is a weighted sum of the region loss and the coordinate loss, and is used to comprehensively evaluate the performance of the model. By simultaneously considering the region loss and the coordinate loss, the model can more accurately predict the position and shape of the target region frame. In each training iteration, the comprehensive loss is calculated, and the model parameters are adjusted according to the loss value. By optimizing the comprehensive loss, the region frame regression module is trained until the comprehensive loss is less than a predetermined loss threshold, and the model training is considered to be completed, obtaining the trained region frame regression module. The introduction of the comprehensive loss enables the model to optimize the shape and position of the region frame simultaneously during the training process, avoiding the underfitting or overfitting problems that may be caused by a single loss function.

[0060] Exemplarily, the region frame regression module belongs to a VLA model for regressing the operation region of a Web GUI. The overall architecture of the VLA model includes five core modules, realizing a complete process from interface images and user instructions to operation instructions, including a visual encoder, a language encoder, a multi-modal fusion module, a region frame regression module, and an operation execution module.

[0061] The visual encoder is used to process the input GUI screen image, and extracts visual features such as icon shape, color, layout position, etc. in the image through a visual model such as a convolutional neural network (CNN). The language encoder is used to parse the user instruction text, and can extract semantic features by using natural language processing (NLP) technology to convert natural language instructions into semantic vectors, and explicitly express the user interaction intent. The multi-modal fusion module is used to fuse the visual features output by the visual encoder and the semantic features output by the language encoder, and generates unified fusion features through splicing or attention mechanism, etc. The unified fusion features can be used to eliminate the semantic gap between the visual and language modalities, so that the model can associate the semantic correspondence between “the button in the image” and the user instruction. The region frame regression module is used to predict the boundary frame coordinates of the interactive elements in the interface based on the multi-modal fusion features, instead of predicting traditional single-point coordinates. The region frame covers the complete interactive region (such as the clickable range of the button). The operation execution module is used to generate specific operation instructions according to the predicted boundary frame coordinates, combined with the element type (such as button, slider) and the operation type (such as click, drag), which can directly drive RPA robots, browser plug-ins and other automation tools to perform interactive operations on the operation interface.

[0062] Exemplarily, Intersection over Union (IoU) can be used to calculate the region loss, which is the ratio of the intersection area of the predicted region box and the true region box to the union area. Generalized IoU (GIoU), Distance-IoU (DIoU) can also be used for loss calculation. The goal of GIoU is to make the predicted region box not only close to the true box in area, but also closer in shape and size by introducing an additional penalty term. The goal of DIoU is to make the region box closer in position to the true box by penalizing the distance between the center points of the region box and the true box.

[0063] Exemplarily, Mean Squared Error (MSE) or Smooth L1 Loss is used to calculate the coordinate loss. Mean Squared Error measures the difference between the two by calculating the square mean of the difference between the reference coordinates and the true coordinates. Smooth L1 Loss is an improved version of L1 Loss, which approximates MSE when the error is small and approximates L1 when the error is large, avoiding the problem of gradient explosion.

[0064] Exemplarily, the parameters of the region box regression module are updated using backpropagation algorithms and optimizers such as Adaptive Moment Estimation (Adam) or Stochastic Gradient Descent (SGD). Adam dynamically adjusts the learning rate based on the historical gradient of the parameters, eliminating the need for manual adjustment of the learning rate of each parameter, and performs well on sparse data. SGD updates the model parameters of the region box regression module in the opposite direction of the gradient by calculating the gradient of each sample, and when the data volume is large, it can effectively converge to the vicinity of the global optimal solution.

[0065] Exemplarily, first, the visual and language encoders are pre-trained on a large-scale general dataset, then fine-tuned in the Web GUI dataset, and finally fine-tuned specifically for the region box regression task. A general dataset containing a large number of images and text, such as image-text pair data, is used to preliminarily train the visual encoder used by the model to extract image features and the language encoder used to understand text semantics, allowing the model to learn basic visual feature extraction capabilities, such as object recognition, color, shape, and basic language understanding capabilities, such as understanding text meaning and keyword extraction.

[0066] Further adjustment training is performed using a dataset specifically for web graphical user interfaces (Web GUI), such as buttons, input boxes, menus, and other interface elements in web pages. By applying a dataset specific to the interface, the model is adapted to the visual characteristics of web interfaces, with better recognition capabilities for interface layouts, icon styles, and the like. In combination with interactive language instructions such as "click the search button" and "slide the scrollbar", the model can better understand and process elements and instructions in web interfaces.

[0067] After completing the first two steps of training, the model is further optimized for the region box regression task, i.e., predicting the location region of interactive elements in the interface, with a region box labeling the clickable range of the button. Through a large amount of data labeled with interface element region boxes, the model is fine-tuned to accurately output the coordinate range of the element, improving the positioning accuracy and ultimately meeting the needs of automated interaction, interface analysis, and other tasks. Through the step-by-step training of "general pre-training → domain fine-tuning → task fine-tuning", the model gradually develops from basic capabilities to specialized capabilities for positioning web interface elements, ensuring accurate understanding of instructions and positioning of interactive regions in actual applications.

[0068] To improve training efficiency, the VLA model uses mixed precision training techniques, which reduce memory usage and computation time while maintaining model accuracy. Mixed precision training breaks the limitations of single data types by combining the use of 16-bit floating-point numbers (float16) and 32-bit floating-point numbers (float32) during model training. In forward propagation calculations, most convolution, matrix multiplication, and other operations can be performed using float16 data types. float16 occupies only half the memory of float32, is faster to calculate, and can significantly reduce computation time to speed up the training process. Since float16 has a smaller representation range, it is prone to numerical overflow or underflow issues, so float32 data types are still used in gradient calculation and normalization layers with high precision requirements to ensure the stability and final accuracy of the model training are not affected. By combining both, efficient training is achieved while maintaining model accuracy.

[0069] For the convenience of understanding, the visual encoder is preferably set to float16 precision, the convolutional neural network is not sensitive to precision, float16 can accelerate matrix operations, and the last fully connected layer of the visual encoder is preferably set to float32 to avoid loss of feature map details of the webpage screenshot. The language encoder is preferably set to float16 precision, and the text embedding layer (Embedding) is preferably used for user instruction text to maintain float32 to avoid distortion of word meaning. The Transformer layer used for natural language processing can use float16 to accelerate attention calculation. The multi-modal fusion module is preferably set to float32 precision, because cross-modal feature alignment is sensitive to small changes, and attention score calculation requires high precision to prevent overflow. The region regression module is preferably set to float32, which can be understood as that coordinate prediction requires sub-pixel level precision, and loss calculation requires float32 precision to ensure numerical stability. The precision of the operation execution module is preferably float16, the operation execution module determines the operation strategy based on the element type and the operation type, and high precision calculation is not required.

[0070] The model performance of the above target region frame depends on the richness of the training set. In order to improve the performance of the region frame regression module, the embodiments provided in the present application expand the training set through a data enhancement method, thereby improving the generalization ability and robustness of the model. According to the embodiments provided in the present application, the method can include the following steps:

[0071] Based on the reference interface element labeled with the reference region frame, an initial training set is constructed;

[0072] Adjust the reference region frame to obtain a plurality of data enhancement frames, and the plurality of data enhancement frames are bounded by the reference region frame as the maximum range boundary;

[0073] Based on the plurality of data enhancement frames, the initial training set is expanded to obtain a target training set for training the region frame regression module.

[0074] In the embodiments provided in the present application, the reference interface element labeled with the reference region frame is used as the initial training set, which serves as the basis for model training. By adjusting the reference region frame, a plurality of data enhancement frames are generated. The optional adjustments include translation, scaling, rotation, etc. All generated data enhancement frames are bounded by the reference region frame as the maximum range boundary, which is considered as the boundary of the interactive range. The plurality of data enhancement frames are all within the interactive range, which ensures that the agent can normally interact within the range selected by the interface element frame. The plurality of generated data enhancement frames are added to the initial training set as an expansion means to form a target training set.

[0075] By generating diverse training samples through data augmentation, the model can learn more variable region box shapes and positions, thereby improving its adaptability to different interface elements. The introduction of data augmentation boxes makes the model more robust in the face of noise and changes, and better able to handle complex scenarios in actual applications.

[0076] For example, for the above-mentioned multiple adjustment methods, the translation method is to randomly select a point in the reference region box as a new center point to generate a new region box. The scaling method is to randomly select a scaling factor in the reference region box to scale the reference region box. The rotation method is to randomly select an angle in the reference region box to rotate the reference region box, which can have good data augmentation effect for non-centrally symmetric reference region boxes.

[0077] For example, the enhancement processing of the training data is a key technical means to improve the generalization ability of the model. By simulating the diversity and variability of the interface in the real scene, the model can adapt to interface elements of different sizes, layouts, and visual styles. The Web GUI interface can present different visual forms due to factors such as device screen size (such as PC, tablet, mobile phone), browser scaling ratio, theme style (such as night mode, day mode, eye protection mode), etc. By simulating these changes through data augmentation, the model is prevented from over-relying on fixed button sizes, single color schemes, etc., which can result in training data only including specific features.

[0078] Therefore, other data augmentation processing can also be performed on the training data, such as random scaling, cropping, color jittering, blurring, etc. Random scaling randomly scales the interface image by a certain ratio (such as 0.8-1.2 times) to simulate the screen resolution of different devices or the scaling effect of the browser, so that the model can adapt to changes in element size and avoid memorizing only buttons of a fixed pixel size. Random cropping randomly crops a local region from the interface image as a new sample to simulate interface scrolling or user perspective changes, so that the model can locate elements from local visual features and improve its retrieval ability for target elements in complex interfaces. Color jittering can be brightness / contrast adjustment, which randomly adjusts the brightness and contrast of the image according to a predetermined ratio to simulate different lighting conditions or theme modes, so that the model is not sensitive to changes in color brightness and focuses on the shape and structural features of the elements. Gaussian blur can also be added to add Gaussian noise or blur effects to the image to simulate low-resolution screens or visual blurring of interface elements, such as background blurring caused by popup overlays, to enhance the robustness of the model to low-quality images or visual disturbances.

[0079] In the embodiments provided in the present application, based on the reference interface elements labeled with reference region boxes, an initial training set is constructed, including:

[0080] Based on the historical user instructions, historical semantic features are extracted;

[0081] According to the element type and the operation type of the reference interface element, a reference interface element marked with a type is obtained;

[0082] According to the reference interface element marked with a type and a reference region frame, and the historical semantic features, an initial training set is generated.

[0083] In the embodiments provided in the present application, in order to improve the performance of the region frame regression module, not only the reference region frame needs to be marked, but also the semantic features of the historical user instructions, and the type and operation type of the interface element need to be combined to construct a more rich and representative initial training set. The instructions issued by the user are extracted from the historical interaction records, and these instructions can be natural language description operations (such as “click the login button”, “input the username”, etc.). The user instructions are converted into semantic feature vectors. These feature vectors can capture the semantic information of the user instructions. The reference interface element is classified, and its type (such as button, text box, slider, etc.) and the operation type related to the interface element (such as click, input text, drag, etc.) are marked. The interface element marked with the type is combined with the corresponding operation type to form the marked data. The interface element marked with the type and the reference region frame is combined with the historical semantic features to form the initial training set. Each training sample can include the visual features of the interface element, the marked type and operation type, the coordinates of the reference region frame, and the historical semantic features.

[0084] By combining the historical semantic features and the type and operation type of the interface element, the model can learn more rich context information, thereby improving the adaptability to different scenarios. The rich annotation information enables the model to better understand the semantics and operation requirements of the interface element, thereby being more robust in the face of noise and changes.

[0085] Exemplarily, a language encoder is trained using natural language processing (NLP) techniques such as Word Embedding, BERT (Bidirectional Encoder Representations from Transformers), etc., to obtain semantic features. Text information can be converted into numerical features that can be understood and processed by machines, thereby providing support for subsequent tasks (such as training of the region box regression module). Word Embedding is a technique of mapping words to high-dimensional vector spaces, which can capture semantic relationships between words. Common word embedding methods include Word2Vec, GloVe, etc. Word2Vec learns vector representations of words through contextual information. GloVe (Global Vectors for Word Representation) learns vector representations of words through global word frequency statistics, combining local contextual information and global word frequency statistics to better capture the semantics of words.

[0086] The language encoder of the BERT network can also be pre-trained through a large amount of text data to capture deep semantic information of language. BERT uses a bidirectional Transformer encoder that can simultaneously consider forward and backward information in the context, learns language representations by randomly masking some words in the input sentence and predicting these masked words. This is conducive to the language encoder adapting to various natural language tasks and has strong generalization ability.

[0087] According to the embodiments provided in the present application, the method further comprises:

[0088] Based on the element type of the reference interface element, the visual boundary and the interactable area of the reference interface element are determined.

[0089] According to the visual boundary and the interactable area, the reference region box is labeled for the reference interface element.

[0090] In the embodiments provided in the present application, the display range of the element in the interface is determined according to the visualization feature of the element (such as the icon range of the button, the border of the input box). For example, the visual boundary of the button is the icon pixel range, and the visual boundary of the slider includes the track and the handle. According to the overlapping part of the visual boundary and the interactable area, the interactable range of the element is labeled using a rectangular box. By judging the interaction characteristics of the element according to the element type, the labeled reference region box not only covers the visual display range, but also accurately reflects the actual operable area, thereby providing accurate labels for the training of the subsequent region box regression module.

[0091] According to the embodiments provided in the present application, in step S103, generating an operation instruction based on the target region box to perform an operation can include the following specific steps:

[0092] determine an operation strategy for the target area frame based on the element type of the interface element and the operation type, the operation strategy including an operation position and / or an operation direction for the target area frame;

[0093] generate an operation instruction based on the target area frame according to the operation strategy.

[0094] In the embodiments provided in the present application, according to the element type (such as button, text box, slider, etc.) of the interface element and the operation type (such as clicking, inputting text, dragging, etc.) specified by the user, a specific operation strategy can be determined. It can be understood that for the operation position, the center position is usually selected for clicking for the button; the upper left corner position can be selected for text input for the text box. For the operation direction, if the operation needs directionality, such as dragging the slider, the direction of the operation is determined, such as left, right, up, down, and from one specified direction to another specified direction. According to the determined operation strategy, a specific operation instruction is generated for guiding subsequent automated operation. Different interface elements can require different operation strategies. By dynamically adjusting the operation strategy according to the element type and the operation type, various complex scenarios can be better adapted to, and the robustness in different interfaces and operation tasks can be improved.

[0095] By determining the operation strategy, the most suitable operation position and direction can be selected according to the type of the interface element and the operation type. Blind selection of the operation point can be avoided, which can cause incorrect operation, so that the automated operation is more in line with the interaction habits of human users, for example, avoiding clicking on the edge of the button which can cause operation failure.

[0096] Illustratively, the above operation strategy generally selects the center point of the area frame as the operation point, that is, the operation strategy of the area frame is set in a default form, which is realized by global setting. Specifically, the operation strategy generated based on the element type and the operation type can include the following forms, for example: for the clicking operation, the center point of the area frame is used to perform the clicking processing. For the dragging operation, the center points of the starting area frame and the target area frame are used to determine the starting point and the ending point of the dragging. For text input, a predetermined position in the input box area is selected for input, such as the default cursor position or the upper left position. For the scrolling operation, according to the scrolling direction (such as vertical or horizontal) and the range of the area frame, the starting point and the ending point of the scrolling are determined.

[0097] According to the above embodiments and optional embodiments, the application further provides an optional implementation. For the VLA model of the Web GUI regression operation area, the method of region box regression is used to replace the traditional point regression mode. The overall architecture of the VLA model includes the following main parts, including a visual encoder (Visual Encoder) responsible for processing the input GUI screen image and extracting visual features; a language encoder (Language Encoder) processing user instructions and extracting semantic understanding; a multimodal fusion module (Multimodal Fusion Module) fusing visual and language features; a bounding box regression module (Bounding Box Regression Module) predicting the boundary box of the operation area instead of single point prediction; and an operation execution module (Action Execution Module) generating actual operation instructions according to the predicted region box.

[0098] Unlike the traditional regression point mode, the optional implementation requires the model to output a region box, which is represented by two corner point coordinates with four coordinate values, denoted as (x_min, y_min, x_max, y_max), representing the coordinates of the upper left corner and the lower right corner of the region box.

[0099] For the data labeling strategy used by the region box regression module, the GUI elements in the training data are no longer labeled with a single point, but with the boundary box of the entire operable region. This labeling method is more consistent with the semantics of actual operation and is also more consistent. For different types of interface elements such as buttons, links, and input boxes, the corresponding boundary box is determined according to their visual boundaries and interactive regions.

[0100] The output layer of the region box regression module is designed as four regression heads, which respectively predict the four coordinate values of the region box (x_min, y_min, x_max, y_max). These regression heads can be implemented using fully connected layers or convolutional layers, and the output values are normalized to the image size range.

[0101] The loss function of the region box regression adopts the Intersection over Union (IoU) loss or its variants (such as GIoU, DIoU, etc.). These loss functions directly optimize the overlap between the predicted box and the true box, which is more consistent with the region semantics. At the same time, the Smooth L1 Loss can be combined to optimize the accuracy of the coordinate values.

[0102] The optional embodiment optimizes the model performance in the following way: the training data is enhanced, including random scaling, cropping, color transformation, etc., to increase the generalization ability of the model. In particular, the size and position of the bounding box are randomly adjusted (but kept within the valid area) to enhance the model's understanding of the region rather than specific points.

[0103] A multi-stage training strategy is adopted: first, pre-train the visual and language encoders on a large-scale general dataset, then fine-tune on the Web GUI dataset, and finally fine-tune specifically for the region box regression task.

[0104] To improve training efficiency, mixed precision training technology is used to reduce memory usage and computation time while maintaining model accuracy.

[0105] In practical applications, the predicted region box obtained after model inference needs to be converted into specific operation instructions. The general strategy is to select the center point of the region box as the operation point, but more detailed adjustments can be made according to the specific interface element type and operation type. For click operations, the center point of the region box is used to perform click processing. For drag operations, the center points of the starting region box and the target region box are used to determine the starting point and ending point of the drag. For text input, a predetermined position within the input box region is selected for input, such as the default cursor position or the top-left position. For scrolling operations, the starting point and ending point of the scroll are determined based on the scroll direction (e.g., vertical or horizontal) and the range of the region box.

[0106] Experimental results show that the region box regression method improves the average accuracy of the model by 15%-20% compared to point regression, especially in complex interfaces and small element positioning.

[0107] Compared to traditional regression point methods, the VLA model based on region box regression proposed in this application has the following significant advantages: the region box regression method directly corresponds to the operable region of the GUI element, with clear semantic meaning and more consistent with the actual operation logic, eliminating the problem of ambiguous semantics in the point regression method; by labeling consistent region boxes, the problem of jittering of the same element annotation points in different samples is avoided, providing a more stable and consistent learning goal for the model, reducing training noise, and improving convergence speed and stability.

[0108] The region box prediction provides more spatial information, and even if the predicted region is partially deviated from the real region, the operation can still be successful as long as there is sufficient overlap, greatly enhancing the robustness of the model in actual application. The region box method can better adapt to interface elements of different sizes and shapes, effectively processing from small icons to large regions, and providing a foundation for subsequent expansion to more complex interactive operations (such as dragging, region selection, etc.). The method is not only suitable for Web GUI interaction, but can also be easily extended to mobile applications, desktop software and other GUI environments, and has wide application prospects.

[0109] By innovatively using the region box regression method to replace the traditional point regression method, the problems of ambiguous semantics and true value jitter in the prior art are effectively solved, and the training efficiency, prediction accuracy and operation robustness of the model are significantly improved. The technology has important application value and industrialization prospects in the fields of computer vision, natural language processing and human-computer interaction.

[0110] Figure 3 The schematic diagram of the interface operation instruction generation device of the embodiment of the present application is shown in Figure 3 According to the application scenario and method of the embodiment of the present application, the embodiment of the present application further provides an interface operation instruction generation device, which comprises:

[0111] The feature generation module 301 is configured to obtain fused features based on the visual features of the operation interface and the current semantic features of the user instruction.

[0112] The box regression module 302 is configured to generate a target region box corresponding to the current interface element by using a region box regression module according to the fused features, the current interface element being an interactive element in the operation interface, and the target region box being used to represent the interactive region of the current interface element, the region box regression module being trained based on reference interface elements labeled with reference region boxes.

[0113] The instruction generation module 303 is configured to generate an operation instruction for executing an operation based on the target region box.

[0114] The functions of each module in each device of the embodiment of the present application can be referred to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be described here.

[0115] Figure 4 The block diagram of an electronic device for implementing the embodiment of the present application is shown in Figure 4As shown, the electronic device includes a memory 401 and a processor 402, and the memory 401 stores a computer program capable of running on the processor 402. The processor 402 implements the method in the above embodiments when executing the computer program. The number of the memory 401 and the processor 402 can be one or more. In a specific implementation, the electronic device can further include a communication interface 403 for communicating with external devices and transmitting data.

[0116] In a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are independently implemented, the memory 401, the processor 402 and the communication interface 403 can be connected to each other through a bus and complete communication therebetween. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0117] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete communication therebetween through an internal interface.

[0118] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0119] The embodiment of the present application provides a computer program product, which includes a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0120] The embodiment of the present application further provides a chip, which includes a processor, is used for calling and running instructions stored in a memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0121] The embodiment of the present application further provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0122] It should be appreciated that the above processor can be a Central Processing Unit (CPU), but can also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or the like. The general purpose processor can be a microprocessor or any conventional processor, and the like. It is worthy to note that the processor can be a processor that supports an Advanced RISC Machines (ARM) architecture.

[0123] Further, the above memory can optionally include a Read-Only Memory (ROM) and a Random Access Memory (RAM). The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can include a Random Access Memory (RAM) used as an external cache memory. By way of example, but not limitation, many forms of RAM are available. For example, a Static Random Access Memory (SRAM), a Dynamic Random Access Memory (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Sync Link DRAM (SLDRAM), and a Direct Rambus RAM (DR RAM) can be used.

[0124] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0125] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0126] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0127] Any process or method described in the flowchart or otherwise described herein can be understood as a representation of code including one or more executable instructions for performing a specific logical function or process. Also, the scope of the preferred embodiments of the present disclosure includes additional implementations, in which the functions can be performed in an order different from that shown or discussed, including functions performed in a substantially simultaneous manner, or in reverse order.

[0128] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can take instructions from an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0129] It should be understood that each part of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium, and the program includes one of the steps of the method embodiment or a combination thereof when executed.

[0130] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0131] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An interface operation instruction generation method characterized by comprising: The method comprises: obtaining a fusion feature based on a visual feature of an operation interface and a current semantic feature of a user instruction; generating a target bounding box corresponding to a current interface element in the operation interface by using a bounding box regression module according to the fusion feature, the current interface element being an element in the operation interface that can be interacted with, and the target bounding box being used to represent an interactable region of the current interface element, the bounding box regression module being trained based on a reference interface element with a labeled reference bounding box; generating an operation instruction for performing an operation based on the target bounding box; the method further comprises: determining a region loss according to an overlap degree between the reference bounding box and a real bounding box; determining a coordinate loss according to a deviation degree between a reference coordinate of the reference bounding box and a real coordinate of the real bounding box; determining a comprehensive loss based on the region loss and the coordinate loss; training the bounding box regression module until the comprehensive loss is less than a predetermined loss threshold to obtain a trained bounding box regression module; wherein the region loss is obtained using an intersection-over-union ratio, the intersection-over-union ratio being a ratio of an intersection area of a predicted bounding box to a union area of the predicted bounding box and a real bounding box; or by introducing a penalty term to make the predicted bounding box close to the real bounding box in area, and close to the real bounding box in shape and size; or by penalizing the distance between the center points of the predicted bounding box and the real bounding box to make the predicted bounding box closer to the real bounding box in position; the method further comprises: constructing an initial training set based on the reference interface element with the labeled reference bounding box; adjusting the reference bounding box to obtain a plurality of data enhancement boxes, the plurality of data enhancement boxes being bounded by the reference bounding box as the maximum range; expanding the initial training set based on the plurality of data enhancement boxes to obtain a target training set for training the bounding box regression module.

2. The method of claim 1, wherein, The output layer of the bounding box regression module comprises a plurality of regression heads, different regression heads corresponding to the corner points of the target bounding box, and the bounding box regression module generating the target bounding box corresponding to the current interface element according to the fusion feature comprises: predicting initial coordinates of the corresponding corner points by using the plurality of regression heads according to the fusion feature; mapping the initial coordinates to the corresponding corner point coordinates by a fully connected layer in the bounding box regression module to normalize the plurality of corner point coordinates to a predetermined size range; obtaining the target bounding box according to the plurality of corner point coordinates.

3. The method of claim 1, wherein, The method of generating an operation instruction for performing an operation based on the target bounding box comprises: determining an operation strategy for the target bounding box based on an element type and an operation type of the interface element, the operation strategy comprising an operation position and / or an operation direction for the target bounding box; generating the operation instruction based on the target bounding box according to the operation strategy.

4. The method of claim 1, wherein, The method of constructing an initial training set based on the reference interface element with the labeled reference bounding box comprises: extracting historical semantic features based on historical user instructions; obtaining the reference interface element with the labeled type according to an element type and an operation type of the reference interface element. According to the reference interface element labeled with the type and the reference region box, and the historical semantic features, the initial training set is generated.

5. The method of claim 1, wherein, The method further comprises: determining a visual boundary and an interactive region of the reference interface element based on an element type of the reference interface element; labeling the reference region box for the reference interface element according to the visual boundary and the interactive region. 6.An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1 to 5 when executing the computer program. 7.A computer readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 5. 8.A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 5.

Images