User interface positioning method, apparatus, device, and storage medium

Abstract

The application provides a user interface positioning method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a natural language instruction and a user interface image; generating a query key according to the natural language instruction and the user interface image; querying the feature cache database by using the query key; and if a cache feature corresponding to the target element is matched in the feature cache database, a first positioning mode is used for target element positioning. Resource consumption is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a user interface positioning method, apparatus, device, and storage medium. Background Technology

[0002] In scenarios such as User Interface (UI) automation testing and Robotic Process Automation (RPA), it is typically necessary to input natural language commands and the current user interface image into a large-scale Artificial Intelligence (AI) model. The AI ​​model then uses its visual understanding capabilities to identify target elements and return their screen coordinates, achieving UI element localization driven by natural language. Existing technologies also include localization schemes based on multimodal fusion. These schemes simultaneously acquire UI hierarchy structure or Document Object Model (DOM) tree information along with screenshots and feed them into a large-scale multimodal model for comprehensive reasoning and localization. By fusing visual features, structural information, and semantic understanding, they improve localization accuracy.

[0003] However, all of the above solutions require calling large AI models for inference calculations each time a location is located, resulting in high resource consumption. Summary of the Invention

[0004] This application provides a user interface positioning method, apparatus, device, and storage medium to solve the problem of high resource consumption caused by calling a large AI model for inference calculations every time positioning is performed in the prior art, thereby reducing resource consumption.

[0005] This application provides a user interface positioning method, including the following steps: Acquire natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface; Based on the natural language instructions and the user interface image, a query key is generated. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Query the feature cache database using the query key; If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

[0006] In some embodiments, it also includes: If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0007] In some embodiments, generating a query key based on the natural language instruction and the user interface image includes: The natural language instruction is semantically encoded to generate a semantic vector, which is used to indicate the semantic information of the natural language instruction; A page signature is calculated on the user interface image to generate a page signature, which is used to indicate the visual features of the user interface image; The query key is generated based on the semantic vector and the page signature.

[0008] In some embodiments, the step of performing page signature calculation on the user interface image to generate a page signature includes: Edge detection is performed on the user interface image to obtain the edge density distribution; The edge density distribution is meshed to obtain meshed edge density features; Generate a feature vector based on the meshed edge density features; The feature vector is encoded using local sensitive hashing and perceptual hashing to generate the page signature.

[0009] In some embodiments, generating the query key based on the semantic vector and the page signature includes: The semantic vector is binarized or hash-encoded to obtain a semantic hash value; The semantic hash value and the page signature are combined to generate the query key.

[0010] In some embodiments, querying the feature cache database using the query key includes: Perform an exact match query in the feature cache database using the query key; and, Based on the semantic vector of the natural language instruction, a semantic similarity matching query is performed in the feature cache database.

[0011] In some embodiments, the positioning of the target element using the first positioning method includes: Obtain the first local feature descriptor of the target element from the matched cached features; Local feature extraction is performed on the user interface image to obtain a second local feature descriptor; The first local feature descriptor and the second local feature descriptor are matched to obtain matching point pairs; Calculate the geometric transformation matrix based on the matching point pairs; The positioning coordinates of the target element in the user interface image are calculated based on the geometric transformation matrix.

[0012] In some embodiments, matching the first local feature descriptor and the second local feature descriptor to obtain a matching point pair includes: Calculate the distance between each feature point in the first local feature descriptor and the second local feature descriptor; Initial matching point pairs are selected based on the distance using a ratio test; Spatial consistency verification is performed on the initial matching point pairs to eliminate erroneous matching point pairs and obtain the final matching point pairs.

[0013] In some embodiments, the second positioning method for locating the target element includes: The AI ​​big model is invoked to identify the target element based on the natural language instructions and the user interface image, and the location coordinates of the target element returned by the AI ​​big model are obtained; Based on the positioning coordinates, the image region of the target element is cropped from the user interface image, and local feature extraction is performed on the image region to generate the cached features of the target element; The cache features of the target element are stored in the feature cache database along with the target element.

[0014] In some embodiments, it also includes: After locating the target element using the second positioning method, in response to obtaining a user interface image containing the target element again, the feature cache database is queried using the query key, and if a cache feature corresponding to the target element is matched, the target element is located using the first positioning method.

[0015] In some embodiments, it also includes: When using the first positioning method to locate the target element, if it is determined that the matched cached feature is invalid, the second positioning method is switched to re-acquire the positioning coordinates and cached features of the target element, and the feature cache database is updated according to the degree of change of the target element.

[0016] In some embodiments, updating the feature cache database based on the degree of change of the target element includes: If the degree of change is lower than a preset threshold, the newly acquired cached features will be fused with the existing cached features in the feature cache database to update the feature cache database in an incremental manner. If the degree of change is equal to or greater than the preset threshold, the existing cached features in the feature cache database are replaced with newly acquired cached features.

[0017] In some embodiments, the cache feature is determined to be invalid by at least one of the following methods: After performing local feature extraction on the user interface image, the number of matching point pairs obtained by matching with the cached features does not reach a preset threshold; or, The location coordinates of the target element calculated based on the matching point pair exceed the display range of the user interface image, or the area of ​​the bounding box corresponding to the location coordinates changes by more than a preset proportion compared to the area of ​​the historical bounding box associated with the cached feature; or, The difference between the page signature in the query key and the current page signature calculated from the user interface image exceeds a preset threshold; or... The cached feature exists for a period exceeding the preset validity period.

[0018] In some embodiments, it also includes: When locating multiple target elements using the first positioning method, a local topology map is constructed based on the relative spatial relationships between the multiple target elements; The current relative spatial relationships between target elements in the current positioning results are compared with the corresponding historical relative spatial relationships in the local topology map; When the current relative spatial relationship deviates from the historical relative spatial relationship by more than a preset threshold, relocation is triggered or the confidence level of the positioning result is reduced.

[0019] This application also provides a user interface positioning device, including the following modules: An information acquisition module is used to acquire natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface; The query generation module is used to generate a query key based on the natural language instruction and the user interface image. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. The information query module is used to query the feature cache database using the query key; The first processing module is configured to locate the target element using a first positioning method if a cached feature corresponding to the target element is matched in the feature cache database. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

[0020] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the user interface positioning method as described above.

[0021] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the user interface positioning method as described above.

[0022] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the user interface positioning method as described above.

[0023] The user interface positioning method, apparatus, device, and storage medium provided in this application cache the features of the target element after the initial location by calling the AI ​​large model, and calculate the positioning coordinates of the target element by using the cached visual features through local feature matching during subsequent location, thereby reducing the number of calls to the AI ​​large model and thus reducing resource consumption. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the user interface positioning method provided in this application.

[0026] Figure 2 This is a system architecture diagram of the intelligent caching middleware provided in this application.

[0027] Figure 3 This is a complete workflow diagram of the intelligent caching middleware provided in this application.

[0028] Figure 4 This is a schematic diagram of the user interface positioning device provided in this application.

[0029] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] The user interface positioning method, apparatus, device, and storage medium of this application are described below with reference to the accompanying drawings.

[0032] Figure 1 This is a flowchart illustrating the user interface positioning method provided in this application. Figure 1 As shown, the method includes the following: Step 101: Obtain natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface.

[0033] Natural language instructions refer to user commands issued in natural language, such as "click the login button," "enter your phone number in the search box," and "click the plus icon in the upper right corner of the page." User interface images are screenshots of the current display of the application being operated on, such as an application (APP), a mini-program, or a World Wide Web (WEB) page. These images serve as visual references for subsequently locating target elements.

[0034] It should be noted that natural language instructions can originate from preset operation steps in automated test scripts or from control commands input by the user in real time; this application does not impose any limitations on this. User interface images can be obtained by calling screenshot interfaces provided by the operating system (such as Android's Screenshot API, iOS's UIScreen snapshot, and the web's html2canvas, etc.), and the obtained image format can be common formats such as JPEG and PNG; this application also does not impose any limitations on this.

[0035] Step 102: Generate a query key based on the natural language instruction and the user interface image. The query key is used to query the feature cache database, which stores feature information corresponding to different user interface elements.

[0036] The query key is a crucial identifier used to retrieve cached features of a target element from the feature cache database. The feature cache database is a pre-built database for storing UI element feature information. This stored feature information includes at least local feature descriptors (such as ORB features, SIFT features, or SURF features), and may also include semantic vectors and other information.

[0037] It should be noted that the query key is generated based on both the semantic information of the natural language instruction and the visual information of the user interface image. This ensures that the query key reflects both the semantic meaning of the instruction and the visual context of the current interface, thereby improving the accuracy of cached queries.

[0038] Step 103: Query the feature cache database using the query key.

[0039] Among them, querying refers to using the generated query key to perform a retrieval operation in the feature cache database to determine whether there is a cache record in the feature cache database that matches the query key.

[0040] It should be noted that querying the feature cache database can employ a strategy combining multiple matching methods to balance the accuracy of exact matching with the flexibility of semantic generalization. Matching results are obtained by matching the query key with records in the cache database, and these results are used for decision-making in subsequent steps.

[0041] Step 104: If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

[0042] The first positioning method refers to the fast positioning method used when the cache is hit. The cache features store the historical visual feature information of the target element.

[0043] It should be noted that when a cached feature is matched, the first positioning method is adopted, and feature matching is performed on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

[0044] The above embodiments implement a cache reuse mechanism by introducing a feature cache database. That is, the target element features are learned and cached through the AI ​​large model in the first step, and the localization is completed through local feature matching in the subsequent step, thereby reducing the number of times the AI ​​large model is called and reducing resource consumption.

[0045] In some embodiments, it also includes: If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0046] The second positioning method refers to the positioning method used when the cache is missed. The AI ​​large model refers to a multimodal large model with visual understanding capabilities.

[0047] It should be noted that when no cached feature is matched, a second positioning method is adopted, which calls the AI ​​large model to identify the target element to obtain the positioning coordinates, and generates feature information based on the positioning coordinates and stores it in the feature cache database.

[0048] The above embodiments achieve "learn once, reuse many times" by calling the large model to learn features and storing them in the database on demand when the cache is missed. This effectively reduces repeated calls to the large model, lowers resource consumption and response latency.

[0049] In some embodiments, generating a query key based on the natural language instruction and the user interface image includes: The natural language instruction is semantically encoded to generate a semantic vector, which is used to indicate the semantic information of the natural language instruction; A page signature is calculated on the user interface image to generate a page signature, which is used to indicate the visual features of the user interface image; The query key is generated based on the semantic vector and the page signature.

[0050] Semantic vectors are numerical representations of natural language instructions mapped to a vector space through a semantic model. They characterize the semantic meaning of the instructions, ensuring that different expressions of the same intent are closely related in the vector space. Page signatures are identifying information used to characterize the visual features of user interface images, reflecting the layout and visual structure of the current page.

[0051] It should be noted that when semantically encoding natural language instructions, lightweight semantic models (such as Sentence-BERT or a general sentence encoder) can be used to encode the instruction text into fixed-dimensional semantic vectors, and L2 normalization can be performed to make subsequent similarity calculations more accurate. When calculating page signatures for user interface images, edge detection and gridded edge density calculation can be performed on the image, followed by locality-sensitive hashing and perceptual hashing encoding to generate the page signature. The query key can be a combination of the semantic vector and the page signature; for example, the hash value of the semantic vector can be concatenated with the page signature to form a unique query key. Other combinations are also possible, and this application does not limit this approach.

[0052] In the above embodiments, the query key integrates the semantic information of natural language instructions and the visual information of user interface images, enabling cached queries to balance the accuracy of semantic understanding and the adaptability to the page context.

[0053] In some embodiments, the step of performing page signature calculation on the user interface image to generate a page signature includes: Edge detection is performed on the user interface image to obtain the edge density distribution; The edge density distribution is meshed to obtain meshed edge density features; Generate a feature vector based on the meshed edge density features; The feature vector is encoded using local sensitive hashing and perceptual hashing to generate the page signature.

[0054] Edge density distribution refers to the distribution of edge intensity in each pixel region after edge detection of a user interface image, used to characterize the image's contour information. Meshable edge density features refer to the distribution of edge density within each grid region after dividing the image into multiple grid regions, used to characterize the image's spatial layout features. The feature vector is a numerical vector form converted from the meshable edge density features. Locality-Sensitive Hashing (LSH) is a technique that maps high-dimensional data to hash values, enabling similar data to remain adjacent in the hash space. Perceptual Hashing (pHash) is an algorithm that generates hash values ​​based on image frequency domain features, capable of resisting slight image transformations.

[0055] It should be noted that edge detection can be implemented using algorithms such as Canny edge detection and the Sobel operator. This involves detecting locations in the image where pixel values ​​change drastically to extract the contour information of UI elements. Mesh calculation refers to dividing the image into multiple grids (e.g., 8×8 or 16×16 grids), calculating the edge density within each grid region, and generating meshed edge density features. These features are then unfolded into a one-dimensional feature vector, and subsequently subjected to both locality-sensitive hashing (LSH) and perceptual hashing. The combined hash results generate the final page signature.

[0056] In the above embodiments, through multi-layer processing of edge detection, meshing, locality-sensitive hashing, and perceptual hashing, the visual features of the user interface image are compressed into a page signature, so that the page signature can maintain the recognizability of the page structure while having a certain tolerance for minor changes in the UI.

[0057] In some embodiments, generating the query key based on the semantic vector and the page signature includes: The semantic vector is binarized or hash-encoded to obtain a semantic hash value; The semantic hash value and the page signature are combined to generate the query key.

[0058] The semantic hash value is a hash value obtained by converting the semantic vector through binarization or hash encoding, and is used for efficient retrieval during cache queries. The page signature is a visual identifier of the user interface image.

[0059] It should be noted that, in generating semantic hash values, one implementation can compare each dimension of the semantic vector with a preset threshold; values ​​greater than the threshold are assigned a value of 1, and values ​​less than the threshold are assigned a value of 0, resulting in a binary semantic hash value. In another implementation, Locality Sensitive Hashing (LSH) can be used to encode the semantic vector to obtain the semantic hash value. After obtaining the semantic hash value and the page signature, the two can be combined to generate a query key. For example, in one implementation, the semantic hash value can be used as a prefix and the page signature as a suffix for string concatenation to form the query key; alternatively, a bitwise XOR operation can be performed between the two to generate the query key; or the two can be concatenated and hashed again to generate the query key. The specific combination method of the query key can be flexibly selected according to the actual application scenario, as long as the same semantic vector and page signature can generate the same query key; this application does not impose any limitations on this.

[0060] In the above embodiments, by hashing the semantic vector and combining it with the page signature to generate a query key, high-dimensional semantic information and page visual information are compressed into a key-value form that can be used for fast retrieval, thereby improving the efficiency of cached queries.

[0061] In some embodiments, querying the feature cache database using the query key includes: Perform an exact match query in the feature cache database using the query key; and, Based on the semantic vector of the natural language instruction, a semantic similarity matching query is performed in the feature cache database.

[0062] Among them, exact match query refers to searching for cached records with identical key values ​​in the feature cache database using the query key as an index. Semantic similarity match query refers to calculating the similarity between the semantic vector of a natural language instruction and the semantic vector associated with an existing cached record in the feature cache database, based on the semantic vector of the instruction. A match is considered successful when the similarity exceeds a preset threshold.

[0063] It should be noted that the two matching query modes described above can be executed sequentially. In one implementation, an exact match query is executed first; if the exact match fails, a semantic similarity match query is then executed. In another implementation, both matching queries can be executed simultaneously, and the decision is made by combining the results of the two matches; this application does not limit this approach. Semantic similarity can be measured using methods such as cosine similarity or Euclidean distance. For example, for the instruction "Click the login button," if a cached record of the "login" instruction already exists in the feature cache database, and the cosine similarity between the semantic vectors of the two is greater than a preset threshold, then a semantic similarity match is achieved, thereby enabling cache reuse of different representations of the same element.

[0064] In the above embodiments, the dual-mode query strategy that combines exact matching and semantic similarity matching can ensure high efficiency when using exact matching, and also support semantic generalization and reuse of the same element in different ways, thereby improving the cache hit rate.

[0065] In some embodiments, the positioning of the target element using the first positioning method includes: Obtain the first local feature descriptor of the target element from the matched cached features; Local feature extraction is performed on the user interface image to obtain a second local feature descriptor; The first local feature descriptor and the second local feature descriptor are matched to obtain matching point pairs; Calculate the geometric transformation matrix based on the matching point pairs; The positioning coordinates of the target element in the user interface image are calculated based on the geometric transformation matrix.

[0066] Local feature descriptors are numerical vectors used to characterize visual features such as texture, edges, and gradients in local regions of an image. Common local feature descriptors include ORB (Oriented Fast and Rotated BRIEF), SIFT (Scale-Invariant Feature Transform), and SURF (Speeded Up Robust Features). The first local feature descriptor is a historical feature obtained from cached features, while the second local feature descriptor is a real-time feature extracted from the current user interface image. Matching point pairs refer to the correspondence between matching feature points in the first and second local feature descriptors. Geometric transformation matrices describe the spatial mapping relationship between two planar images, such as homography matrices, affine transformation matrices, or similarity transformation matrices.

[0067] It should be noted that the specific implementation of feature matching can be as follows: Calculate the distance (e.g., Hamming distance or Euclidean distance) between each feature point in the first local feature descriptor and the second local feature descriptor. Based on the distance magnitude, use a ratio test to select initial matching point pairs. Then, perform spatial consistency verification on the initial matching point pairs, eliminate mismatched point pairs, and obtain the final matching point pairs. The geometric transformation matrix can be calculated using the RANSAC (Random Sample Consensus) algorithm to iteratively solve for the matching point pairs to obtain the optimal geometric transformation matrix. After obtaining the geometric transformation matrix, substitute the historical bounding box coordinates of the target element recorded in the cached features into the geometric transformation matrix for coordinate mapping, and the precise positioning coordinates of the target element in the current user interface image can be calculated.

[0068] In the above embodiments, fast and accurate localization based on cached features is achieved through local feature matching and geometric transformation matrix calculation, which can obtain the precise coordinates of the target element without calling a large AI model.

[0069] In some embodiments, matching the first local feature descriptor and the second local feature descriptor to obtain a matching point pair includes: Calculate the distance between each feature point in the first local feature descriptor and the second local feature descriptor; Initial matching point pairs are selected based on the distance using a ratio test; Spatial consistency verification is performed on the initial matching point pairs to eliminate erroneous matching point pairs and obtain the final matching point pairs.

[0070] Here, the distance between feature points refers to a similarity measure between two local feature descriptors. Ratio testing is a strategy that filters reliable matching point pairs by comparing the ratio of the nearest neighbor distance to the second nearest neighbor distance for each feature point, used to initially eliminate ambiguous matches (i.e., ambiguous matches caused by multiple similar feature points). Spatial consistency verification is a strategy that further eliminates mismatched point pairs by checking whether the geometric spatial relationship between matching point pairs is consistent, used to ensure that the remaining matching point pairs conform to a uniform geometric transformation law in spatial distribution.

[0071] It should be noted that the distance measurement method between feature points is related to the type of feature used. For example, when the local feature descriptor is a binary feature descriptor such as ORB or BRIEF, the distance can be measured using Hamming distance; when the local feature descriptor is a floating-point feature descriptor such as SIFT or SURF, the distance can be measured using Euclidean distance. In the ratio test, for each feature point in the first local feature descriptor, the nearest neighbor and the second nearest neighbor are found in the second local feature descriptor. The ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. If the ratio is less than a preset ratio threshold (e.g., 0.7 or 0.8), the matching point pair is considered reliable and is retained; otherwise, it is discarded. In spatial consistency verification, one implementation method can use the Random Sample Consensus (RANSAC) algorithm to perform spatial geometric verification on the initial matching point pairs. Based on the spatial distribution consistency between the matching point pairs, outliers (mismatched point pairs) are discarded, and the retained inliers (correctly matched point pairs) constitute the final matching point pairs.

[0072] In the above embodiments, a three-level screening mechanism of distance calculation, ratio testing and spatial consistency verification is used to gradually eliminate mismatched point pairs, improve the accuracy and reliability of matching point pairs, and lay the foundation for subsequent calculation of accurate geometric transformation matrices.

[0073] For example, a cache hit scenario serves as the fast location channel. When a query key has a matching record in the cache database (exact match or semantically similar match), the system enters the fast location channel, completely bypassing the upstream large model and achieving millisecond-level location.

[0074] 1. Workflow: Retrieve pre-stored local feature descriptors and semantic vectors from the cache database based on the query key; Real-time multi-scale local feature extraction is performed on the current screen to generate a global feature point set; Feature matching is performed using Hamming distance or Euclidean distance, and high-quality matching point pairs are selected through ratio testing and spatial consistency verification. Calculate the precise bounding box coordinates of the target elements using the geometric transformation matrix; Perform topology consistency verification in multi-element location scenarios; Update hit statistics and return the location results.

[0075] 2. Performance advantages: Fast positioning speed: No need to call the large model application programming interface (API), positioning time is reduced from seconds to milliseconds; High positioning accuracy: Based on the matching of local features and geometric transformation matrices, the positioning accuracy can reach the sub-pixel level; Low resource consumption: It does not consume token resources, only uses local computing resources, and has no network requests; High stability: It is not sensitive to image rotation, scaling, brightness changes, etc., and the positioning results are stable and reliable; Semantic generalization: Supports the reuse of different representations of the same element, reducing repetitive learning.

[0076] 3. Technical parameters: Average positioning time: 20-50ms Positioning accuracy: ≤0.5 pixels Resource consumption: CPU usage <10%, no network requests. Semantic matching accuracy: >92%.

[0077] In some embodiments, the second positioning method for locating the target element includes: The AI ​​big model is invoked to identify the target element based on the natural language instructions and the user interface image, and the location coordinates of the target element returned by the AI ​​big model are obtained; Based on the positioning coordinates, the image region of the target element is cropped from the user interface image, and local feature extraction is performed on the image region to generate the cached features of the target element; The cache features of the target element are stored in the feature cache database along with the target element.

[0078] Among them, the AI ​​large model refers to a multimodal large model with visual understanding capabilities (such as GPT-4V, Tongyi Qianwen VL, Doubao, etc.), which can receive natural language commands and image input, understand the semantics of the commands, locate target elements in the image, and return the bounding box coordinates of the target elements. The location coordinates refer to the positional information of the target element in the user interface image, usually represented in the form of a bounding box, such as using the coordinates of the top-left and bottom-right corners, or using the center point coordinates plus width and height. Cropping refers to extracting a sub-image of the region containing the target element from the user interface image based on the location coordinates. Cached features refer to the set of feature descriptors obtained after extracting local features from the cropped target element image region, used for subsequent fast localization and reuse.

[0079] It should be noted that when calling the AI ​​large model, the natural language commands and user interface images can be encapsulated into a request format that conforms to the requirements of the large model API through the AI ​​large model adapter, sent to the large model service, and the positioning coordinates returned by the large model can be received. When cropping the target element image region, the local image region where the target element is located can be segmented from the original user interface image based on the bounding box coordinates returned by the large model. When performing local feature extraction on the cropped image region, the feature extraction algorithm can be adaptively selected according to the texture characteristics of the target element. For example, ORB features are used by default, and SIFT or SURF features are switched when the target element texture is sparse. In extreme scenarios, features based on Convolutional Neural Network (CNN) can also be used, which is not limited in this application. After extraction, the cached features are stored in the feature cache database along with the target elements. In one implementation, the cached features can be stored in association with a query key so that the corresponding cached features can be quickly retrieved by query key during subsequent positioning. Once stored in the database, the target element has completed its "learning" process. Subsequent location attempts can be made directly using the first location method (i.e., local feature matching based on cached features), without needing to call the large AI model again.

[0080] In the above embodiments, the target element is initially identified by an AI large model, and the identification result is converted into reusable cacheable features and stored in the database, realizing the transformation from "zero knowledge" to "learned" and providing a data foundation for subsequent rapid reuse.

[0081] For example, a cache miss scenario corresponds to the learning and localization channel. When a query key does not have a matching record in the cache database, the system enters the learning and localization channel, calls the large model for initial localization, and performs cache learning.

[0082] First, let's introduce the workflow: The AI ​​large model adapter calls the upstream large model to send commands and screenshots; Receive the bounding box coordinates returned by the large model; Based on the returned coordinates, crop the target element image from the original screenshot; Adaptive local feature extraction (ORB / SIFT / CNN) is performed on the cropped element image. Semantic encoding is performed on the instruction text to generate a semantic vector; The feature descriptors, semantic vectors, and query keys are associated and stored in the cache database.

[0083] Next, let's discuss the learning outcomes: Establish visual fingerprints: Create unique local feature descriptors for each UI element; Semantic indexing: Establishing a semantic mapping from natural language instructions to visual features; Learn once, reuse multiple times: After the large model is used for learning for the first time, subsequent localization does not require calling it again; Progressive optimization: As usage increases, cache hit rate continuously improves, and overall location efficiency is continuously optimized.

[0084] Technical parameters: Initial location time: 1-3 seconds (including large model API call time) Large model token consumption: approximately 500-1000 tokens per location. Cache learning time: 50-100ms.

[0085] In some embodiments, it also includes: After locating the target element using the second positioning method, in response to obtaining a user interface image containing the target element again, the feature cache database is queried using the query key, and if a cache feature corresponding to the target element is matched, the target element is located using the first positioning method.

[0086] "Re-acquiring" refers to initiating a location request again for the same UI element (such as the "login button" on the same app page) during subsequent test executions. The user interface image acquired at this time may have slight differences from the image acquired during the initial learning due to factors such as page rendering and device resolution. "Responding to" refers to the location request triggered in response to this re-acquired user interface image.

[0087] It's important to note that after the initial "learning" of a UI element using the second positioning method (calling the AI ​​large model for recognition and storage), the feature cache database already stores the corresponding cached features of that element. When the target element needs to be located again subsequently, the system receives a natural language command and the current user interface image from the upper-layer application, regenerates a query key, and uses this query key to query the feature cache database. Since the element's cached features were associated with the query key and stored in the database during the initial learning, the query key can match the corresponding cached record in the feature cache database. This allows the system to automatically use the first positioning method (i.e., local feature matching of the current user interface image based on cached features) to complete the location, without needing to call the AI ​​large model again. For example, in an automated testing scenario, when the test script is executed for the first time, the features of the "login button" are learned and cached using the second positioning method; when the same test script is executed again subsequently, the "login button" can be quickly located using the first positioning method, without repeatedly calling the large model API.

[0088] In the above embodiments, through the mechanism of "learn once, reuse many times", the subsequent localization of the same element after the initial localization is completed by local feature matching, without the need to repeatedly call the large AI model, thus reducing resource consumption.

[0089] In some embodiments, it also includes: When using the first positioning method to locate the target element, if it is determined that the matched cached feature is invalid, the second positioning method is switched to re-acquire the positioning coordinates and cached features of the target element, and the feature cache database is updated according to the degree of change of the target element.

[0090] In this context, cached feature invalidation refers to the inability of cached features to accurately locate the target element in the current user interface image. This can manifest as feature matching failure or abnormal calculated coordinates. The degree of change refers to the magnitude of the difference between the currently re-acquired cached features and historical cached features, used to measure the extent of changes to UI elements or page layout.

[0091] It's important to note that when the UI undergoes version updates, A / B testing, gray-scale releases, or element style adjustments, cached visual features may become inaccurate due to changes in the target element's appearance, position, or surrounding layout. In such cases, if the system determines that the cached features are invalid, it automatically triggers a fallback mechanism, switching from the first positioning method (local feature matching) to the second positioning method (calling the AI ​​large model). The AI ​​large model then re-identifies the target element in the current interface, obtaining new positioning coordinates and cached features. After obtaining the new positioning coordinates and cached features, the system further evaluates the degree of change in the current target element. In one implementation, the newly acquired cached features can be compared with existing cached features in the feature cache database to calculate the matching rate (e.g., feature point matching rate). If the matching rate is high (e.g., greater than 60%), the degree of change is determined to be low, indicating that only minor changes have occurred in the UI (e.g., slight adjustments to element colors, minor changes in size, etc.). In this case, the newly acquired cached features are fused with the existing cached features to update the feature cache database incrementally. If the matching rate is low (e.g., less than or equal to 60%), the degree of change is determined to be high, indicating that significant changes have occurred in the UI (e.g., significant shifts in element positions, complete style redesign, etc.). In this case, the newly acquired cached features replace the existing cached features. Through this mechanism, the system can automatically adapt to UI changes without relying on manual intervention, maintaining the long-term effectiveness of the positioning.

[0092] In the above embodiments, through the cache feature invalidation judgment and automatic rollback update mechanism, the system can automatically detect and adapt when the UI changes, without the need for manual maintenance of cache data, thereby improving the automation level and long-term stability of the system.

[0093] In some embodiments, updating the feature cache database based on the degree of change of the target element includes: If the degree of change is lower than a preset threshold, the newly acquired cached features will be fused with the existing cached features in the feature cache database to update the feature cache database in an incremental manner. If the degree of change is equal to or greater than the preset threshold, the existing cached features in the feature cache database are replaced with newly acquired cached features.

[0094] The degree of change refers to the difference between the visual appearance of the target element in the current interface and its visual appearance recorded in the historical cache, representing the extent of change in the UI element. The preset threshold is a critical value used to distinguish between low and high degrees of change, and can be set according to the actual application scenario and accuracy requirements. Feature fusion refers to merging newly acquired cached features with existing cached features to generate a comprehensive feature description containing both new and old feature information. Incremental update refers to supplementing the database with new features while retaining existing effective features, rather than completely replacing existing data. Full replacement refers to completely overwriting existing cached features with newly acquired cached features.

[0095] It should be noted that the degree of change can be quantified and evaluated in various ways. In one implementation, the newly acquired cached features can be matched with existing cached features in the feature cache database, and the matching rate between the two can be calculated. If the matching rate is high (e.g., greater than 60%), the degree of change is determined to be lower than a preset threshold; if the matching rate is low (e.g., less than or equal to 60%), the degree of change is determined to be equal to or greater than the preset threshold. The specific threshold for the matching rate can be flexibly set according to the actual application scenario, and this application does not limit it. One implementation of feature fusion can be to merge old and new features by weighted averaging of feature descriptors or feature concatenation, so that the fused features retain both historically valid information and incorporate new visual change information, thereby improving the adaptability of cached features to different UI states. Full replacement is suitable for scenarios where the UI undergoes significant changes (such as large-scale movement of element positions or complete style redesign), in which case the old features are basically invalid, and directly replacing the old features with new features can ensure the accuracy of the cached data.

[0096] In the above embodiments, by selecting an update strategy of incremental update or full replacement based on the degree of change, it is possible to retain effective historical features and integrate new features when the UI undergoes minor changes, and to quickly update cached data when the UI undergoes significant changes, thereby achieving intelligent maintenance of cached data.

[0097] In some embodiments, the cache feature is determined to be invalid by at least one of the following methods: After performing local feature extraction on the user interface image, the number of matching point pairs obtained by matching with the cached features does not reach a preset threshold; or, The location coordinates of the target element calculated based on the matching point pair exceed the display range of the user interface image, or the area of ​​the bounding box corresponding to the location coordinates changes by more than a preset proportion compared to the area of ​​the historical bounding box associated with the cached feature; or, The difference between the page signature in the query key and the current page signature calculated from the user interface image exceeds a preset threshold; or... The cached feature exists for a period exceeding the preset validity period.

[0098] Among these criteria, "number of matching point pairs not reaching the preset threshold" means that after matching the local features extracted from the current user interface image with the cached local features, the number of reliable matching point pairs obtained is too small (e.g., less than 4 pairs), indicating that the visual appearance of the target element in the current interface has changed significantly compared to when it was cached. "Location coordinates exceeding the display range of the user interface image" means that the calculated position of the target element is outside the image area, indicating a significant anomaly in the location result. "Boundary box area change" means that the difference between the currently calculated boundary box area of ​​the target element and the boundary box area recorded in the historical cache is too large (e.g., a change exceeding ±50%), indicating a significant change in the size of the target element. "Page signature difference" means that the difference between the page signature calculated from the current user interface image and the page signature recorded in the query key exceeds the preset threshold (e.g., LSH Hamming distance greater than 10), indicating a significant change in the overall page layout. "Cached feature existence time exceeding the preset validity period" means that the time elapsed since the cached record was created or last updated exceeds the preset validity period (e.g., 7 days), and it is judged as invalid for data freshness considerations.

[0099] It should be noted that the above four determination methods can be used individually or in combination. For example, it can be set that the cache feature is invalidated as long as any one of the methods is met (i.e., an "OR" relationship); it can also be set that multiple methods must be met simultaneously to determine invalidation (i.e., an "AND" relationship). The specific determination strategy adopted and the preset thresholds in each determination condition (such as the threshold for the number of matching point pairs, the threshold for the proportion of area change, the threshold for page signature differences, the validity period, etc.) can all be configured according to the actual application scenario and accuracy requirements, and this application does not limit them.

[0100] In one implementation, if the number of matching point pairs is less than four, it indicates that the appearance of the target element in the current interface has changed significantly compared to when it was cached, and the cached feature is determined to be invalid. In another implementation, if the Hamming distance of the page signature exceeds 10, it indicates that the overall page layout has changed significantly. In this case, even if the feature of a single element matches successfully, the cached feature should still be determined to be invalid because the page context is different. By combining the above multiple determination methods, the system can accurately identify whether the cached feature is still valid.

[0101] In the above embodiments, by combining multiple judgment methods, the system can comprehensively judge whether cache features are invalid from multiple dimensions such as feature matching quality, coordinate rationality, page layout changes, and time validity, thereby improving the accuracy and comprehensiveness of cache invalidation perception.

[0102] For example, when the UI undergoes significant changes, the system can automatically detect and adapt, ensuring long-term effectiveness. This mechanism includes two core components: cache invalidation detection and automatic rollback, as well as incremental updates.

[0103] Cache invalidation detection. The system accurately identifies cache invalidation situations through a multi-layered detection mechanism, including four detection conditions: Judgment condition 1: Feature matching failed.

[0104] The number of high-quality matching pairs has not reached the preset threshold (<4 pairs); The distribution of matching point pairs is unreasonable, resulting in low geometric consistency. Multi-scale matching failed (there were insufficient matching point pairs at all scales).

[0105] Judgment condition 2: Coordinate calculation is abnormal.

[0106] The coordinates calculated based on the homography matrix are outside the screen area; The calculated bounding box area is abnormal (too large or too small).

[0107] Judgment condition 3: Topological consistency violation.

[0108] In multi-element positioning scenarios, the relative positional relationship between the target element and the anchor element deviates from the historical constraints by more than a threshold. Angle difference exceeding 15 degrees or distance ratio exceeding 1.2 times.

[0109] Judgment condition 4: Time expired.

[0110] The cached record has exceeded its preset validity period (e.g., 7 days); The page signature has undergone a significant change (LSH Hamming distance > 10).

[0111] Automatic rollback and incremental updates. When the cache is determined to be invalid, the system automatically triggers the rollback mechanism, re-calls the large model for location, and updates the cached data.

[0112] First, let's introduce the rollback process: Expiration trigger: A cache expiration signal was detected; Automatic rollback: Seamlessly switch to the large model positioning channel; Relocation: Call the upstream large model to re-identify the current screen; Change assessment: Compare the old and new features to assess the degree of change. If the degree of change is low (match rate > 60%): perform incremental feature update and merge the old and new features; If the degree of change is high (match rate ≤ 60%): perform full feature replacement; Version management: Update feature versions and maintain version history; Learning complete: Completes a "self-repair" and adapts to the new UI environment.

[0113] To further explain the technological advantages: No human intervention required: The system automatically completes the entire rollback and learning process, reducing maintenance costs; Progressive adaptation: Retaining valid historical information through incremental updates to adapt to the gradual evolution of the UI; Seamless transition: User experience remains unaffected, and the continuity of the testing process is guaranteed; Rapid Adaptation: Able to quickly learn and adapt after UI redesign, maintaining positioning accuracy; Continuous optimization: Through continuous failure detection and relearning, the system has the ability to self-evolve and maintain high-efficiency operation in the long term.

[0114] In some embodiments, it also includes: When locating multiple target elements using the first positioning method, a local topology map is constructed based on the relative spatial relationships between the multiple target elements; The current relative spatial relationships between target elements in the current positioning results are compared with the corresponding historical relative spatial relationships in the local topology map; When the current relative spatial relationship deviates from the historical relative spatial relationship by more than a preset threshold, relocation is triggered or the confidence level of the positioning result is reduced.

[0115] In this context, multiple target elements refer to multiple UI elements that need to be located simultaneously in a single location request, such as a "login button," "username input box," and "password input box" on a page. Relative spatial relationships refer to the positional constraints between multiple target elements, which can include geometric relationships such as distance, angle, arrangement order, and relative direction. A local topology graph is a graph structure with each target element as a node and the relative spatial relationships between elements as edges, used to represent the spatial distribution and interdependencies among multiple elements. Confidence level is a quantitative assessment value of the reliability of the current location result, used to reflect the reliability of the location result.

[0116] It's important to note that in multi-element localization scenarios, even if each target element is successfully located individually, spatial inconsistencies may exist between the localization results. For example, after a UI redesign, the positions of some elements may shift overall while their relative relationships remain unchanged. In this case, the absolute coordinates of each element change, but their relative spatial relationships (such as A being to the upper left of B, or the distance ratio between A and B) remain stable. Therefore, relatively stable elements on the page can be selected as anchor elements. A relative coordinate system can be established based on the anchor elements, and spatial constraints such as relative distances and relative angles between other target elements and the anchor elements can be calculated to construct a local topology map. During subsequent localization, the current relative spatial relationships between target elements in the current localization result are compared with the corresponding historical relative spatial relationships in the local topology map. If the current relative spatial relationship deviates from the historical relative spatial relationship by more than a preset threshold (e.g., an angle difference exceeding 15 degrees or a distance ratio exceeding 1.2 times), it indicates the presence of abnormal elements in the current localization result. In this case, re-localization (i.e., re-executing the localization process for that element) or the confidence level of the localization result can be reduced (i.e., assigning a lower weight to the localization result of that element in subsequent processing) can be implemented. By using the above methods, the spatial constraints between elements can be used to cross-validate the positioning results, thereby improving the overall reliability of multi-element positioning.

[0117] In the above embodiments, by constructing a local topology map and using the relative spatial relationships between multiple target elements for cross-validation, it is possible to discover and correct the positioning anomalies of a single element, thereby improving the positioning reliability and accuracy in multi-element positioning scenarios.

[0118] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0119] This application constructs an intelligent image feature caching middleware with self-learning and evolution capabilities, and integrates it as the core engine into a UI automation testing framework based on a large AI model. This middleware is designed to be model-independent and adaptable to any upstream large AI model that provides visual positioning capabilities (such as OpenAI GPT-4V, ByteDance Doubao, Tongyi Qianwen VL, etc.). Its core mission is to significantly reduce the frequency of calls to the large model by introducing a caching mechanism without sacrificing or even improving positioning accuracy, thereby achieving a leap in testing efficiency and optimizing resource costs.

[0120] Figure 2 This is a system architecture diagram of the intelligent caching middleware provided in this application. (For example...) Figure 2 As shown, the intelligent caching middleware occupies a crucial position in the entire system, bridging the gap between upstream and downstream processes. Upstream integration: Integrates with upper-layer applications (such as automated test scripts, continuous integration / continuous deployment (CI / CD) systems), receives natural language commands and screenshots, provides a unified location service interface, and is transparent to upper-layer applications.

[0121] Downstream integration: Integrate with AI large model services (such as cloud APIs) and execution layer applications (such as automated testing frameworks and device operation executors). Based on cache hits, intelligently select to call the large model or use the local feature matching engine.

[0122] The overall architecture of the intelligent caching middleware consists of six core modules: a request scheduling and semantic-aware cache query module, a multi-scale local feature fast localization engine, an AI large model adapter, a feature extraction and cache management module, a topology consistency verification module, and a performance monitoring and adaptive tuning module. These six modules work together to achieve efficient UI element localization.

[0123] This section introduces the request scheduling and semantic-aware cache query module.

[0124] This module is the "brain sentinel" of the intelligent caching middleware, responsible for handling all location requests from upper-layer applications and making intelligent decisions.

[0125] Core features include: Receive natural language commands and screenshots sent by upper-layer applications; Generate a unique element lookup key, which is composed of a semantic embedding vector and a current page context signature; The query feature cache database supports both exact matching and semantic similarity matching modes. Based on the cached query results, intelligent routing is used to either the multi-scale local feature fast localization engine or the AI ​​large model adapter.

[0126] Semantic-aware query key generation algorithms include: The system generates a unique element lookup key through the following steps: Semantic embedding extraction: Lightweight semantic models (such as Sentence-BERT or a general sentence encoder) are used to encode the instruction text into fixed-dimensional semantic vectors, so that different expressions such as "login button", "click to log in", and "sign in" are mapped to similar vector spaces.

[0127] Semantic normalization: L2 normalization is performed on semantic vectors to eliminate differences in vector magnitudes and retain only directional information for similarity calculation.

[0128] Generate page context signature: Generate page signatures through edge detection, meshed edge density calculation, Locality Sensitive Hash (LSH), and Perceptual Hash (pHash), tolerating minor layout changes.

[0129] Combined query key: A unique identifier is formed by concatenating the hash value of the semantic embedding vector with the page signature. A semantic index table is also maintained to support fuzzy queries based on cosine similarity.

[0130] Intelligent decision-making logic includes: Cache hit (semantic similarity > threshold): Invoke the multi-scale local feature fast localization engine. Cache miss: The AI ​​large model adapter is invoked, triggering the large model localization and cache learning process. This paper introduces a fast localization engine for multi-scale local features.

[0131] This engine is the "core executor" of the intelligent caching middleware, responsible for high-speed, high-precision element location when a cache hit occurs.

[0132] Core functions: Multi-scale local feature extraction: Construct an image pyramid to extract and merge local features at multiple scaling ratios, enhancing adaptability to different resolutions; Adaptive Feature Selection: Dynamically selects the optimal feature algorithm based on the texture characteristics of UI elements (ORB by default, switches to SIFT / SURF when the texture is sparse, and enables CNN features in extreme scenarios). Feature matching and filtering: False matches are eliminated through distance metrics, ratio tests, and spatial consistency verification; Geometric verification and coordinate calculation: Solve the homography matrix using RANSAC (if it fails, backtrack to the least squares method or affine transformation) to calculate the precise bounding box coordinates of the target elements; Topology consistency verification: Cross-validation using relative positional relationships in multi-element scenarios.

[0133] Introducing the AI ​​Large Model Adapter.

[0134] This module is the "learning engine" of the intelligent caching middleware. It is responsible for calling the upstream large model and caching the learning results when the cache misses.

[0135] Core functions: It encapsulates upstream large model API calls and supports various large model interfaces (such as OpenAI GPT-4V, Alibaba Cloud Bailian, ByteDance Doubao, etc.). Encapsulate commands and screenshots into a request format that conforms to the requirements of the large model API; Receive and parse the bounding box coordinates returned by the large model; The feature extraction and cache management module is triggered to perform cache learning.

[0136] This section introduces the feature extraction and cache management module.

[0137] This module is the "knowledge base manager" of the intelligent caching middleware, responsible for maintaining the feature caching database and ensuring the efficiency and accuracy of cached data.

[0138] Core functions: Element image cropping: Accurately crop the target element image based on the coordinates returned by the large model; Adaptive feature extraction: Selecting algorithms such as ORB / SIFT / CNN to extract local feature descriptors based on element characteristics; Cache storage: Associate feature descriptors, semantic vectors, and query keys and store them in the cache database; Incremental feature update: When there are minor changes in the UI, the effective historical features are retained, and the features are merged for version updates (multi-version maintenance + time decay weighting). Cache invalidation management: Multi-level cache invalidation determination, supporting automatic rollback and relearning; Cache statistics analysis: Statistics on key indicators such as cache hit rate, average response time, and semantic matching success rate.

[0139] This section introduces the topology consistency verification module.

[0140] This module is responsible for cross-validation when locating multiple elements by utilizing the relative spatial relationships between the elements.

[0141] Core functions: Constructing a local topology graph: Establishing a relative coordinate system based on stable anchor point elements; Calculate relative position constraints: Verify whether the relative distance and angle between the target element and the anchor element conform to the historical topological relationship; Anomaly detection and correction: When the positioning result deviates from the topological constraints by more than a threshold, repositioning or weight reduction processing is triggered.

[0142] This section introduces the performance monitoring and adaptive tuning module.

[0143] This module tracks key system metrics in real time and automatically optimizes performance based on monitoring data.

[0144] Core functions: Monitoring metrics collected include: total number of requests, cache hits / misses, semantic match hits, large model calls, response time, token consumption, feature matching errors, cache invalidations, and incremental updates. Adaptive tuning: dynamically adjust cache size and semantic similarity threshold based on hit rate; switch feature extraction algorithms or adjust parameters based on error rate; optimize algorithm complexity based on response time.

[0145] Figure 3 This is a complete workflow diagram of the intelligent caching middleware provided in this application. For example... Figure 3 As shown, this flowchart details the complete closed loop from receiving a location request to outputting the location result, highlighting the role of the caching middleware in the AI ​​large-scale model UI location. It includes: 1. Receive natural language commands and screenshots sent by the upper-layer application; 2. Perform semantic embedding encoding on the instruction text to generate a semantic vector; 3. Calculate the page context signature (LPH-based perceptual hashing); 4. Combine and generate query keys to query the cached database; 5. Semantic similarity matching: If a cached record has a semantic similarity exceeding the threshold (e.g., 0.85), it is considered a hit; 6. Cache Hit Path: Load cached features → Full-screen multi-scale feature extraction → Feature matching and spatial consistency verification → Geometric transformation to calculate accurate bounding boxes → Topological consistency verification (multi-element scenes) → Update hit statistics → Return localization results 7. Cache Missed Path: Call the AI ​​Large Model Adapter → Receive Bounding Box Coordinates → Prune Element Properties → Adaptive Feature Extraction → Semantic Vector Input → Feature Descriptor Input → Return Localization Result 8. Cache invalidation and evolution: When feature matching fails, coordinates are abnormal, or topology is inconsistent, invalidation is determined → automatic rollback to large model localization → assess the degree of change → perform incremental update (low change) or full replacement (high change) → update cache database.

[0146] To verify the effectiveness of this application's solution in improving positioning accuracy and reducing resource consumption, a comparative experiment was conducted on a test set covering 500 common UI elements (buttons, input boxes, text links, icons, floating windows, dynamic lists, etc.) across Android, iOS, and WEB platforms. The results are shown in Table 1 below. Table 1. Comparison results between the pure visual solution and traditional template matching.

[0147] Experiments show that this application maintains sub-pixel accuracy (≤0.5 pixels) and a success rate of >97% in six typical scenarios: standard scenes, slight geometric transformations, lighting changes, semantic generalization, progressive UI evolution, and UI redesign, demonstrating significant advantages in accuracy, efficiency, and robustness. The semantic generalization capability increases the cache hit rate from 50% for exact matches to 85%, further reducing the frequency of large model calls.

[0148] Compared with existing technologies, this application has the following advantages: 1. Significantly improved positioning accuracy and stability. By introducing intelligent caching middleware, the responsibilities of "semantic reasoning" and "geometric matching" are separated, greatly improving positioning accuracy and stability. The high-cost "semantic recognition" task is decoupled from the high-precision "coordinate calculation" task. Subsequent positioning uses ORB features that are insensitive to image rotation, scaling, and brightness changes for accurate geometric matching, eliminating positioning failures caused by instantaneous understanding biases of the large model from the root. 2. Significantly improved resource utilization and execution efficiency. Through caching and reuse mechanisms, the transformation from "every calculation" to "learn once, reuse multiple times" is achieved. This application adopts a "stateful" mode. For stable and unchanging UI elements (such as main navigation bar buttons, login boxes, etc.), only the large model needs to be called once to learn features, and the positioning result can be reused at zero cost through efficient local feature matching. 3. Positioning accuracy reaches sub-pixel level. The positioning method based on local feature matching and geometric transformation matrix has a much higher accuracy than schemes that directly return bounding boxes based on large models. Feature point pairs are obtained through local feature matching, and the accurate boundary of the target element on the current screen is accurately calculated using the geometric transformation matrix, thus achieving sub-pixel level precise geometric transformation.

[0149] The user interface positioning device provided in this application is described below. The user interface positioning device described below can be referred to in correspondence with the user interface positioning method described above.

[0150] Figure 4 This is a structural schematic diagram of the user interface positioning device provided in this application. Figure 4 As shown, the user interface positioning device includes: Information acquisition module 210 is used to acquire natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface; The query generation module 220 is used to generate a query key based on the natural language instruction and the user interface image. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Information query module 230 is used to query the feature cache database using the query key; The first processing module 240 is configured to locate the target element using a first positioning method if a cached feature corresponding to the target element is matched in the feature cache database. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

[0151] In some embodiments, it also includes: The second processing module is used to locate the target element by using a second positioning method if no cached feature corresponding to the target element is matched in the feature cache database. The second positioning method is to call an AI big model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0152] In some embodiments, the query generation module 220 includes: A vector generation unit is used to semantically encode the natural language instruction and generate a semantic vector, the semantic vector being used to indicate the semantic information of the natural language instruction; A signature generation unit is used to perform page signature calculation on the user interface image and generate a page signature, wherein the page signature is used to indicate the visual features of the user interface image; The query generation unit is used to generate the query key based on the semantic vector and the page signature.

[0153] In some embodiments, the signature generation unit is specifically used for: Edge detection is performed on the user interface image to obtain the edge density distribution; The edge density distribution is meshed to obtain meshed edge density features; Generate a feature vector based on the meshed edge density features; The feature vector is encoded using local sensitive hashing and perceptual hashing to generate the page signature.

[0154] In some embodiments, the query generation unit is specifically used for: The semantic vector is binarized or hash-encoded to obtain a semantic hash value; The semantic hash value and the page signature are combined to generate the query key.

[0155] In some embodiments, the information query module 230 is specifically used for: Perform an exact match query in the feature cache database using the query key; and, Based on the semantic vector of the natural language instruction, a semantic similarity matching query is performed in the feature cache database.

[0156] In some embodiments, the positioning of the target element using the first positioning method includes: Obtain the first local feature descriptor of the target element from the matched cached features; Local feature extraction is performed on the user interface image to obtain a second local feature descriptor; The first local feature descriptor and the second local feature descriptor are matched to obtain matching point pairs; Calculate the geometric transformation matrix based on the matching point pairs; The positioning coordinates of the target element in the user interface image are calculated based on the geometric transformation matrix.

[0157] In some embodiments, matching the first local feature descriptor and the second local feature descriptor to obtain a matching point pair includes: Calculate the distance between each feature point in the first local feature descriptor and the second local feature descriptor; Initial matching point pairs are selected based on the distance using a ratio test; Spatial consistency verification is performed on the initial matching point pairs to eliminate erroneous matching point pairs and obtain the final matching point pairs.

[0158] In some embodiments, the second positioning method for locating the target element includes: The AI ​​big model is invoked to identify the target element based on the natural language instructions and the user interface image, and the location coordinates of the target element returned by the AI ​​big model are obtained; Based on the positioning coordinates, the image region of the target element is cropped from the user interface image, and local feature extraction is performed on the image region to generate the cached features of the target element; The cache features of the target element are stored in the feature cache database along with the target element.

[0159] In some embodiments, it also includes: The relocation module is used to, after locating the target element using the second location method, in response to obtaining a user interface image containing the target element again, query the feature cache database using the query key, and if a cache feature corresponding to the target element is matched, locate the target element using the first location method.

[0160] In some embodiments, it also includes: The positioning switching module is used to switch to the second positioning method to re-acquire the positioning coordinates and cache features of the target element when the target element is located using the first positioning method. If the matched cached features are determined to be invalid, the module updates the feature cache database according to the degree of change of the target element.

[0161] In some embodiments, the positioning switching module is specifically used for: If the degree of change is lower than a preset threshold, the newly acquired cached features will be fused with the existing cached features in the feature cache database to update the feature cache database in an incremental manner. If the degree of change is equal to or greater than the preset threshold, the existing cached features in the feature cache database are replaced with newly acquired cached features.

[0162] In some embodiments, the cache feature is determined to be invalid by at least one of the following methods: After performing local feature extraction on the user interface image, the number of matching point pairs obtained by matching with the cached features does not reach a preset threshold; or, The location coordinates of the target element calculated based on the matching point pair exceed the display range of the user interface image, or the area of ​​the bounding box corresponding to the location coordinates changes by more than a preset proportion compared to the area of ​​the historical bounding box associated with the cached feature; or, The difference between the page signature in the query key and the current page signature calculated from the user interface image exceeds a preset threshold; or... The cached feature exists for a period exceeding the preset validity period.

[0163] In some embodiments, it also includes: The topology graph construction module is used to construct a local topology graph based on the relative spatial relationships between the multiple target elements when locating multiple target elements using the first positioning method. The comparison module is turned off to compare the current relative spatial relationship between each target element in the current positioning result with the corresponding historical relative spatial relationship in the local topology map. The re-triggering module is used to trigger repositioning or reduce the confidence level of the positioning result when the current relative spatial relationship deviates from the historical relative spatial relationship by more than a preset threshold.

[0164] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in this application. For example... Figure 5As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a user interface positioning method, which includes: acquiring natural language instructions and a user interface image, wherein the natural language instructions are used to indicate a target element to be positioned in the user interface; Based on the natural language instructions and the user interface image, a query key is generated. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Query the feature cache database using the query key; If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element. If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0165] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the user interface positioning method provided by the above methods. The method includes: acquiring natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface. Based on the natural language instructions and the user interface image, a query key is generated. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Query the feature cache database using the query key; If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element. If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0167] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the user interface positioning method provided by the above methods, the method comprising: acquiring natural language instructions and a user interface image, wherein the natural language instructions are used to indicate a target element to be located in the user interface; Based on the natural language instructions and the user interface image, a query key is generated. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Query the feature cache database using the query key; If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element. If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A user interface positioning method, characterized in that, include: Acquire natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface; Based on the natural language instructions and the user interface image, a query key is generated. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. Query the feature cache database using the query key; If a cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a first positioning method. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

2. The method according to claim 1, characterized in that, Also includes: If no cached feature corresponding to the target element is matched in the feature cache database, the target element is located using a second positioning method. The second positioning method is to call an AI large model to identify the target element to obtain positioning coordinates, and generate feature information based on the positioning coordinates and store it in the feature cache database.

3. The method according to claim 1, characterized in that, The step of generating a query key based on the natural language instruction and the user interface image includes: The natural language instruction is semantically encoded to generate a semantic vector, which is used to indicate the semantic information of the natural language instruction; A page signature is calculated on the user interface image to generate a page signature, which is used to indicate the visual features of the user interface image; The query key is generated based on the semantic vector and the page signature.

4. The method according to claim 3, characterized in that, The step of performing page signature calculation on the user interface image to generate a page signature includes: Edge detection is performed on the user interface image to obtain the edge density distribution; The edge density distribution is meshed to obtain meshed edge density features; Generate a feature vector based on the meshed edge density features; The feature vector is encoded using local sensitive hashing and perceptual hashing to generate the page signature.

5. The method according to claim 3, characterized in that, The step of generating the query key based on the semantic vector and the page signature includes: The semantic vector is binarized or hash-encoded to obtain a semantic hash value; The semantic hash value and the page signature are combined to generate the query key.

6. The method according to claim 1, characterized in that, The step of querying the feature cache database using the query key includes: Perform an exact match query in the feature cache database using the query key; and, Based on the semantic vector of the natural language instruction, a semantic similarity matching query is performed in the feature cache database.

7. The method according to claim 1, characterized in that, The method of locating the target element using the first positioning method includes: Obtain the first local feature descriptor of the target element from the matched cached features; Local feature extraction is performed on the user interface image to obtain a second local feature descriptor; The first local feature descriptor and the second local feature descriptor are matched to obtain matching point pairs; Calculate the geometric transformation matrix based on the matching point pairs; The positioning coordinates of the target element in the user interface image are calculated based on the geometric transformation matrix.

8. The method according to claim 7, characterized in that, The step of matching the first local feature descriptor and the second local feature descriptor to obtain matching point pairs includes: Calculate the distance between each feature point in the first local feature descriptor and the second local feature descriptor; Initial matching point pairs are selected based on the distance using a ratio test; Spatial consistency verification is performed on the initial matching point pairs to eliminate erroneous matching point pairs and obtain the final matching point pairs.

9. The method according to claim 2, characterized in that, The second positioning method for locating the target element includes: The AI ​​big model is invoked to identify the target element based on the natural language instructions and the user interface image, and the location coordinates of the target element returned by the AI ​​big model are obtained; Based on the positioning coordinates, the image region of the target element is cropped from the user interface image, and local feature extraction is performed on the image region to generate the cached features of the target element; The cache features of the target element are stored in the feature cache database along with the target element.

10. The method according to claim 9, characterized in that, Also includes: After locating the target element using the second positioning method, in response to obtaining a user interface image containing the target element again, the feature cache database is queried using the query key, and if a cache feature corresponding to the target element is matched, the target element is located using the first positioning method.

11. The method according to claim 2, characterized in that, Also includes: When using the first positioning method to locate the target element, if it is determined that the matched cached feature is invalid, the second positioning method is switched to re-acquire the positioning coordinates and cached features of the target element, and the feature cache database is updated according to the degree of change of the target element.

12. The method according to claim 11, characterized in that, The step of updating the feature cache database based on the degree of change of the target element includes: If the degree of change is lower than a preset threshold, the newly acquired cached features will be fused with the existing cached features in the feature cache database to update the feature cache database in an incremental manner. If the degree of change is equal to or greater than the preset threshold, the existing cached features in the feature cache database are replaced with newly acquired cached features.

13. The method according to claim 11, characterized in that, The cached feature is determined to be invalid using at least one of the following methods: After performing local feature extraction on the user interface image, the number of matching point pairs obtained by matching with the matched cached features did not reach the preset threshold. or, The location coordinates of the target element calculated based on the matching point pair exceed the display range of the user interface image, or the area of ​​the bounding box corresponding to the location coordinates changes by more than a preset ratio compared with the area of ​​the historical bounding box associated with the cached feature. or, The difference between the page signature in the query key and the current page signature calculated from the user interface image exceeds a preset threshold; or... The cached feature exists for a period exceeding the preset validity period.

14. The method according to claim 1, characterized in that, Also includes: When locating multiple target elements using the first positioning method, a local topology map is constructed based on the relative spatial relationships between the multiple target elements; The current relative spatial relationships between target elements in the current positioning results are compared with the corresponding historical relative spatial relationships in the local topology map; When the current relative spatial relationship deviates from the historical relative spatial relationship by more than a preset threshold, relocation is triggered or the confidence level of the positioning result is reduced.

15. A user interface positioning device, characterized in that, include: An information acquisition module is used to acquire natural language instructions and user interface images, wherein the natural language instructions are used to indicate the target element to be located in the user interface; The query generation module is used to generate a query key based on the natural language instruction and the user interface image. The query key is used to query the feature cache database, which is used to store feature information corresponding to different user interface elements. The information query module is used to query the feature cache database using the query key; The first processing module is configured to locate the target element using a first positioning method if a cached feature corresponding to the target element is matched in the feature cache database. The first positioning method is to perform feature matching on the user interface image based on the matched cached feature to calculate the positioning coordinates of the target element.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the user interface positioning method as described in any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the user interface positioning method as described in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the user interface positioning method as described in any one of claims 1 to 14.

Images