A UI element positioning method, device, equipment and storage medium

CN117407106BActive Publication Date: 2026-07-07SHANGHAI YISAIQI SOFTWARE CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
SHANGHAI YISAIQI SOFTWARE CO LTD
Filing Date
2023-10-19
Publication Date
2026-07-07

AI Technical Summary

Technical Problem

In existing technologies, UI element localization methods based on image matching are easily interfered with in practical applications, resulting in the inability to accurately match target UI elements and affecting the design and running efficiency of RPA processes.

Method used

By generating amplitude maps of a baseline UI element and a UI element to be matched, the similarity between the two is calculated. Spatial pyramid pooling and high-pass filtering techniques are used, combined with low-frequency, mid-frequency, and high-frequency feature weights, to achieve accurate positioning of the UI element.

Benefits of technology

Even with interference such as background color changes and scale scaling, it can accurately match target UI elements, improve positioning accuracy, reduce computational load, and avoid using neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117407106B_ABST
    Figure CN117407106B_ABST
Patent Text Reader

Abstract

The application discloses a UI element positioning method, device and equipment and a storage medium, the method comprising: generating a corresponding first amplitude graph based on an image block comprising a reference UI element; wherein the image block comprising the reference UI element is extracted from a first interface; the first interface represents an interface in which a target UI element is located in an RPA flow design stage; in response to receiving an instruction to position the target UI element, an image block comprising a to-be-matched UI element is extracted from a second interface, and a corresponding second amplitude graph is generated based on the image block comprising the to-be-matched UI element; wherein the second interface represents an interface in which the target UI element is located in a running stage; the second interface is at least partially the same as the first interface; the similarity between the first amplitude graph and the second amplitude graph is calculated, and the to-be-matched UI element with the highest similarity is determined as the target UI element, thereby achieving accurate positioning of the UI element.
Need to check novelty before this filing date? Find Prior Art