Visual anchor-based swing interface recognition and automated positioning method and system

Abstract

This invention relates to the field of software automation testing technology, specifically to a Swing interface recognition and automated positioning method and system based on visual anchor points. In this invention, visual anchor point components with component type stability and text saliency are located, and their top-left corner coordinates and boundary dimensions are extracted as unified technical features. The dominant direction is determined based on the aspect ratio of the visual anchor point component, and an alignment region is defined within its parent container to generate a local candidate component list. Using the top-left corner coordinates in the unified technical features as the origin of the local coordinate system, the relative offset of each component is calculated to form a set of spatial offset constraints. A set of effective candidate components is obtained by combining region attribution and dynamic tolerance range filtering. A comprehensive correlation score is generated by fusing spatial distance weight, anchor point saliency weight, and alignment attribute weight. Based on this score, the optimal target component is determined after type priority disambiguation, completing the binding with the visual anchor point component.

Description

Technical Field

[0001] This invention relates to the field of software automated testing technology, specifically to a method and system for Swing interface recognition and automated localization based on visual anchor points. Background Technology

[0002] In the current field of automated testing for graphical user interfaces, component identification and localization technologies for Java Swing applications still face structural challenges. Traditional methods mainly rely on two paths: one is a logical localization method based on component tree traversal and attribute matching, which achieves target identification by reading the programming attributes of controls (such as name, text, class); the other is to use visual means such as screen coordinates or image template matching for coarse-grained localization. The former is highly coupled to the internal implementation details of the application. When the interface structure undergoes minor adjustments (such as component rearrangement, dynamic attribute updates, or changes in nesting levels), the original localization logic is prone to failure, leading to a significant increase in the maintenance cost of test scripts. The latter, although possessing a certain degree of visual robustness, is easily affected by changes in interface appearance style (such as theme switching, differences in font rendering, and changes in color brightness), and has a large computational resource overhead, making it difficult to meet the needs of high-frequency, low-latency automated interaction.

[0003] Furthermore, existing technologies generally lack the ability to model semantic layout relationships within an interface, making it difficult to effectively establish logical connections between label-type components and their corresponding input controls. For example, the spatial and semantic correspondence between text labels (such as "Settlement Method") and their right-side dropdown boxes often requires manual configuration and is difficult to maintain adaptively after partial interface changes. Although some frameworks attempt to introduce static layout analysis or simple geometric constraints, they still lack a systematic strategy to uniformly support high-precision and loosely coupled component binding and operation execution when dealing with complex nested structures, dynamically generated content, and multi-pane switching scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for Swing interface recognition and automated localization based on visual anchors, in order to solve the problems mentioned in the background art. Specifically, the technical problem involves how to accurately identify and locate target components that have semantic and geometric relationships with the anchors by constructing a relative spatial association model centered on visual anchors and combining it with a dynamic weight fusion mechanism, without relying on the stable internal identifiers of Swing components. This solves the problem of inaccurate automated localization caused by dynamic changes in the interface structure.

[0005] To achieve the above objectives, one of the objectives of this invention is a Swing interface recognition and automated localization method based on visual anchor points, comprising the following method steps: S1. Starting from the root container of the Swing interface, perform a depth-first traversal. Combining the component type stability and text saliency selection criteria, locate a visual anchor component and extract its top-left corner coordinates and boundary dimensions as unified technical features. During the depth-first traversal, a pruning strategy is implemented simultaneously to skip invisible, unusable, or invalid nodes with zero boundary dimensions and all their descendant subtrees. The component type stability selection criteria include prioritizing static layout components and excluding volatile interactive components. The text saliency selection criteria are based on evaluating the visual contrast score between the component's foreground and background colors and its hierarchical position importance in the interface layout to comprehensively determine whether the component meets the visual identifier feature requirements.

[0006] Step S1 performs a depth-first traversal starting from the root container of the Swing interface, and simultaneously applies a pruning strategy to skip invisible, unusable, or invalid nodes with zero boundary size and their descendant subtrees, which can effectively reduce traversal overhead and improve search efficiency. On this basis, by combining the dual screening criteria of component type stability (prioritizing static layout components and excluding volatile interactive components) and text saliency (based on the visual contrast score of foreground and background colors and the importance of layout hierarchy), the identification and selection of components with reliable visual identification capabilities are realized, thereby locating a suitable visual anchor component as a benchmark, and extracting its top-left corner coordinates and boundary size as unified technical features. This process makes subsequent positioning no longer dependent on the internal attributes of the component (such as name and text fields), but instead shifts to modeling external observable features based on visual stability and layout semantics.

[0007] S2. Determine the dominant direction based on the aspect ratio of the visual anchor component; using the top-left corner coordinates of the screen in the unified technical features as a reference, define the corresponding alignment region within its parent container; perform a second round of depth-first traversal only on components sharing the same parent container level within this alignment region to generate a local candidate component list; the process of defining the alignment region specifically includes: Compare the relative size of the width and height of the visual anchor point component. If the width is greater than the height, the dominant direction is determined to be horizontal; if the height is greater than the width, the dominant direction is determined to be vertical. Using the direct parent container of the visual anchor component as the scope, extend the defined range along the determined dominant direction to construct a local spatial range that is at the same parent container level and adjacent to the visual anchor component.

[0008] Step S2 determines the dominant direction (horizontal or vertical) of the visual anchor component based on its aspect ratio. Using the coordinates of the top-left corner of the screen in the unified technical features as a reference, it defines an alignment area along the dominant direction within its direct parent container. Only components sharing the same parent container level within this area undergo a second round of depth-first traversal to generate a local candidate component list. This design converges the global search to the local proximity space, reflecting the modeling of common layout patterns in human-computer interfaces (such as labels and their corresponding input controls often being arranged horizontally or vertically adjacent). By limiting the scope to the parent container level, it avoids the risk of mismatch caused by adjustments to the overall interface structure, thus supporting a spatial constraint mechanism oriented towards semantic proximity.

[0009] S3. Set the top-left corner coordinates of the screen in the unified technical features as the origin of the local coordinate system. For each component in the local candidate component list, calculate its horizontal and vertical offsets relative to the origin of the local coordinate system, and generate a set of spatial offset constraints, specifically including: The origin of the local coordinate system is used as the reference point for the relative position description system; Obtain the top-left corner coordinates of each local candidate component in the local candidate component list, and calculate its horizontal and vertical displacement values ​​relative to the origin. By calculating the center coordinates of the visual anchor point component and the center coordinates of each local candidate component, the corrected horizontal and vertical offsets are obtained and summarized into a set of spatial offset constraints.

[0010] Step S3 sets the top-left corner coordinates of the screen in the unified technical features as the origin of the local coordinate system, using this as a reference for the relative position description system. It calculates the horizontal and vertical offsets of each component in the local candidate component list relative to this origin, and further corrects them with the center coordinates to obtain offset values ​​that are more in line with visual perception habits, ultimately forming a set of spatial offset constraints. This process establishes a relative pose description system with the visual anchor point as the origin, so that the positional relationship between components no longer depends on the absolute screen coordinates, but is transformed into a reusable and transferable relative dimension, providing structured input for subsequent filtering and association judgment, and enhancing the model's adaptability to transformations such as interface scaling and translation.

[0011] S4. Perform intersection filtering on the local candidate component list and the spatial offset constraint set, retaining components that simultaneously satisfy both the regional attribution and offset tolerance ranges, generating a valid candidate component set, specifically including: The offset tolerance range is dynamically set based on the height of the visual anchor component itself; each component in the local candidate component list is compared with the offset parameters in the spatial offset constraint set, and components that simultaneously satisfy the region attribution and offset tolerance range are retained. The determination of region attribution indicates that the layout boundary of the component and the alignment area have spatial overlap; the determination of offset tolerance range indicates that both the horizontal offset and the vertical offset of the component fall within the offset tolerance range; the components that simultaneously pass the above two determination conditions are summarized to form a valid candidate component set.

[0012] Step S4 filters the local candidate component list and the set of spatial offset constraints by intersecting them. It retains components that simultaneously satisfy the regional attribution attribute (i.e., the component layout boundary and the alignment area have spatial overlap) and the offset tolerance range (both horizontal and vertical offsets fall within the tolerance threshold dynamically set by the visual anchor component height), generating a valid candidate component set. This process achieves a secondary screening from potential proximity to reasonable association, eliminating interference items that deviate from the logical layout path even though they are in the same parent container, while retaining components with small displacements due to rendering errors or layout fine-tuning. This improves the semantic relevance and geometric rationality of the candidate set, laying a high-quality input foundation for subsequent multi-dimensional weight evaluation.

[0013] S5. For each component in the set of valid candidate components, calculate its spatial distance weight, anchor salience weight, and alignment attribute weight with the visual anchor component, and then fuse them to generate a comprehensive relevance score, specifically including: For each candidate component in the set of valid candidate components, calculate the spatial pixel distance between its center point and the center point of the visual anchor component, and calculate the spatial distance weight according to the non-linear decay law that the farther the distance, the lower the score. Based on the basic priority of the visual anchor point components, quantification and normalization are performed to obtain the anchor point salience weight that reflects the priority of the visual anchor point components themselves. Based on whether the candidate component and the visual anchor component have a geometric alignment relationship of right-alignment or bottom-alignment, and whether they are in the same direct parent container panel, a hierarchical value is assigned to obtain the alignment attribute weight. By using a pre-defined weighted fusion strategy, the scores of spatial distance weight, anchor point salience weight, and alignment attribute weight are accumulated and calculated to generate a comprehensive relevance score.

[0014] Step S5 calculates the spatial distance weight (reflecting the influence of center spacing according to a non-linear decay law), anchor point salience weight (quantitative normalization result based on the basic priority of the visual anchor point component), and alignment attribute weight (assigned hierarchically according to the alignment relationship to the right / bottom and the belonging relationship to the same direct parent container panel) between each component in the effective candidate component set and the visual anchor point component. The weights are then accumulated through a preset weighted fusion strategy to generate a comprehensive relevance score. This multi-dimensional dynamic weight fusion mechanism collaboratively models geometric distance, visual importance, and layout logic, so that the final score not only reflects spatial proximity but also semantic tightness and structural consistency, thus maintaining a stable ability to recognize the target component even when there are local changes in the interface.

[0015] S6. Based on the comprehensive relevance score, select the component with the highest score as the optimal target component, and establish a binding relationship between it and the visual anchor component. This is used to perform automated interactive operations on the optimal target component. The process of determining the optimal target component specifically includes: The comprehensive relevance scores of each component in the effective candidate component set are sorted from high to low, and the component with the highest comprehensive relevance score is selected as the optimal candidate component. When multiple optimal candidate components have the same comprehensive relevance score, a type priority disambiguation mechanism is activated, specifically including: Based on the basic category attributes of each candidate optimal component, they are divided into three predefined priority levels: static layout components, interactive input components, and selection control components. Among the candidate optimal components, only those belonging to the highest priority level are retained. If multiple candidate optimal components still belong to the same highest priority level, the total number of parent node levels of each candidate optimal component in the interface component tree structure up to the root container is further calculated. This total number of parent node levels is used as a quantitative indicator of spatial nesting depth, and the component with the smallest total number of parent node levels is selected as the unique optimal target component. Finally, the unique component is determined as the optimal target component.

[0016] Step S6 sorts the set of valid candidate components based on the comprehensive relevance score, selects the component with the highest score as the optimal candidate component, and initiates a type priority disambiguation mechanism when scores are the same. First, it divides the components into three priority levels based on the basic category attributes: static layout components, interactive input components, and selection control components, retaining the highest-level component. If there are still multiple components at the same level, it further selects the one with the smallest total number of parent node levels to the root container in the interface component tree (i.e., spatial nesting depth) as the unique optimal target component. This hierarchical decision-making logic ensures that even in the absence of a unique high-confidence match, it can still converge to the most likely target object through the dual criteria of component type semantics and structural simplicity. Finally, it completes the binding relationship between the visual anchor and the optimal target component, providing a reliable anchoring basis for automated interactive operations.

[0017] The second objective of this invention is to provide a system for Swing interface recognition and automated localization based on visual anchor points, comprising a visual anchor point extraction module, a local search domain delineation module, a relative spatial measurement module, an attribution and tolerance filtering module, a multidimensional correlation evaluation module, and a target selection and binding module, wherein: The visual anchor point extraction module starts from the root container of the Swing interface and performs a depth-first traversal; combining the component type stability and text saliency selection criteria, it locates a visual anchor point component and extracts its top-left corner coordinates and boundary dimensions as unified technical features. The local search domain delineation module determines the dominant direction based on the aspect ratio of the visual anchor component; using the coordinates of the top left corner of the screen in the unified technical features as a reference, it delineates the corresponding alignment area within its parent container; only components sharing the same parent container level within the alignment area are subjected to a second round of depth-first traversal to generate a local candidate component list. The relative spatial measurement module sets the coordinates of the top left corner of the screen in the unified technical features as the origin of the local coordinate system. For each component in the local candidate component list, it calculates its horizontal and vertical offset relative to the origin of the local coordinate system and generates a set of spatial offset constraints. The attribution and tolerance filtering module performs intersection filtering on the local candidate component list and the set of spatial offset constraints, retaining components that simultaneously satisfy the regional attribution and offset tolerance range, and generating a set of valid candidate components. The multidimensional correlation evaluation module calculates the spatial distance weight, anchor salience weight, and alignment attribute weight between each component in the set of valid candidate components and the visual anchor component, and integrates them to generate a comprehensive correlation score.

[0018] The target selection and binding module sorts the components based on their comprehensive relevance scores, selects the component with the highest score as the optimal target component, and binds it to the visual anchor component to perform automated interactive operations on the optimal target component.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a relative spatial association model centered on visual anchor components and integrates a dynamic evaluation mechanism that combines spatial distance weight, anchor salience weight, and alignment attribute weight. This improves the stability and accuracy of target component identification without relying on the stable internal identifiers of Swing components. Specifically, the selection of visual anchor components considers both component type stability and text salience, giving them strong visual recognizability and serving as a layout benchmark. The local candidate component list generated by combining dominant direction determination and alignment region delineation effectively constrains the search range and enhances semantic proximity modeling capabilities. Furthermore, a set of spatial offset constraints established by the local coordinate system origin and a dynamic tolerance mechanism further filters out components that meet geometric rationality and regional affiliation requirements, forming a high-quality set of effective candidate components. Finally, based on a ranking strategy using comprehensive association scores and a type priority disambiguation strategy, the invention achieves the ability to reliably determine the optimal target component even when the interface structure undergoes local changes. This supports the continuous availability and low maintenance cost of automated interactive operations in scenarios lacking unique identifiers. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall method steps of the present invention; Figure 2 This is a schematic diagram of the core process of step S1 of the present invention; Figure 3 This is a schematic diagram of the core process from step S2 to step S3 of the present invention; Figure 4 This is a schematic diagram of the core process for determining the optimal target component in this invention; Figure 5 This is a comparison chart of the positioning accuracy of the present invention; Figure 6 This is a schematic diagram of the overall module flow of the present invention.

[0021] In the diagram: 100, Visual Anchor Point Extraction Module; 200, Local Search Domain Delineation Module; 300, Relative Spatial Measurement Module; 400, Attribution and Tolerance Filtering Module; 500, Multidimensional Relationship Evaluation Module; 600, Target Selection and Binding Module. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Next, please refer to Figure 1 One of the objectives of this embodiment is to provide a Swing interface recognition and automated localization method based on visual anchor points, which includes the following method steps: S1. Starting from the root container of the Swing interface, perform a depth-first traversal; combining the component type stability and text saliency selection criteria, locate a visual anchor component, and extract its top-left corner coordinates and boundary dimensions as unified technical features. S2. Determine the dominant direction based on the aspect ratio of the visual anchor component; using the coordinates of the top left corner of the screen in the unified technical features as a reference, define the corresponding alignment area within its parent container; only perform the second round of depth-first traversal on components that share the same parent container level within the alignment area to generate a local candidate component list. S3. Set the coordinates of the top left corner of the screen in the unified technical features as the origin of the local coordinate system. For each component in the local candidate component list, calculate its horizontal and vertical offset relative to the origin of the local coordinate system and generate a set of spatial offset constraints. S4. Perform intersection filtering on the local candidate component list and the set of spatial offset constraints, retaining components that simultaneously satisfy the regional attribution and offset tolerance range, and generating a set of valid candidate components. S5. For each component in the set of valid candidate components, calculate the spatial distance weight, anchor point salience weight, and alignment attribute weight between it and the visual anchor point component, and then merge them to generate a comprehensive relevance score.

[0024] S6. Based on the comprehensive relevance score, select the component with the highest score as the optimal target component, and establish a binding relationship between it and the visual anchor component to perform automated interactive operations on the optimal target component.

[0025] The specific steps are as follows: Please see Figure 2 S1. Starting from the root container of the Swing interface, perform a depth-first traversal; combining component type stability and text saliency selection criteria, locate a visual anchor component, and extract its top-left corner coordinates and boundary dimensions as unified technical features, specifically including: Starting with the top-level window object of a Java Swing application (a toolkit for developing Java graphical user interfaces), and using its content panel as the starting node for traversal, the system recursively visits nested component containers and their child components from top to bottom, performing a depth-first traversal. During the traversal, a pruning strategy is implemented simultaneously, automatically skipping... (Invisible) (Unavailable) or invalid nodes with a boundary size of zero and all their descendant subtrees, to avoid invalid computation; For the remaining valid components, a preliminary screening is conducted based on component type stability criteria, with priority given to certain components. (Tag component) Static layout components such as panel components (often used for grouping) are excluded. (Dropdown component) (Text input box component) and other mutable interactive components; among which mutable interactive components include interactive input components (such as...) ) and selection control components (such as ); Subsequently, a secondary judgment is made by combining the text saliency screening criteria. The visual contrast score between the foreground and background colors of the component (values ​​between [0,1], for example, black text on a white background scores above 0.95) and its hierarchical position importance in the interface layout (such as whether it is located at the top of the window or in the main flow area) are evaluated to comprehensively determine whether the component has sufficiently significant visual identification features. Specifically, the foreground and background color attributes of the effective component are first extracted, and the corresponding visual contrast score is calculated based on the degree of color difference between the two. The score is then verified using a preset high contrast safety threshold to ensure that the content of the effective component is clear and readable. At the same time, the screen space coordinate information of the effective component in the overall interface layout is obtained. Based on the space coordinate information, it is determined whether the component is located in the window top area or the core business main flow area, which are designated as high priority, to assess and confirm its hierarchical position importance. Only when the visual contrast score of the effective component is greater than the set threshold requirement, and its space coordinates clearly fall into the area with high-level position importance, does the system comprehensively determine that the effective component has sufficiently significant visual identification features.

[0026] A component is identified as a visual anchor component when it meets both of the above conditions. Then the visual anchor component is obtained. The coordinates of the top left corner in the screen coordinate system and its width With height This forms the unified technical characteristics upon which subsequent processes rely. .

[0027] The specific execution logic of depth-first traversal is as follows: The root container of the Swing interface is set as the initial exploration node. Starting from this initial exploration node, vertical exploration is preferentially performed towards deeper levels of the interface nesting structure. Specifically, after visiting the current component node, the state of its direct child nodes is immediately determined. If there are unvisited child nodes, the exploration continues down a single branch path until it reaches the end node that no longer contains any derived components. When the end node is reached or all child branches within the current node have been explored to exhaustion, the traversal path is immediately backtracked upwards to the direct parent container node. Then, an unvisited node at the same level adjacent to that parent container is selected to restart the downward vertical exploration. The above alternating addressing process of vertical drilling and upward backtracking is strictly and repeatedly executed throughout the entire interface component topology tree until all valid component nodes derived from the root container have been completely searched, ultimately achieving a detailed topological decomposition of the multi-layered nested interface layout.

[0028] Please see Figure 3 S2. Determine the dominant direction based on the aspect ratio of the visual anchor component; using the top-left corner coordinates of the screen in the unified technical features as a reference, define the corresponding alignment area within its parent container; perform a second round of depth-first traversal only on components sharing the same parent container level within this alignment area to generate a local candidate component list, specifically including: In obtaining the visual anchor point component After determining the boundary dimensions, compare their widths. With height The relative size relationship; if If the direction of the component is horizontal, it is determined to be horizontally dominant; otherwise, it is determined to be vertically dominant. Recorded in the unified technical features Using the origin as a reference, in this visual anchor point component direct parent container (through Within the interface (obtained), extend the defined range along the determined dominant direction to construct an alignment region that conforms to spatial logic; this alignment region is limited to the visual anchor point component. Local spatial ranges that are at the same parent container level and are adjacent in position; Subsequently, only within this aligned region, and satisfying The set of components with conditions initiates a second round of depth-first traversal, no longer traversing branches that cross container paths or are far from the region. The local candidate components whose relevance is to be calculated share the same direct parent container as the visual anchor component, meaning they are at the same level in the interface layout nesting structure. This locally restricted traversal strategy efficiently identifies several potential target components that conform to the region affiliation relationship, ultimately forming a list of local candidate components. ,in To meet Total number of conditions Indicates the first A local candidate component whose correlation degree needs to be calculated.

[0029] S3. Set the top-left corner coordinates of the screen in the unified technical features as the origin of the local coordinate system. For each component in the local candidate component list, calculate its horizontal and vertical offsets relative to the origin of the local coordinate system, and generate a set of spatial offset constraints, specifically including: Visual anchor component The corresponding unified technical feature's top-left corner coordinates on the screen The local coordinate system origin is set as the reference point to establish a relative position description system; for each local candidate component in the local candidate component list... Obtain the coordinates of the top left corner of the screen respectively. And calculate its lateral displacement relative to the origin. With longitudinal displacement value ; To improve positioning accuracy, further correction calculations are performed using the component center point, through calculation of the visual anchor point component. center coordinates With local candidate components center coordinates Thus, the corrected lateral offset is obtained. With longitudinal offset The offsets corresponding to all components are organized to generate a set of spatial offset constraints. This is used for subsequent precise filtering and matching verification of candidate components.

[0030] S4. Perform intersection filtering on the local candidate component list and the spatial offset constraint set, retaining components that simultaneously satisfy both the regional attribution and offset tolerance ranges, generating a valid candidate component set, specifically including: First, verify the local candidate component list. Each local candidate component Whether it is indeed located within the alignment area defined in step S2, i.e. There is spatial overlap with the aligned area, ensuring that it has regional ownership attributes; Next, determine the local candidate component. Lateral offset With longitudinal offset Does it fall within the preset offset tolerance range? Within (alignment tolerance pixels), the offset tolerance range depends on the visual anchor point component. Its own dimensions are dynamically set to reflect its tolerance for minor layout changes; the value is its height. 1 / 5, rounded down; Only components that meet both of the above conditions will be retained in the result set and considered as valid candidate components. After this intersection filtering mechanism, interfering items that are in the same parent container but are too far apart or obviously misaligned are eliminated, and the final set is the set of valid candidate components. ,in ,and , This represents the number of valid candidate components.

[0031] S5. For each component in the set of valid candidate components, calculate its spatial distance weight, anchor salience weight, and alignment attribute weight with the visual anchor component, and then fuse them to generate a comprehensive relevance score, specifically including: For the set of valid candidate components Each valid candidate component Three independent weighted indicators were evaluated sequentially. The first was the spatial distance weight, which was assigned different scores based on the spatial distance between the center point of the component and the center point of the visual anchor component, according to a non-linear decay law; the closer the distance, the higher the weight. The second was the anchor point salience weight, which was based on the visual anchor component. The basic priority and its own visual text features in the current business scenario are normalized and evaluated to reflect the reference reliability of the benchmark component in the local area of ​​the current interface; the third item is the alignment attribute weight, which is assigned a graded value based on whether the component and the visual anchor component are aligned to the right, aligned to the bottom, or nested in the same group panel, to reflect the logical consistency in the layout structure. The three weights are weighted and fused to generate a single comprehensive relevance score. This score reflects a quantitative evaluation of the overall relevance strength between each candidate component and the visual anchor component, serving as the core criterion for final selection. The comprehensive relevance score... The calculation formula is as follows: ,in: Spatial distance weighting is based on the distance between the component's center point and the visual anchor point component. pixel Euclidean distance between center points Different scores are assigned according to a non-linear decay law, with higher weights for closer distances. ,in ,Pick and Represents visual anchor component The width and height of the direct parent container. This is an empirical attenuation coefficient, with a default value of 10. This empirical attenuation coefficient can be dynamically adjusted in the system configuration (e.g., set to 5 or 20) to adapt to high-resolution screens or dense layout scenarios. ,in , ; The anchor point saliency weight is determined by the visual anchor point component. Basic priority Directly normalize to the [0,1] interval, the calculation method is as follows: ,in The basic priority of integers is preset for business scenarios (e.g., "key field label" is set to 10, "auxiliary description label" is set to 3). For alignment attribute weight, if right-alignment is satisfied... and Or aligned directly below and , If only the same parent container is shared ( ),but Other cases .

[0032] The overall correlation score The calculation formula embodies a triple credibility constraint mechanism, among which... Ensure local consistency; Prevent low-quality anchors from misleading matching; Injecting prior knowledge of the structure is essential; all three are indispensable. If any one of them is 0, the association is invalid, thus avoiding incorrect binding.

[0033] Please see Figure 4 S6. Based on the comprehensive relevance score, select the component with the highest score as the optimal target component, and establish a binding relationship between it and the visual anchor component. This is used to perform automated interactive operations on the optimal target component, specifically including: Set of valid candidate components The overall correlation score of each component Sort the data from highest to lowest, and select... The component corresponding to the (maximum score) is selected as the candidate optimal component. When multiple candidate optimal components have the same comprehensive relevance score, the system initiates a type priority disambiguation mechanism. First, based on the basic category attributes of the valid candidate components, they are divided into several predefined priority levels, among which static layout components (such as...) ), interactive input components (such as ) and selection control components (such as The following steps are taken to form a three-tiered priority sequence from low to high priority: Among the candidate optimal components, components belonging to higher priority levels are prioritized for retention; if multiple candidate optimal components at different priority levels still have the same comprehensive relevance score, the total number of parent node levels of each candidate optimal component in the interface component tree structure up to the root container is further calculated. This total number of parent node levels is used as a quantitative indicator of spatial nesting depth, and the component with the smallest total number of parent node levels is selected as the unique optimal target component; finally, the component with the highest unique score is determined as the optimal target component. This ensures the uniqueness and certainty of the binding relationship; specifically: For each component in the valid candidate component set, a comprehensive relevance score is calculated, and candidates are generated by sorting them from highest to lowest score. A multi-level disambiguation decision mechanism then proceeds: First, it is determined whether the highest-scoring component is unique; if so, it is directly identified as the optimal target component. If multiple optimal components exist (with the same comprehensive relevance score), the first-level type priority disambiguation mechanism is activated, dividing them into three priority levels based on basic category attributes (static layout components < interactive input components < selection control components), retaining only the subset of components at the highest priority level. If multiple components still exist in this subset, the second-level spatial nesting depth disambiguation is further executed, calculating the total number of parent node levels from each component to the root container (i.e., spatial nesting depth), and selecting the component with the smallest total number of levels as the unique optimal target component. In extreme cases where differentiation is still impossible, an error reporting mechanism is triggered (theoretically, a very small probability). Finally, regardless of which path is used for determination, the resulting unique component is bound to the visual anchor component, supporting subsequent automated interactive operations such as value retrieval, assignment, or clicking. This ensures the uniqueness and engineering robustness of the binding result while maintaining the strength of the relevance.

[0034] The visual anchor component is then constructed. With the optimal target component The binding relationship between the two is stored in a global binding mapping table; this binding relationship supports subsequent interactive behaviors such as retrieving values, assigning values, or clicking on the optimal target component; when the interface changes... (Component movement) (Size change) or When events such as (window scaling) occur, a full recalculation process can be triggered through the event listener mechanism to ensure that the binding relationship is always synchronized with the current interface state.

[0035] Please see Figure 5 The figure shows the comparison of the accuracy of the optimal target component positioning between the method of this invention (solid dot) and the traditional positioning method (dashed square) under the condition that the degree of local change in the interface structure gradually increases (from level 1 to level 5). The traditional method, due to its high dependence on the stable internal identifier of the Swing component, has a sharp decrease in accuracy as the change level increases (from 95% to 8%). In contrast, the method of this invention, by constructing a relative spatial association model with visual anchor points as the core, maintains a high accuracy (98%→92%), demonstrating excellent robustness. The relative spatial association model refers to a physical pose and semantic association evaluation system centered on visual anchor components and independent of absolute screen coordinates and stable internal identifiers of Swing components. This model first sets the screen coordinates extracted from the visual anchors as the origin of the local coordinate system, transforming the positional relationships between components into reusable and transferable relative offset dimensions. Then, combining the region attribution calculation of the local search domain with a multi-dimensional dynamic evaluation mechanism based on spatial distance weights, anchor saliency weights, and alignment attribute weights, a structured spatial association between components is established. By applying this relative spatial association model, the algorithm can effectively resist interference from interface translation, scaling, and local nesting changes, converging globally uncontrollable changes into relatively stable local spatial relationships.

[0036] This invention is specifically implemented through the following mechanisms Figure 5 Technical effects: First, in S1, visual anchor components that combine type stability and text saliency are selected as the layout benchmark. Then, in S2 to S4, the alignment area is defined and a local candidate list is generated, with the top left corner of the screen as the origin of the local coordinate system. Based on horizontal / vertical offset constraints and dynamic tolerance range, a set of effective candidate components that meet the region belonging relationship is filtered out. Finally, in S5 and S6, the spatial distance weight, anchor saliency weight, and alignment attribute weight are combined to calculate the comprehensive relevance score. Combined with the dual disambiguation mechanism of type priority and nesting depth, the unique optimal target component is accurately determined and a binding relationship is established. The whole process effectively avoids the dependence on the internal identifier of Swing components, so as to maintain stable recognition and positioning capabilities even under local changes in the interface.

[0037] As can be seen from the above description, the Swing interface recognition and automated localization method based on visual anchor points provided in this embodiment has the following technical effects: Step S1 locates visual anchor components by combining component type stability and text saliency screening criteria, and extracts their top-left corner coordinates and boundary dimensions as unified technical features, thus freeing the positioning benchmark from dependence on Swing component internal identifiers; Step S2 determines the dominant direction based on the aspect ratio of the visual anchor component, delineates the alignment region within its parent container, and limits the range of the second round of depth-first traversal, generating a local candidate component list, effectively focusing on semantically adjacent regions; Step S3 uses the top-left corner coordinates from the unified technical features as the origin of the local coordinate system, calculates the horizontal and vertical offsets of each local candidate component to form a set of spatial offset constraints, and constructs a relative position description system; Step S4, through region attribution and dynamically set... The offset tolerance range is used for intersection filtering to generate a set of valid candidate components and exclude irrelevant interference items; step S5 integrates spatial distance weight, anchor point salience weight and alignment attribute weight to generate a comprehensive relevance score, realizing a quantitative evaluation of the multi-dimensional relationship between candidate components and visual anchor components; step S6 sorts according to the comprehensive relevance score and combines it with the type priority disambiguation mechanism to determine the unique optimal target component, and then establishes a binding relationship with it to support subsequent automated interactive operations; the whole process can be recalculated by event listeners when the interface layout is slightly adjusted, and the reliable binding between the visual anchor component and the optimal target component can be continuously maintained, thereby improving the adaptability and robustness of automated positioning without relying on the stable internal identifier of Swing components.

[0038] Please see Figure 6 The second objective of this embodiment is to provide a system for Swing interface recognition and automated localization based on visual anchor points, including a visual anchor point extraction module 100, a local search domain delineation module 200, a relative spatial measurement module 300, a classification and tolerance filtering module 400, a multidimensional correlation evaluation module 500, and a target selection and binding module 600, wherein: The visual anchor point extraction module 100 starts from the root container of the Swing interface and performs a depth-first traversal; combining the component type stability and text saliency selection criteria, it locates a visual anchor point component and extracts its screen top-left corner coordinates and boundary dimensions as unified technical features. The local search domain delineation module 200 determines the dominant direction based on the aspect ratio of the visual anchor component; using the coordinates of the upper left corner of the screen in the unified technical features as a reference, it delineates the corresponding alignment area within its parent container; and only performs a second round of depth-first traversal on components that share the same parent container level within the alignment area to generate a local candidate component list. The relative spatial measurement module 300 sets the coordinates of the upper left corner of the screen in the unified technical features as the origin of the local coordinate system, calculates the horizontal and vertical offsets of each component in the local candidate component list relative to the origin of the local coordinate system, and generates a set of spatial offset constraints. The attribution and tolerance filtering module 400 performs intersection filtering between the local candidate component list and the set of spatial offset constraints, retaining components that simultaneously satisfy the regional attribution and offset tolerance range, and generating a set of valid candidate components. The multidimensional correlation evaluation module calculates the spatial distance weight, anchor point salience weight, and alignment attribute weight between each component in the set of 500 valid candidate components and the visual anchor point component, and then integrates them to generate a comprehensive correlation score. The target selection and binding module 600 sorts the components based on their comprehensive relevance scores, selects the component with the highest score as the optimal target component, and binds it to the visual anchor component to perform automated interactive operations on the optimal target component.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Swing interface recognition and automated localization method based on visual anchor points, characterized in that, The methods and steps include the following: S1. Starting from the root container of the Swing interface, perform a depth-first traversal; combining the component type stability and text saliency selection criteria, locate a visual anchor component, and extract its top-left corner coordinates and boundary dimensions as unified technical features. S2. Determine the dominant direction based on the aspect ratio of the visual anchor point component; using the screen top-left corner coordinates in the unified technical features as a reference, define the corresponding alignment area within its parent container; perform a second round of depth-first traversal only on components sharing the same parent container level within the alignment area to generate a local candidate component list. S3. Set the coordinates of the upper left corner of the screen in the unified technical feature as the origin of the local coordinate system. For each component in the local candidate component list, calculate its horizontal offset and vertical offset relative to the origin of the local coordinate system to generate a set of spatial offset constraints. S4. Perform intersection filtering on the local candidate component list and the spatial offset constraint set, retaining components that simultaneously satisfy the region attribution and offset tolerance range, and generating a valid candidate component set. S5. For each component in the set of effective candidate components, calculate the spatial distance weight, anchor point salience weight, and alignment attribute weight between it and the visual anchor point component, and fuse them to generate a comprehensive correlation score. S6. Based on the comprehensive correlation score, select the component with the highest score as the optimal target component, and establish a binding relationship between it and the visual anchor component for performing automated interactive operations on the optimal target component.

2. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, In step S1, when performing a depth-first traversal, a pruning strategy is implemented simultaneously to skip invisible, unusable, or invalid nodes with zero boundary size and all their descendant subtrees.

3. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The selection criteria for component type stability include prioritizing static layout components and excluding volatile interactive components.

4. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The text saliency screening criteria are based on evaluating the visual contrast score between the component's foreground and background colors and its importance in the interface layout, to comprehensively determine whether the component meets the visual identifier feature requirements.

5. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The process of defining the alignment region specifically includes: By comparing the relative size of the width and height of the visual anchor point component, if the width is greater than the height, the dominant direction is determined to be horizontal; if the height is greater than the width, the dominant direction is determined to be vertical. Using the direct parent container of the visual anchor component as the scope, extend the set range along the determined dominant direction to construct a local spatial range that is at the same parent container level and adjacent to the visual anchor component.

6. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The process of generating the set of spatial offset constraints specifically includes: The origin of the local coordinate system is used as the reference point for the relative position description system; Obtain the screen top-left corner coordinates of each local candidate component in the local candidate component list, and calculate its horizontal and vertical displacement values ​​relative to the origin. By calculating the center coordinates of the visual anchor point component and the center coordinates of each local candidate component, the corrected lateral offset and longitudinal offset are obtained and summarized into a set of spatial offset constraints.

7. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The process of generating the set of valid candidate components specifically includes: The offset tolerance range is dynamically set based on the height dimension of the visual anchor point component itself; each component in the local candidate component list is compared with the offset parameters in the spatial offset constraint set, and components that simultaneously satisfy both the region attribution and the offset tolerance range are retained. The determination of satisfying the region attribution indicates that the layout boundary of the component has spatial overlap with the alignment area; the determination of satisfying the offset tolerance range indicates that both the horizontal offset and the vertical offset of the component fall within the offset tolerance range; components that simultaneously pass the above two determination conditions are summarized to form a valid candidate component set.

8. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The process of generating the comprehensive correlation score specifically includes: For each candidate component in the set of effective candidate components, calculate the spatial pixel distance between its center point and the center point of the visual anchor component, and calculate the spatial distance weight according to the nonlinear decay law that the farther the distance, the lower the score. Based on the basic priority of the visual anchor point component, quantification and normalization are performed to obtain the anchor point salience weight that reflects the priority of the visual anchor point component itself. Based on whether the candidate component and the visual anchor component have a geometric alignment relationship of right-side alignment or bottom-side alignment, and whether they are in the same direct parent container panel, a hierarchical value is assigned to obtain the alignment attribute weight. By using a preset weighted fusion strategy, the scores of the spatial distance weight, anchor point salience weight, and alignment attribute weight are accumulated and calculated to generate a comprehensive correlation score.

9. The Swing interface recognition and automated localization method based on visual anchor points according to claim 1, characterized in that, The process of determining the optimal target component specifically includes: The comprehensive relevance scores of each component in the effective candidate component set are sorted from high to low, and the component with the highest comprehensive relevance score is selected as the optimal candidate component. When multiple optimal candidate components have the same comprehensive relevance score, a type priority disambiguation mechanism is activated, specifically including: Based on the basic category attributes of each candidate optimal component, they are divided into three predefined priority levels: static layout components, interactive input components, and selection control components. Among the candidate optimal components, only those belonging to the highest priority level are retained. If multiple candidate optimal components still belong to the same highest priority level, the total number of parent node levels of each candidate optimal component in the interface component tree structure up to the root container is further calculated. This total number of parent node levels is used as a quantitative indicator of spatial nesting depth, and the component with the smallest total number of parent node levels is selected as the unique optimal target component. Finally, the unique component is determined as the optimal target component.

10. A system using the Swing interface recognition and automated localization method based on visual anchors as described in any one of claims 1-9, characterized in that, It includes a visual anchor point extraction module, a local search domain delineation module, a relative spatial measurement module, an attribution and tolerance filtering module, a multidimensional correlation evaluation module, and a target selection and binding module, among which: The visual anchor point extraction module starts from the root container of the Swing interface and performs a depth-first traversal; combining the component type stability and text saliency screening criteria, it locates a visual anchor point component and extracts its screen top-left corner coordinates and boundary dimensions as unified technical features. The local search domain delineation module determines the dominant direction based on the aspect ratio of the visual anchor point component; using the screen top-left corner coordinates in the unified technical features as a reference, it delineates the corresponding alignment area within its parent container; only components sharing the same parent container level within the alignment area are subjected to a second round of depth-first traversal to generate a local candidate component list. The relative spatial measurement module sets the top-left corner coordinate of the screen in the unified technical feature as the origin of the local coordinate system, calculates the horizontal and vertical offsets of each component in the local candidate component list relative to the origin of the local coordinate system, and generates a set of spatial offset constraints. The attribution and tolerance filtering module performs intersection filtering on the local candidate component list and the spatial offset constraint set, retaining components that simultaneously satisfy the regional attribution and offset tolerance range, and generating a valid candidate component set. The multidimensional correlation evaluation module calculates the spatial distance weight, anchor salience weight, and alignment attribute weight between each component in the set of effective candidate components and the visual anchor component, and integrates them to generate a comprehensive correlation score. The target selection and binding module sorts the components according to the comprehensive relevance scores, selects the component with the highest score as the optimal target component, and binds it to the visual anchor component to perform automated interactive operations on the optimal target component.

Images