A mobile user interface detection method and system

Abstract

The application belongs to the technical field of human-computer interaction interface analysis and visual detection, and specifically discloses a mobile user interface detection method and system. First, OCR recognition and initial candidate region generation are performed on a mobile user interface screenshot, and according to the calculated positioning uncertainty, local magnification cutting and secondary candidate region resampling are performed on the initial candidate region with high positioning uncertainty to obtain a secondary candidate region. Then, a corresponding prompt word input is constructed for each final candidate region, a large language model is used to generate multiple candidate prompt words, and the candidate prompt words are self-checked to obtain standardized prompt words. Graphical visual, text semantic and spatial layout three-channel GUI perception features are constructed, and dynamic gating weights are used for fusion to obtain GUI perception fusion features, and finally the final candidate region detection result is output. The method improves the detection accuracy, semantic recognition ability and generalization ability of icons and text controls in the mobile user interface.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction interface analysis and visual inspection technology, and relates to a mobile user interface detection method and system, which is particularly suitable for control recognition, interface semantic understanding and interactive element localization tasks in mobile application interfaces. Background Technology

[0002] With the rapid development of mobile internet applications, mobile user interfaces have been widely used in e-commerce, social networking, office work, travel, and lifestyle services. Mobile user interface detection, as a fundamental step in automated interface testing, accessibility assessment, automated interface generation, intelligent agent execution, and mobile application quality analysis, aims to automatically identify key interface elements such as buttons, input boxes, icons, list items, pop-ups, and navigation bars from screenshots, text, and layout information, and further obtain the location and semantic functions of these elements.

[0003] Existing mobile user interface detection methods typically employ a single visual model or object detection network, relying solely on pixel information from screenshots to locate and classify controls. While these methods can achieve some success in rule-based interface scenarios, they are prone to issues such as inaccurate candidate region localization, confusion between similar icons, and insufficient semantic recognition when dealing with small icons, dense lists, weak text controls, composite controls, pop-up occlusion, and cross-application style transfer.

[0004] Mobile user interfaces exhibit typical multimodal characteristics. Their interface elements include not only local graphic appearances but also OCR text, page orientation, adjacent controls, layout structure, and contextual semantics. While traditional image-text models can establish similarities between images and text, their spatial layout modeling capabilities are often insufficient, making it difficult to explicitly utilize strong spatial priors in mobile interfaces such as "top back," "bottom navigation," "top right close," and "list item entry."

[0005] In recent years, large language models have been used to generate richer text descriptions to enhance the semantic input of multimodal models. However, if only the location of the candidate region and the OCR text are input into the large language model to generate a single prompt word, problems such as unverifiable prompt words, susceptibility to OCR misidentification, and unstable semantic inference of icon functions still exist. Especially in mobile user interfaces, some icon controls themselves do not contain directly identifiable text information, or their functional semantics need to be judged in combination with nearby text labels, page position, and contextual information; if semantic inference is based solely on a single prompt word, semantic bias may occur, thus affecting subsequent detection results.

[0006] Furthermore, existing lightweight fine-tuning methods typically only insert adapters into the visual or text branches of the visual language model, mainly to achieve parameter adaptation from the general domain to the target domain. They have not yet modeled the three types of features of mobile user interfaces—graphical appearance, textual semantics, and spatial layout—separately, nor have they fully constructed and utilized hard negative samples of easily confused interface elements that are similar in appearance but different in function, have the same text but different spatial positions, have the same icons but different page contexts, or have similar OCR text but different control types, in order to enhance the model's ability to distinguish similar interface elements.

[0007] Therefore, how to improve the localization accuracy, semantic reliability, and cross-page generalization ability in mobile user interface detection while retaining the advantages of training cost has become an urgent technical problem to be solved.

[0008] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0009] To address the aforementioned problems in existing technologies, this invention proposes a mobile user interface detection method to improve the detection accuracy, semantic recognition capability, and generalization capability of icons and text controls in mobile user interfaces.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A mobile user interface detection method includes the following steps: Step 1. Collect screenshot data of the mobile user interface, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, perform initial candidate region generation on it to obtain one or more preliminary candidate regions; Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions with a positioning uncertainty greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot; The secondary candidate regions and the initial candidate regions that did not trigger resampling of the secondary candidate regions are combined to form the final candidate region set; For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text within the region are determined according to the spatial relationship between the final candidate region and the OCR text region. Step 2. Construct structured prompt words to generate input based on the region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information of the final candidate region; A large language model is used to generate candidate prompt words for each final candidate region, and standardized prompt words are generated through self-verification. Step 3. Input the region image of the final candidate region into the visual encoding branch of the visual language model to obtain the graphic visual features, and input the standardized prompt words and OCR text into the text encoding branch to obtain the text semantic features; Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding. Weighted fusion of graphic visual features, text semantic features, and spatial layout features yields GUI perception fusion features. Step 4. Input the GUI-aware fusion features into the multi-channel fusion detection model. The model outputs the interface element category, location coordinates, functional semantics, and detection confidence of each final candidate region.

[0011] Furthermore, based on the aforementioned mobile user interface detection method, this invention also proposes a corresponding mobile user interface detection system, which adopts the following technical solution: A mobile user interface detection system includes the following modules: The candidate region generation module is used to collect screenshot data of mobile user interfaces, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, it performs initial candidate region generation on it to obtain one or more preliminary candidate regions. Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions with a positioning uncertainty greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot; The secondary candidate regions and the initial candidate regions that did not trigger resampling of the secondary candidate regions are combined to form the final candidate region set; For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text within the region are determined according to the spatial relationship between the final candidate region and the OCR text region. The standardized prompt word generation module is used to construct structured prompt word generation input based on the attribute information of the final candidate region; The attribute information of the final candidate region includes region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information; A large language model is used to generate candidate prompt words for each final candidate region, and standardized prompt words are generated through self-verification. The fusion feature generation module is used to input the region image of the final candidate region into the visual encoding branch of the visual language model to obtain graphic visual features, and input the standardized prompt words and OCR text into the text encoding branch to obtain text semantic features; Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding. Weighted fusion of graphic visual features, text semantic features, and spatial layout features yields GUI perception fusion features. It also includes a mobile user interface detection module, which inputs GUI-aware fusion features into a multi-channel fusion detection model. The model outputs the interface element category, location coordinates, functional semantics, and detection confidence for each final candidate region.

[0012] The present invention has the following advantages: To address the shortcomings of existing mobile user interface (User Interface) detection methods, such as coarse candidate region localization, unverifiable prompt word generation, insufficient spatial layout modeling, and weak ability to distinguish similar controls, this invention proposes a mobile User Interface (User Interface) detection method based on large language model self-verification prompt enhancement and multi-channel fusion. This method, building upon existing mobile User Interface (User Interface) screenshots and OCR text input, introduces a coarse-to-fine candidate region resampling mechanism driven by localization uncertainty to improve the quality of candidate boxes for small-sized controls and densely populated areas. Simultaneously, it introduces a multi-candidate prompt word generation and consistency self-verification mechanism to enhance the reliability of semantic descriptions generated by the large language model. Furthermore, it incorporates a three-channel GUI perception mechanism encompassing graphical vision, textual semantics, and spatial layout, and fuses different modal features through dynamic gating. Finally, it enhances the model's adaptability to the mobile User Interface domain through multi-branch adapter fine-tuning and hard negative sample comparison training, ultimately improving the stability and accuracy of mobile User Interface element localization results. Attached Figure Description

[0013] Figure 1 This is a flowchart of the mobile user interface detection method in this embodiment of the invention; Figure 2 This is a schematic diagram of CLIP model comparison learning in an embodiment of the present invention; Figure 3 This is a schematic diagram of three-channel GUI perception and dynamic gating fusion in an embodiment of the present invention; Figure 4 This is a schematic diagram of the internal structure of the lightweight adapter in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This invention provides a mobile user interface detection method, targeting tasks such as control recognition, interactive element localization, interface semantic parsing, and interface layout understanding in mobile application interfaces. It uses mobile user interface screenshot data, OCR text data, and candidate region information as basic inputs, and completes interface element detection through large language model self-verification and prompt enhancement, multi-channel GUI perception fusion, and adapter fine-tuning. Specifically, this invention first performs OCR recognition and preliminary candidate region generation on the mobile user interface screenshot, and calculates the localization uncertainty of the preliminary candidate regions based on the region size uncertainty factor, candidate region neighborhood density factor, OCR text recognition confidence, candidate region overlap factor, and initial detection confidence. For preliminary candidate regions with high localization uncertainty, local magnification and cropping and secondary candidate region resampling are performed to obtain secondary candidate regions, which are then combined with the preliminary candidate regions that did not trigger secondary candidate region resampling to form the final candidate region set. Subsequently, a structured prompt word generation input is constructed for each final candidate region, including region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, relative positional relationship, and page context information. Multiple candidate prompt words are generated using a large language model, and self-validation of these prompt words is performed based on OCR consistency, spatial consistency, image-text consistency, and control prior consistency to obtain standardized prompt words. Three-channel GUI perception features—graphical vision, textual semantics, and spatial layout—are constructed and fused using dynamic gating weights. Lightweight adapters are set in the visual encoding branch, text encoding branch, spatial layout encoding branch, and feature fusion layer, and fine-tuned using hard negative sample contrast loss, classification loss, image-text contrast loss, and IoU perception localization loss. The final output includes interface element categories, position coordinates, functional semantics, and detection confidence.

[0015] like Figure 1 As shown, the mobile user interface detection method in this embodiment includes the following steps: Step 1. Collect screenshot data of the mobile user interface, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, perform initial candidate region generation on it to obtain one or more preliminary candidate regions.

[0016] Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions that are greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot.

[0017] The secondary candidate regions and the initial candidate regions that did not trigger secondary candidate region resampling are combined to form the final candidate region set.

[0018] For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text corresponding to that final candidate region are determined based on the spatial relationship between the final candidate region and the OCR text region.

[0019] Step 1.1. Obtain screenshot data of the mobile user interface and perform preprocessing.

[0020] The system acquires screenshots of the mobile application's running page and performs size unification, pixel normalization, and noise filtering on the original screenshots to obtain preprocessed screenshots of the mobile user interface.

[0021] Mobile user interface screenshots can be taken from real device screenshots or page images saved during automated testing.

[0022] Step 1.2. Perform OCR recognition on the preprocessed mobile user interface screenshot. This invention uses an OCR recognition tool (such as PaddleOCR) to perform text detection and recognition on the preprocessed interface screenshot.

[0023] The PaddleOCR tool can locate text areas in a screenshot and output the corresponding text bounding boxes for the OCR text areas. It can also perform character recognition on the image content within the text bounding boxes and output the text content and text recognition confidence score.

[0024] For the j-th OCR text region, the output result is represented as follows: .

[0025] in Represents a text bounding box. Represents text content, This indicates the confidence level of text recognition.

[0026] For text regions whose text recognition confidence is lower than the preset text recognition confidence threshold (e.g., set to 0.8), they can be used as low-confidence text for subsequent verification instead of being directly used as strong semantic evidence.

[0027] Step 1.3. Perform initial candidate region generation.

[0028] Specifically, a Region Proposal Network (RPN) is used to generate candidate regions from the preprocessed mobile user interface screenshots. The generated candidate regions are called preliminary candidate regions.

[0029] The region proposal network adopts a conventional RPN structure, which includes several parts: a feature extraction subnetwork, an anchor box generation subnetwork, a candidate region classification subnetwork, and a candidate region regression subnetwork.

[0030] The feature extraction subnetwork is used to extract multi-scale interface features from mobile user interface screenshots. The anchor box generation subnetwork is used to generate anchor boxes with different sizes and aspect ratios at different spatial locations in the multi-scale interface feature map.

[0031] The candidate region classification subnetwork is used to predict the foreground probability of each anchor box belonging to the interface element region, and the candidate region regression subnetwork is used to predict the coordinate offset of each anchor box relative to the actual interface element boundary.

[0032] Based on the foreground probabilities output by the candidate region classification subnetwork and the coordinate offsets output by the candidate region regression subnetwork, anchor boxes are filtered and their positions corrected. Duplicate candidate regions are then removed using non-maximum suppression, resulting in a preliminary candidate region set. : ; in These are the first, second, ..., nth candidate regions, where n is a natural number.

[0033] Step 1.4. Calculate the positioning uncertainty.

[0034] In this embodiment, the positioning uncertainty of the initial candidate region can be determined based on the region size uncertainty factor, the candidate region neighborhood density factor, the OCR text recognition confidence, the candidate region overlap factor, and the initial detection confidence.

[0035] For the Preliminary candidate areas Calculate its positioning uncertainty. The formula is as follows: .

[0036] in Indicates the uncertainty factor of the region size. Represents the neighborhood density factor of the candidate region. This indicates the confidence level of OCR text recognition associated with the candidate region. Indicates the candidate box overlap factor. This indicates the initial detection confidence level.

[0037] to To satisfy the normalization constraint, the non-negative weight coefficients can be determined through pre-setting, validation set search, or training / learning methods. In this embodiment, to For example, values ​​such as 0.20, 0.20, 0.20, 0.15, and 0.25 can be used.

[0038] I. Uncertainty factor of region size Sure.

[0039] Let the first The initial candidate areas are Its width is The height is The width of the mobile user interface screenshot is The height is The normalized area of ​​the preliminary candidate region is then... It can be represented as: .

[0040] For the first The coordinates of the top left corner of the initial candidate region, For the first The coordinates of the lower right corner of the initial candidate region.

[0041] When the When the normalized area of ​​a preliminary candidate region is small, it usually corresponds to a small icon, status bar control, navigation bar control, or local sub-control in a list item in a mobile user interface screenshot.

[0042] In such initial candidate regions, slight offsets in the region boundaries may cause the icon body to be truncated or mistakenly include neighboring regions. Therefore, region size is considered an important factor affecting positioning uncertainty. Represented as: .

[0043] in For the preset area threshold, This indicates that the minimum value is taken. If the normalized area of ​​the initial candidate region is less than the preset area threshold... ,but A larger value indicates that the preliminary candidate area has a higher risk of small target location.

[0044] Conversely, if the normalized area of ​​the initial candidate region is greater than or equal to the preset area threshold... ,but A value close to 0 indicates that the initial candidate region has low positioning uncertainty due to its small size.

[0045] In this embodiment, The value can be set according to the resolution of the screenshot and the size distribution of the controls, ranging from 0.005 to 0.02.

[0046] II. Neighborhood Density Factor of Candidate Regions Sure.

[0047] For the A preliminary candidate region (let the width and height of the preliminary candidate region be respectively...). and The neighborhood search range is constructed using the center point of the initial candidate region as the center and its maximum side length as the benchmark.

[0048] The longest side length is Then A rectangular neighborhood is constructed using the side lengths of the neighborhood search range. The preset neighborhood expansion coefficient is used, and Greater than 1. In this embodiment, Version 2.0 is acceptable.

[0049] The number of other preliminary candidate regions whose center point lies within the neighborhood search range is used to calculate the neighborhood density factor of the candidate regions. The neighborhood density factor of the candidate regions can then be expressed as: .

[0050] in This is a region density threshold, which can be set based on the density of candidate regions in the mobile user interface. In one embodiment, Take 5. This represents the number of other preliminary candidate regions within the neighborhood of the preliminary candidate region.

[0051] When there are many other preliminary candidate regions within the neighborhood search range of a certain preliminary candidate region, it indicates that the preliminary candidate region may be located in interface areas such as dense lists, grid pages, toolbars, or pop-up buttons.

[0052] In such regions, overlapping, boundary misclassification, or duplicate detection are likely to occur between adjacent preliminary candidate regions.

[0053] As the number of other preliminary candidate regions within the neighborhood search range increases, the neighborhood density factor of the candidate regions increases accordingly; since the neighborhood density factor of the candidate regions is a component of the positioning uncertainty, it participates in the calculation.

[0054] Therefore, its increase will correspondingly increase the positioning uncertainty of the initial candidate region, making it easier for the initial candidate region in the dense region to trigger subsequent local magnification and cropping and secondary candidate region resampling.

[0055] III. Confidence Level of OCR Text Recognition Sure.

[0056] a. If for the first If a preliminary candidate region contains one or more OCR text regions, the average of the text recognition confidence scores corresponding to each OCR text region within that region is taken as the OCR text recognition confidence score of that preliminary candidate region.

[0057] b. If there is no OCR text region within the preliminary candidate region, but there is an OCR text region within its neighborhood, then the OCR text region located within the neighborhood that is not within the preliminary candidate region is identified as the neighboring OCR text region.

[0058] The text recognition confidence score of the preliminary candidate region is obtained by weighting the distances between the preliminary candidate region and the neighboring OCR text regions. The formula is expressed as follows: ; in Indicates the first The confidence level of OCR text recognition corresponding to each preliminary candidate region. Indicates the relationship with the first The number of neighboring OCR text regions associated with each preliminary candidate region. Indicates the first The text recognition confidence of each neighboring OCR text region.

[0059] To prevent the denominator from being zero, an extremely small positive number is set; in this embodiment, the value is, for example, 10. -8 .

[0060] Indicates the first The distance weights corresponding to each neighboring OCR text region are expressed as follows: .in, Indicates the first The first preliminary candidate region and the first Normalized minimum boundary distance between neighboring OCR text regions.

[0061] Therefore, the closer the neighboring OCR text region is to the preliminary candidate region, the greater its distance weight (i.e., the neighboring OCR text region) and the greater its contribution to the confidence of OCR text recognition.

[0062] c. If there are no OCR text regions within or near the preliminary candidate region, then the OCR text recognition confidence of the preliminary candidate region is set to 0.

[0063] Therefore, 1- It is used to characterize the uncertainty caused by the lack of textual evidence and the unreliability of OCR recognition.

[0064] IV. Candidate Region Overlap Factor Sure.

[0065] For the A preliminary candidate region is obtained, and the preliminary candidate region is calculated. Other preliminary candidate areas The maximum crossover ratio between the regions yields the overlap factor of the initial candidate region. .

[0066] in Indicates the solution and The maximum crossover-union ratio. When Greater than or equal to the preset overlap threshold If the preliminary candidate region is determined to have a high risk of overlap, it means that multiple preliminary candidate regions may point to different sub-regions of the same interface element or the same composite control, or that the boundaries of the preliminary candidate regions are unstable.

[0067] If the preliminary candidate region is directly classified and semantically recognized at this point, it is easy to cause duplicate detection or incorrect control hierarchy judgment. Therefore, it is necessary to further refine the preliminary candidate region by combining local magnification and resampling.

[0068] in, In this embodiment, it can be determined by setting or verifying a set search method. For example, the value is 0.7.

[0069] V. Initial detection confidence Sure.

[0070] Initial detection confidence Derived from the initial candidate region generation in step 1.3, it is used to characterize the reliability of the preliminary candidate region as a real interface element region.

[0071] When the initial candidate region is generated by the region proposal network, the network outputs the foreground probability of that initial candidate region along with its coordinates. The foreground probability represents the likelihood that the initial candidate region belongs to a UI element region. This represents the initial candidate region prospect probability or detection score.

[0072] At this point, the initial detection confidence level It is generated by the candidate region classification subnetwork of the region proposal network.

[0073] The text recognition confidence level compared to the OCR recognition tool output in step 1.2 The reliability of the identification of relevant text content used to characterize preliminary candidate regions varies. Used to characterize the detection reliability of the initial candidate region itself as an interface element region.

[0074] The lower the value, the less reliable the initial candidate region is as a real interface element. Therefore, 1- The larger.

[0075] Step 1.5. Set a preset positioning uncertainty threshold. Perform local magnification and cropping and secondary resampling on the preliminary candidate regions that are greater than the preset positioning uncertainty threshold after the positioning uncertainty calculation in Step 1.4.

[0076] First, using the center of the initial candidate region as the cropping center, obtain the corresponding local image patch according to the preset cropping range, and then scale it according to the preset scaling ratio. Scaling of local image patches.

[0077] Candidate region generation is performed again within the scaled local image patch to obtain one or more secondary candidate regions. The secondary candidate regions are refined candidate regions obtained after local magnification cropping and re-generation.

[0078] In this embodiment, the positioning uncertainty is no longer calculated repeatedly for the secondary candidate regions. Instead, they are used as candidate regions for subsequent OCR text spatial mapping and feature extraction.

[0079] Let the offset of the local image patch from the top left corner of the original screenshot be . The scaling ratio is Then the coordinates of the secondary candidate region in the local image patch Inverse transformation to the original interface screenshot coordinates: .

[0080] For preliminary candidate regions with positioning uncertainty not exceeding a preset positioning uncertainty threshold, local zoom-out and secondary candidate region resampling are not performed; the preliminary candidate region is directly retained.

[0081] The secondary candidate regions obtained through local zoom-out cropping and secondary candidate region resampling, together with the initial candidate regions that were directly retained without triggering local zoom-out cropping and secondary candidate region resampling, form the final candidate region set for subsequent processing.

[0082] For ease of subsequent description, all candidate regions in the final candidate region set will be referred to as the final candidate regions.

[0083] In this embodiment, the original interface screenshot refers to the original mobile user interface screenshot.

[0084] Step 1.6. Establish the spatial mapping relationship between the final candidate regions and the OCR text.

[0085] Based on each final candidate region and the OCR text region Text bounding box The relationship between the center points, the proportion of overlapping areas, and the boundary distance are used to determine the OCR text within the region and the neighboring OCR text corresponding to the final candidate region.

[0086] Among them, OCR text area It is calculated through step 1.2 above. The OCR text within the region refers to the OCR text content that has an internal spatial relationship with the final candidate region.

[0087] Specifically, when the center point of a certain OCR text region is located inside the final candidate region, or when the overlap area between the OCR text region and the final candidate region is not less than the area of ​​the OCR text region, the OCR text content corresponding to the OCR text region is determined as the OCR text within the final candidate region.

[0088] Neighboring OCR text refers to OCR text that is not located within the region but is spatially adjacent to the final candidate region.

[0089] For any OCR text region, calculate the minimum boundary distance between the OCR text region and the final candidate region.

[0090] When the minimum boundary distance is not greater than the preset proximity distance threshold, the OCR text content corresponding to the OCR text region is determined as the neighboring OCR text of the final candidate region.

[0091] For the final candidate region associated with multiple OCR text regions, each OCR text region is first divided into an intra-region OCR text region and a neighboring OCR text region according to the determination criteria of intra-region OCR text and neighboring OCR text.

[0092] For OCR text regions within a region, they are sorted from high to low according to their overlap ratio with the final candidate regions. When the overlap ratios are the same, they are sorted from high to low according to the text recognition confidence.

[0093] For adjacent OCR text regions, they are sorted from near to far according to the minimum boundary distance between them and the final candidate regions. When the minimum boundary distances are the same, they are sorted from high to low according to the text recognition confidence.

[0094] Based on the above division and sorting results, the set of OCR texts within the region, the set of neighboring OCR texts, and the corresponding set of text recognition confidence scores are obtained for the final candidate region.

[0095] Based on the coordinate difference between the center point of each OCR text region and the center point of the final candidate region, the relative positional relationship of each region with respect to the final candidate region (left, right, top, bottom, etc.) is determined, which is used for subsequent prompt word generation and consistency self-verification.

[0096] Therefore, using each final candidate region as an index, the OCR text regions within that region, the neighboring OCR text regions, and the corresponding OCR text content, text recognition confidence, and relative positional relationships are associated and recorded.

[0097] Through the above operations, a spatial mapping relationship between the final candidate region and the OCR text was established.

[0098] Step 1 of this invention calculates the localization uncertainty of the initial candidate region based on the candidate region size, candidate region density, OCR text recognition confidence, candidate box overlap, and initial detection confidence. Based on this localization uncertainty, it determines the target region requiring local magnification cropping and final candidate region resampling. Therefore, for small-sized controls, densely arranged controls, pop-up edge controls, and interface elements with severely overlapping candidate boxes, candidate boxes can be regenerated in the magnified image region. This reduces the impact of initial candidate region boundary offset, target omission, and duplicate box interference on the detection results, improving the stability and accuracy of mobile user interface element localization.

[0099] Step 2. Generation of multiple candidate prompts and self-verification of consistency.

[0100] The structured prompt word is constructed based on the region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information of the final candidate region to generate input.

[0101] The system uses a large language model to generate candidate prompts for each final candidate region and generates standardized prompts through self-verification.

[0102] Step 2.1. Construct prompt words to generate input for each final candidate region.

[0103] The region location, size, and aspect ratio of the final candidate region are calculated based on its coordinates.

[0104] Based on the spatial mapping relationship established in step 1.6, the OCR text within the region corresponding to the final candidate region, the neighboring OCR text, and the corresponding text recognition confidence are obtained. Based on the coordinate difference between the center point of each OCR text region and the center point of the final candidate region, the relative positional relationship of the OCR text region with respect to the final candidate region is determined.

[0105] When the x-coordinate of the center point of the OCR text region is less than the x-coordinate of the center point of the final candidate region, the relative position is determined to be on the left; when it is greater, it is determined to be on the right.

[0106] When the ordinate of the center point of the OCR text region is less than the ordinate of the center point of the final candidate region, the relative position is determined to be above; when it is greater, it is determined to be below.

[0107] The page orientation is determined based on the coordinates of the final candidate region in the mobile user interface screenshot.

[0108] Specifically, based on the normalized coordinates of the center point of the final candidate region in the screenshot of the mobile user interface, the mobile user interface is divided into three vertical regions (top, middle, and bottom) and three horizontal regions (left, middle, and right). The page orientation is determined according to the region to which the center point of the final candidate region belongs.

[0109] For example, the normalized x-axis and y-axis can be used separately. and As the boundary for dividing the area, the final candidate area is determined to be located in the upper left, upper middle, upper right, left middle, middle, right middle, lower left, lower middle, or lower right area of ​​the page.

[0110] Page context information is determined based on the distribution of other final candidate regions around the final candidate region, the distribution of OCR text regions, and the corresponding spatial layout relationships. Specifically, taking the final candidate region as the center, other final candidate regions and OCR text regions are obtained within a preset context neighborhood, their numbers are counted, and their direction, normalized distance, and horizontal or vertical arrangement relationship relative to the final candidate region are calculated based on the coordinates of the center point of each region. The above neighborhood region distribution information and the page orientation of the final candidate region are used together as page context information.

[0111] The above-mentioned region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information are combined into a structured prompt word generation input.

[0112] When there is no OCR text within the final candidate region but there is neighboring OCR text, the field of OCR text within the region in the structured prompt word generation input is set to empty, while the neighboring OCR text, the corresponding text recognition confidence, and its direction and distance information relative to the final candidate region are retained for the large language model to generate candidate prompt words.

[0113] When neither the OCR text within the final candidate region nor the adjacent OCR text exists, the two types of OCR text fields in the structured prompt word generation input are set to empty, while retaining the region location, region size, aspect ratio, page orientation, and page context information. If the existing information is insufficient to support a definite candidate description, then according to the structured prompt template requirements in step 2.2, the large language model outputs an uncertain identifier.

[0114] This enables large language models to infer the possible control types and functional semantics of the final candidate region by combining spatial location and layout context when generating candidate prompts, even when textual evidence is insufficient.

[0115] Step 2.2. Use a large language model to generate multiple candidate prompts for the same final candidate region.

[0116] The structured prompt words are generated as input to the large language model, and the output format of the large language model is constrained by the structured prompt template.

[0117] The candidate prompts generated by the large language model include: control category candidate prompts, function action candidate prompts, visual appearance candidate prompts, and spatial relationship candidate prompts.

[0118] When generating candidate prompts, the large language model simultaneously outputs the evidence source for each candidate prompt.

[0119] A structured prompt template may include the following: "You are a mobile user interface analysis expert. Given information about a final candidate region, including region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, relative positional relationships, and page context information, please provide candidate control categories, candidate functions, candidate visual appearances, and candidate spatial relationships for this final candidate region. For each candidate result, please provide the source of evidence supporting that candidate result; if the current information is insufficient to determine the outcome, please output an uncertainty flag and explain the reason for the uncertainty."

[0120] Unlike methods that only generate a single description, this invention requires a large language model to generate M candidate prompt words for the same final candidate region. M is a preset quantity or a quantity determined based on the completeness of the final candidate region information.

[0121] Each candidate prompt word All are structured text descriptions, including control categories, functional semantics, appearance attributes, spatial relationships, and interactivity fields; at the same time, each candidate prompt word has a corresponding evidence field.

[0122] The evidence field is determined by the OCR text within the region, the neighboring OCR text, the text recognition confidence, the page orientation, the relative positional relationship, and the page context information in step 2.1, and is used to identify the basis for generating the candidate prompt word.

[0123] Step 2.3. Perform a consistency self-check on multiple candidate prompt words.

[0124] To reduce the risk of semantic illusions generated by large language models, a consistency self-check is performed on each candidate prompt word. This invention performs self-checks on each candidate prompt word based on OCR consistency, spatial consistency, image-text consistency, and control prior consistency.

[0125] For the j-th candidate prompt word Its verification score Represented as: .

[0126] in This indicates the OCR consistency between the candidate prompts and the OCR text; This indicates the spatial consistency between candidate prompts and spatial layout priors; This indicates the consistency between the candidate prompt words and the candidate region image;

[0127] This indicates the consistency between the candidate suggestion and the control category prior.

[0128] β1 to β4 are non-negative weighting coefficients. .

[0129] I. OCR Consistency It is used to measure the degree of semantic consistency between candidate prompts and the OCR text within the final candidate region and adjacent OCR text.

[0130] For example, when the text inside the candidate area is "search", the candidate suggestions "search button" or "search input box" will be displayed. The higher the score, the higher the score for the candidate suggestion "close button". Lower.

[0131] In one embodiment, It is obtained through keyword matching, edit distance similarity, text vector cosine similarity, or a weighted combination of the above methods, and its formula is as follows: .

[0132] in, Indicates the first OCR text content, This indicates the confidence level of the corresponding text recognition. Indicates the first The candidate prompt and the first Text similarity between OCR texts This indicates the weight of the OCR text.

[0133] This represents the OCR text within the area. This indicates the adjacent OCR text.

[0134] The smallest positive number set to prevent the denominator from being 0 can be taken as... .

[0135] For OCR text within a region, its text weight is set to 1; for neighboring OCR text, its text weight is determined based on the normalized minimum boundary distance between the OCR text region and the final candidate region. The closer the distance, the greater the text weight.

[0136] Both the OCR text within the region and the neighboring OCR text are obtained from the spatial mapping relationship established in step 1.6.

[0137] II. This is used to measure whether candidate prompts conform to the spatial layout priors of the mobile user interface, based on candidate prompts. The degree of matching between the control category or functional semantics in the text and the page location of the final candidate area is calculated.

[0138] Set candidate prompts The corresponding candidate control category is The normalized coordinates of the center point of the final candidate region are: Control categories The corresponding preset spatial prior region is ,but It can be represented as: ; in Indicates the distance from the center point of the final candidate area to the control category. The normalized distance corresponding to the prior region of the space, σ represents the distance decay parameter, which controls the degree to which spatial distance affects the spatial consistency score. The smaller σ is, the faster the spatial consistency score decreases with increasing distance. σ can be determined by pre-setting or by validation set search.

[0139] In this implementation, σ is, for example, 0.2.

[0140] The spatial prior area can be preset according to common layout rules of mobile user interfaces, for example: The "Back" control corresponds to the top left area of ​​the page, the "Close" control corresponds to the top right area of ​​the page or the top right area of ​​the pop-up window, the "Bottom Navigation" control corresponds to the bottom area of ​​the page, and the "Search Input Box" corresponds to the top or upper-middle horizontal area of ​​the page.

[0141] Therefore, when the control category or functional semantics in the candidate suggestion words are consistent with the page location of the candidate area, High; when the two are clearly mismatched, Lower.

[0142] III. This is used to measure the degree of cross-modal matching between candidate prompts and the final candidate region image. Specifically, the final candidate region image can be input into the visual encoding branch of the visual language model to obtain image features. : ;in, Represents a visual encoder. This is the final candidate region image.

[0143] The candidate prompt words are input into the text encoding branch to obtain text features. : ;in, Indicates a text encoder. These are candidate suggestion words.

[0144] And calculate the cosine similarity between image features and text features as... : .in, and These represent the normalized image features and text features, respectively. This method avoids incorporating erroneous prompts inferred solely from OCR text or page location into the subsequent fusion process.

[0145] IV. This is used to measure whether candidate suggestions conform to the shape and semantic patterns of common controls in mobile user interfaces.

[0146] .

[0147] in, , , and These represent the form matching score, text matching score, context matching score, and interaction attribute matching score, respectively. to These are non-negative weighting coefficients, and .

[0148] In this embodiment, , , , The values ​​can be 0.25, 0.20, 0.35, and 0.20 respectively.

[0149] Indicates the first The candidate control category corresponding to each candidate prompt word.

[0150] Indicates the first The morphological information of each final candidate region, including region shape, region size, and aspect ratio; Indicates the first The text information corresponding to each final candidate region includes the OCR text within the region and the neighboring OCR text.

[0151] Indicates the first The page context information corresponding to each final candidate region includes the page orientation, the distribution of neighboring candidate regions, and their relative positional relationship with the OCR text region; Indicates the first Interactive attribute information corresponding to each final candidate region.

[0152] Four categories of prior rules are pre-established for each control category: shape, text, context, and interaction attributes. Based on the information of the final candidate area, each rule is checked to see if it is satisfied; rules with insufficient information to make a judgment are marked as undecidable.

[0153] The four types of rules are set and judged as follows: (1) Morphological matching: Based on the labeled areas of the corresponding control categories in the training set, the 5th and 95th percentiles of the normalized area and aspect ratio are calculated respectively to form the value range of the two morphological rules.

[0154] shape information The corresponding value in the interval is compared with the above interval. If it falls into the interval, the rule is satisfied; otherwise, it is not satisfied.

[0155] (2) Text matching: Preset keywords and synonym sets for each control category.

[0156] For example, button categories can be set with keywords such as "OK", "Cancel", "Submit", and "Back".

[0157] If text information If the text contains any word from the keyword set of that category, the text rule is satisfied; if there is OCR text but no keyword is matched, the rule is not satisfied; if there is no OCR text, the rule cannot be determined.

[0158] (3) Context matching: Set page orientation and adjacent area arrangement rules for control categories with clear layout characteristics, and match them according to page context information. Judge each item individually.

[0159] For example, the bottom navigation control can be configured with two rules: "located in the bottom third of the page" and "there are horizontally arranged adjacent candidate areas." For categories without explicitly defined layout rules, the matching score is treated as the default.

[0160] (4) Interaction attribute matching: interaction attribute information The interaction type is inferred from the OCR text within the region, neighboring OCR text, and a preset action dictionary. For example, "click" and "tap" correspond to the click type, "input" and "fill in" correspond to the input type, and "swipe" and "scroll" correspond to the scroll type. The inferred interaction type is compared with the preset allowed interaction types for the candidate control category. If it is an allowed type, the rule is satisfied; otherwise, it is not satisfied. When there are insufficient clues or the inference is ambiguous, the interaction attribute is marked as unknown, and the rule cannot be determined. The above interaction attributes represent the inference results based on text clues.

[0161] The matching scores for each type of match are calculated based on the proportion of rules that have been satisfied in the decisionable rules: ; .

[0162] in, These correspond to the four matching scores mentioned above; Indicates the first The number of rules that can be determined in a class of rules. This indicates the number of rules that have been satisfied; rules that cannot be determined are not included in the above count.

[0163] If no definable rule exists, the corresponding matching score is set to 0.5, which is the neutral default score agreed upon in this embodiment. All four matching scores are between zero and one, and then weighted according to the aforementioned weights to obtain the control prior consistency score.

[0164] Step 2.4. Filter candidate prompts based on the verification score.

[0165] If the verification score of a candidate suggestion word is lower than the preset verification score threshold, the candidate suggestion word will be removed.

[0166] For candidate prompts with a verification score not lower than the preset verification score threshold, they are sorted from high to low according to their verification scores, and the control category field and the functional semantic field in the candidate prompts are compared in turn.

[0167] When a candidate suggestion word and a higher-scoring candidate suggestion word ranked before it provide mutually exclusive control categories or functional actions for the same final candidate area, it is determined that there is a functional semantic conflict between the two.

[0168] At this point, candidate suggestions with low verification scores are removed. For example: For the same final candidate area, if two candidate prompts are defined as 'open' and 'close' respectively, or as 'return control' and 'search control' respectively, then it is considered that there is a functional semantic conflict between the two.

[0169] In this embodiment, a preset verification score threshold is used. The possible value is 0.6.

[0170] If there are multiple verification scores that are not lower than the preset verification score threshold Candidate suggestions that do not have functional semantic conflicts are sorted according to their verification scores from highest to lowest, and the top ones are selected. The candidate prompts were selected and merged into standardized prompts.

[0171] Where K is the preset retention quantity, and .

[0172] The total number of candidate suggestions generated for the same final candidate region; when the number of candidate suggestions meeting the above conditions is less than When that happens, select all candidate prompts that meet the criteria and merge them.

[0173] If no candidate prompts are retained after filtering, the fixed text "Candidate area of ​​interface with undetermined category and function" is used as a standardized prompt to retain the final candidate area, and subsequent feature encoding and detection are continued.

[0174] This placeholder text does not indicate that the control category or functional semantics of this area have been determined.

[0175] Candidate suggestion word fusion employs methods such as selecting the highest-scoring candidate suggestion word, weighted text concatenation, or merging structured fields. Specifically, selecting the highest-scoring candidate suggestion word means directly using the candidate suggestion word with the highest validation score as the standardized suggestion word.

[0176] Weighted text concatenation refers to determining the weight of each candidate prompt word based on its verification score, and then combining the text descriptions in the candidate prompt words according to their weights to obtain standardized prompt words.

[0177] Structured field merging refers to comparing the control category, functional semantics, appearance attributes, spatial relationship, and interactivity fields in each candidate prompt word, prioritizing the retention of fields with higher validation scores and no semantic conflicts, and combining the retained fields to form standardized prompt words.

[0178] By using the above method, the standardized prompts obtained in the final result retain the control categories, functional semantics, and contextual information with high credibility in the verification results.

[0179] This invention generates multiple candidate prompts for the same final candidate region using a large language model, and performs self-verification on each candidate prompt based on OCR consistency, spatial consistency, image-text consistency, and control prior consistency to determine the degree of matching between the candidate prompts and the region text information, spatial layout information, region image information, and control category prior.

[0180] Candidate prompts with a matching degree lower than the verification score threshold or semantic conflicts with other candidate prompts are removed. This reduces the impact of OCR misidentification, text missing, and instability of semantic inference in large language models on the detection results, and improves the credibility of standardized prompts and the stability of subsequent text semantic encoding.

[0181] This invention can reduce semantic bias caused by illusions and OCR misidentification in descriptions generated by large language models.

[0182] Step 3. Fusion of three-channel GUI perception and dynamic gating, such as... Figure 3 As shown.

[0183] The region image of the final candidate region is input into the visual encoding branch of the visual language model to obtain the graphic visual features, and the standardized prompt words and OCR text are input into the text encoding branch to obtain the text semantic features.

[0184] Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding.

[0185] The GUI perception fusion features are obtained by weighted fusion of graphic visual features, text semantic features and spatial layout features.

[0186] Step 3.1. Graphic visual channel encoding.

[0187] Graphical visual features are used to characterize the local visual appearance of candidate regions. The final candidate region image is then used. The visual encoding branch of the input visual language model is used to obtain the visual features of the image. The preferred visual language model is the CLIP model, such as... Figure 2 As shown.

[0188] The graphic visual features obtained through feature extraction via the visual encoding branch include icon shape, border structure, button background, input box edge, pop-up outline, color distribution, and local texture information.

[0189] For preliminary candidate regions whose positioning uncertainty exceeds a preset positioning uncertainty threshold and triggers secondary candidate region resampling, the region image corresponding to the final candidate region obtained by resampling is used as the visual input.

[0190] For preliminary candidate regions whose positioning uncertainty is not greater than the preset positioning uncertainty threshold and which have not been touched by secondary candidate region resampling, the region image corresponding to the preliminary candidate region is directly used as the visual input step 3.2 text semantic channel encoding.

[0191] Textual semantic features are used to characterize the text content, functional actions, and semantic cue words of candidate regions.

[0192] For the The final candidate regions will contain the standardized prompts obtained in step 2. Step 1.6 yields the OCR text within the region and the set of neighboring OCR text. The text is concatenated and input into the text encoding branch of the visual language model.

[0193] Text semantic features are obtained by extracting features from text-encoded text. .

[0194] Step 3.3. Spatial layout channel coding.

[0195] For the There are 1 final candidate regions, whose coordinates are 1 / 2. If the width of a mobile user interface screenshot is and the height is , then its normalized coordinates are... Normalized width Normalization height and aspect ratio They are represented as follows: ; ; ; .

[0196] Simultaneously obtain the page orientation of the final candidate area, its distance from neighboring final candidate areas, and its relative positional relationship with the OCR text area; combine the above information with... , , and Combined as spatial layout input .

[0197] First, determine the page orientation based on the normalized coordinates of the center point of the final candidate region in the screenshot of the mobile user interface.

[0198] Secondly, based on the normalized Euclidean distance between the center point of the final candidate region and the center points of other final candidate regions, the minimum value among them is determined as the distance between the center point and the neighboring final candidate regions.

[0199] For the For each of the 10 final candidate regions, calculate the normalized Euclidean distance between its center point and the center points of other final candidate regions, and take the minimum value as the distance between the final candidate region and its neighboring final candidate regions. : ; in, and They represent the first The and the first Normalized coordinates of the center point of each final candidate region.

[0200] Then, based on the spatial mapping relationship between the final candidate region and the OCR text established in step 1, and based on the coordinate difference between the center point of the OCR text region and the center point of the final candidate region, the relative positional relationship between the two is determined.

[0201] Specifically, based on the coordinate difference between the center point of the OCR text region and the center point of the final candidate region, the location of the OCR text region to the left, right, above, or below the final candidate region is determined.

[0202] Finally, the coordinates of the final candidate region's center point are subtracted from the coordinates of the OCR text region's center point, and the result is calculated using the width of the mobile user interface screenshot. and height The differences in the horizontal and vertical coordinates are normalized to obtain a two-dimensional relative position vector. This vector is used as a numerical representation of the relative positional relationship between the final candidate region and the OCR text region, and is incorporated into the spatial layout input. .

[0203] The normalized coordinates, normalized width, normalized height, aspect ratio, page orientation, distance from neighboring final candidate regions, and relative positional relationship with the OCR text region of the final candidate region are combined into a spatial layout input. .

[0204] And Input spatial layout encoding branch to obtain spatial layout features : ; in This indicates a spatial layout coding branch, which can be constructed using a Transformer encoder.

[0205] Step 3.4. Generate dynamic gating weights.

[0206] For the The final candidate regions are selected based on their graphic visual characteristics. Textual semantic features Spatial layout characteristics and candidate region quality factor Generate dynamic gating weights The calculation formula is as follows: ; in and These are trainable parameters.

[0207] For the first The three-channel dynamic gating weight vector corresponding to each final candidate region is expressed as follows: ,in, , and These represent the gating weights corresponding to graphic visual features, text semantic features, and spatial layout features, respectively.

[0208] Candidate region quality factor For the first The candidate region quality factor vector corresponding to each final candidate region; Candidate region quality factor This includes the OCR text recognition confidence score, normalized area, candidate region neighborhood density factor, localization uncertainty, image-text matching score, and prompt word verification score corresponding to the final candidate region.

[0209] .

[0210] in Indicates the first The OCR confidence scores for each final candidate region; the OCR confidence scores for each final candidate region. The spatial mapping relationship between the final candidate regions and the OCR text established in step 1.6 is used to determine the region.

[0211] Specifically, based on the spatial mapping relationship between the final candidate region established in step 1 and the OCR text, the region corresponding to the first candidate region is obtained. The final candidate region is associated with the OCR text within the region, the neighboring OCR text, and their text recognition confidence.

[0212] If there are OCR texts within one or more regions, the average text recognition confidence score of the OCR texts within each region is taken as the OCR confidence score of the final candidate region. .

[0213] If no OCR text exists within the region but neighboring OCR text exists, then the text recognition confidence scores of each neighboring OCR text are weighted and averaged based on the distance between the neighboring OCR text regions and the final candidate region, to obtain the final candidate region. .

[0214] Calculate the weighted average of the confidence scores for neighboring OCR text recognition using the distance-weighted method described in step 1.4.

[0215] If neither the OCR text within the region nor the neighboring OCR text exists, the OCR confidence of the final candidate region is set to 0.

[0216] The normalized area of ​​the final candidate region is calculated from the coordinates of the final candidate region obtained in step 1.5.

[0217] Let the width and height of the final candidate region be respectively... and The width and height of the mobile user interface screenshot are respectively and ,but Represented as: ; in Indicates the first The preliminary candidate region neighborhood density factor corresponding to the preliminary candidate region of the final candidate region.

[0218] Indicates the first The positioning uncertainty corresponding to each final candidate region.

[0219] For the final candidate region obtained through local magnification cropping and secondary candidate region resampling and The neighborhood density factor and location uncertainty of the candidate region calculated in step 1.4 are used respectively for the corresponding preliminary candidate region.

[0220] For preliminary candidate regions that are directly retained without triggering local zoom-out clipping and final candidate region resampling, the neighborhood density factor and location uncertainty of the candidate region calculated in step 1.4 are directly used.

[0221] Indicates the first The final candidate region corresponds to the image-text consistency score. For the candidate prompts retained after filtering in step 2.4, the candidate prompt with the highest verification score is selected based on the image-text consistency score calculated in step 2.3. .

[0222] Indicates the first The prompt word verification score corresponding to each final candidate region.

[0223] For the candidate prompts retained after filtering in step 2.4, the highest verification score is taken as... .

[0224] If no candidate suggestions are retained after filtering, the suggestion validation score will be calculated. Consistency score between text and images All are set to 0, which is the default value when no valid candidate suggestions are obtained.

[0225] Step 3.5. Perform three-channel feature fusion.

[0226] Setting the output feature dimensions of the three encoding branches to be consistent makes the graphic visual features Textual semantic features and spatial layout features All are feature vectors of the same dimension.

[0227] Based on the dynamic gating weights, the three feature vectors are summed element-wise with weights to obtain the GUI-perceptual fusion features. : .in , and These represent the gating weights corresponding to the graphic visual features, text semantic features, and spatial layout features, respectively. + + =1.

[0228] Through a dynamic gating mechanism, the model can adaptively adjust the contribution ratio of different modalities according to the actual situation of the candidate region.

[0229] For example, for candidate regions without text icons or with low OCR confidence, the gating weights corresponding to graphic visual features and spatial layout features can be increased; for buttons or input boxes with explicit text labels, the gating weights corresponding to text semantic features can be increased; and for candidate regions located in navigation bars, title bars, pop-up areas, or list areas, the gating weights corresponding to spatial layout features can be increased. This enhances the model's ability to recognize weak text controls, spatially dependent controls, and composite controls.

[0230] This invention constructs three-channel GUI perception features: graphic vision, text semantics, and spatial layout. It also utilizes dynamic gating weights to adaptively fuse these three types of features, which can automatically adjust feature contributions based on the modal dependency differences of different controls, thereby improving the recognition accuracy of weak text controls, icon controls, and spatially dependent controls.

[0231] Step 4. Fine-tune the multi-channel fusion detection model and output the detection results.

[0232] This invention inputs GUI-perceptual fusion features into a multi-channel fusion detection model. The model outputs the interface element category, position coordinates, functional semantics, and detection confidence score corresponding to each final candidate region. The multi-channel fusion detection model consists of a visual encoding branch, a text encoding branch, a spatial layout encoding branch, a feature fusion layer, and a detection head.

[0233] The visual encoding branch, text encoding branch, and spatial layout encoding branch are used to extract graphic visual features, text semantic features, and spatial layout features, respectively, while the feature fusion layer is used to generate GUI perception fusion features.

[0234] The detection head is used to output the detection results of the final candidate region based on the GUI-perceptual fusion features.

[0235] Step 4 includes three parts: multi-branch adapter fine-tuning, hard negative sample training, and detection output.

[0236] Step 4.1. Configure the multi-branch lightweight adapter.

[0237] Lightweight adapters are set in the visual encoding branch, text encoding branch, spatial layout encoding branch, and feature fusion layer, respectively. The lightweight adapters include the following four: The visual branch adapter is used to adapt the appearance features of icons, borders, and controls in mobile interfaces; The text branch adapter is used to adapt the semantics of control functions in OCR text and standardized prompt words; The spatial branch adapter is used to adapt the candidate region coordinates, size, and neighboring control relationships; The fusion layer adapter is used to adapt the GUI-aware fusion features after the three-channel feature fusion.

[0238] Step 4.2. Fine-tune the lightweight adapter based on training samples from the mobile user interface domain.

[0239] To reduce training costs and maintain the original cross-modal representation capabilities of the visual language model, it is preferable to freeze the main pre-training parameters of the visual language model and jointly update the parameters of each lightweight adapter, dynamic gating module and detection head.

[0240] The trainable parameters of the dynamic gating module include: and If the spatial layout coding branch uses random initialization, then its encoder parameters are included in the training.

[0241] like Figure 4 As shown, for the input features in any coding branch Lightweight adapter It can be represented as: .

[0242] in This is the downprojection matrix, used to map the input features to a low-dimensional space; This is the up-projection matrix, used to map low-dimensional features back to the original feature dimensions; It is a non-linear activation function.

[0243] By introducing residual connections into the lightweight adapter, mobile user interface domain features can be learned without compromising the original pre-trained representation capabilities of the visual language model.

[0244] Step 4.3. Construct hard negative samples of easily confused interface elements.

[0245] During the fine-tuning process, hard negative samples of easily confused interface elements are constructed, including: Control samples that are similar in appearance but different in function, control samples with the same text but different spatial positions, control samples with the same icon but different page contexts, and control samples with similar OCR text but different control types.

[0246] During the training batch construction process, for each positive sample's final candidate region: Candidate regions that are similar in appearance but have different functional labels are selected as visual hard negative samples; Candidate regions that are the same as or similar to the OCR text but have different spatial locations are selected as hard negative samples of the text; Candidate regions with the same expected control category but different page contexts are selected as context hard negative samples.

[0247] Select the final candidate region that is similar to its OCR text but has a different control type as the hard negative sample of the text.

[0248] Hard negative samples and positive samples are used together for comparative learning training of the multi-channel fusion detection model, which enables the multi-channel fusion detection model to narrow the distance between correct image-text pairs in the feature space and widen the distance between positive samples and easily confused hard negative samples.

[0249] Step 4.4. Construct a joint optimization objective.

[0250] Training loss function It can be represented as: .

[0251] in This represents the loss for classifying UI element categories. This indicates the loss in image-text contrast. This represents the hard negative sample contrast loss. This represents the IoU-aware localization loss. 1 to 3 represents the weighting coefficient, which is set to 0.4, 0.4, and 0.2 respectively.

[0252] 1 to 3 represents the weighting coefficients for image-text contrast loss, hard negative sample contrast loss, and IoU perceptual localization loss, respectively.

[0253] The calculation process for each loss function is as follows: .

[0254] in, This indicates the number of samples in the training batch. Indicates the number of categories of interface elements. Indicates the first The training sample belongs to the first... The actual category label of the class, The model predicts the first... The training sample belongs to the first... The probability of a class.

[0255] ; in, and They represent the first The final candidate regions correspond to the graphic visual features and textual semantic features. Represents cosine similarity. This represents the temperature coefficient, with an initial value set to 0.07.

[0256] ; in, For step 4.3 and the The semantic features of the hard negative sample text corresponding to each positive sample. This indicates the preset interval parameter.

[0257] When the similarity between positive sample image and text features is less than the preset interval relative to the similarity between hard negative samples, a loss occurs, thereby causing the model to bring the correct image and text pairs closer and pull away easily confused image and text pairs.

[0258] .

[0259] in, Indicates the first The predicted region corresponding to each training sample This represents the actual area of ​​the interface elements. This represents the intersection-union ratio between the predicted region and the actual interface element region.

[0260] This joint optimization objective enables the multi-channel fusion detection model to simultaneously learn control category determination, image-text semantic alignment, differentiation of easily confused controls, and precise positioning of interface elements.

[0261] Step 4.5. Output the detection results.

[0262] The trained multi-channel fusion detection model is used to identify the final candidate regions. The multi-channel fusion detection model outputs the interface element category, position coordinates, functional semantics and detection confidence of the final candidate regions.

[0263] The training samples in the mobile user interface domain consist of screenshots of mobile user interfaces and their corresponding annotation information.

[0264] For mobile user interface elements in the training samples, label their interface element category, real location coordinates, and functional semantics, and use the corresponding labeling information as supervision labels for model training.

[0265] Among them, the interface element category label is used to calculate the interface element category classification loss, the real location coordinates are used to calculate the IoU perception positioning loss, and the functional semantic label is used to characterize the functional meaning of the interface element.

[0266] Based on the actual position coordinates of the interface elements, the candidate regions generated during the training process are matched with the actual interface element regions. The candidate regions that meet the preset intersection-union ratio (IUU) conditions are determined as positive samples of the corresponding interface elements, and the remaining candidate regions are used as ordinary negative samples or constructed as hard negative samples in accordance with the method described in step 4.3.

[0267] After the multi-channel fusion detection model outputs the results, post-processing is performed on the above detection results, as follows: First, the detection results are sorted from high to low confidence. Then, non-maximum suppression is performed on the detection results that overlap in position, and low confidence detection results that overlap with high confidence detection results by more than a preset threshold are suppressed. Subsequently, the remaining detection results that are spatially adjacent and have the same interface element category and functional semantics are merged to obtain the final detection result of the current mobile user interface screenshot.

[0268] The above post-processing can reduce duplicate detection results caused by multiple final candidate regions corresponding to the same interface element, and retain detection results with high detection confidence and semantic consistency.

[0269] This invention packages the adapter fine-tuning into a branch adaptation and fusion adaptation mechanism oriented towards multi-channel GUI features, and combines it with hard negative sample contrast training, enabling the model to better distinguish interface elements that are similar in appearance but different in function, have the same text but different in position, or have the same icon but different in context, thereby improving domain adaptation capability without significantly increasing training cost.

[0270] This invention addresses typical scenarios in mobile user interfaces, such as small icons, weak text controls, composite controls, spatially dependent controls, and semantically confusing similar icons. Through a coarse-to-fine candidate region resampling mechanism driven by positioning uncertainty, this invention can locally enlarge and reposition candidate regions with low initial detection confidence, severe candidate box overlap, small region size, or insufficient OCR text evidence, thereby improving the quality of candidate boxes for complex interface elements.

[0271] This invention utilizes a large language model to generate multiple candidate prompts and performs self-verification on these prompts through OCR consistency, spatial layout consistency, image-text consistency, and prior consistency of control categories. This avoids semantic illusions and erroneous function inferences that may result from directly using a single prompt. Therefore, the system can obtain more reliable textual semantic input even when candidate areas lack explicit text, icon semantics are implicit, or contextual relationships are complex.

[0272] Simultaneously, this invention constructs a three-channel GUI perception feature set encompassing graphic vision, text semantics, and spatial layout, and adaptively adjusts the feature contributions of different channels through dynamic gating weights. For buttons or input boxes with explicit OCR text, the model increases the weight of text semantic features; for icons without text or small-sized controls, the model increases the weight of graphic visual features and spatial layout features; for interface elements in navigation bars, title bars, pop-up areas, or list areas, the model combines spatial layout features for semantic determination. This mechanism enables the model to adaptively recognize different interface elements based on modal reliability and page context differences.

[0273] Furthermore, this invention sets lightweight adapters in the visual encoding branch, text encoding branch, spatial layout encoding branch, and feature fusion layer, and performs joint optimization by combining hard negative samples of easily confused interface elements, image-text contrast loss, classification loss, and IoU perceptual localization loss. This enables the model to learn the correspondence between icon shapes, text semantics, spatial layout, and control functions unique to mobile user interface scenarios without significantly increasing the model training cost, thereby improving the model's adaptability to different page structures, control styles, and application interface styles.

[0274] In summary, this invention constructs a multi-channel fusion detection framework for mobile user interface detection tasks based on mobile user interface screenshot data, OCR text data, candidate region images, spatial layout information, and self-verification prompts generated by a large language model. It fully integrates the position, size, visual appearance, text content, neighboring control relationships, and page context information of candidate regions, thereby enhancing the detection accuracy, semantic recognition capability, and cross-page generalization capability for complex mobile user interface elements.

[0275] Example 2 This embodiment 2 describes a mobile user interface detection system, which is based on the same inventive concept as the mobile user interface detection method in embodiment 1 above.

[0276] The mobile user interface detection system in this embodiment includes the following modules: The candidate region generation module is used to collect screenshot data of mobile user interfaces, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, it performs initial candidate region generation on it to obtain one or more preliminary candidate regions. Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions with a positioning uncertainty greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot; The secondary candidate regions and the initial candidate regions that did not trigger resampling of the secondary candidate regions are combined to form the final candidate region set; For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text within the region are determined according to the spatial relationship between the final candidate region and the OCR text region. The standardized prompt word generation module is used to construct structured prompt word generation input based on the attribute information of the final candidate region; The attribute information of the final candidate region includes region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information; A large language model is used to generate candidate prompt words for each final candidate region, and standardized prompt words are generated through self-verification. The fusion feature generation module is used to input the region image of the final candidate region into the visual encoding branch of the visual language model to obtain graphic visual features, and input the standardized prompt words and OCR text into the text encoding branch to obtain text semantic features; Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding. Weighted fusion of graphic visual features, text semantic features, and spatial layout features yields GUI perception fusion features. And a mobile user interface detection module, which is used to input GUI perception fusion features into a multi-channel fusion detection model, and the model outputs the interface element category, position coordinates, functional semantics and detection confidence of each final candidate region; It should be noted that any content not mentioned in the above-described functional modules of the system described in Embodiment 2 can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0277] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for detecting mobile user interfaces, characterized in that, Includes the following steps: Step 1. Collect screenshot data of the mobile user interface, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, perform initial candidate region generation on it to obtain one or more preliminary candidate regions; Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions with a positioning uncertainty greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot; The secondary candidate regions and the initial candidate regions that did not trigger resampling of the secondary candidate regions are combined to form the final candidate region set; For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text within the region are determined according to the spatial relationship between the final candidate region and the OCR text region. Step 2. Construct structured prompt words to generate input based on the region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information of the final candidate region; A large language model is used to generate candidate prompt words for each final candidate region, and standardized prompt words are generated through self-verification. Step 3. Input the region image of the final candidate region into the visual encoding branch of the visual language model to obtain the graphic visual features, and input the standardized prompt words and OCR text into the text encoding branch to obtain the text semantic features; Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding. Weighted fusion of graphic visual features, text semantic features, and spatial layout features yields GUI perception fusion features. Step 4. Input the GUI-aware fusion features into the multi-channel fusion detection model. The model outputs the interface element category, location coordinates, functional semantics, and detection confidence of each final candidate region.

2. The mobile user interface detection method according to claim 1, characterized in that, In step 1, the positioning uncertainty The calculation process is as follows: ； in Indicates the uncertainty factor of the region size. Represents the neighborhood density factor of the candidate region. Indicates the confidence level of OCR text recognition. Indicates the overlap factor of candidate regions. This indicates the initial detection confidence level. to These are the weighting coefficients; I. The formula for the uncertainty factor of the region size is as follows: ； in For the first Normalized area of ​​the initial candidate regions For the preset area threshold, This indicates taking the minimum value; II. The neighborhood density factor of the candidate region is expressed as: ； in This represents the region density threshold. This represents the number of other preliminary candidate regions within the neighborhood of the preliminary candidate region. III. The process for determining the confidence level of OCR text recognition is as follows: a. If the first If a preliminary candidate region contains one or more OCR text regions, then the average value of the text recognition confidence scores corresponding to the OCR text regions is taken as the OCR text recognition confidence score of the preliminary candidate region. b. If there is no OCR text region within the initial candidate region but there are adjacent OCR text regions, then: Based on the distance between the neighboring OCR text regions and the preliminary candidate region, the text recognition confidence scores corresponding to each neighboring OCR text region are weighted and averaged to obtain the OCR text recognition confidence score of the preliminary candidate region. c. If there are no OCR text regions within or near the preliminary candidate region, then the OCR text recognition confidence of the preliminary candidate region is set to 0; IV. Candidate Region Overlap Factor By calculating the first Preliminary candidate areas Other preliminary candidate areas The intersection-union ratio between the two is used to determine the maximum value, and it is expressed as: ； in Indicates the solution and Maximum crossover ratio between them; V. Initial detection confidence Obtained from the initial candidate region generation process.

3. The mobile user interface detection method according to claim 1, characterized in that, In step 1, the process of locally zooming out and cropping the initial candidate regions with positioning uncertainty greater than a preset positioning uncertainty threshold, resampling the secondary candidate regions, and mapping the coordinates of the secondary candidate regions back to the original interface screenshot is as follows: First, the center of the initial candidate region is used as the cropping center. Local image blocks are obtained according to the preset cropping range, and the local image blocks are scaled according to the preset scaling ratio. Candidate region generation is performed again in the scaled local image patch to obtain one or more secondary candidate regions; the secondary candidate regions are candidate regions obtained after local magnification and cropping and re-generation of candidate regions. Let the offset of the local image patch from the top left corner of the original screenshot be . The scaling ratio is The coordinates of the secondary candidate region in the local image patch, after inverse transformation, are as follows: ； in This represents the coordinates of the secondary candidate region within the local image patch.

4. The mobile user interface detection method according to claim 1, characterized in that, In step 2, the candidate prompts generated by the large language model include candidate prompts for control categories, candidate prompts for functional actions, candidate prompts for visual appearance, and candidate prompts for spatial relationships.

5. The mobile user interface detection method according to claim 1, characterized in that, In step 2, the self-verification process is as follows: Define the verification score The formula is as follows: ; in This indicates the OCR consistency between the candidate prompts and the OCR text; This indicates the spatial consistency between candidate prompts and spatial layout priors; This indicates the consistency between the candidate prompts and the region image; This indicates the consistency between the candidate suggestion words and the control category priors; to To satisfy the non-negative weight coefficients of the normalization constraint, ; If the verification score of a candidate suggestion word is lower than the preset verification score threshold, the candidate suggestion word will be removed. For candidate suggestion words with a verification score not lower than the preset verification score threshold, sort them from highest to lowest verification score: When a candidate suggestion word and a candidate suggestion word ranked before it mutually exclude each other for the same final candidate area, it is determined that there is a functional semantic conflict between the two, and the candidate suggestion word with the lower verification score is removed. If there are multiple verification scores that are not lower than the preset verification score threshold Candidate suggestions that do not have functional semantic conflicts are sorted according to their verification scores from highest to lowest, and the top ones are selected. 10 candidate prompt words were selected and merged into standardized prompt words; in To retain the preset quantity, , The total number of candidate suggestions generated for the same final candidate region; when the number of candidate suggestions meeting the above conditions is less than When that happens, select all candidate prompts that meet the criteria and merge them.

6. The mobile user interface detection method according to claim 5, characterized in that, The OCR consistency The calculation formula is as follows: ； in Indicates the first OCR text content, This indicates the confidence level of the corresponding text recognition. Indicates the first The candidate prompt and the first Text similarity between OCR texts Indicates the weight of the OCR text; For OCR text within a region, its text weight is set to 1; for neighboring OCR text, its text weight is determined based on the normalized minimum boundary distance between the OCR text region and the final candidate region. The closer the distance, the greater the text weight. Both the OCR text within the region and the neighboring OCR text are obtained from the spatial mapping relationship established in step 1.6; in the formula... A very small positive number is set to prevent the denominator from being 0; This represents the OCR text within the area. Indicates adjacent OCR text; Spatial consistency The calculation process is as follows: Set candidate prompts The corresponding candidate control category is The normalized coordinates of the center point of the final candidate region are: Control categories The corresponding preset spatial prior region is ,but Represented as: ； in, Indicates the distance from the center point of the final candidate area to the control category. The normalized distance corresponding to the prior spatial region, where σ represents the distance decay parameter; Consistency between text and images The calculation process is as follows: The final candidate region image is input into the visual encoding branch of the visual language model to obtain image features. : ;in, Represents a visual encoder. Candidate region image; The candidate prompt words are input into the text encoding branch to obtain text features. : ;in, Indicates a text encoder. These are candidate suggestions; And calculate the cosine similarity between image features and text features as... : ； in, and These represent the normalized image features and text features, respectively; Control Prior Consistency The calculation process is as follows: ； in , , and These represent the form matching score, text matching score, context matching score, and interaction attribute matching score, respectively. to These are non-negative weighting coefficients, and ; Indicates the first The candidate control categories corresponding to each candidate prompt word; Indicates the first The morphological information of each final candidate region, including region shape, region size, and aspect ratio; Indicates the first Text information corresponding to each final candidate region; For the first The page context information corresponding to each final candidate region includes the page orientation, the distribution of neighboring candidate regions, and their relative positional relationship with the OCR text region; For the first Interactive attribute information corresponding to each final candidate region.

7. The mobile user interface detection method according to claim 1, characterized in that, In step 3, the visual language model is the CLIP model, the visual encoding branch corresponds to the image encoder of the CLIP model, and the text encoding branch corresponds to the text encoder of the CLIP model. The spatial layout coding branch is composed of a Transformer encoder; the spatial layout channel coding process is as follows: For the There are 1 final candidate regions, whose coordinates are 1 / 2. The width of the mobile user interface screenshot is The height is Then its normalized coordinates Normalized width Normalization height and aspect ratio They are represented as follows: ； ； ； ； in, A very small positive number is set to prevent the denominator from being 0; Simultaneously obtain the page orientation of the final candidate area, its distance from neighboring final candidate areas, and its relative positional relationship with the OCR text area; combine the above information with... , , and Combined as spatial layout input ; Input the spatial layout Input spatial layout encoding branch to obtain spatial layout features : ; in This represents the spatial layout coding branch, which is composed of a Transformer encoder.

8. The mobile user interface detection method according to claim 7, characterized in that, In step 3, the process of generating dynamic gating weights and performing weighted fusion is as follows: For the Each final candidate region is assigned a dynamic gating weight based on its graphic visual features, textual semantic features, spatial layout features, and candidate region quality factor. The calculation formula is as follows: ； in, For the first The three-channel dynamic gating weight vector corresponding to each final candidate region is expressed as follows: ,in , and These represent the gating weights corresponding to graphic visual features, text semantic features, and spatial layout features, respectively. in For the first The candidate region quality factor vector corresponding to each final candidate region; The metrics include OCR text recognition confidence, normalized area, candidate region neighborhood density factor, localization uncertainty, image-text matching score, and prompt word verification score, expressed as: ; in Indicates the first OCR confidence scores for each final candidate region; The normalized area of ​​the final candidate region is calculated from its coordinates. Indicates the first The neighborhood density factor of each final candidate region; Indicates the first The positioning uncertainty corresponding to each final candidate region; Indicates the first The image-text consistency score corresponding to each final candidate region; Indicates the first The prompt word verification score corresponding to each final candidate region; and These are trainable parameters; GUI-aware fusion features are obtained based on dynamic gating weights. : ;in , and These represent the gating weights corresponding to the graphic visual features, text semantic features, and spatial layout features, respectively. + + =1.

9. The mobile user interface detection method according to claim 1, characterized in that, In step 4, a lightweight adapter is set in the visual encoding branch, text encoding branch, spatial layout encoding branch and feature fusion layer, and the lightweight adapter is fine-tuned based on training samples from the mobile user interface domain. During the fine-tuning process, construct hard negative samples of easily confused interface elements; The model is optimized by combining classification loss, image-text comparison loss, hard negative sample comparison loss, and IoU perceptual localization loss. The optimized model is then used to output the interface element category, location coordinates, functional semantics, and detection confidence of the final candidate region.

10. A mobile user interface detection system, characterized in that, Includes the following modules: The candidate region generation module is used to collect screenshot data of mobile user interfaces, perform text detection and recognition on it to obtain one or more OCR text regions; at the same time, it performs initial candidate region generation on it to obtain one or more preliminary candidate regions. Calculate the positioning uncertainty of each preliminary candidate region, perform local zoom-out and cropping on the preliminary candidate regions with a positioning uncertainty greater than the preset positioning uncertainty threshold, and resample the secondary candidate regions, and map the coordinates of the secondary candidate regions back to the original interface screenshot; The secondary candidate regions and the initial candidate regions that did not trigger resampling of the secondary candidate regions are combined to form the final candidate region set; For each final candidate region determined in the final candidate region set, the OCR text within the region and the adjacent OCR text within the region are determined according to the spatial relationship between the final candidate region and the OCR text region. The standardized prompt word generation module is used to construct structured prompt word generation input based on the attribute information of the final candidate region; The attribute information of the final candidate region includes region location, region size, aspect ratio, OCR text within the region, neighboring OCR text, text recognition confidence, page orientation, and page context information; A large language model is used to generate candidate prompt words for each final candidate region, and standardized prompt words are generated through self-verification. The fusion feature generation module is used to input the region image of the final candidate region into the visual encoding branch of the visual language model to obtain graphic visual features, and input the standardized prompt words and OCR text into the text encoding branch to obtain text semantic features; Spatial layout features of the final candidate region are obtained by using spatial layout coding branch to perform spatial layout channel coding. Weighted fusion of graphic visual features, text semantic features, and spatial layout features yields GUI perception fusion features. It also includes a mobile user interface detection module, which inputs GUI-aware fusion features into a multi-channel fusion detection model. The model outputs the interface element category, location coordinates, functional semantics, and detection confidence for each final candidate region.

Images