Method and system for positioning interface display defects related to settings in an android application

Abstract

The application discloses an interface display defect positioning method and system related to settings in an Android application, and the method comprises the following steps: storing defect reports in an empty set BUGS through initialization, and obtaining all UI pages by injecting a test Activity; traversing each pair of UI components in each UI page, comparing their visual feature relationships after default settings and system setting changes, including consistency of color, alignment mode, distance and inclusion relationship; if any inconsistency is detected, a defect report is generated and added to the BUGS set; finally, a report set containing all detected style-related and semantic-related defects is returned. The application obtains UI pages by injecting a test Activity, compares visual feature relationships of components under different settings, and realizes automatic detection and report generation of setting-related interface display defects.

Description

Technical Field

[0001] This invention relates to the field of software engineering technology, and specifically to a method and system for locating interface display defects related to settings in Android applications. Background Technology

[0002] Android provides a powerful and flexible framework that allows developers to build beautiful and highly interactive UIs. Views are the fundamental building blocks of the Android UI, and the UI rendered by an Android application consists of a view tree. Initially, developers use XML files to define the tree structure of the UI. These XML files are located in the res / layout directory and are converted into actual UI components at runtime. The attributes of views defined in the XML configuration file (such as view ID, layout height, and text content) determine their initial appearance and behavior, such as the position and size of buttons, text, and image views.

[0003] As a popular mobile operating system, Android comes pre-loaded with various application components for developers to build application UIs using XML. A study of 200 top-performing applications found that each application had an average of 663.1 layout files and 25,991.6 XML elements, demonstrating the complexity and breadth of XML in defining mobile application UIs.

[0004] Screen User Design (SUD) defects typically stem from developers failing to adapt XML configuration files to different system settings. This is primarily because Android application UI layouts are defined through static XML files. When faced with diverse device environments (different screen sizes, resolutions, operating system versions, etc.), if the XML configuration doesn't handle these changes correctly, display issues will occur. Adaptation is complex and time-consuming, requiring UI layout design and testing for multiple scenarios. This is easily simplified or ignored when resources are limited or time is tight. Furthermore, the lack of effective automation tools in the past made manual adjustments and testing both error-prone and inefficient. With the continuous release of new features and APIs, the ongoing updates and maintenance of applications can also easily introduce new SUD defects.

[0005] Android provides a range of system settings that allow users to customize their devices and applications according to their personal preferences. These system settings cover various aspects, such as language, font size, brightness, and color mode. Users might change the system language from English to Arabic. While system settings meet users' needs in different environments, they also place a burden on developers. Developers need to adjust their applications to adapt to various system settings; otherwise, it can lead to settings-related defects, referred to in this article as "Setting-related UI Display Bugs" (SUD defects). Developing automated methods to detect SUD defects faces two key challenges: covering all potential UI components and building a universal test oracle. The first ensures the breadth of defect detection, while the second is used to verify whether UI components have SUD defects. To ensure the accuracy of verification, each system setting requires a customized test oracle, taking into account the application's specific contextual information. SUD defects are mainly located in XML files and involve the hierarchy of XML elements and attribute configurations; existing dynamic-based methods rely on random testing to generate test cases, making it difficult to cover all UI components. Furthermore, their test predictions do not fully cover the expected adaptation results of the system settings, leading to false negative results. After the system language is changed, the layout alignment does not meet the expected results, and existing work cannot identify this defect. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a method and system for locating UI display defects related to settings in Android applications. Through research, common patterns of unexpected UI adaptations that lead to SUD defects have been identified, and a method for automatically detecting SUD defects in Android applications has been proposed.

[0007] The specific plan is as follows:

[0008] On the one hand, methods for locating defects in settings-related interfaces in Android applications include:

[0009] S1, initialize an empty set BUGS for storing detected defect reports;

[0010] S2, Inject the Activity function for testing to obtain all UI pages in the Android application; the input parameters of the Activity function include the ID of the UI page;

[0011] S3, iterate through all UI pages in sequence, get all UI components in each UI page, combine the UI components under the same UI page in pairs to get multiple UI component pairs, and get the relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed.

[0012] S4. Based on the relationships under the default settings and the relationships after the Android system settings are modified, check each UI page for defects in turn. If a UI component pair on a certain UI page does not maintain the same relationship after the Android system settings are modified as the default relationship, then a defect is detected. After all UI component pairs on that UI page have been checked, a defect report is generated based on all detected defects and added to the BUGS collection.

[0013] S5, after all UI pages have been inspected, returns the collection of detected problem reports (BUGS).

[0014] Furthermore, in S4, the detected defect types include style-related SUD defects and semantic-related SUD defects; the style-related SUD defects include inconsistencies in color modification, inconsistencies in alignment, inconsistencies in distance, and inconsistencies in containment relationships; the semantic-related SUD defects include unexpected UI visibility, incorrect data processing, and inconsistent translations.

[0015] Furthermore, the method for determining inconsistencies in color modifications is as follows:

[0016] The system detects two UI components that have the same color under default settings and checks if their colors remain consistent after the settings are changed. If they do not remain consistent, the color modification is considered inconsistent, as follows:

[0017] u1.color = u2.color and u'1.color ≠ u'2.color;

[0018] u1 and u2 represent two UI components in their original state; u1.color and u2.color represent the color attribute values ​​of these two UI components under default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.color and u'2.color represent the color attribute values ​​of these two UI components after the settings have been changed.

[0019] Furthermore, the method for determining inconsistencies in alignment is as follows:

[0020] This function checks whether two UI components that are aligned under default settings still maintain alignment after the settings are changed. If they are not aligned, it is determined that the alignment methods are inconsistent. Specifically:

[0021] Compare the six key points of the visual boundaries of UI components u1 and u2: left (L), right (R), top (T), bottom (B), vertical center (VC), and horizontal center (HC). Define u1 and u2 as aligned if the following conditions are met:

[0022]

[0023] If the following conditions are met after system settings are modified, the alignment is considered inconsistent:

[0024]

[0025] Here, u1 and u2 represent two UI components in their original state; u1.align and u2.align represent the alignment of these two UI components under the default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.align and u'2.align represent the alignment of these two UI components after the settings have been changed.

[0026] Furthermore, the alignment detection methods specifically include:

[0027] S11: Initialize an empty collection `align` to store the alignment of a pair of UI components;

[0028] S12: Obtain the visible rectangular range of each UI component from the UI page and calculate their respective center points;

[0029] S13: Based on the top-left, bottom-right, and center coordinates of the rectangular visible area of ​​the UI components, detect the alignment of the pair of UI components: If the x-coordinates of the top-left corners are equal, the pair of UI components are left-aligned, and a left alignment identifier L is added to the align set; if the x-coordinates of the bottom-right corners are equal, the pair of UI components are right-aligned, and a right alignment identifier R is added to the align set; if the x-coordinates of the center coordinates are equal, the pair of UI components are horizontally center-aligned, and a horizontal center alignment identifier HC is added to the align set; if the y-coordinates of the top-left corners are equal, the pair of UI components are top-aligned, and a top alignment identifier T is added to the align set; if the y-coordinates of the bottom-right corners are equal, the pair of UI components are bottom-aligned, and a bottom alignment identifier B is added to the align set; if the y-coordinates of the center coordinates are equal, the pair of UI components are vertically center-aligned, and a vertical center alignment identifier VC is added to the align set.

[0030] S14: After detecting the alignment, return the set of alignment options for this pair of UI components.

[0031] Furthermore, the method for determining inconsistencies in distance is as follows:

[0032] Calculate the horizontal Euclidean distance dh(u1,u2) and the vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set horizontal threshold th or vertical threshold tv before and after the system settings change, it is determined to be an inconsistency in distance. The specific formula is as follows:

[0033] ∣dh(u1′,u2′)-dh(u1,u2)∣≥th or ∣dv(u1′,u2′)-dv(u1,u2)∣≥tv;

[0034] Here, u1' and u2' represent the two UI components u1 and u2 after their settings have been changed, respectively.

[0035] Furthermore, the method for determining inconsistencies in inclusion relationships is as follows:

[0036] Check the overlap between the visible boundaries of components u1 and u2, denoted as overlap(u1, u2); if the overlap changes after system settings are changed, it is determined that the containment relationship is inconsistent, as shown in the following formula:

[0037] overlap(u1,u2)≠overlap(u1',u2');

[0038] Here, u1' and u2' represent the two UI components after the settings of u1 and u2 have been changed, respectively.

[0039] On the other hand, the Android application's settings-related interface displays a defect location system, including:

[0040] The initialization module is used to initialize an empty set BUGS for storing detected defect reports;

[0041] The page retrieval module is used to retrieve all UI pages in an Android application by injecting an Activity for testing.

[0042] The page traversal module is used to traverse all UI pages sequentially, obtain all UI components in each UI page, combine UI components under the same UI page in pairs to obtain multiple UI component pairs, and obtain the relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed.

[0043] The defect detection module is used to detect whether there are defects in each UI page based on the relationships under the default settings and the relationships after the Android system settings are modified. If a UI component on a UI page does not maintain the same relationship after the Android system settings are modified as the default relationship, a defect is detected. After all UI components on that UI page have been detected, a defect report is generated based on all detected defects and added to the BUGS collection.

[0044] The defect return module is used to return a set of bug reports (BUGS) after all UI pages have been inspected.

[0045] The present invention adopts the above technical solution and has the following beneficial effects:

[0046] (1) By comparing the default settings and the modified relationships, this invention can accurately determine inconsistencies such as color, alignment, distance and containment relationship, and automatically generate detailed defect reports to help developers quickly locate and fix interface display defects and improve application quality.

[0047] (2) This invention obtains all UI pages by injecting a test Activity and iterates through and compares the visual feature relationships of each UI component under different settings, thereby achieving comprehensive automated detection of style-related and semantic-related SUD defects, which significantly improves detection efficiency and coverage.

[0048] (3) This invention designs and implements an automated detection tool for the interface display defect localization method (SUDFinder) related to settings in Android applications. It uses the extracted SUD defect pattern as a test oracle and performs defect detection directly in the application UI's XML configuration file. This achieves efficient and accurate identification of previously unknown SUD defects, which not only improves test coverage but also successfully overcomes the inefficiency and high false negative rate problems caused by existing methods relying on random exploration. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method for locating interface display defects related to settings in an Android application according to an embodiment of the present invention;

[0050] Figure 2(a) is a schematic diagram of the behavior of an embodiment of the present invention under default settings;

[0051] Figure 2(b) is a schematic diagram of the SUD defect shown in the embodiment of the present invention when switching to Arabic;

[0052] Figure 2(c) is an illustration of the effect of repairing the SUD defect according to an embodiment of the present invention;

[0053] Figure 3This is a schematic diagram of the interface display defect localization algorithm related to settings in an Android application according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of a test Activity injected for defect detection according to an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of a patch for repairing SUD defects related to the detection style in an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of a prompt word template according to an embodiment of the present invention;

[0057] Figure 7 This is a schematic diagram of two true positive cases of SUDFinder on AnkiDroid according to an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the alignment detection algorithm according to an embodiment of the present invention;

[0059] Figure 9 This is a diagram of a defect location system for the interface display related to settings in an Android application according to an embodiment of the present invention. Detailed Implementation

[0060] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0061] like Figure 1 As shown, the method for locating interface display defects related to settings in an Android application according to the present invention includes:

[0062] S1, initialize an empty set BUGS for storing detected defect reports.

[0063] Specifically, the detected defect types include style-related SUD defects and semantic-related SUD defects; style-related SUD defects include inconsistencies in color modification, alignment, distance, and containment relationships; semantic-related SUD defects include unexpected UI visibility, incorrect data processing, and inconsistent translations.

[0064] Specifically, the method for determining inconsistencies in color modifications is as follows:

[0065] The system detects two UI components that have the same color under default settings and checks if their colors remain consistent after the settings are changed. If they do not remain consistent, the color modification is considered inconsistent, as follows:

[0066] u1.color = u2.color and u'1.color ≠ u'2.color;

[0067] u1 and u2 represent two UI components in their original state; u1.color and u2.color represent the color attribute values ​​of these two UI components under default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.color and u'2.color represent the color attribute values ​​of these two UI components after the settings have been changed.

[0068] Specifically, the method for determining inconsistencies in alignment is as follows:

[0069] This function checks whether two UI components that are aligned under default settings still maintain alignment after the settings are changed. If they are not aligned, it is determined that the alignment methods are inconsistent. Specifically:

[0070] Compare the six key points of the visual boundaries of UI components u1 and u2: left (L), right (R), top (T), bottom (B), vertical center (VC), and horizontal center (HC). Define u1 and u2 as aligned if the following conditions are met:

[0071]

[0072] If the following conditions are met after system settings are modified, the alignment is considered inconsistent:

[0073]

[0074] Here, u1 and u2 represent two UI components in their original state; u1.align and u2.align represent the alignment of these two UI components under the default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.align and u'2.align represent the alignment of these two UI components after the settings have been changed.

[0075] Specifically, the alignment detection method, such as Figure 8 As shown, it specifically includes:

[0076] S11: Initialize an empty collection `align` to store the alignment of a pair of UI components;

[0077] S12: Obtain the visible rectangular range of each UI component from the UI page and calculate their respective center points;

[0078] S13: Based on the top-left, bottom-right, and center coordinates of the rectangular visible area of ​​the UI components, detect the alignment of the pair of UI components: If the x-coordinates of the top-left corners are equal, the pair of UI components are left-aligned, and a left alignment identifier L is added to the align set; if the x-coordinates of the bottom-right corners are equal, the pair of UI components are right-aligned, and a right alignment identifier R is added to the align set; if the x-coordinates of the center coordinates are equal, the pair of UI components are horizontally center-aligned, and a horizontal center alignment identifier HC is added to the align set; if the y-coordinates of the top-left corners are equal, the pair of UI components are top-aligned, and a top alignment identifier T is added to the align set; if the y-coordinates of the bottom-right corners are equal, the pair of UI components are bottom-aligned, and a bottom alignment identifier B is added to the align set; if the y-coordinates of the center coordinates are equal, the pair of UI components are vertically center-aligned, and a vertical center alignment identifier VC is added to the align set.

[0079] S14: After detecting the alignment, return the set of alignment options for this pair of UI components.

[0080] Specifically, this embodiment mainly detects style-related SUD defects (T1 type). When the settings change, UI components with similar visual characteristics should have the same adaptation method to maintain a consistent visual effect after the system settings change. As shown in Figures 2(a), 2(b), and 2(c), Figure 2(a) is a schematic diagram of the behavior of this embodiment under the default settings; Figure 2(b) is a schematic diagram of the SUD defects displayed when switching to Arabic; Figure 2(c) is an effect diagram after the SUD defects are repaired in this embodiment. When the system is set to a right-to-left language, the title and text area of ​​the text should be aligned and close to the icon, prompting the use of different metrics to evaluate the relationship between UI components on the page, identifying UI components with defects that cannot maintain these relationships, and finally generating a defect report. In this embodiment, Figures 2(a) and 2(c) are aligned, while Figure 2(b) is not aligned because the system language of Figures 2(b) and 2(c) is an RTL language read from right to left, so the alignment direction is right alignment.

[0081] Specifically, the method for determining inconsistencies in distance is as follows:

[0082] Calculate the horizontal Euclidean distance dh(u1,u2) and the vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set horizontal threshold th or vertical threshold tv before and after the system settings change, it is determined to be an inconsistency in distance. The specific formula is as follows:

[0083] ∣dh(u1′,u2′)-dh(u1,u2)∣≥th or ∣dv(u1′,u2′)-dv(u1,u2)∣≥tv;

[0084] Here, u1' and u2' represent the two UI components u1 and u2 after their settings have been changed, respectively.

[0085] Specifically, the method for determining inconsistencies in inclusion relationships is as follows:

[0086] Check the overlap between the visible boundaries of components u1 and u2, denoted as overlap(u1, u2); if the overlap changes after system settings are changed, it is determined that the containment relationship is inconsistent, as shown in the following formula:

[0087] overlap(u1,u2)≠overlap(u1',u2');

[0088] Here, u1' and u2' represent the two UI components after the settings of u1 and u2 have been changed, respectively.

[0089] S2 obtains all UI pages in the Android application by injecting the Activity used for testing;

[0090] Specifically, SUDFinder injects a test Activity into the application to display the visual appearance of all UI pages. Figure 4 A test Activity injected to detect the defect shown in Figure 2(b) is illustrated. Specifically, the test Activity calls the `setContentView` method, passing the UI ID as a parameter. This design choice was made because XML configuration files are widely used to build application UIs, and many SUD defects reside in XML configuration files. This allows for automatic detection of SUD defects without the need to build test cases to dynamically explore UI components, achieving higher UI coverage. When running the application under test, SUDFinder detects whether the currently visible content supports horizontal or vertical scrolling and performs the corresponding scrolling operation to load and display new UI elements. This step is crucial for revealing views that were part of the screen layout under the original settings (e.g., default font size) but have been pushed out of view due to settings changes (e.g., increased font size). Only after obtaining these new visible views can a comprehensive comparison of all displayed views before and after the system settings modification be performed.

[0091] Specifically, the application's UI pages contain text retrieved from the network, rather than loaded from predefined strings. If this text is missing, the TextView's UI will not display correctly. For example... Figure 5 As shown, the text content of `text_title` and `text_domain` in the TextView is not specified in the XML configuration file, but is retrieved from the server at runtime. Most of these TextView views lack relevant test cases, making it difficult for the system to obtain text that matches real-world application scenarios.

[0092] To address this issue, SUDFinder utilizes Large Language Models (LLMs) to generate missing text in the XML configuration file. Trained on a large corpus of internet data, these LLMs acquire rich domain knowledge. In this embodiment, the LLM is provided with contextual information about the UI component being tested, allowing it to infer text that corresponds to the TextView view. This method aims to generate realistic and context-appropriate text for subsequent testing. In this way, the accuracy of the UI display is improved, and potential problems arising from different text inputs can be identified.

[0093] This embodiment employs an LLM-based text input generation method, with prompt word templates such as... Figure 6 As shown. Specifically, the template takes the application name, XML configuration file name, and XML code as input, allowing the LLM to infer the context information of the TextView and EditText views in the XML code, thereby generating appropriate text content. This ensures that the LLM only outputs XML code without including additional information, and replaces the content of the original XML configuration file with the generated XML code, ultimately building the application for testing. By injecting test Activities, 35 additional SUD defects were discovered, and 34 of these SUD defects were located in the text generated by the LLM.

[0094] S3, iterate through all UI pages in sequence, get all UI components in each UI page, combine the UI components under the same UI page in pairs to get multiple UI component pairs, and get the relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed.

[0095] Specifically, SUDFinder will examine the following three visual feature relationships R(u1,u2) for each pair of UI components u1 and u2.

[0096] The three visual feature relationships include: inconsistency in color changes, inconsistency in alignment changes, and inconsistency in distance.

[0097] To detect defects of inconsistent color changes, SUDFinder assumes that two UI components with the same color under default settings will maintain that color even after the settings are changed. Therefore, SUDFinder will generate a defect report when the following conditions are met:

[0098] u1.color = u2.color and u'1.color ≠ u'2.color;

[0099] To detect inconsistencies in alignment, SUDFinder assumes that two UI components that remain aligned under default settings will remain aligned after the settings are changed. Specifically, SUDFinder compares the six key points of the visual boundaries (bndvis) of UI components u1 and u2: left (L), right (R), top (T), bottom (B), vertical center (VC), and horizontal center (HC). u1 and u2 are defined as aligned when the following conditions are met:

[0100]

[0101] After system settings are modified, SUDFinder will generate a defect report if it detects that the following conditions are met:

[0102]

[0103] To detect inconsistencies in distance, SUDFinder assumes that the distance between the visual content of two UI components should remain as consistent as possible with the default settings unless a change in system settings causes a screen change (e.g., screen rotation). Therefore, SUDFinder calculates the horizontal Euclidean distance dh(u1,u2) and the vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries bndvis of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set thresholds (horizontal threshold th and vertical threshold tv) before and after a change in system settings, SUDFinder will generate a defect report. See below for details:

[0104] ∣dh(u1′,u2′)-dh(u1,u2)∣≥th or ∣dv(u1′,u2′)-dv(u1,u2)∣≥tv;

[0105] To detect inconsistencies related to overlap, SUDFinder checks whether the overlap between the visible boundaries (bndvis) of components u1 and u2 (denoted as overlap(u1,u2)) changes after system settings are modified. If the overlap changes, SUDFinder will report a defect, specifically:

[0106] overlap(u1,u2)≠overlap(u1',u2');

[0107] S4. Based on the relationship between the two UI components under the default settings and the relationship after the Android system settings are modified, determine whether there is a defect in the UI page; if the relationship between the two UI components after the system settings are modified is not consistent with the relationship under the default settings, then a defect is detected and a defect report is generated and added to the BUGS collection.

[0108] S5, after all UI pages have been inspected, returns the collection of detected problem reports (BUGS).

[0109] Specifically, such as Figure 3 As shown, Algorithm 1 illustrates the overall process of SUDFinder. For each UI page in an Android application, SUDFinder first extracts the visual features of the UI components on the page. In formal representation, a UI component u is defined as a tuple.<bndvh,bndvis,color> Specifically: bnd vh This represents the layout boundary of component u in the view hierarchy, as shown by the black dashed line in Figure 2, denoted as <(x l vh ,y l vh ),(x r vh ,y r vh )>;(x l vh ,y l vh (x) represents the coordinates of the top-left corner. r vh ,y r vh () represents the coordinates of the bottom right corner; bnd vis This represents in bnd vh The smallest rectangular area that can cover all non-transparent pixels, i.e., the visible content within component u, such as... Figure 1 The red solid line portion in the image is represented as <(x) l vis ,y l vis ),(x r vis ,y r vis )>, where (x l vis ,y l vis (x) represents the coordinates of the top-left corner.r vis ,y r vis The coordinates of the bottom right corner are ; color represents the two main colors displayed by component u, expressed as .<c1,c2> SUDFinder determines colors by calling the `getcolors()` method from the Python Image Library. While UI components may contain multiple colors, their color schemes are usually simple, involving only background and text / image colors. Then, for two UI components u1 and u2, SUDFinder extracts the relationship between u1 and u2 under default settings, denoted as R(u1, u2), and the relationship after settings modification, R(u'1, u'2). If R(u1, u2) ≠ R(u'1, u'2), SUDFinder will generate a defect report.

[0110] Specifically, this embodiment is based on an empirical study of 218 real-world SUD defects. By utilizing common patterns discovered in the empirical study as test predictions, and the design principle that "UI components with similar visual characteristics tend to adapt to changes in settings in similar ways," SUDFinder performs defect detection in the application UI's XML configuration file to cover more UI components during testing. This avoids dynamically exploring the UI interface through test case construction to achieve high coverage. Although this process ignores application code that may dynamically change the UI, the empirical study shows that SUD defects are often caused by developers failing to successfully adapt the XML configuration file to different system settings. SUDFinder was implemented and experimentally evaluated on 28 popular open-source Android applications in F-Droid. The results show that SUDFinder successfully detected 84 defects with 80% accuracy and found 51 defects that state-of-the-art methods failed to detect. Reports of these previously unknown SUD defects were submitted to application developers, of which 67 defects have been confirmed by developers, and 37 have been fixed and merged.

[0111] The dataset used in this embodiment contains 1074 settings-related defects from open-source Android applications. Among them, 218 defects are related to SUD (Surface Adaptive Defects), which affect the UI display of the application. Table 1 shows the statistics of these SUD defects and their corresponding system settings, with 81.6% of the SUD defects related to screen, language, and theme settings.

[0112] Table 1. Classification and subclassification of system settings

[0113]

[0114] The data analysis process is as follows: First, 109 SUD defects were randomly selected as samples (accounting for 50% of all SUD defects), and the code revisions and corresponding defect reports of each sample were checked to identify the code segments related to the patch; and a preliminary classification method was formulated to classify the remaining 109 defects by label in turn.

[0115] Specifically, this embodiment uses a server with a 64-core Intel Xeon CPU E5-2683 v4@2.10GHz processor and 192GB of memory. An Android device (Galaxy S8) configured with API level 34 and a 1440×2960 pixel display resolution was used. The horizontal threshold th and vertical threshold tv were set to 144 and 296, respectively, representing 10% of the screen width and height. The screen, scaling, language, and theme settings shown in Table 1 were evaluated. SUDFinder was evaluated under three different scaling settings: default display size and default font size, maximum display size and default font size, and maximum display size and maximum font size. Regarding system language settings, English was used as the default language, and Spanish, Hindi, and Arabic were selected as target languages. These languages ​​were derived from a list of languages ​​with the most native speakers worldwide, and IBM Translation Services provided translation from English to these languages. The selected languages ​​covered both left-to-right (LTR) and right-to-left (RTL) writing styles.

[0116] Specifically, in this embodiment, each defect report generated by SUDFinder and the benchmark method was manually reproduced and verified. The defects were categorized as follows: T1: Style-related SUD defects. These defects are related to the style of the UI (e.g., color, layout structure, space size, etc.) and affect the consistency of the UI interface under different system settings. Further, the following four subtypes were identified: T1.1: Inconsistent color modifications. This type of defect makes the color scheme of application UI components inconsistent under different system settings, sometimes even making the components invisible to the user; T1.2: Inconsistent alignment; T1.3: Inconsistent distance. Developers often place semantically related UI components together to achieve visual consistency. The distance between these components is only expected to change under specific system settings related to screen size (e.g., screen rotation). However, modifying other system settings may also cause these components to deviate from each other, resulting in SUD defects; T1.4: Inconsistent containment relationships. This type of SUD defect manifests as inconsistent containment and overlap relationships between two UI components under different system settings. For example, a container view might not adjust its size or position after settings are modified, making it unable to properly accommodate the contained views. T2: Semantic SUD defects. These defects occur when developers fail to properly adapt system settings changes, leading to functional failures. Subtypes include: T2.1: Unexpected UI visibility, such as UI components not updating correctly, causing notifications or dialogs to persist and incorrectly displaying the application's current state; T2.2: Incorrect data handling, such as application crashes due to developers' failure to properly manage data displayed by UI components (e.g., data loss); T2.3: Inconsistent translation, where text translation does not meet language requirements. Based on these defect types, test cases are first constructed to access Activities that use the XML elements mentioned in the defect report. Then, the visual effects of the defective UI components are presented through the Activity's code logic. If unexpected visual effects are confirmed, and these effects may affect the application's visual consistency and functionality, the defect report is considered a true positive (TP); otherwise, it is considered a false positive (FP).

[0117] Table 2 Results of SUDFinder and Benchmark Methods

[0118] method total T1.1 type T1.2 type T1.3 type T1.4 type SUDFinder 84 / 107(0.80) 15 / 22 46 / 60 26 / 34 11 / 13 <![CDATA[SUDFinder r ]]> 49 / 62(0.79) 9 / 14 28 / 35 4 / 4 8 / 9 SetDroid 10 / 12(0.83) 10 / 12 - - - dVermin 11 / 13(0.85) - - - 11 / 13 ITDroid 17 / 25(0.68) - 12 / 20 - 5 / 5

[0119] As shown in Table 2, SUDFinder detected 107 bugs in 11 applications, of which 84 were true positives (accuracy rate 0.80). This indicates that SUDFinder can accurately detect SUD defects in Android applications. Table 2 shows that the SUD defects detected by SUDFinder are diverse: these applications are affected by different settings and produce different consequences. True positive (TP) defect reports identified and verified by SUDFinder were submitted to application developers. When submitting merge requests and defect reports, the application project's contribution guidelines and license requirements were strictly followed. To date, application developers have confirmed 67 vulnerabilities, and 37 vulnerabilities have been merged and fixed, proving the effectiveness of SUDFinder. Further analysis of the applications with true positives (TPs) at the application code level revealed two root causes of SUD defects. These two root causes accounted for 81.0% of all identified true positives. 45 true positives (TPs) occurred because application developers assigned fixed values ​​to the visual appearance, preventing adjustments based on system settings. For example, SUDFinder identified a defect in AnkiDroid where the "Default" view was assigned the fixed attribute android:textAlignment="inherit". This caused the view to fail to right-align the text content correctly when the system language was switched to Arabic, resulting in a visual inconsistency with the "Basic" view. Figure 7 (a) The Android framework provides various attributes for UI components that can automatically adjust their visual appearance based on different system settings. Twenty-three cases were found where application developers used incorrect attributes, causing the visual appearance of UI components to fail to change as expected. For example, SUDFinder found a defect in AnkiDroid where, when the font size was set to maximum, a portion of the view became invisible because the developer used the attribute android:layout_height="36dp". Figure 7 (b)

[0120] Specifically, this embodiment evaluates the effectiveness of the test application generation process. To achieve this goal, SUDFinder and SUDFinder... r A comparison was made. SUDFinder r The test prediction of SUDFinder is retained, but the test application generation strategy is changed to random exploration, and SUDFinder is set. rThe random exploration strategy was adopted, generating 20 seed tests, each containing 100 events. SUDFinder was then compared with publicly available benchmark tools that could run on API 35 (the latest Android framework version as of August 2024). The following benchmark tools were also compared: SetDroid, dVermin, and ITDroid. For dVermin, experiments were conducted using its original configuration; for SetDroid, system settings other than display and language settings were excluded, and SetDroid was only configured to detect whether the UI remained consistent after a given system setting change and restoration. SetDroid was not configured to detect SUD defects that caused unexpected changes when settings were not restored, because SetDroid did not consider what adaptive changes the UI should make when a T1 type defect occurs. The results are shown in Table 2. SUDFinderr generated 29 TPs and 7 FPs, with an accuracy of 0.81. Of the 29 detected test cases (TPs), SUDFinder covered 25 (25 / 29 = 86.2%). The remaining 4 true cases were not detected, mainly because some runtime information is defined in the application code and cannot be obtained from the XML configuration file. However, the remaining 59 TPs detected by SUDFinder could not be covered by SUDFinder. This is mainly because the randomization strategy failed to generate valid test cases, thus failing to explore all possible UI components defined in the Android application. These results indicate that SUDFinder's test application generation strategy helps to discover more defective UI components that are difficult to reach through dynamic randomization. Table 2 also shows the effectiveness of the benchmark method. For SetDroid, it generated 10 TPs across 4 applications with an accuracy of 0.83. Of these 10 TPs, 5 were defects that caused application crashes. SUDFinder could not detect these crashes because they were located in the application code rather than in the XML file. The remaining 5 defects were inconsistencies in visual features caused by specific system events. For example, the keyboard of the droid-ify app disappeared after rotating the screen, resulting in an inconsistent UI interface between the two devices. These 5 TPs also failed to be detected by SUDFinder because its defect detection process did not involve system / user events. However, none of the 84 TPs found by SUDFinder could be detected by SetDroid because: (1) SetDroid's random exploration strategy did not cover defective UI pages; and (2) SetDroid did not model expected changes related to UI styles. For dVermin, it successfully detected 9 TPs (T1.4 type defects) related to inconsistencies in inclusion relationships with an accuracy of 0.75.Of the 11 problematic areas (TPs) detected by SUDFinder, dVermin was able to detect 6, while the remaining 5 TPs failed to be detected due to the limitations of random exploration. On the other hand, dVermin detected 3 TPs not covered by SUDFinder, mainly because SUDFinder failed to accurately recover the boundaries of visible content. ITDroid successfully detected 17 TPs, all of which were also identified by SUDFinder. Twelve of these were T1.2 type defects caused by layout boundary drift, while the remaining five were related to T1.4 type defects. These results demonstrate that SUDFinder addresses the shortcomings of existing benchmark methods. First, SUDFinder focuses on detecting T1 type defects, which are mainly caused by inconsistent UI styles and require understanding the relationship between UI components before and after settings changes. Second, by directly analyzing the XML configuration files in the application project, SUDFinder can comprehensively test the UI contained in the application without relying on random testing or manually built test cases as in existing methods, thus helping developers discover more previously undiscovered SUD defects.

[0121] Specifically, this embodiment also evaluated the effectiveness of using GPT-4o to detect SUD defects. For each application and system setting combination, five screenshots with different interface states were randomly selected as test inputs for GPT-4o. These 1,540 screenshots (28 applications × 11 settings × 5) were used as a benchmark dataset, and SUD defects in this benchmark dataset were manually labeled, ultimately confirming 54 SUD defects. This benchmark dataset included 50 TPs (covering all types of defects from T1.1 to T1.4) reported by randomized benchmarking methods, as well as 4 SUD defects that SUDFinder and other benchmarking methods failed to detect. These 4 SUD defects do not cause style inconsistencies but affect the visibility of UI components. To reduce the impact of randomness, the GPT-4o detection process was repeated three times, and the best result was statistically analyzed.

[0122] Table 3. Benchmark datasets for evaluating GPT-4o

[0123] type The number of SUD defects Number of screenshots accuracy T1.1 15 5 0.6(3 / 5) T1.2 42 12 0.5(12 / 24) T1.3 8 8 0.75(6 / 8) T1.4 7 4 0.75(3 / 4) T1 72 26 0.58(15 / 26) No defects - 1485 0.05(76 / 1485)

[0124] As shown in Table 3, GPT-4o successfully detected 15 screenshots containing SUD defects in the benchmark dataset, with an accuracy of 0.58. For the remaining 11 screenshots containing SUD defects, due to interference from other unrelated UI components on the same UI page, the generated defect reports failed to locate the defective UI component or identify the root cause. On the other hand, for the remaining 1,485 screenshots that did not contain SUD defects, GPT-4o incorrectly identified 1,409 of them as containing SUD defects. Although GPT-4o can detect defects based on the application context provided in the screenshots, the knowledge it learns from the large-scale corpus does not model SUD defects well, leading it to incorrectly identify normal application runtime behavior as SUD defects and report them to the user. These results indicate that while GPT-4o has the ability to detect SUD defects, it also generates a large number of false positives, which affects its effectiveness in detecting real SUD defects in practical applications.

[0125] This invention, based on an understanding of the common root causes and patterns of UI defects (SUD defects) related to settings in Android applications, proposes and implements SUDFinder, an automated tool for detecting SUD defects in Android applications. SUDFinder encodes common manifestation patterns of SUD defects, thereby automatically detecting potential SUD defects in Android applications. Experimental results show that SUDFinder can accurately identify real and previously undiscovered SUD defects, outperforming existing benchmark methods.

[0126] like Figure 9 As shown, this embodiment also discloses a system for locating interface display defects related to settings in an Android application, including:

[0127] Initialization module 91 is used to initialize an empty set BUGS for storing detected defect reports;

[0128] Page acquisition module 92 is used to obtain all UI pages in an Android application by injecting an Activity for testing;

[0129] Page traversal module 93 is used to traverse all UI pages sequentially, obtain all UI components in each UI page, combine UI components under the same UI page in pairs to obtain multiple UI component pairs, and obtain the relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed.

[0130] The defect detection module 94 is used to detect whether there are defects in each UI page based on the relationship under the default settings and the relationship after the Android system settings are modified. If a UI component on a certain UI page does not maintain the same relationship after the Android system settings are modified as the default settings, a defect is detected. After all UI components on that UI page have been detected, a defect report is generated based on all detected defects and added to the BUGS collection.

[0131] The defect return module 95 is used to return the set of detected problem reports (BUGS) after all UI pages have been inspected.

[0132] The specific implementation of the interface display defect localization system related to settings in Android applications is the same as the interface display defect localization method related to settings in Android applications, and will not be described again in this embodiment.

[0133] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for locating display defects in settings-related interfaces in an Android application, characterized in that, include: S1, initialize an empty set BUGS for storing detected defect reports; S2, Inject the Activity function for testing to obtain all UI pages in the Android application; the input parameters of the Activity function include the ID of the UI page; S3, iterate through all UI pages sequentially, retrieve all UI components on each UI page, and combine the UI components under the same UI page. The components are combined in pairs to obtain multiple UI component pairs. The relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed are obtained. S4 checks for defects in each UI page based on the relationship under default settings and the relationship after the Android system settings are modified. If a UI component on a certain UI page has a relationship that is not consistent with the default relationship after the Android system settings are modified, a defect is detected. After all UI components on that UI page have been detected, a defect report is generated based on all detected defects and added to the BUGS collection. The detected defect types include style-related SUD defects and semantic-related SUD defects; the style-related SUD defects include inconsistencies in color modification, alignment, distance, and containment relationships; the semantic-related SUD defects include unexpected UI visibility, incorrect data processing, and inconsistent translation; the SUD defects are settings-related user interface display defects. S5, after all UI pages have been inspected, returns the collection of detected problem reports (BUGS).

2. The method for locating interface display defects related to settings in an Android application according to claim 1, characterized in that, The method for determining inconsistencies in color modifications is as follows: The system detects two UI components that have the same color under default settings and checks if their colors remain consistent after the settings are changed. If they do not remain consistent, the color modification is considered inconsistent, as follows: u1.color = u2.color and u'1.color ≠ u'2.color; u1 and u2 represent two UI components in their original state; u1.color and u2.color represent the color attribute values ​​of these two UI components under default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.color and u'2.color represent the color attribute values ​​of these two UI components after the settings have been changed.

3. The method for locating interface display defects related to settings in an Android application according to claim 1, characterized in that, The method for determining inconsistencies in alignment is as follows: This function checks whether two UI components that are aligned under default settings still maintain alignment after the settings are changed. If they are not aligned, it is determined that the alignment methods are inconsistent. Specifically: Compare the six key points of the visual boundaries of UI components u1 and u2: left (L), right (R), top (T), bottom (B), vertical center (VC), and horizontal center (HC). Define u1 and u2 as aligned if the following conditions are met: u1.align ∩ u2.align ≠ ∅; If the following conditions are met after system settings are modified, the alignment is considered inconsistent: u'1.align ∩ u'2.align = ∅; Here, u1 and u2 represent two UI components in their original state; u1.align and u2.align represent the alignment of these two UI components under the default settings; u'1 and u'2 represent two UI components after the settings have been changed, and u'1.align and u'2.align represent the alignment of these two UI components after the settings have been changed.

4. The method for locating interface display defects related to settings in an Android application according to claim 3, characterized in that, Alignment detection methods specifically include: S11: Initialize an empty collection `align` to store the alignment of a pair of UI components; S12: Obtain the visible rectangular range of each UI component from the UI page and calculate their respective center points; S13: Based on the top-left, bottom-right, and center coordinates of the rectangular visible area of ​​the UI components, detect the alignment of the pair of UI components: If the x-coordinates of the top-left corners are equal, the pair of UI components are left-aligned, and a left alignment identifier L is added to the align set; if the x-coordinates of the bottom-right corners are equal, the pair of UI components are right-aligned, and a right alignment identifier R is added to the align set; if the x-coordinates of the center coordinates are equal, the pair of UI components are horizontally center-aligned, and a horizontal center alignment identifier HC is added to the align set; if the y-coordinates of the top-left corners are equal, the pair of UI components are top-aligned, and a top alignment identifier T is added to the align set; if the y-coordinates of the bottom-right corners are equal, the pair of UI components are bottom-aligned, and a bottom alignment identifier B is added to the align set; if the y-coordinates of the center coordinates are equal, the pair of UI components are vertically center-aligned, and a vertical center alignment identifier VC is added to the align set. S14: After detecting the alignment, return the set of alignment options for this pair of UI components.

5. The method for locating interface display defects related to settings in an Android application according to claim 1, characterized in that, The method for determining inconsistencies in distance is as follows: Calculate the horizontal Euclidean distance dh(u1,u2) and the vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set horizontal threshold th or vertical threshold tv before and after the system settings change, it is determined to be an inconsistency in distance. The specific formula is as follows: ∣dh(u1′,u2′)−dh(u1,u2)∣≥ th or ∣dv(u1′,u2′)−dv(u1,u2)∣≥ tv; Here, u1' and u2' represent the two UI components u1 and u2 after their settings have been changed, respectively.

6. The method for locating interface display defects related to settings in an Android application according to claim 1, characterized in that, The method for determining inconsistencies in inclusion relationships is as follows: Check the overlap between the visible boundaries of components u1 and u2, denoted as overlap(u1, u2); if the overlap changes after system settings are changed, it is determined that the containment relationship is inconsistent, as shown in the following formula: overlap(u1, u2) ≠ overlap(u1', u2'); Here, u1' and u2' represent the two UI components after the settings of u1 and u2 have been changed, respectively.

7. A system for locating defects in the interface display related to settings in an Android application, characterized in that, include: The initialization module is used to initialize an empty set BUGS for storing detected defect reports; The page retrieval module is used to retrieve all UI pages in an Android application by injecting an Activity for testing. The page traversal module is used to traverse all UI pages sequentially, obtain all UI components in each UI page, combine UI components under the same UI page in pairs to obtain multiple UI component pairs, and obtain the relationship between each UI component pair under the default settings and the relationship after the Android system settings are changed. The defect detection module is used to detect whether there are defects in each UI page based on the relationship under the default settings and the relationship after the Android system settings are modified. If a UI component on a certain UI page has a relationship that is not consistent with the default relationship after the Android system settings are modified, a defect is detected. After all UI components on that UI page have been detected, a defect report is generated based on all detected defects and added to the BUGS collection. The detected defect types include style-related SUD defects and semantic-related SUD defects; the style-related SUD defects include inconsistencies in color modification, alignment, distance, and containment relationships; the semantic-related SUD defects include unexpected UI visibility, incorrect data processing, and inconsistent translation; the SUD defects are settings-related user interface display defects. The defect return module is used to return a set of bug reports (BUGS) after all UI pages have been inspected.

Images