Setup-related interface display defect positioning method and system in Android application

Abstract

The invention discloses a setting-related interface display defect positioning method and system in an Android application, and the method comprises the steps: storing a defect report through an initialized null set BUGS, and injecting a test Activity to obtain all UI pages; traversing each pair of UI components in each UI page, and comparing visual feature relationships of the UI components after default setting and system setting change, including color, alignment mode, distance and inclusion relationship consistency; if any inconsistency is detected, generating a defect report and adding the defect report to a BUGS set; and finally, returning a report set containing all detected style-related and semantic-related defects. According to the method, the UI page is obtained by injecting the test Activity, and the visual feature relationships of the components under different settings are traversed and compared, so that the automatic detection and report generation of the interface display defects related to the settings are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of software engineering, and particularly to a method and system for locating display defects related to settings in Android applications. Background Art

[0002] Android provides a powerful and flexible framework that allows developers to build beautiful and interactive UI interfaces. A view is the basic building block in the Android UI, and the UI interface rendered by an Android application consists of a view tree. Initially, developers used XML files to define the tree structure of the UI. These XML files are located in the res / layout directory and are converted into actual UI components at runtime. The attributes of the views defined in the XML configuration file (such as the ID of the view, layout height, text content) determine their initial appearance and behavior, such as the position and size of buttons, text, and image views.

[0003] As a popular mobile operating system, Android presets a variety of application components for developers to build the UI interface of an application through XML. A study of 200 top-ranked applications found that each application has an average of 663.1 layout files and 25,991.6 XML elements, which is sufficient to demonstrate the complexity and extensiveness of XML when defining the UI interface of mobile applications.

[0004] SUD (Screen User Design) defects usually stem from the developer's failure to successfully adapt the XML configuration file to different system settings. This is mainly because the UI layout of an Android application is defined by static XML files. When facing diverse device environments (different screen sizes, resolutions, operating system versions, etc.), if the XML configuration does not correctly handle these changes, display problems will occur. The adaptation work is complex and time-consuming, requiring designing and testing UI layouts for multiple scenarios, which is likely to be simplified or overlooked in cases where resources are limited or time is tight. In addition, the lack of effective automated tool support in the past has made manual adjustment and testing error-prone and inefficient. With the continuous introduction of new features and APIs, new SUD defects are also likely to be introduced during the process of continuously updating and maintaining the application.

[0005] Android provides a series of system settings that allow users to customize their devices and applications according to their personal preferences. These system settings cover different aspects, such as language, font size, brightness, color mode, etc. Users may change the system language from English to Arabic. While system settings meet the needs of users in different environments, they also bring a burden to developers. Developers need to adjust their applications to adapt to various system settings; otherwise, it will lead to some settings-related defects, namely the "Setting-related UI Display Bug" (SUD Bug) referred to in this article, abbreviated as "SUD defect". Developing an automated method to detect SUD defects faces two key challenges: covering all potential UI components and constructing a universal test oracle. The first ensures the comprehensiveness of defect detection, while the second is used to verify whether there are SUD defects in the UI components. To ensure the accuracy of verification, each system setting requires a customized test oracle and takes into account the specific context information of the application. SUD defects are mainly located in XML files and involve the hierarchical structure of XML elements and attribute configurations; existing dynamic-based methods rely on random testing to generate test cases, making it difficult to cover all UI components. In addition, their test oracles do not fully cover the expected adaptation results of system settings, resulting in false negative results. After the system language is changed, the alignment of the layout does not meet the expected results, and the existing work fails to identify this defect. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a method and system for locating setting-related interface display defects in Android applications. By research, common patterns of unexpected UI adaptation that cause SUD defects are determined, and a method for automatically detecting SUD defects in Android applications is proposed.

[0007] The specific solutions are as follows:

[0008] On the one hand, a method for locating setting-related interface display defects in Android applications includes:

[0009] S1, initialize an empty set BUGS for storing detected defect reports;

[0010] S2, inject Activity functions for testing to obtain all UI pages in the Android application; the input parameters of the Activity function include the ID of the UI page;

[0011] S3, traverse all UI pages in sequence, obtain all UI components in each UI page, combine the UI components under the same UI page in pairs to obtain multiple pairs of UI components, and obtain the relationships of each pair of UI components under the default settings and the relationships after the Android system settings are changed;

[0012] S4. Based on the relationships under the default settings and the relationships after modification in the Android system settings, detect whether there are defects in each UI page in turn; if there is a pair of UI components on a certain UI page whose relationship after modification in the Android system settings does not match the relationship under the default settings, it is determined that a defect is detected; after all pairs of UI components on this UI page are detected, generate a defect report based on all the detected defects and add it to the set BUGS.

[0013] S5. After all UI pages are detected, return the set BUGS of the detected problem reports.

[0014] Further, in S4, the detected defect types include SUD defects related to styles and SUD defects related to semantics; the SUD defects related to styles include inconsistencies in color modification, alignment, distance, and containment relationships; the SUD defects related to semantics include unexpected UI visibility, incorrect data processing, and inconsistent translations.

[0015] Further, the determination method for the inconsistency in color modification is specifically as follows:

[0016] Detect two UI components that have the same color under the default settings to see if they still have the same color after the setting change; if they do not, it is determined that there is an inconsistency in color modification, as follows:

[0017] u1.color = u2.color and u’1.color ≠ u’2.color;

[0018] u1 and u2 respectively represent two UI components in the original state; u1.color and u2.color respectively represent the color attribute values of these two UI components under the default settings; u’1 and u’2 respectively represent the two UI components after the setting change, and u’1.color and u’2.color respectively represent the color attribute values of these two UI components after the setting change.

[0019] Further, the determination method for the inconsistency in alignment is specifically as follows:

[0020] Detect two UI components that are aligned under the default settings to see if they are still aligned after the setting change; if they are not, it is determined that there is an inconsistency in alignment, specifically as follows:

[0021] Compare the six key points of the visual boundaries of UI components u1 and u2: left L, right R, top T, bottom B, vertical center VC, and horizontal center HC. When the following conditions are met, u1 and u2 are defined as aligned:

[0022]

[0023] After the system settings are modified, if the following conditions are detected to be satisfied, it is determined that the alignment methods are inconsistent:

[0024]

[0025] Among them, u1 and u2 respectively represent two UI components in the original state; u1.align and u2.align respectively represent the alignment methods of these two UI components under the default settings; u'1 and u'2 respectively represent the two UI components after the settings are changed, and u'1.align and u'2.align respectively represent the alignment methods of these two UI components after the settings are changed.

[0026] Furthermore, the method for detecting the alignment method specifically includes:

[0027] S11: Initialize an empty set align to store the alignment methods of a pair of UI components;

[0028] S12: Obtain the respective rectangular visible ranges of this pair of UI components from the UI page and calculate their respective center points;

[0029] S13: Based on the left upper corner coordinates, right lower corner coordinates and center point coordinates of the rectangular visible ranges of the UI components, detect the alignment methods of this pair of UI components: If the abscissas of their left upper corner coordinates are equal, this pair of UI components is left-aligned, and add the left alignment identifier L to the align set; if the abscissas of their right lower corner coordinates are equal, this pair of UI components is right-aligned, and add the right alignment identifier R to the align set; if the abscissas of their center point coordinates are equal, this pair of UI components is horizontally center-aligned, and add the horizontal center alignment identifier HC to the align set; if the ordinates of their left upper corner coordinates are equal, this pair of UI components is top-aligned, and add the top alignment identifier T to the align set; if the ordinates of their right lower corner coordinates are equal, this pair of UI components is bottom-aligned, and add the bottom alignment identifier B to the align set; if the ordinates of their center point coordinates are equal, this pair of UI components is vertically center-aligned, and add the vertical center alignment identifier VC to the align set;

[0030] S14: After detecting the alignment method, return the alignment method set align of this pair of UI components.

[0031] Furthermore, the determination method for the distance inconsistency is specifically:

[0032] Calculate the horizontal Euclidean distance dh(u1, u2) and the vertical Euclidean distance dv(u1, u2) between the center points of the visible boundaries of computing components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set horizontal threshold th or vertical threshold tv before and after the system settings change, it is determined that the distance is inconsistent. The specific formula is as follows:

[0033] |dh(u1′, u2′) - dh(u1, u2)| ≥ th or |dv(u1′, u2′) - dv(u1, u2)| ≥ tv;

[0034] Among them, u1’ and u2’ respectively represent the two UI components after the settings of component u1 and component u2 are changed.

[0035] Furthermore, the determination method for the inconsistency of the inclusion relationship is specifically as follows:

[0036] Check the overlapping part between the visible boundaries of components u1 and u2, denoted as overlap(u1, u2); after the system changes the settings, if the overlapping part changes, it is determined that the inclusion relationship is inconsistent. The specific formula is as follows:

[0037] overlap(u1, u2) ≠ overlap(u1’, u2’);

[0038] Among them, u1’ and u2’ respectively represent the two UI components after the settings of u1 and u2 are changed.

[0039] On the other hand, an interface display defect localization system related to settings in an Android application includes:

[0040] An initialization module for initializing an empty set BUGS for storing detected defect reports;

[0041] A page acquisition module for obtaining all UI pages in the Android application by injecting the Activity for testing;

[0042] A page traversal module for sequentially traversing all UI pages, obtaining all UI components in each UI page, combining the UI components under the same UI page in pairs to obtain multiple UI component pairs, and obtaining the relationships of each UI component pair under the default settings and the relationships after the Android system settings are changed;

[0043] The defect detection module is used to detect whether there are defects in each UI page in turn based on the relationships under the default settings and the relationships after the Android system settings are modified; if there is a UI component pair in a certain UI page whose relationship after the Android system settings are modified does not match the relationship under the default settings, it is determined that a defect is detected; after all UI component pairs in the UI page are detected, a defect report is generated based on all the detected defects and added to the set BUGS;

[0044] The defect return module is used to return the set BUGS of the detected problem reports after all UI pages are detected.

[0045] The present invention adopts the above technical solutions and has the following beneficial effects:

[0046] (1) By comparing based on the default settings and the modified relationships, the present invention can accurately determine inconsistent problems such as color, alignment, distance, and inclusion relationship, automatically generate a detailed defect report, help developers quickly locate and fix interface display defects, and improve the application quality;

[0047] (2) By injecting the test Activity to obtain all UI pages and traversing and comparing the visual feature relationships of each UI component under different settings, the present invention realizes the comprehensive automatic detection of style-related and semantic-related SUD defects, significantly improving the detection efficiency and coverage rate.

[0048] (3) The present invention designs and implements an automatic detection tool for the method (SUDFinder) of locating interface display defects related to settings in an Android application, uses the extracted SUD defect patterns as test oracles, and directly performs defect detection in the XML configuration file of the application UI, realizing the efficient and accurate identification of previously unknown SUD defects, not only improving the test coverage rate, but also successfully overcoming the inefficiency and high false negative rate problems caused by the existing methods relying on random exploration. Description of the Drawings

[0049] Figure 1 It is a flowchart of the method for locating interface display defects related to settings in an Android application according to an embodiment of the present invention;

[0050] Figure 2(a) is a schematic diagram of the behavior under the default settings according to an embodiment of the present invention;

[0051] Figure 2(b) is a schematic diagram of the SUD defects shown when switching to Arabic according to an embodiment of the present invention;

[0052] Figure 2(c) is an effect diagram after repairing the SUD defects according to an embodiment of the present invention;

[0053] Figure 3Schematic diagram of the interface display defect location algorithm related to settings in the Android application of the embodiment of the present invention;

[0054] Figure 4 Schematic diagram of the test Activity injected for defect detection in the embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the repair patch for detecting SUD defects related to the detection style in the embodiment of the present invention;

[0056] Figure 6 Schematic diagram of the prompt word template in the embodiment of the present invention;

[0057] Figure 7 Schematic diagram of two true positive cases of SUDFinder on AnkiDroid in the embodiment of the present invention;

[0058] Figure 8 Schematic diagram of the detection method algorithm for the alignment method in the embodiment of the present invention;

[0059] Figure 9 System diagram of the interface display defect location related to settings in the Android application of the embodiment of the present invention. Detailed implementation manners

[0060] The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0061] As Figure 1 shown, the method for locating the interface display defect related to settings in the Android application of the present invention includes:

[0062] S1, initialize an empty set BUGS for storing the detected defect reports.

[0063] Specifically, the detected defect types include SUD defects related to styles and SUD defects related to semantics; the SUD defects related to styles include inconsistent color modification, inconsistent alignment, inconsistent distance, and inconsistent inclusion relationship; the SUD defects related to semantics include unexpected UI visibility, incorrect data processing, and inconsistent translation.

[0064] Specifically, the determination method for inconsistent color modification is as follows:

[0065] Detect whether two UI components with the same color under the default settings still maintain the same color after the setting change; if not, it is determined as inconsistent color modification, as follows:

[0066] u1.color = u2.color and u’1.color ≠ u’2.color;

[0067] u1 and u2 respectively represent two UI components in the original state; u1.color and u2.color respectively represent the color attribute values of these two UI components under the default settings; u'1 and u'2 respectively represent two UI components after the setting change, and u'1.color and u'2.color respectively represent the color attribute values of these two UI components after the setting change.

[0068] Specifically, the determination method for the inconsistency of the alignment method is as follows:

[0069] Detect whether two UI components that are aligned under the default settings remain aligned after the setting change; if they do not remain aligned, it is determined that there is an inconsistency in the alignment method, specifically as follows:

[0070] Compare the six key points of the visual boundaries of UI components u1 and u2: left L, right R, top T, bottom B, vertical center VC, and horizontal center HC. When the following conditions are met, it is defined that u1 and u2 are aligned:

[0071]

[0072] After the system settings are modified, if it is detected that the following conditions are met, it is determined that the alignment method is inconsistent:

[0073]

[0074] Among them, u1 and u2 respectively represent two UI components in the original state; u1.align and u2.align respectively represent the alignment methods of these two UI components under the default settings; u'1 and u'2 respectively represent two UI components after the setting change, and u'1.align and u'2.align respectively represent the alignment methods of these two UI components after the setting change.

[0075] Specifically, the detection method for the alignment method is as Figure 8 shown, specifically including:

[0076] S11: Initialize an empty set align to store the alignment method of a pair of UI components;

[0077] S12: Obtain the respective rectangular visible ranges of this pair of UI components from the UI page and calculate their respective center points;

[0078] S13: Based on the upper-left coordinate, lower-right coordinate, and center point coordinate of the rectangular visible range of the UI components, detect the alignment mode of this pair of UI components: If the abscissas of the upper-left coordinates of the two are equal, then this pair of UI components is left-aligned, and add the left-alignment identifier L to the align set; if the abscissas of the lower-right coordinates of the two are equal, then this pair of UI components is right-aligned, and add the right-alignment identifier R to the align set; if the abscissas of the center point coordinates of the two are equal, then this pair of UI components is horizontally center-aligned, and add the horizontal center-alignment identifier HC to the align set; if the ordinates of the upper-left coordinates of the two are equal, then this pair of UI components is top-aligned, and add the top-alignment identifier T to the align set; if the ordinates of the lower-right coordinates of the two are equal, then this pair of UI components is bottom-aligned, and add the bottom-alignment identifier B to the align set; if the ordinates of the center point coordinates of the two are equal, then this pair of UI components is vertically center-aligned, and add the vertical center-alignment identifier VC to the align set.

[0079] S14: After detecting the alignment mode, return the alignment mode set align of this pair of UI components.

[0080] Specifically, in this embodiment, the SUD defects (T1 type) related to the style are mainly detected. When the setting changes, the UI components with similar visual features should have the same adaptation method, so as to maintain a consistent visual effect after the system setting changes. As shown in Figures 2(a), 2(b), and 2(c), Figure 2(a) is a schematic diagram of the behavior under the default setting of the embodiment of the present invention; Figure 2(b) is a schematic diagram of the SUD defect shown when switching to Arabic in the embodiment of the present invention; Figure 2(c) is the effect diagram after repairing the SUD defect in the embodiment of the present invention; when the system is set to a right-to-left language, the title and text area of the text should be aligned and close to the icon, which prompts the use of different measurement criteria to evaluate the relationship between UI components on the page, identify the defective UI components that cannot maintain these relationships, and finally generate a defect report; in this embodiment, Figure 2(a) and Figure 2(c) are aligned, and Figure 2(b) is not aligned. Because the system languages of Figure 2(b) and Figure 2(c) are RTL languages read from right to left, the alignment direction is right-aligned.

[0081] Specifically, the determination method for the inconsistency of the distance is as follows:

[0082] Calculate the horizontal Euclidean distance dh(u1, u2) and the vertical Euclidean distance dv(u1, u2) between the center points of the visible boundaries of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set horizontal threshold th or vertical threshold tv before and after the system setting changes, then it is determined that the distance is inconsistent. The specific formula is as follows:

[0083] |dh(u1′, u2′) - dh(u1, u2)| ≥ th or |dv(u1′, u2′) - dv(u1, u2)| ≥ tv;

[0084] Wherein, u1’ and u2’ respectively represent two UI components after the settings of component u1 and component u2 are changed.

[0085] Specifically, the determination method for the inconsistency of the inclusion relationship is as follows:

[0086] Check the overlapping part between the visible boundaries of components u1 and u2, denoted as overlap(u1, u2); after the system settings are changed, if the overlapping part changes, it is determined that the inclusion relationship is inconsistent. The specific formula is as follows:

[0087] overlap(u1, u2) ≠ overlap(u1’, u2’);

[0088] Wherein, u1’ and u2’ respectively represent two UI components after the settings of u1 and u2 are changed.

[0089] S2. By injecting the Activity for testing, obtain all UI pages in the Android application;

[0090] Specifically, SUDFinder injects an Activity for testing into the application to display the visual appearance of all UI pages. Figure 4 Shows a test Activity injected for detecting the defect in Figure 2(b). Specifically, the test Activity calls the setContentView method and passes the ID of the UI interface as a parameter. This design choice is adopted because the XML configuration file is widely used to build the UI of the application, and many SUD defects are located in the XML configuration file. Thus, it is possible to automatically detect SUD defects without building test cases to dynamically explore UI components and achieve a higher UI coverage rate. When running the application under test, SUDFinder will detect whether the visible content on the current screen supports horizontal or vertical scrolling and perform the corresponding scrolling operations to load and display new UI elements. This step is crucial for revealing views that are part of the screen layout under the original settings (such as the default font size) but are pushed out of the view due to setting changes (such as increasing the font size). Only after obtaining these new visible views can a comprehensive comparison be made between all the views displayed before and after the system settings are modified.

[0091] Specifically, the UI pages in the application contain some text obtained from the network rather than being loaded from predefined strings. If these texts are missing, it will cause the UI interface of the TextView to not be displayed correctly. As Figure 5 shown, the text contents of text_title and text_domain in the TextView are not specified in the XML configuration file but are obtained from the server at runtime. Most of these TextView views lack relevant test cases, making it difficult for the system to obtain texts that match the actual application scenarios.

[0092] To solve this problem, SUDFinder utilizes large language models (LLMs) to generate the missing texts in the XML configuration file. Through the training of large language models with a large amount of Internet corpora, rich domain knowledge can be obtained. In this embodiment, the context information of the UI components to be tested is provided to the LLM, allowing it to infer texts that match the TextView views. This method aims to generate practical and context-appropriate texts for subsequent testing. In this way, the accuracy of UI display is improved, and potential problems caused by different text inputs can be discovered.

[0093] In this embodiment, a text input generation method based on LLM is adopted, and the prompt template is as Figure 6 shown. Specifically, the template takes the name of the application, the XML configuration file name, and the XML code as inputs, enabling the LLM to infer the context information of the TextView and EditText views in the XML code, thereby generating appropriate text contents. Ensure that the LLM only outputs the XML code without including additional information, and use the generated XML code to replace the content of the original XML configuration file, thereby finally constructing the application for testing. By injecting the test Activity, 35 additional SUD defects were discovered, and 34 SUD defects were located in the texts generated by the LLM.

[0094] S3. Traverse all UI pages in sequence, obtain all UI components in each UI page, combine the UI components under the same UI page in pairs to obtain multiple pairs of UI components, and obtain the relationships of each pair of UI components under the default settings and the relationships after the Android system settings are changed;

[0095] Specifically, SUDFinder will check the following three visual feature relationships R(u1, u2) for each pair of UI components u1 and u2.

[0096] The three visual feature relationships include: inconsistent color changes, inconsistent alignment changes, and inconsistent distances.

[0097] To detect defects of inconsistent types in color changes, SUDFinder assumes that two UI components with the same color under default settings will still maintain consistent colors after the setting changes. Therefore, when the following conditions are met, SUDFinder will generate a defect report:

[0098] u1.color = u2.color and u’1.color ≠ u’2.color;

[0099] To detect defects of inconsistent types in alignment changes, SUDFinder assumes that two UI components that are aligned under default settings will still maintain alignment after the setting changes. Specifically, SUDFinder compares six key points of the visual boundaries bndvis of UI components u1 and u2: left (L), right (R), top (T), bottom (B), vertical center (VC), and horizontal center (HC). When the following conditions are met, u1 and u2 are defined as aligned:

[0100]

[0101] After the system settings are modified, if SUDFinder detects that the following conditions are met, it will generate a defect report:

[0102]

[0103] To detect defects of inconsistent types in distance, SUDFinder assumes that unless the change in system settings causes a screen change (such as screen rotation), the distance between the visual contents of two UI components should be as consistent as possible with the default settings. Therefore, SUDFinder will calculate the horizontal Euclidean distance dh(u1,u2) and the vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries bndvis of components u1 and u2. If the change in the horizontal or vertical Euclidean distance exceeds the set thresholds (horizontal threshold th and vertical threshold tv) before and after the system settings change, SUDFinder will generate a defect report. Specifically, it is as follows:

[0104] |dh(u1′,u2′) - dh(u1,u2)| ≥ th or |dv(u1′,u2′) - dv(u1,u2)| ≥ tv;

[0105] To detect defects of inconsistent types in overlap, SUDFinder checks whether the overlap part (denoted as overlap(u1,u2)) between the visible boundaries bndvis of components u1 and u2 changes after the system changes the settings. If the overlap part changes, SUDFinder will report a defect, and the specific conditions are:

[0106] overlap(u1, u2) ≠ overlap(u1’, u2’);

[0107] S4. Based on the relationship under the default settings and the relationship of this pair of UI components after the Android system settings are modified, determine whether there are defects in the UI page; if the relationship of this pair of UI components after the system settings are modified does not match the relationship under the default settings, it is determined that a defect is detected, and a defect report is generated and added to the set BUGS.

[0108] S5. After all UI pages are detected, return the set of problem reports BUGS detected.

[0109] Specifically, as Figure 3 shown, Algorithm 1 shows the overall process of SUDFinder. For each UI page in the Android application, SUDFinder first extracts the visual features of the UI components in the page. In formal representation, a UI component u is defined as a tuple <bndvh, bndvis, color>. Specifically: bnd vh represents the layout boundary of component u in the view hierarchy (ViewHierarchy), as shown by the black dotted line in Figure 2, denoted as <(x l vh , y l vh ), (x r vh , y r vh )>; (x l vh , y l vh ) is the coordinate of the upper left corner, and (x r vh , y r vh ) is the coordinate of the lower right corner; bnd vis represents the smallest rectangular range that can cover all non - transparent pixels in bnd vh , that is, the visible content in component u, as Figure 1 shown by the red solid line part in, denoted as <(x l vis , y l vis ), (x r vis , y r vis )>, where (x l vis , y l vis ) is the coordinate of the upper left corner, and (xr vis , y r vis ) is the coordinate of the lower right corner; color represents the two main colors displayed by component u, denoted as <c1, c2>. SUDFinder determines the colors by calling the getcolors() method in the Python Image Library. Although a UI component may have multiple colors, its color scheme is usually relatively simple, only involving the background color and the color of text / images. Then, for two UI components u1 and u2, SUDFinder extracts the relationship between u1 and u2 under the default settings, denoted as R(u1, u2), and the relationship R(u'1, u'2) after the setting modification. If R(u1, u2) ≠ R(u'1, u'2), then SUDFinder will generate a defect report.

[0110] Specifically, this embodiment is based on an empirical study on 218 real-world SUD defects collected. By using the common patterns found in the empirical study as test oracles, and the design that "UI components with similar visual characteristics tend to adapt to setting changes in a similar way"; in order to cover more UI components during testing, SUDFinder performs defect detection in the XML configuration file of the application UI, avoiding dynamically exploring the UI interface by constructing test cases to achieve high coverage. Although this process ignores the application code that may dynamically change the UI, the empirical study shows that SUD defects are usually because developers fail to successfully adapt the XML configuration file to different system settings. SUDFinder was implemented and experimentally evaluated on 28 popular open-source Android applications in F-Droid. The results show that SUDFinder successfully detected 84 defects with an accuracy of 80%, and found 51 defects that the state-of-the-art methods failed to detect. Reports of these previously unknown SUD defects were submitted to the application developers, and 67 of the defects have been confirmed by the developers, and 37 have been fixed and merged.

[0111] The dataset used in this embodiment contains 1074 setting-related defects in open-source Android applications. 218 of these defects are related to SUD, that is, they affect the UI display of the application. Table 1 shows the statistical data of these SUD defects and the corresponding system settings, and 81.6% of the SUD defects are related to the settings of the screen, language, and theme.

[0112] Table 1 Classification and Subclassification of System Settings

[0113]

[0114] The process of data analysis is as follows: First, 109 of the SUD defects were randomly selected as samples (50% of all SUD defects), and the code revisions and corresponding defect reports of each sample were self-checked to determine the code fragments related to the patches; and a preliminary classification method was developed to label and classify the remaining 109 defects in sequence.

[0115] Specifically, the operating environment selected in this embodiment was a server with a 64-core Intel Xeon CPU E5-2683 v4 @ 2.10 GHz processor and 192 GB of memory, and an Android device (Galaxy S8) with a configuration of API level 34 and a display resolution of 1440×2960 pixels was used. The horizontal threshold th and the vertical threshold tv were set to 144 and 296, respectively, which represented 10% of the screen width and height. The screens, scaling ratios, languages, and theme settings shown in Table 1 were evaluated, and SUDFinder was evaluated under 3 different ratio settings: default display size and default font size, maximum display size and default font size, maximum display size and maximum font size. In terms of the system language settings, English was used as the default language, and Spanish, Hindi, and Arabic were selected as the target languages. These languages were from the list of languages with the largest number of native speakers globally, and IBM translation services provided translations from English to these languages. The selected languages covered languages written from left to right (LTR) and from right to left (RTL).

[0116] Specifically, in this embodiment, manual reproduction and verification were performed for each defect report generated by SUDFinder and the benchmark method. And the types of defects were divided into T1: style-related SUD defects, which are related to the style of the UI (such as color, layout structure, space size, etc.) and affect the consistency of the UI interface under different system settings. The following four subtypes were further confirmed: T1.1: inconsistent color modification, this type of defect will make the color matching of the application UI component inconsistent under different system settings, and sometimes even make the component invisible to the user; T1.2: inconsistent alignment; T1.3: inconsistent distance, developers usually place semantically related UI components together to achieve visual consistency, and the distance between these components is only expected to change under specific system settings related to the screen size (such as screen rotation). However, modifying other system settings may also cause these components to deviate from each other and cause SUD defects; T1.4: inconsistent inclusion relationship, this type of SUD defect is manifested as inconsistent inclusion and overlap relationships between two UI components in different system settings. For example, a container view may not adjust its size or position accordingly after the setting is modified, so that the contained view cannot be properly accommodated. T2: Semantic-related SUD defects. In this type of defect, the developer does not correctly adapt to the modification of system settings, resulting in functional failures. The subtypes are classified as follows: T2.1: Unexpected UI visibility, such as UI components not updating correctly, resulting in notifications or dialog boxes not disappearing, incorrectly presenting the current state of the application; T2.2: Incorrect data processing, such as the developer's failure to properly manage the data displayed by the UI component (such as data loss), causing the application to crash. T2.3: Inconsistent translation, the text translation does not meet the language requirements. Based on the above defect types, first build a test case to access the Activity that uses the XML elements mentioned in the defect report. After that, the visual effects of the defective UI component are presented through the code logic of the Activity. If unexpected visual effects are confirmed and these effects may affect the visual consistency and functionality of the application, the defect report is considered a true positive (TP); otherwise, it is considered a false positive (FP).

[0117] Table 2 Results of SUDFinder and baseline methods

[0118] Method Total Type T1.1 Type T1.2 Type T1.3 Type T1.4 SUDFinder 84 / 107(0.80) 15 / 22 46 / 60 26 / 34 11 / 13 <![CDATA[SUDFinder r > 49 / 62(0.79) 9 / 14 28 / 35 4 / 4 8 / 9 SetDroid 10 / 12(0.83) 10 / 12 - - - dVermin 11 / 13(0.85) - - - 11 / 13 ITDroid 17 / 25(0.68) - 12 / 20 - 5 / 5

[0119] As shown in Table 2, SUDFinder detected 107 bugs in 11 applications, 84 of which were true positives (accuracy rate of 0.80). This indicates that SUDFinder can accurately detect SUD defects in Android applications. As can be seen from Table 2, the SUD defects detected by SUDFinder are diverse: these applications are affected by different settings and have different consequences. True positive (TP) defect reports identified and verified by SUDFinder were submitted to the application developers. When submitting merge requests and defect reports, the contribution guidelines and license requirements of the application project were strictly followed. So far, 67 of these vulnerabilities have been confirmed by the application developers, and 37 vulnerabilities have been merged and fixed, proving the effectiveness of SUDFinder. The true positive (TP) applications were further analyzed at the application code level, and two root causes of SUD defects were summarized. These two root causes account for 81.0% of all identified true positives. 45 true positives (TP) occurred because the application developers assigned fixed values to the visual appearance, resulting in its inability to make corresponding adjustments according to the system settings. Taking a defect identified by SUDFinder in AnkiDroid as an example, the "Default" view was assigned the fixed attribute android:textAlignment = "inherit", which caused the view to fail to right-align the text content correctly when the system language was switched to Arabic, resulting in inconsistent visual performance with the "Basic" view as Figure 7 (a). The Android framework provides various properties for UI components that can automatically adjust the visual appearance according to different system settings. 23 cases were found where the application developers used incorrect properties, resulting in the visual appearance of the UI components not changing as expected. Taking a defect found by SUDFinder in AnkiDroid as an example, when the font size was set to the maximum, due to the developer using the property android:layout_height = "36dp", a part of the view became invisible as Figure 7 (b).

[0120] Specifically, this embodiment evaluated the effectiveness of the test application generation process. To achieve this goal, SUDFinder was compared with SUDFinder r SUDFinder r retained the test oracle of SUDFinder, but changed the test application generation strategy to random exploration, and set SUDFinder rThe random exploration strategy is to generate 20 seed tests, with each test containing 100 events. Subsequently, SUDFinder is compared with those publicly available benchmark tools that can run on API 35 (the latest Android framework version as of August 2024). And it is compared with the following benchmark tools: SetDroid, dVermin, and ITDroid. For dVermin, experiments are conducted according to its original configuration; for SetDroid, other system settings except display and language settings are excluded, and SetDroid is only configured to detect whether the UI remains consistent after given system setting changes and recoveries, without configuring SetDroid to detect SUD defects that cause unexpected changes when the settings are not recovered. This is because SetDroid does not consider what adaptive changes the UI should take when defects of type T1 occur. The results are shown in Table 2. Among them, SUDFinderr generated 29 true positives (TPs) and 7 false positives (FPs), with an accuracy of 0.81. Among the 29 detected TPs, SUDFinder covered 25 of them (25 / 29 = 86.2%), and the remaining 4 true positives were not detected, mainly because some runtime information is defined in the application code and cannot be obtained from the XML configuration file; however, the remaining 59 TPs detected by SUDFinder cannot be covered by SUDFinderr. This is mainly because the random strategy failed to generate effective test cases and thus could not explore all possible UI components defined in the Android application. These results indicate that the test application generation strategy of SUDFinder helps to discover more defective UI components that are difficult to reach by dynamic random exploration. The results in Table 2 also show the effectiveness of the benchmark methods. For SetDroid, it generated 10 TPs in 4 applications, with an accuracy of 0.83. Among these 10 TPs, 5 are defects that cause the application to crash. SUDFinder cannot detect these crashes because they are located in the application code rather than the XML file. The remaining 5 defects are inconsistencies in visual features caused by specific system events. For example, the keyboard of the droid-ify application disappears after the screen is rotated, resulting in an inconsistent UI interface between two devices. These 5 TPs also cannot be detected by SUDFinder because its defect detection process does not involve system / user events. However, the 84 TPs discovered by SUDFinder cannot be detected by SetDroid for the following reasons: (1) The random exploration strategy of SetDroid does not cover defective UI pages; (2) SetDroid does not model the expected changes related to UI styles. For dVermin, it successfully detected 9 TPs related to inconsistent containment relationships (defects of type T1.4), with an accuracy of 0.75.Among the 11 TPs detected by SUDFinder, dVermin was able to detect 6 of them, while the remaining 5 TPs were not detected due to the limitations of random exploration. On the other hand, dVermin detected 3 TPs that were not covered by SUDFinder, mainly because SUDFinder failed to accurately recover the boundaries of the visible content. ITDroid successfully detected 17 TPs, and all of these defects were also identified by SUDFinder. 12 of them were T1.2 type defects caused by layout boundary drift, and the remaining 5 were related to T1.4 type defects. The above results show that SUDFinder makes up for the deficiencies of existing benchmark methods. First, SUDFinder focuses on detecting T1 type defects, which are mainly caused by inconsistent UI styles and require understanding the relationship between UI components before and after setting changes. Second, by directly analyzing the XML configuration files in the application project, SUDFinder can comprehensively test the UIs included in the application without relying on random testing or manually constructed test cases like existing methods, thus helping developers discover more previously undetected SUD defects.

[0121] Specifically, this embodiment also evaluated the effectiveness of using GPT-4o to detect SUD defects. For each application and system setting combination, 5 screenshots with different interface states were randomly selected as the test input for GPT-4o. These 1,540 screenshots (28 applications × 11 settings × 5) were used as the benchmark dataset, and the SUD defects in this benchmark dataset were manually labeled. Finally, 54 SUD defects were confirmed. This benchmark dataset contains 50 TPs reported by the random-based benchmark method (covering all types of defects from T1.1 to T1.4), as well as 4 SUD defects that were not detected by SUDFinder and other benchmark methods. These 4 SUD defects do not cause style inconsistencies but affect the visibility of UI components. To reduce the influence of randomness, the detection process of GPT-4o was repeated three times, and the best results were counted.

[0122] Table 3 Benchmark dataset for evaluating GPT-4o

[0123] Type Number of SUD Defects Number of Screenshots Accuracy Rate T1.1 15 5 0.6(3 / 5) T1.2 42 12 0.5(12 / 24) T1.3 8 8 0.75(6 / 8) T1.4 7 4 0.75(3 / 4) T1 72 26 0.58(15 / 26) No Defects - 1485 0.05(76 / 1485)

[0124] As shown in Table 3, GPT-4o successfully detected 15 screenshots containing SUD defects in the benchmark dataset, with an accuracy of 0.58. For the remaining 11 screenshots containing SUD defects, due to interference from other irrelevant UI components in the same UI page, the generated defect reports failed to locate the defective UI components and also failed to identify the root cause. On the other hand, for the remaining 1,485 screenshots without SUD defects, GPT-4o incorrectly considered 1,409 of them to contain SUD defects. Although GPT-4o can detect defects based on the application context provided in the screenshots, the knowledge it learned from the large-scale corpus did not well model the SUD defects, resulting in misidentifying normal behaviors during application runtime as SUD defects and reporting them to users. The above results indicate that GPT-4o has the ability to detect SUD defects but also generates a large number of false positives, which affects its effectiveness in detecting real SUD defects in practical applications.

[0125] Based on understanding the common root causes and patterns of settings-related UI defects (SUD defects) in Android applications, the present invention proposes and implements an automated tool SUDFinder for SUD defects in Android applications. It encodes the common manifestation patterns of SUD defects, enabling automated detection of possible SUD defects in Android applications. Experimental results show that SUDFinder can accurately identify real and previously undetected SUD defects and outperforms existing benchmark methods.

[0126] As Figure 9 shown, this embodiment also discloses a system for locating display defects related to settings in an Android application, including:

[0127] An initialization module 91 for initializing an empty set BUGS for storing detected defect reports;

[0128] A page acquisition module 92 for obtaining all UI pages in the Android application by injecting an Activity for testing;

[0129] A page traversal module 93 for sequentially traversing all UI pages, obtaining all UI components in each UI page, combining the UI components in the same UI page in pairs to obtain multiple pairs of UI components, and obtaining the relationships of each pair of UI components under default settings and after changes in Android system settings;

[0130] The defect detection module 94 is used to sequentially detect whether there are defects in each UI page based on the relationship under the default settings and the relationship after the modification of the Android system settings; if there is a UI component pair in a certain UI page whose relationship after the modification of the Android system settings does not match the relationship under the default settings, it is determined that a defect is detected; after all UI component pairs in the UI page are detected, a defect report is generated based on all the detected defects and added to the set BUGS;

[0131] The defect return module 95 is used to return the set BUGS of the detected problem reports after all UI pages are detected.

[0132] The specific implementation of the interface display defect localization system related to settings in the Android application is the same as the interface display defect localization method related to settings in the Android application, and this embodiment will not be repeated.

[0133] Although the present invention is specifically shown and described in conjunction with the preferred embodiments, those skilled in the art should understand that various changes can be made to the present invention in terms of form and details without departing from the spirit and scope of the present invention defined by the appended claims, and all of them are within the protection scope of the present invention.

Claims

1. A method for locating interface display defects related to settings in an Android application, characterized in that: Including: S1. Initialize an empty set BUGS for storing detected defect reports; S2. Inject Activity functions for testing to obtain all UI pages in the Android application; the input parameters of the Activity function include the IDs of the UI pages; S3. Traverse all UI pages in sequence, obtain all UI components in each UI page, combine the UI components under the same UI page in pairs to obtain multiple UI component pairs, and obtain the relationships of each UI component pair under the default settings and the relationships after the Android system settings are changed; S4. Based on the relationships under the default settings and the relationships after the Android system settings are modified, detect whether there are defects in each UI page in sequence; If there is a UI component pair under a certain UI page whose relationship after the Android system settings are changed does not remain the same as the relationship under the default settings, it is determined that a defect is detected; after all UI component pairs under this UI page are detected, generate a defect report based on all detected defects and add it to the set BUGS; S5. After all UI pages are detected, return the set BUGS of the detected problem reports.

2. The method for locating interface display defects related to settings in an Android application according to claim 1, characterized in that: In S4, the detected defect types include SUD defects related to styles and SUD defects related to semantics; the SUD defects related to styles include inconsistencies in color modification, alignment, distance, and inclusion relationship; the SUD defects related to semantics include unexpected UI visibility, incorrect data processing, and inconsistent translation.

3. The method for locating interface display defects related to settings in an Android application according to claim 2, characterized in that: The determination method for the inconsistency in color modification is specifically as follows: Detect two UI components with the same color under the default settings to see if they still have the same color after the settings are changed; if not, it is determined that there is an inconsistency in color modification, as follows: u1.color = u2.color and u’1.color ≠ u’2.color; u1 and u2 respectively represent two UI components in the original state; u1.color and u2.color respectively represent the color attribute values of these two UI components under the default settings; u’1 and u’2 respectively represent the two UI components after the settings are changed, and u’1.color and u’2.color respectively represent the color attribute values of these two UI components after the settings are changed.

4. The method for locating interface display defects related to settings in an Android application according to claim 2, characterized in that: The determination method for the inconsistency in alignment is specifically as follows: Detect two UI components that are aligned under the default settings to see if they are still aligned after the settings are changed; if not, it is determined that there is an inconsistency in alignment, specifically: Compare the six key points of the visual boundaries of UI components u1 and u2: left L, right R, top T, bottom B, vertical center VC, and horizontal center HC. When the following conditions are met, it is defined that u1 and u2 are aligned: After the system settings are modified, if it is detected that the following conditions are met, it is determined that the alignment is inconsistent: Among them, u1 and u2 represent the two UI components in the original state; u1.align and u2.align represent the alignment of the two UI components under the default settings; u'1 and u'2 represent the two UI components after the settings are changed, and u'1.align and u'2.align represent the alignment of the two UI components after the settings are changed.

5. The method for locating interface display defects related to settings in an Android application according to claim 4, characterized in that: Alignment detection methods include: S11: Initialize an empty collection align to store the alignment of a pair of UI components; S12: Obtain the respective rectangular visible ranges of the pair of UI components from the UI page, and calculate their respective center points; S13: Based on the upper left corner coordinates, lower right corner coordinates and center point coordinates of the rectangular visible range of the UI components, detect the alignment of the pair of UI components: if the horizontal coordinates of the upper left corner coordinates of the two are equal, the pair of UI components is left-aligned, and the left alignment mark L is added to the align set; if the horizontal coordinates of the lower right corner coordinates of the two are equal, the pair of UI components is right-aligned, and the right alignment mark R is added to the align set; if the horizontal coordinates of the center point coordinates of the two are equal, the pair of UI components is horizontally center-aligned, and the horizontal center alignment mark HC is added to the align set; if the vertical coordinates of the upper left corner coordinates of the two are equal, the pair of UI components is top-aligned, and the top alignment mark T is added to the align set; if the vertical coordinates of the lower right corner coordinates of the two are equal, the pair of UI components is bottom-aligned, and the bottom alignment mark B is added to the align set; if the vertical coordinates of the center point coordinates of the two are equal, the pair of UI components is vertically center-aligned, and the vertical center alignment mark VC is added to the align set; S14: After detecting the alignment, return the alignment set align of the pair of UI components.

6. The method for locating interface display defects related to settings in an Android application according to claim 2, characterized in that: The method for determining the inconsistency of distance is as follows: Calculate the horizontal Euclidean distance dh(u1,u2) and vertical Euclidean distance dv(u1,u2) between the center points of the visible boundaries of components u1 and u2. If the change in the horizontal or vertical Euclidean distance before and after the system setting changes exceeds the set horizontal threshold th or vertical threshold tv, it is determined to be a distance inconsistency. The specific formula is as follows: ∣dh(u1′,u2′)-dh(u1,u2)∣≥th or ∣dv(u1′,u2′)-dv(u1,u2)∣≥tv; Among them, u1' and u2' represent two UI components after the settings of component u1 and component u2 are changed respectively.

7. The method for locating interface display defects related to settings in an Android application according to claim 2, characterized in that: The inconsistency determination method of the inclusion relationship is as follows: Check the overlap between the visible boundaries of components u1 and u2, recorded as overlap(u1,u2); if the overlap changes after the system changes the settings, it is determined to be inconsistent with the containment relationship. The specific formula is as follows: overlap(u1,u2)≠overlap(u1',u2'); Among them, u1' and u2' represent the two UI components after the settings of u1 and u2 are changed respectively.

8. A system for locating interface display defects related to settings in an Android application, characterized in that: include: The initialization module is used to initialize an empty collection BUGS for storing detected defect reports; The page acquisition module is used to obtain all UI pages in the Android application by injecting the Activity used for testing; The page traversal module is used to traverse all UI pages in sequence, obtain all UI components in each UI page, combine the UI components in the same UI page in pairs, obtain multiple UI component pairs, and obtain the relationship between each UI component pair under the default settings and after the Android system settings are changed; A defect detection module is used to detect whether there are defects in each UI page in turn based on the relationship under the default settings and the relationship modified by the Android system settings; If the relationship of a UI component pair under a certain UI page after the modification in the Android system settings is not consistent with the relationship under the default settings, it is determined that a defect is detected; after all UI component pairs under the UI page are detected, a defect report is generated based on all detected defects and added to the collection BUGS; The defect return module is used to return the detected problem report set BUGS after all UI pages are detected.

Images