A method and electronic device for detecting interface scaling defects in mobile applications

Abstract

The present invention discloses a method and electronic device for detecting interface scaling defects for mobile applications. The method is as follows: 1) scaling the user interface of a target mobile application at different scaling ratios to obtain the interface at each scaling ratio; using the components in the interface as the granularity, establishing a correspondence between the same components in different scaling ratio interfaces; 2) based on the corresponding relationship, performing inter-component analysis, that is, comparing the positional relationship between components of the same component in different scaling ratio interfaces, and detecting scaling defects caused by changes in the positional relationship between components; 3) based on the corresponding relationship, performing intra-component analysis, comparing and analyzing the content displayed by the component itself, and detecting scaling defects caused by inconsistencies in component content. The present invention can more comprehensively and accurately discover problems, quickly and automatically complete scaling accessibility testing of applications, and ensure the development quality of mobile applications.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and electronic device for automatically detecting interface scaling defects of mobile applications. Background Art

[0002] Nowadays, mobile smart terminal devices are very popular, and the Android operating system has become one of the most common systems. So far, there are at least 2.6 million mobile applications published in the Android application market, covering various functions.

[0003] Users with visual impairments, such as middle-aged and elderly people, tend to increase the font size or display size of the screen when using mobile applications to more easily navigate the application.

[0004] However, the graphical user interfaces of many applications are designed only for the default font or display size settings, because most developers are sighted and have given little or no consideration to users with visual impairments. This leads to inconsistencies in the user interface when switching from the default settings to larger fonts or display sizes. This inconsistency can cause scaling defects in user interface components, such as component overlap, component missing, and component occlusion. These three types of scaling defects are also the most common types of defects. The user interface contains components, which are the basic units that make up the user interface. There are many types of components, such as text components, image components, drop-down box components, and custom components. The large variety of components also makes the manifestation of scaling problems diverse and increases the difficulty of detecting scaling problems.

[0005] In the prior art, there are mainly the following methods for detecting scaling issues:

[0006] Manual inspection. The basic idea is to rely on developers to manually explore the application, record the screen display at the default zoom level, and the corresponding screen display at a higher zoom level. By comparing the component display results, they detect inconsistencies and record the components with display defects. However, manual inspection is time-consuming and labor-intensive.

[0007] Screenshot-based deep learning detection. The basic idea is to directly detect scaling issues on a large-scale interface by taking screenshots. Its core detection methods include, but are not limited to, deep learning image feature detection methods. However, detecting scaling defects only on a single screenshot without considering the dynamic properties of the layout can lead to misjudgments and missed detections. Currently, there is an urgent need for an automated and accurate technology to detect scaling defects in mobile application interfaces. Summary of the Invention

[0008] In view of this, the present invention provides a method for automatically detecting scaling defects in mobile application interfaces, addressing issues that current detection solutions cannot address. The present invention provides a comparison-based detection method, which detects scaling issues by comparing the positional relationships and content of layout files and components within the layouts at different scaling ratios, including:

[0009] Step S1: scaling a user interface of a target mobile application at different scaling ratios to obtain interfaces at each scaling ratio; establishing a correspondence between the same components in the interfaces at different scaling ratios, using components as granularity;

[0010] Step S2: Based on the correspondence between the components, perform inter-component analysis, and detect scaling defects caused by changes in the positional relationships between components by comparing the positional relationships between components at different scaling ratios.

[0011] Step S3: Based on the correspondence between components, perform intra-component analysis, compare and analyze the content displayed by the component itself, and detect scaling defects caused by inconsistency of component content.

[0012] In one embodiment, step S1 specifically includes:

[0013] Step S1.1: Obtain all layout files in the installation package of the mobile application, and inject unique component identifiers into the components contained in the layout files to distinguish different types of components.

[0014] Step S1.2: Package the application injected with the component unique identifier into an installation package and automatically install it on the mobile phone.

[0015] Step S1.3: Install and operate the application on mobile phones with different zoom ratios, so that the application reaches the same interface p, and obtains the layout file of interface p and the image information of the components in interface p, uses the component identifier in the layout file and the contained string to generate the component ID of the component, and identifies the component pairs corresponding to the same content.

[0016] In one embodiment, step S2 specifically includes:

[0017] Step S2.1: Perform inter-component analysis based on the component tree of the interface, that is, analyze whether there is a change in the positional relationship of a component relative to other components, which causes a change in the display result of this component and a scaling problem. Specifically, analyze whether each non-root component in the component tree is visible in the component that directly contains it in the order of pre-root traversal, and analyze the visibility, including the visible ratio, through the image and visible area of ​​the non-root component and the image and visible area of ​​the component that contains this component. At the same time, if the non-root component has sibling components, analyze whether the sibling components obstruct this component based on the drawing order of the non-root components and sibling components, as well as the visible areas and images of these components, and record the component ID of the sibling component that obscures this component to form an occlusion ID set. Analyze scaling problems (visibility problems, overlapping problems) based on this.

[0018] Step S2.2: After performing inter-component analysis on the interfaces of two different zoom ratios, based on the visibility of the component in the component pair and the ID of the sibling component that occludes it, compare and analyze whether the two parts of the content of the components change at different zoom ratios, that is, the changes in the visibility and occlusion relationship. If changes occur, it is determined that there is a zoom problem.

[0019] In one embodiment, step S3 specifically includes:

[0020] Step S3.1: Classify the component pairs consisting of leaf nodes in the component tree at two scaling ratios into text component pairs and non-text component pairs.

[0021] Step S3.2: For the text component pair, use the binarization method to extract the image of the text part of the component from the image of the component at different scaling ratios, and perform statistics on the connected domains of the text part images. Compare the connected domains, count the areas where the areas have changed, and sum the areas where the areas have changed as the text area. If the proportional change of the text area at two different scaling ratios is inconsistent with the scaling ratio, it is determined that there is a scaling problem.

[0022] Step S3.3: For non-text component pairs, use binarization to extract the foreground of the component and count the number of connected domains. If the components at two scaling ratios have the same connected domains, it is determined that there is no scaling problem. Otherwise, call the SSIM algorithm to identify the similarity of the images. If the similarity is lower than 0.9, it is determined that there is a scaling problem.

[0023] The present invention also includes a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute the above method when running.

[0024] Meanwhile, the present invention includes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer to execute the method described above.

[0025] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:

[0026] To detect scaling issues, this invention first identifies component pairs within an application. Based on these pairs, it then performs both inter-component analysis and intra-component analysis, examining the positional relationships and visibility between components, as well as changes in their internal content, to detect scaling issues. This ultimately forms a complete component scaling detection solution.

[0027] Due to the method of the present invention, the detection solution for scaling problems solves the problems of high cost and low efficiency of manual detection, and can also discover problems more comprehensively and accurately through comparison. Users can use this solution to quickly and automatically complete scaling accessibility testing of applications, and ensure the quality of development by prompting users of components and areas with scaling problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of an automatic detection technology for interface scaling defects in mobile applications provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to achieve the above technical effects, the main concepts of the present invention are as follows:

[0031] The present invention aims to provide a detection scheme for detecting interface defects of mobile applications caused by adjusting the zoom settings of mobile devices, including component overlap, component missing and component occlusion problems. The method involves three steps: injecting unique identifiers into components in the layout file to identify corresponding components in the component tree at different scales, i.e., component pairs. To analyze scaling issues within a component pair, subsequent steps require obtaining the component's visibility at the corresponding scale, as well as any components that obscure it, and its content. By comparing these inconsistencies, it is determined whether the component at a given scale has scaling issues. Specifically, the method first analyzes scaling issues between components using the positional relationship between the component pair and the components in the component tree, as well as the component image and its own properties (such as drawing order). Specifically, the method obtains component visibility and the identifiers of any components that obscure it. Based on the component pair, these inconsistencies are compared. If any discrepancies are found, a scaling issue (visibility or occlusion) is reported. Finally, the method uses the component pair and its own content to analyze scaling issues within the component. Specifically, the method obtains the component's content and analyzes any discrepancies based on the component's properties. If any discrepancies are found, a scaling issue (missing content) is reported. Based on these three steps, a complete and automated scaling defect detection solution is ultimately formed.

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0033] This embodiment provides a method for detecting interface defects in mobile applications caused by adjusting the zoom settings of mobile devices. Figure 1 , the method comprising:

[0034] Step S1: Establishing a correspondence between components in the interface at different zoom ratios with the component as the granularity.

[0035] Since the purpose of this method is to detect scaling problems accurately and fine-grained, this method uses components as the minimum detection granularity, and detects scaling problems by comparing the display of a component at different scaling ratios. However, under normal circumstances, because there is no guarantee that the component contains a unique ID identifier, it is difficult to accurately and quickly find components corresponding to the same content in the interface at different scaling ratios, and it is difficult to establish a corresponding relationship between components. At the same time, the component tree may also contain scrollable list components. The list items in the list component are structurally repetitive, and these list items may have the same ID identifier. It is difficult to distinguish these list items simply by using the ID identifier, and it is difficult to determine the list items containing the same content at different scaling ratios and establish a component pair relationship for them. In order to solve this problem, this method injects a unique component identifier in step S1, combines the text inside the component, and uses a heuristic method to establish a corresponding relationship between component pairs. Specifically, it includes the following sub-steps:

[0036] Step S1.1: Obtain all layout files within the mobile application and inject unique component identifiers into the components contained in the layout files to distinguish different types of components. A single interface may use multiple layout files, and a layout file may be used multiple times by one or more interfaces.

[0037] The layout file is a file that controls the display of the mobile application interface. Each widget exists in the layout file in the form of a tree structure (i.e., the widget tree widgetTree). The widget i in the widget tree widgetTree may contain K (K>0) components {widget j ,widget j+1 ,…,widget j+K} or zero components. In a component tree, a component can only be included by one other component, forming a strict tree structure. The display of a component in a component tree is controlled by the component that directly includes it. In a component tree, most of the time, only leaf components (components that don't contain any other components) carry content, such as text or images.

[0038] Because the component widget in the layout file i There may be a lack of unique identification, making it difficult to correctly distinguish different types of components. In this step, the attribute auxiliaryID i , that is, a unique string identifier injected into all widgets i In each widget i All have auxiliaryID i , so, after injecting the attributes, according to the widget iauxiliaryID i Information, this method can distinguish components.

[0039] Step S1.2: Package the application program injected with the component unique identifier into an installation package and automatically install it on the device under test.

[0040] The specific implementation of the installation package packaging in this step can be completed from two perspectives. One is to directly inject attributes into the components of the layout file from the source code before packaging the application into the installation package, and compile the source code into the installation package after injection; the other is to decompile the packaged installation package, inject attributes from the decompiled layout file, and package it into the installation package together with the source code and various resources after injection.

[0041] After the installation package is packaged, it will be processed as necessary, such as signing, byte alignment, etc., and then installed on the device under test.

[0042] Step S1.3: Install and operate the application on mobile phones with different zoom ratios so that the application reaches the same interface, and obtain the layout file of the interface and the image information of the component, use the component identifier of the component in the layout file and the string contained in the component to generate the component ID of the corresponding component, and identify the component pairs corresponding to the same content.

[0043] After installing the corresponding program on the device to be tested, use the test script or other means to operate the program to adjust the program to a certain interface p to be tested and the scale size s, and obtain the component tree PageWidgetTree of the interface p p,s and all its components Image Components in the component tree Contains a series of attributes For example, height width Rendering Order text Scrollability Scaling Whether to use ellipsis to replace omitted text after text overflows In addition, the component itself also contains images in RGBA format (common PNG format images are in RGBA format, containing four channels) That is, a width of Height is A multi-channel image with a depth of 4. The last channel of this image is the transparency channel image. The first three channels are red, green and blue channel images use Image, you can grayscale the image to get the corresponding grayscale image

[0044] Collect the component tree PageWidgetTree p,s Then, for each component Generate component ID attribute value And use the ID to find the component pair Where s represents the normal scaling ratio and s′ represents a larger scaling ratio. The component pairs are generated as follows:

[0045] 1. Traverse all components in the component tree Record the value of its attribute auxiliaryID If this component It is unique, that is, it is not the same as the auxiliaryID value of any other component, that is, it is used as the ID attribute value of the component. Then directly add the component of Values ​​as components The component ID value.

[0046] 2. For other cases, determine this component Whether it is its sibling component (having the same parent component) have the same auxiliaryID value, if so, traverse this component in a certain order of traversal tree All descendant components The order of traversal can be one of the first root traversal, the middle root traversal and the last root traversal. In the process of generating component pairs, the traversal order must be consistent. In the process of traversal, according to the traversed components, a string is generated. If you encounter a descendant component And this descendant component The auxiliaryID value of its sibling components The auxiliaryID value of this descendant component is the same as and All descendant components do not participate Generation; for other traversed components Concatenate the taxt attribute value and auxiliaryID value into a string in the traversal order After going through all After that, the components of The value and Connect together to form components Because a layout file may be used multiple times by a page, the auxiliaryID injected into the layout file may appear repeatedly in the page component tree, so a new ID needs to be generated to distinguish components in the page component tree; because they may be reused, sibling components may come from the same layout file, so they may also contain the same auxiliaryID value.

[0047] 3. Based on the generated ID value, establish the component pair relationship:

[0048] a) For the component injected with auxiliaryID in step 1 Directly search for components with the same ID in another component tree with a different scale s′ If the corresponding components are found, a component pair relationship is established for this pair of components.

[0049] b) For the component injected with auxiliaryID in step 2 Directly search for components with the same ID in another component tree with a different scale s′ If the corresponding component is found, the component pair relationship is directly established And for the descendant components of these two components and Establish a corresponding relationship. Specifically, first record the components The ID value is Traverse the descendant components of this component During the traversal process, if you encounter a descendant component And this descendant component The auxiliaryID value of its sibling components If the auxiliaryID value is the same as the auxiliaryID value of and components The method of establishing a relationship between the descendant components (according to step b) is to Establish a component pair relationship with its descendant components and the components in the component tree under s′, because the component The auxiliaryID is also injected by step 2); for other descendant components Then and this descendant component The auxiliaryID value is connected together as this descendant component If you can find a component with the same ID in another component tree Establish a corresponding relationship between this pair of components, that is, generate a component pair

[0050] Step S2: Based on the correspondence between components, perform inter-component analysis, and detect scaling defects caused by changes in the positional relationship and visibility between components by comparing the positional relationship between components at different scaling ratios.

[0051] In the component tree, components are organized together according to the tree structure and are also rendered on the interface according to the tree structure. Depending on the drawingOrder of the components, there may be occlusion issues between components. Due to the tree structure relationship between components, the parent component controls the visibility of its direct child components. If the visibility changes, the child component may be partially or completely invisible. Therefore, based on the tree structure and the relationship between parent and child components, this step performs inter-component analysis and analyzes scaling defects based on the positional relationship between components and changes in visibility. Specifically, this step includes the following sub-steps.

[0052] Step S2.1: Perform inter-component analysis based on the component tree of the interface, that is, analyze whether there is a change in the positional relationship of a component relative to other components, which causes a change in the display result of this component and a scaling problem. Specifically, analyze whether each non-root component in the component tree is visible in the component that directly contains it in the order of pre-root traversal, and analyze the visibility, including the visible ratio, through the image and visible area of ​​the non-root component and the image and visible area of ​​the component that contains this component. At the same time, if the non-root component has sibling components, analyze whether the sibling components obstruct this component based on the drawing order of the non-root components and sibling components, as well as the visible areas and images of these components, and record the component ID of the sibling component that obscures this component to form an occlusion ID set. Analyze scaling problems (visibility problems, overlapping problems) based on this.

[0053] For a non-root component For example, its parent component Controls its visibility (visible range) and its rendering order relative to its sibling components. This step analyzes each non-root component in the component tree in a pre-root traversal manner. Visibility, and its sibling component collection The case of blocking it, that is, the overlapping case, where Corresponding components

[0054] We use the visible matrix visible p,s,i To describe the component Visibility, for components For example, the visible matrix is ​​initialized to a width of Height is A two-dimensional matrix, the values ​​in this matrix are either 0 or 1, 0 means invisible, 1 means visible. The alpha channel p,s,i The data used to initialize the visible matrix is:

[0055]

[0056] The alpha channel is a width Height is The two-dimensional matrix alpha p,s,i , the size is consistent with the size of the visible matrix. For the element in the mth row and nth column of the visible matrix, if the two-dimensional matrix alpha p,s,i If the value of the element in the mth row and nth column is greater than 0, the value of the element in the mth row and nth column in the visible matrix is ​​1, otherwise it is 0.

[0057] This step traverses the component tree PageWidgetTree in a pre-root traversal manner p,s Specifically, after traversing a node, if the node has child components, the child components are traversed in descending order according to their drawing order. When traversing each non-root component, two parts need to be analyzed, as shown below:

[0058] Part 1: For each non-root component We need to determine whether it is in the parent component that directly contains it It is not visible, whether there is a parent component In the case of occlusion, calculate the attributes The corresponding judgment method is as follows:

[0059] 1. First determine the non-root component Whether the area on the screen is within the parent component Outside the area of ​​the screen, if it is outside the area and the parent component cannot be scrolled, then the non-root component is judged to be invisible and the component is set of The property is false;

[0060] 2. If the area of ​​the non-root component is within the area of ​​the parent component, obtain the cropped image of the non-root component within the area of ​​the parent component And the corresponding clipping coordinate area

[0061] 3. Get the image of the parent component and use the template matching algorithm in the image field to crop the non-root component Use the template to search in the parent component's image. If the area found is And the clipping coordinate area of ​​the subcomponent If the intersection-and-union ratio is equal to 1, it means that the non-root component is visible in the parent component, and the component is recorded. of The property is false, otherwise it is invisible and the component is recorded of The property is true.

[0062] 4. Get the parent component Visible matrix visible p,s,j , get the non-root component Visible matrix visible p,s,i , if the component of If the property is false, the visible matrix will be p,s,i Assign all values ​​in to 0. Otherwise, go to step 5.

[0063] 5. Based on non-root components Visible matrix visible p,s,i , get the corresponding visible area According to the clipping coordinate area Clip the visible matrix of the non-root component and the visible matrix of the parent component (clipping the visible matrix means obtaining the matrix block of the visible matrix inside the clipping coordinate area) to obtain croppedVisible p,s,i and croppedVisible p,s,j and get cropedANDVisible p,s,i ,Right now:

[0064]

[0065] In short, it is to croppedVisible p,s,i and cropedANDVisible p,s,j Perform a bitwise AND operation to get cropedANDVisible p,s,i . After that, according to the clipping coordinate area cropedANDVisible p,s,i Replace it with visible p,s,i In the clipping area, get the updated visible p,s,i , and calculate the new visible area like equal Record component Whether the property is blocked by the parent component false if yes, true otherwise.

[0066] Part 2: At the same time, for all Non-root components with this property set to true If its brother component set Not an empty collection, using its drawing order in the parent component Analyze the occlusion relationship between non-root components and their sibling components, and record the occlusion of non-root components The overlapping set of sibling components The details are as follows:

[0067] 1. Because components with larger drawingOrder may be drawn on top of components with smaller drawingOrder, non-root components and their sibling components are sorted in descending order according to the drawing order in the parent component, and their sibling components are filtered in descending order according to the drawing order. Component If the brother component Component of Greater than and true, this component Add to sorted set among Corresponding to the brother component Component

[0068] 2. Traverse the ordered set of Peers in descending order according to the drawing order p,s,i . According to the component Position coordinates and the currently traversed component Calculate the intersection and union ratio of the two components based on their position coordinates. If the two components intersect on the screen, update the component. The visibility matrix of the two components. That is, calculate the intersection area overlapRegion of the two components on the screen p,s,i,k The width of this intersection area is pverlapRegionWidth p,s,i,k , the height is overlapRegionHeight p,s,i,k The coordinates of its upper left corner on the user interface are (overlapRegionX p,s,i,k ,overlapRegionY p,s,i,k ), based on this intersection area and its position on the screen and components Visible matrix visible p,s,i , the visible matrix visible p,s,i After cutting, we get the components The clipping visible matrix overlapVisible p,s,i and components The clipping visible matrix overlapVisible p,s,k . At this time, calculate the visible matrix overlapVisible p,s,i Visible area According to these two clipping visible matrices, the intersection occlusion matrix overlapBlockedVisible is calculated p,s,i,k :

[0069]

[0070] Component The parts blocked by the visible area are invisible, and the visibility of other unblocked parts is not affected.

[0071] According to the intersection area overlapRegion p,s,i,k , overlapBlockedVisible p,s,i,k Matrix replacement to overlapVisible p,s,i In the corresponding area, overlapVisible is updated p,s,i At this point, calculate the intersection occlusion matrix overlapBlockedVisible p,s,i,k Visible area like The component Blocked components And the components The ID attribute value Add to the sibling component overlap set Then analyze the components Whether to block the component Until the ordered set Peer p,s,i All analysis completed.

[0072] Step S2.2: After performing inter-component analysis on the interfaces of two different zoom ratios, based on the visibility of the component in the component pair and the ID of the sibling component that occludes it, compare and analyze whether the two parts of the content of the components change at different zoom ratios, that is, the changes in the visibility and occlusion relationship. If changes occur, it is determined that there is a zoom problem.

[0073] After performing component analysis on the interface p with two scaling ratios s and s′, in the component tree PageWidgetTree p,s In each non-root component Both have two attribute values and collection In the component tree PageWidgetTree p,s′ In each non-root component Both have two attribute values and collection According to the components like or or Then it is determined that there is a scaling problem.

[0074] Step S3: Based on the correspondence between components, perform intra-component analysis, compare and analyze the content displayed by the component itself, and detect scaling defects caused by inconsistency of component content.

[0075] Since leaf components carry content in the component tree, this step analyzes whether leaf components exhibit scaling defects, such as missing content, at different zoom levels. Components can be broadly categorized as text components and other components. Because text in text components wraps due to scaling adjustments, conventional OCR methods can reduce recognition accuracy and increase computational load. Therefore, the following steps are designed to perform component-level analysis. Specifically, these steps include the following substeps.

[0076] Step S3.1: Classify the component pairs consisting of leaf nodes in the component tree at two scaling ratios into text component pairs and non-text component pairs.

[0077] For component widgets, we classify them as text components or non-text components based on whether they contain the text attribute. If they contain the text attribute, then this component is a text component, otherwise it is a non-text component.

[0078] Step S3.2: For each text component pair, use a binarization method to extract images of the text portion of the component from the images at different scales. Count the connected domains of the text portion images, compare the connected domains, count the regions where the visible area has changed, and sum the regions where the visible area has changed as the visible area of ​​the text. If the ratio of the visible area change at two different scales is inconsistent with the scale ratio, then a scaling issue is determined. This step has the following sub-steps:

[0079] 1. For text components According to its image p,s,i Get RGB p,s,i Image, alpha p,s,i Image and gray p,s,i image.

[0080] 2. Obtain binary image binaryp,s,i and binary ^p,s,i :

[0081] 1) If gray p,s,i If the image is all 0, then alpha p,s,i The image is binarized using the Otsu algorithm to obtain the binarization threshold And obtain the binary image binary according to the threshold p,s,i and binary ^p,s,i ,Right now:

[0082]

[0083]

[0084] 2) If gray p,s,i The image is not an image with all zeros. The gray image is processed as follows to obtain gary ^p,s,i image:

[0085]

[0086] Then, in Gary ^p,s,i In the image, the Otsu algorithm is used to calculate the non-1 area And use this threshold to complete the binarization of the gray image using the method in step 2 to obtain binary p,s,i and binary ^p,s,i image.

[0087] 3. Get binary p,s,i and binary ^p,s,i After the image is generated, the binary is analyzed using the connected domain analysis method. p ,s,i and binary ^p,s,i The 8-connected domains with a value of 1 in the image are obtained, and the area of ​​each connected domain is obtained, that is, the number of pixels with a value of 1. The set of corresponding connected domains is obtained and in, For binary p,s,i The area of ​​the t-th connected component in the image, For binary ^p,s,i The area of ​​the e-th connected component in the image.

[0088] 4. After analyzing the text components of the interface p at two scaling ratios s and s′, use the method in step S1.3 to obtain the text components under the two scaling ratios. Binary image, binary p,s,iand binary p,s′,j , and get its corresponding connected domain set conn p,s,i and conn p,s′,j According to the area of ​​the connected domain obtained, the connected domains with equal areas in the two components are screened out, and the areas of the remaining connected domains are summed up to obtain connSum p,s,i and connSum p,s′,j ,if This means that the text scaling inside the text component is legal. Otherwise, if If true, it is also legal, otherwise, use binary ^p,s,i and binary ^p,s′,j , calculate connSum ^p,s,i and connSum ^p,s′,j ,like Then the text scaling inside the text component is legal, otherwise a scaling problem is reported.

[0089] Step S3.3: For non-text component pairs, use binarization to extract the foreground of the component and count the number of connected domains. If the components at two scaling ratios have the same connected domains, it is determined that there is no scaling problem. Otherwise, call the SSIM algorithm. If the similarity is lower than 0.9, it is determined that there is a scaling problem.

[0090] 1. For non-text components According to its image p,s,i Get RGB p,s,i Image, alpha p,s,i Image and gray p,s,i image.

[0091] 2. Gray p,s,i The image is calculated using the Otsu algorithm And perform binarization to obtain a binary image binary p,s,i ,Right now:

[0092]

[0093] 3. Get binary p,s,i After the image is generated, the binary is analyzed using the 8-connected domain analysis method. p,s,i The connected domain with a value of 1 in the image is obtained, and the area of ​​each connected domain is obtained, that is, the number of pixels with a value of 1, and the number of connected domains is obtained. p,s,i .

[0094] 4. Use the method in step S1.3 to obtain the non-text component pairs under two scaling ratios. Binary image, binary p,s,i and binary p,s′,j , and get the corresponding number of connected domains count p,s,i and count p,s,j If count p,s,i =count p,s,j , it means that the content scaling inside the non-text component is legal, otherwise, proceed to step 5.

[0095] 5. Gray p,s,i and gray p,s′,j Crop and get croppedGray p,s,i and croppedGray p,s′,j , to crop gray p,s,i For example, croppedGray p,s,i Must meet:

[0096]

[0097] croppedGray p,s,i The width and height are W p,s,i and H p,s,i

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] croppedGray p,s′,j Also following the same approach, by gray p,s′,j It can be cut.

[0104] 6. Calculate croppedGray using the SSIM algorithm p,s,i and croppedGray p,s′,j If the similarity value is greater than 0.9, the content scaling is legal, otherwise a scaling problem is reported.

[0105] Once those skilled in the art are aware of the basic creative concept, they may make additional changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.

[0106] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for detecting interface scaling defects in mobile applications, comprising the following steps: 1) Scaling a user interface of a target mobile application at different scaling ratios to obtain the interface at each scaling ratio; Using the components in the interface as the granularity, establish the corresponding relationship between the same components in interfaces with different zoom ratios; 2) Based on the corresponding relationships, performing inter-component analysis, i.e., comparing the positional relationships between components of the same component under different scaling interfaces, and detecting scaling defects caused by changes in the positional relationships between components; 3) Based on the corresponding relationship, perform intra-component analysis to compare and analyze the content displayed by the component itself, and detect scaling defects caused by inconsistency of component content; The method for establishing the corresponding relationship is as follows: obtaining all layout files of the target mobile application, injecting a unique component identifier into the components contained in each layout file to distinguish different types of components; packaging the target mobile application injected with the component unique identifier into an installation package and installing it on multiple smartphones with different zoom ratio displays and operating the target mobile application to reach the same interface p; obtaining the layout file of interface p and the image information of the components in interface p, using the component identifier in the layout file and the character string contained in the component to generate the component ID of the corresponding component, treating the components with the same component ID as a component pair, and establishing a corresponding relationship for the same component in interfaces with different zoom ratios.

2. The method according to claim 1, characterized in that Each component widget in the layout file forms a component tree WidgetTree in the form of a tree structure; the i-th component widget in the component tree WidgetTree i Contains K components {widget j ,widget j+1 ,…,widget j+K } or zero components; generate a unique string identifier for each component as the ID attribute auxiliaryID of the corresponding component; operate the target mobile application to reach the interface p, and obtain the component tree PageWidgetTree of the interface p corresponding to the zoom ratio s p,s and an image of each of its components, where For PageWidgetTree p,s The i-th component in image; according to Find the scaling factor by the component ID s ′ The components in the corresponding interface p Get component pair 3. The method according to claim 2, characterized in that Generate Components The component ID value method is: 11) Traverse each component in the component tree WidgetTree and record its ID attribute auxiliaryID; if The ID attribute value Is unique, then use it as a component The ID attribute value of the component The ID attribute value of the component Component ID value, otherwise proceed to step 12); 12) Determine the component Whether it has the same auxiliaryID value as its sibling components, if so Brother components with the same auxiliaryID value; traverse all descendant components of the brother component. If a descendant component with a non-unique auxiliaryID value is encountered during the traversal process, stop traversing the brother component and its descendant components, concatenate the text attribute values ​​of the traversed components into a string postfix, and concatenate the auxiliaryID value and postfix of the traversed components together to form a component The component ID value.

4. The method according to claim 2, characterized in that The components The properties include height width Rendering Order text Scrollability Scaling The image For width high A multi-channel image with a depth of 4. The last channel is the transparency channel image The first three channels are red, green and blue channel images 5. The method according to claim 4, characterized in that The method for performing inter-component analysis is: 21) For the component tree PageWidgetTree of the corresponding interface p when the scaling ratio is s p,s , traverse the component tree PageWidgetTree in a pre-rooted traversal manner p,s For each component in, if the current component is a non-root component Then use non-root components Image The alpha channel p,s,i Initialize a width of Height is The two-dimensional visible matrix visible p,s,i ; 22) Determine non-root components Whether the area on the screen is within its parent component Outside the area on the screen, if it is outside the area and the parent component If it cannot be scrolled, then it is determined that this is not the root component Invisible, and set the component of The property is false; if it is within the area, the non-root component is obtained In its parent component The cropped image inside the region And the corresponding clipping coordinate area 23) Crop the image As a template, in the parent component If the searched area is and the clipping coordinate area If the intersection-and-union ratio is equal to 1, it is determined to be a non-root component In the parent component As can be seen in the record component of The property is false, otherwise it is invisible and the component is recorded of The property is true; 24) Get the parent component Visible matrix visible p,s,j , get the non-root component Visible matrix visible p,s,i , if the component of If the property is false, the visible matrix will be p,s,i All values ​​in are assigned to 0; otherwise, go to step 25); 25) Based on non-root components Visible matrix visible p,s,i , get the corresponding visible area ; According to the clipping coordinate area For non-root components The visible matrix and parent component The visible matrix is ​​cropped and the cropping result is bitwise ANDed to get croppedANDVisible p,s,i ; Then according to the clipping coordinate area , cropedANDVisible p,s,i Replace it with visible p,s,i In the clipping area, get the updated visible p,s,i , and calculate the new visible area ;like equal , then record the component Whether the property is blocked by the parent component is false, otherwise true; 26) For Non-root components with this property set to true If its brother component set Not empty, Then use its drawing order in the parent component Analyzing non-root components The occlusion relationship with its sibling component set, and records the occlusion of non-root components The overlapping set of sibling components 27) Follow steps 21 to 26 to adjust the scaling factor s. ′ The component tree PageWidgetTree of the corresponding interface p p,s′ Process and get PageWidgetTree p,s′ Each non-root component Attribute value Overlapping collections with sibling components 28) For component pairs like or or but It is determined that there is a scaling problem.

6. The method according to claim 5, characterized in that Get the overlapping set of sibling components The method is: 261) Non-root components and its sibling components based on the parent component Sort the drawing order in descending order, and filter its sibling components in descending order of drawing order; among them, non-root components The qth oldest sibling component For the component tree PageWidgetTree p,s The kth component in If the brother component of Greater than and If true, the component Add to sorted set Among them Corresponding to the brother component Component 262) Traverse the ordered set of Peers in descending order according to the drawing order p,s,i , according to the component Position coordinates and the currently traversed component Calculate the intersection ratio of the two components based on their position coordinates. If the two components intersect on the screen, calculate the intersection area of ​​the two components on the screen. Based on the intersection area and its position on the screen, Visible matrix visible p,s,i Cut and get components The clipping visible matrix overlapVisible p,s,i and components The clipping visible matrix overlapVisible p,s,k ; Calculate the visible matrix overlapVisible p,s,i Visible area And the intersection occlusion matrix overlapBlockedVisible p,s,i,k ; 263) The intersection occlusion matrix overlapBlockedVisible is calculated based on the intersection area p,s,i,k Replaced with overlapVisible p,s,i In the corresponding area, overlapVisible is updated p,s,i ; Calculate the intersection occlusion matrix overlapBlockedVisible p,s,i,k Area like Then determine the component Occlusion Component And the components The ID attribute value is added to the sibling component overlap set middle.

7. The method according to claim 1, characterized in that The method for performing intra-component analysis is: 31) classifying component pairs consisting of leaf nodes in the component tree at two different scaling ratios into text component pairs and non-text component pairs; 32) For a text component pair, extract images of the text portion of the component from images of the component at different scaling ratios. If the proportional change in the text area at two different scaling ratios is inconsistent with the scaling ratio, it is determined that there is a scaling problem; 33) For non-text component pairs, extract the foreground of the component and count the number of connected domains. If the components at two scaling ratios have the same connected domains, it is determined that there is no scaling problem. Otherwise, calculate the image similarity of the two components. If the similarity is lower than the set threshold, it is determined that there is a scaling problem.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images