Software interface defect detection method and device based on visual marking and thinking chain prompt

Abstract

The invention provides a software interface defect detection method and device based on visual marking and thinking chain prompt, and relates to the field of artificial intelligence. The method comprises the steps of obtaining multi-state UI data of a target application program in different interface display states; preprocessing the component information file set data to filter redundant component data; constructing an adjacency relation graph, and traversing the adjacency relation graph based on a deep optimization algorithm to output a component level view sorted according to the adjacency relation; drawing a visual mark set based on the UI screenshot set data; constructing a structured text description, and inputting the visual mark set and the structured text description into a multi-modal large model; and executing software interface defect detection through the multi-modal large model based on a thinking chain prompt mechanism and outputting a defect detection result. The problem that false detection and missing detection are easily generated in a complex scene due to the fact that a traditional static feature extraction method cannot effectively capture a defect mode caused by dynamic changes of a UI component is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a software interface defect detection method and device based on visual markers and thought chain prompts. BACKGROUND

[0002] In recent years, emerging device form factors are redefining the boundaries of mobile interaction. The maturity of folding screen technology enables a single device to dynamically switch between phone and tablet, with the unfolded and folded states corresponding to completely different screen ratios and display areas. At the same time, split-screen multitasking has become a standard feature of modern mobile operating systems, with users expecting to be able to run multiple applications simultaneously, which requires each application to maintain good usability within the limited screen space. While the development of these technologies has greatly improved the flexibility of user experience, it has also brought tremendous adaptation pressure to application developers.

[0003] Existing detection methods such as OwlEyes use computer vision techniques to identify specific types of UI defects by training classifiers, but such methods have several key limitations. First, traditional deep learning models usually require a large amount of labeled data for training, and the diversity of UI defects and device fragmentation make it extremely difficult to build a comprehensive training set. Second, these methods mainly rely on visual features and are difficult to understand the semantic relationships and contextual information between UI components, leading to false positives and false negatives in complex scenarios. In particular, in dynamic scenarios such as split-screen and folding, the layout and rendering of UI components will change significantly, and traditional static feature extraction methods often cannot effectively capture the defect patterns caused by these dynamic changes.

[0004] Therefore, there is an urgent need for a software interface defect detection method and device based on visual markers and thought chain prompts. SUMMARY

[0005] The present application provides a software interface defect detection method and device based on visual markers and thought chain prompts, which solves the problem that traditional static feature extraction methods often cannot effectively capture the defect patterns caused by the dynamic changes of UI components, leading to false positives and false negatives in complex scenarios.

[0006] In the first aspect of the application, a software interface defect detection method based on visual markers and thought chain prompts is provided. The method includes: obtaining multi-state UI data of a target application in different interface display states, the multi-state UI data including UI screenshot set data in different interface display states and component information file set data corresponding to each screenshot; performing preprocessing operations on the component information file set data through a component extraction algorithm, and filtering redundant component data in the component information file set data; constructing an adjacency graph based on the retained component information file set data, and traversing the adjacency graph based on a depth optimization algorithm to output a component hierarchical view sorted by proximity; based on the UI screenshot set data, drawing a set of visual markers by traversing the component hierarchical view; constructing a structured text description based on the component hierarchical view, and inputting the set of visual markers and the structured text description into a pre-constructed multi-modal large model; performing software interface defect detection based on a thought chain prompt mechanism through the multi-modal large model, and outputting a defect detection result.

[0007] Optionally, the component extraction algorithm is used to preprocess the component information file set data and filter redundant component data in the component information file set data, specifically including: obtaining component basic attributes corresponding to a target component, the target component being any one of the component information files in the set; inputting the component basic attributes into a component retention decision function and outputting a retention decision result; determining whether the retention decision result satisfies any one of the filtering conditions, the filtering conditions including a component size preset threshold, a component being a decorative element, and a component being a non-clickable container component; if the retention decision result satisfies any one of the filtering conditions, the component information file data corresponding to the target component is regarded as redundant component data for filtering.

[0008] Optionally, the adjacency graph is constructed, specifically including: calculating the center point distance between any two different components in the retained component information file set data; calculating the horizontal overlap length between the two different components through the left boundary x coordinate and the right boundary x coordinate of the two different components; calculating the vertical overlap length between the two different components through the upper boundary y coordinate and the lower boundary y coordinate of the two different components; determining the adjacency relationship of the two different components based on the center point distance, the vertical overlap length, and the horizontal overlap length; and constructing the adjacency graph based on the determination result of the adjacency relationship determination.

[0009] Optionally, the depth-first optimization algorithm is used to traverse the adjacency graph to output a component hierarchical view sorted according to proximity relationships, specifically including: selecting a component node corresponding to the upper left corner of the interface in the adjacency graph as the starting point of traversal and a component node corresponding to the lower right corner of the interface as the ending point of traversal to construct a natural browsing path traversal process; using a depth-first traversal algorithm to sequentially access a plurality of neighbor component nodes in the adjacency graph that are spatially connected to the current component node, and preferentially accessing the component node with the smallest distance from the center point of the current component node when multiple neighbor component nodes are available; recording the sequence of component nodes generated in the natural browsing path traversal process in the order of access as an initial access sequence; after the traversal is completed, appending the isolated component nodes that have not been accessed to the end of the initial access sequence to form a complete access sequence; and generating a component hierarchical view sorted according to spatial proximity relationships based on the complete access sequence.

[0010] Optionally, based on the UI screenshot set data, a set of visual markers is drawn by traversing the component hierarchical view, specifically including: based on the UI screenshot set data, the coordinate information and the category information of each component are obtained by traversing the component hierarchical view; based on the coordinate information, a visual marker layer is drawn on the UI screenshot set data; based on the category information, color coding is performed on the visual marker layer by using a pre-defined color mapping function; and the set of visual marker layers after color coding is taken as the set of visual markers.

[0011] Optionally, a structured text description is constructed based on the component hierarchical view, specifically including: based on the component hierarchical view, metadata information is parsed, the metadata information including application name information, device model information, and operating system version information; based on the metadata information, a structured text description is generated, the structured text description including system instruction defect type descriptions and user input descriptions, the system instruction defect type descriptions including text overlap descriptions, picture missing descriptions, null value display descriptions, text truncation descriptions, component overlap descriptions, button and navigation bar overlap descriptions, application crash descriptions, folding screen incompatibility descriptions, and split screen incompatibility descriptions; and the user input descriptions including number descriptions, component type descriptions, category name descriptions, component coordinate descriptions, and component content descriptions.

[0012] Optionally, before constructing a structured text description based on the component hierarchical view and inputting the set of visual markers and the structured text description into a pre-constructed multi-modal large model, the method further includes: constructing a labeled sample set based on historical multi-state UI data; and based on the labeled sample set, adjusting the parameters of the multi-modal large model by using a Few-shot adjustment strategy.

[0013] Optionally, the software interface defect detection is performed based on a thought chain prompting mechanism by the multi-modal large model, and a defect detection result is output, specifically including: mapping relationships between visual elements and component coordinate data in the set of visual markers in the component hierarchical view are constructed; the set of visual markers and the structured text description are combined and encoded into multi-modal input based on the mapping relationships, and step-by-step reasoning information is generated by the multi-modal large model; based on the step-by-step reasoning information, a defect detection result is output by a preset defect judgment condition, and the preset defect judgment condition includes a picture missing defect judgment condition, a null value display defect judgment condition, and a text truncation defect judgment condition.

[0014] In a second aspect of the present application, a software interface defect detection device based on visual markers and thought chain prompts is provided. The device includes an acquisition module and a processing module, wherein,

[0015] The acquisition module is configured to acquire multi-state UI data of a target application in different interface display states, and the multi-state UI data includes UI screenshot set data in different interface display states and a set of component information files corresponding to each screenshot.

[0016] The processing module is configured to perform preprocessing operations on the set of component information files by a component extraction algorithm, and filter redundant component data in the set of component information files; an adjacency graph is constructed based on the retained set of component information files, and the adjacency graph is traversed based on a depth optimization algorithm to output a component hierarchical view sorted by proximity; based on the set of UI screenshots, a set of visual markers is drawn by traversing the component hierarchical view; a structured text description is constructed based on the component hierarchical view, and the set of visual markers and the structured text description are input into a pre-constructed multi-modal large model; software interface defect detection is performed based on a thought chain prompting mechanism by the multi-modal large model, and a defect detection result is output.

[0017] In a third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface, and a network interface. The memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method of any one of the above aspects.

[0018] In a fourth aspect of the present application, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to perform the method of any one of the above aspects.

[0019] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] 1、obtain multi-state UI data of the target application in different interface display states; preprocess the component information file set data to filter redundant component data; construct an adjacency relation graph, and traverse the adjacency relation graph based on a depth optimization algorithm to output a component hierarchical view sorted by proximity relations; draw a set of visual markers based on the UI screenshot set data; construct a structured text description, and input the set of visual markers and the structured text description into a multimodal large model; the multimodal large model performs software interface defect detection based on a thought chain prompting mechanism to output a defect detection result, thereby being able to capture the spatial distribution and hierarchical relationship of interface components of the target application in multiple display states in real time, significantly improving the recognition ability of subtle defect patterns in dynamic interfaces, and greatly reducing the false detection and missed detection rates of static feature extraction methods in complex scenarios; at the same time, relying on the depth optimization traversal algorithm based on the adjacency relation graph, the component hierarchical view is efficiently constructed, the preprocessing and feature generation time is shortened, and the cross-state adaptability and robustness of the defect detection model are enhanced, thereby realizing high-precision and high-recall-rate software interface defect detection.

[0021] 2、construct a labeled sample set based on historical multi-state UI data; based on the labeled sample set, adjust the parameters of the multimodal large model through a Few-shot adjustment strategy, thereby realizing the rapid self-adaptation and accurate recognition of the multimodal large model to multi-state UI defect features under limited labeled sample conditions, effectively improving the accuracy and recall rate of defect detection, and significantly enhancing the robustness and generalization ability of the model in complex interface scenarios.

[0022] 3、construct a mapping relationship between component coordinate data in the component hierarchical view and in the set of visual markers; based on the mapping relationship, merge and encode the set of visual markers and the structured text description into multimodal input, and generate step-by-step reasoning information through the multimodal large model; based on the step-by-step reasoning information, output the defect detection result through the preset defect judgment condition, thereby realizing accurate recognition and explainable positioning of interface defects through multimodal step-by-step reasoning of visual markers and structured semantics, and greatly improving the accuracy, positioning accuracy, and robustness of defect detection in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of a software interface defect detection method based on visual markers and thought chain prompts provided by an embodiment of the present application;

[0024] Figure 2 is a multi-application window mode schematic diagram provided by an embodiment of the present application;

[0025] Figure 3 is an adjacency relation judgment schematic diagram provided by an embodiment of the present application;

[0026] Figure 4 is a depth-first traversal schematic diagram based on a component adjacency graph provided by an embodiment of the present application.

[0027] Figure 5 is a visualization marker layer superimposition schematic diagram based on a component hierarchical view provided by an embodiment of the present application.

[0028] Figure 6 is a thought chain prompting process schematic diagram provided by an embodiment of the present application.

[0029] Figure 7 is a defect report result schematic diagram provided by an embodiment of the present application.

[0030] Figure 8 is a module schematic diagram of a software interface defect detection device based on visualization markers and thought chain prompting provided by an embodiment of the present application.

[0031] Figure 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0032] Marker legend: 81, acquisition module; 82, processing module; 901, processor; 902, communication bus; 903, user interface; 904, network interface; 905, memory. DETAILED DESCRIPTION

[0033] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all embodiments.

[0034] The terms used in the following embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to be limiting to the present application. As used in the specification of the present application, the singular expression "one", "a", "said", "the above", "the" and "this" are intended to also include the plural expression, unless there is clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application means any or all possible combinations of one or more listed items.

[0035] Hereinafter, the terms "first", "second" are only for the purpose of description, and cannot be understood as implying or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0036] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0037] Please refer to Figure 1 It shows a flowchart of a software interface defect detection method based on visual markers and thought chain prompts provided by an embodiment of the present application. The flowchart mainly includes the following steps: S101 to S106.

[0038] Step S101, obtaining multi-state UI data of a target application in different interface display states.

[0039] Specifically, the application window mode is the display mode when the application main window is started, which determines how the application occupies the screen space, how to coexist with other applications, and how to respond to changes in device state, etc. With the rapid development of mobile device technology, the window mode of modern mobile applications has evolved from the early single full-screen mode to a variety of complex display forms, which has brought unprecedented adaptation challenges to application interface design. Please refer to Figure 2 It shows a multi-application window mode diagram provided by an embodiment of the present application. The traditional full-screen mode is the most basic window mode, and the application occupies the entire screen space. The split-screen multi-window mode divides the screen space into multiple areas, and each application occupies a part of the area. The picture-in-picture floating mode is a special multi-window form that allows a certain application to float above other applications in the form of a small window. The application needs to maintain the availability of core functions in a very small space. The folding screen response mode is the most complex window mode that has appeared in recent years. The application needs to dynamically respond to changes in the physical form of the device, and the interface needs to complete the conversion from the mobile phone interface to the tablet interface within a few hundred milliseconds.

[0040] Based on the above scenario, in order to realize the software interface defect detection in the automated test framework, first, the target application is received, the target application is a mobile terminal application running on the Android platform, which is provided in the form of an Android Package (APK) installation package, wherein the APK installation package encapsulates the executable code, resource files and Manifest list of the application, which is used to describe the component information and permission configuration of the application. For the received target application, the APK package is installed and started through the automated test framework, and then the target application is set to a plurality of different display states in a programmed manner, including but not limited to: a regular split screen mode state, a split screen upper drag state, a split screen lower drag state, and a foldable screen unfolded / folded state. Among them, the regular split screen mode state means that the target application is displayed side by side with another application on the screen through the system split screen mechanism, each occupying a preset space area and completing the interface layout redraw within the split screen width; the split screen upper drag state means that in the split screen multi-window scene, the target application is placed in the upper half of the screen through the drag operation and triggers the UI component adaptive scaling and layout adjustment within the half-screen height; the split screen lower drag state means that in the split screen multi-window scene, the target application is placed in the lower half of the screen through the drag operation and triggers the UI component adaptive scaling and layout adjustment within the half-screen height; the foldable screen unfolded state means that the target application running on the foldable screen device completes the rapid redraw of the interface from the mobile phone level small screen to the tablet level large screen when the device is switched from the folded form to the unfolded form; the foldable screen folded state means that the target application running on the foldable screen device completes the rapid redraw of the interface from the tablet level large screen to the mobile phone level small screen when the device is switched from the unfolded form to the folded form.

[0041] For each state m (m e {regular split screen, split screen upper drag, split screen lower drag, foldable}), the corresponding multi-state UI data D m is collected.

[0042] D m = {I m , J m}

[0043] Wherein, I m is the UI screenshot set data, J m is the JSON component information file set data corresponding to each screenshot, and J m contains the structured information of all UI elements of the interface.

[0044] In step S102, the component information file set data is preprocessed by a component extraction algorithm, and redundant component data in the component information file set data is filtered.

[0045] Specifically, the boundary coordinate size decoration attribute and the basic attribute such as the interactive state of the component are parsed from the component information file, and the basic attribute is transmitted into the component reservation judgment function to calculate the reservation score or the reservation decision. Then, the reservation score or the reservation decision result is evaluated according to the preset size threshold, decoration intensity threshold and interactability condition, and the component that does not meet the reservation condition is identified as a redundant component and is removed from the component information file set.

[0046] In a possible implementation, the step S102 further includes: obtaining a basic attribute of a target component, the target component being any one of the components in the component information file set; inputting the basic attribute into a component reservation judgment function, and outputting a reservation decision result; judging whether the reservation decision result meets any one of filtering conditions, the filtering conditions including a component size preset threshold, the component being a decorative element and the component being a non-clickable container type component; and if the reservation decision result meets any one of the filtering conditions, regarding the component information file data corresponding to the target component as redundant component data for filtering.

[0047] Specifically, this step mainly removes the redundant component data meaningless for UI defect detection through multi-dimensional filtering rules. For the components in the J m The basic attribute set C of the component is defined as follows:

[0048] C={id, type, text, position, width, height, clickable, resource id}

[0049] wherein, id represents a unique identifier of the interface component, used for uniquely locating the component entity in the component information file set; type represents a category attribute of the interface component, used for indicating the functional role of the component in the user interface layout; text represents the text content presented on the interface component, used for capturing the visual label and user visible information of the component; position=[[x1, y1], [x2, y2]] represents the boundary coordinates of the component; width and height respectively represent the width and height of the component; clickable represents the clickable interactive attribute of the component; resource id represents the resource identifier allocated by the operating system or the application framework for the component at runtime. In order to filter irrelevant components, the component reservation judgment function R is defined as follows:

[0050]

[0051] wherein, R decor={scrim,backtround,divider,shadow} is a collection of decorative elements; T container ={FrameLayout,LinearLayout,ViewGroup,...}, which is a collection of container-type components. clickable(c) indicates that component c is clickable. (Explanation) The determination process) through The decision to retain a component is determined based on whether it meets any of the filtering conditions, including but not limited to: a value of zero if the component's width or height is less than 5 pixels; a value of zero if the component's resource identifier belongs to a set of decorative elements; a value of zero if the component's type belongs to a set of container components and the component is not clickable; if none of the above conditions are met, the value is one. When the value is 0, the retention decision meets any of the filtering conditions, and the component information file data corresponding to the target component is filtered as redundant component data. When the value is 1, the retention decision does not meet any of the filtering conditions, and the component information file data corresponding to the target component is retained.

[0052] In addition, the preprocessing operation in this application also implements a deduplication mechanism for state switching controls, which can identify and process state display controls such as "Flashlight,Off", avoiding duplicate detection caused by different state representations of the same function.

[0053] Step S103: Construct an adjacency graph using the retained component information file set data, and traverse the adjacency graph based on a depth optimization algorithm to output a component hierarchy view sorted by proximity.

[0054] Specifically, an adjacency graph is constructed based on the retained component information set. This graph establishes spatial proximity connections between nodes and generates a graph structure based on the center point coordinates of the boundary rectangles of each component and their horizontal and vertical overlap. Then, an optimized depth-first traversal algorithm is applied to the adjacency graph, starting with the component node corresponding to the top left corner of the interface. When multiple neighboring nodes are available, the node with the smallest distance from the center point of the current node is visited first, thus recording each node in the order of visit. After the traversal is complete, any unvisited isolated component nodes are added to the end of the visit sequence, thereby outputting a component hierarchy view sorted by spatial proximity.

[0055] In one possible implementation, step S103 further includes: calculating the center point distance between any two different components in the retained component information file set data; calculating the horizontal overlap length between the two different components using the left boundary x-coordinate and right boundary x-coordinate of the two different components; calculating the vertical overlap length between the two different components using the upper boundary y-coordinate and lower boundary y-coordinate of the two different components; determining the adjacency relationship between the two different components based on the center point distance, vertical overlap length, and horizontal overlap length; and constructing an adjacency relationship graph based on the adjacency relationship determination result.

[0056] Specifically, for the retained component information file set data, an adjacency graph is constructed. graph = (V, E). In this process, first calculate any two components c. i and c j The distance between the center points is calculated as follows:

[0057]

[0058] Among them, euclidean distance (c i ,c j ) represents any two components c i and c j Distance from the center point, (x i ,y i ) and (x j ,y j The coordinates of the centers of the two components are shown below, and they are calculated as follows:

[0059] x i =(x 1i +x 2i ) / 2,y i =(y 1i +y 2i ) / 2

[0060] Where, x 1i and x 2i Component c i The left and right boundaries x-coordinates, y-coordinates 1i and y 2i Component c i The upper and lower boundary y-coordinates are then used. Then, combined with horizontal and vertical overlap detection algorithms, the adjacency relationship between components is determined. Please refer to [reference needed]. Figure 3 The document presents a schematic diagram illustrating an adjacency determination method provided in an embodiment of this application. It defines a function A(c) for determining the adjacency relationship between components. i ,c j ):

[0061]

[0062] Where distanceh(c i ,c j The distance (c) represents the horizontal edge distance. i ,c j () represents the vertical edge distance, and the expression for edge distance is shown in the following formula:

[0063] distanceh(ci,cj)=min(|x 2i -x 1j |,|x 2j -x 1i |)

[0064] distancev(ci,cj)=min(|y 2i -y 1j |,|y 2j -y 1i |)

[0065] Here, min(.) represents finding the minimum value. overlap (c i ,c j ) represents the horizontal overlap determination function, vertical overlap (c i ,c j () represents the vertical overlap determination function, and the expression of the determination function is shown in the following formula:

[0066]

[0067] Among them, Horizontal overlap (c i ,c j ), vertical overlap (c i ,c j This is used to determine whether there is overlap in horizontal or vertical areas based on the boundaries of components. For the threshold of component distance, let θ... h θ is the horizontal threshold. v The threshold is set for the vertical direction. Considering the special characteristics of mobile interfaces, different thresholds should be used for the horizontal and vertical directions. The calculation formula is shown below:

[0068]

[0069] In the horizontal direction, θ min Take 20, θ max Take 120, k is 0.04; in the vertical direction, θ minTake 20, theta max Take 150, k take 0.06. Finally, the depth-first traversal algorithm is used to traverse the component graph, and the component hierarchical view is obtained by removing the interference components and sorting the components according to the proximity relationship.

[0070] In a possible implementation, the step S103 further includes: selecting a component node corresponding to the upper left corner of the interface as a starting point of traversal and a component node corresponding to the lower right corner of the interface as an ending point of traversal in the adjacency graph to construct a natural browsing path traversal process; applying a depth-first traversal algorithm to sequentially access a plurality of neighbor component nodes in the adjacency graph that are spatially connected to a current component node, and preferentially accessing a component node with the smallest distance from the center point of the current component node when the plurality of neighbor component nodes are selectable; recording a sequence of component nodes generated in the natural browsing path traversal process in the order of access as an initial access sequence; appending an isolated component node that is not accessed to the end of the initial access sequence to form a complete access sequence after the traversal is completed; and generating a component hierarchical view sorted according to the spatial proximity relationship according to the complete access sequence.

[0071] Specifically, after the construction of the adjacency graph is completed, the components need to be sequentially processed by the depth-first traversal algorithm to construct a component hierarchical view that conforms to the semantic logic of the UI interface. The core idea of the process is to simulate the natural browsing path of a user on the interface and construct a natural browsing path traversal process. Please refer to Figure 4 , which shows a depth-first traversal diagram based on the component adjacency graph provided by the embodiment of the present application, in Figure 4 , the entire interface is gradually traversed from the upper left corner according to the spatial proximity relationship, and the associated components are ensured to maintain continuity in the final sequence.

[0072] In the natural browsing path traversal process, for the neighbor access order of each component node, a priority sorting strategy based on distance and direction is adopted. The algorithm first needs to determine a suitable traversal starting point. The system uses a spatial position-based sorting strategy to select the starting component, and the specific steps are as follows: sort according to the vertical position (the upper boundary coordinate y1) of the component; if the vertical position is the same, sort according to the horizontal position (the left boundary coordinate x1); and select the first component after sorting as the starting point. The starting point selected in this way is usually located at the upper left corner of the interface, which conforms to the user's visual browsing habit from left to right and from top to bottom.

[0073] For each neighbor component of the component, the Euclidean distance between them is calculated, and the neighbors are sorted by distance from near to far. This sorting method ensures that in the subsequent depth-first traversal process, the algorithm preferentially visits the adjacent components with closer distances, thus maintaining the spatial continuity of the traversal path. Then the depth-first traversal algorithm is executed. First, check if the current component has been visited, if it has been visited, return directly to avoid repeated processing. If not, mark the current component as visited and add it to the result sequence. Next, traverse all neighbor component nodes of the current component in the adjacency graph. Since the neighbor list in the adjacency graph has been sorted by distance, the algorithm will recursively visit each neighbor component node in order from near to far. This access order ensures the spatial continuity of adjacent components. This recursive traversal method ensures that the algorithm will explore along the adjacency relationship between components until all components in a connected region are traversed, and then backtrack to other unvisited regions. Since the neighbor list has been sorted by distance, the algorithm preferentially visits adjacent components with closer distances. The sequence generated at this time is the initial access sequence.

[0074] Finally, after completing the traversal of the connected components, the isolated component nodes that may exist also need to be processed. Isolated component nodes refer to interface elements that have no adjacency relationship with other components, including but not limited to: floating buttons, independent decorative elements, etc. By checking the access state of all components, unvisited isolated component nodes are directly added to the end of the initial access sequence to generate a complete access sequence.

[0075] Step S104, based on the UI screenshot set data, draw the visualization marker set by traversing the component hierarchical view.

[0076] Specifically, based on the UI screenshot set, the component hierarchical view sorted by spatial proximity relationship is traversed in turn, the boundary coordinates and category attributes of each component are accurately obtained, and the visualization marker layer is formed by superimposing the corresponding UI screenshot according to the category attribute through the pre-defined color mapping or graphic style. All marker layers are combined into a complete visualization marker set to intuitively present the spatial distribution and attribute information of the interface components and provide input for subsequent multi-modal defect detection.

[0077] In one possible implementation, step S104 further includes: based on the UI screenshot set data, traversing the component hierarchical view to obtain the coordinate information and category information of each component; based on the coordinate information, drawing the visualization marker layer on the UI screenshot set data; based on the category information, performing color coding on the visualization marker layer through a pre-defined color mapping function; and taking the set of visualization marker layers after color coding as the visualization marker set.

[0078] Specifically, based on the processed component hierarchical view, the category and coordinates of each component are obtained by traversing the entire view, and the structured text description is generated according to the coordinates in the UI screenshot set data I m The visual marking layer is drawn. Please refer to Figure 5 It shows a visual marking layer superimposition diagram based on a component hierarchical view provided by the embodiment of the application. The process draws a color marking box at the boundary position of each valid component by an image processing algorithm, and different types of components are distinguished by different colors: text components are marked with blue, button components are marked with green, and image components are marked with red.

[0079] During the marking process, the ID number of the component can also be displayed in the upper left corner of each component marking box, forming an information-rich visual marking set. This marking set not only retains the visual information of the original screenshot, but also intuitively displays the distribution of UI components through textual annotations, providing a clearer spatial understanding basis for subsequent multi-modal model analysis.

[0080] In step S105, a structured text description is constructed based on the component hierarchical view, and the visual marking set and the structured text description are input into a pre-constructed multi-modal large model.

[0081] Specifically, based on the component hierarchical view, the unique identification, category attribute, and spatial dependency relationship of each component are extracted in sequence, and the information is converted into a structured text description according to a predefined text template; the structured text description and the visual marking set are fused into a model input through a unified multi-modal encoding process, and are fed into a pre-constructed multi-modal large model to trigger a chain thinking prompt mechanism, thereby realizing step-by-step reasoning and final judgment of interface defects.

[0082] In one possible implementation, step S105 further includes: based on the component hierarchical view, analyzing metadata information, the metadata information including application name information, device model information, and operating system version information; based on the metadata information, generating a structured text description, the structured text description including system instruction defect type description and user input description, the system instruction defect type description including text overlap description, picture missing description, null value display description, text truncation description, component overlap description, button and navigation bar overlap description, application crash description, folding screen incompatibility description, and split screen incompatibility description; and the user input description including number description, component type description, category name description, component coordinate description, and component content description.

[0083] Specifically, the structured text description is input into the multi-modal model, and the metadata information such as the application name, the device model, and the operating system version is parsed from the processed component hierarchical view. The metadata information includes but is not limited to the application name information, the device model information, and the operating system version information. Based on the metadata information, the key component information is extracted from the component hierarchical view obtained in the previous step S10, and the structured text description of the multi-modal model input is generated. The structured text description includes system instruction defect type description and user input description, wherein the system instruction defect type description includes text overlap description, picture missing description, null value display description, text truncation description, component overlap description, button and navigation bar overlap description, application crash description, folding screen inadaptation description, and split screen inadaptation description; and the user input description includes number description, component type description, category name description, component coordinate description, and component content description.

[0084] In a possible implementation, the step S105 further includes: constructing a labeled sample set based on historical multi-state UI data; and adjusting parameters of the multi-modal large model through a Few-shot adjustment strategy based on the labeled sample set.

[0085] Specifically, the Few-shot adjustment strategy is a parameter-efficient model adaptation technique. By using the labeled sample set to adjust the pre-trained model, the model quickly obtains professional skills in a specific field. In the UI defect detection task, the Few-shot fine-tuning can enable the general multi-modal large model to learn professional skills such as spatial relationship analysis of UI components, understanding of interface design specifications, and defect pattern recognition, while maintaining the original visual understanding and language reasoning capabilities. The present application adopts a parameter-efficient fine-tuning strategy based on LoRA (Low-Rank Adaptation) to adapt the multi-modal large model.

[0086] In step S106, the multi-modal large model performs software interface defect detection based on a thought chain prompting mechanism and outputs a defect detection result.

[0087] Specifically, after the visual mark set and the structured text description are jointly encoded in the multi-modal large model, the image features and the text features are fused layer by layer relying on the thought chain prompting mechanism to generate stage-by-stage reasoning output. In each reasoning stage, the candidate regions are classified based on a preset defect judgment condition. In the final stage, the spatial coordinate information and the boundary box parameters output by the model are combined to form a defect positioning result, and a complete defect detection result including the defect category and the defect position is uniformly output.

[0088] In one possible implementation, step S106 further includes: constructing a mapping relationship between the visual elements in the component hierarchical view and the component coordinate data in the visualization mark set; merging and encoding the visualization mark set and the structured text description into a multi-modal input based on the mapping relationship, and generating step-by-step reasoning information through the multi-modal large model; and outputting a defect detection result through a preset defect judgment condition based on the step-by-step reasoning information, the preset defect judgment condition including a picture missing defect judgment condition, a null value display defect judgment condition, and a text truncation defect judgment condition.

[0089] Specifically, the fine-tuned multi-modal large model receives the visualization mark image and the structured text description as input, and performs step-by-step analysis through a thought chain prompting mechanism. The thought chain prompting is a prompting engineering technology that aims to improve the reasoning ability of the large model by guiding it to output a series of intermediate reasoning steps. The main idea is to let the multi-modal large model show its thinking process when answering a question, rather than giving a direct answer. This method is particularly suitable for complex problems, as it allows the multi-modal large model to solve problems step by step, providing a clearer reasoning path. Please refer to Figure 6 , which shows a thought chain prompting process diagram provided by an embodiment of the present application. In the diagram, the thought chain prompting process adopts a four-layer progressive analysis architecture, each layer assuming a specific cognitive function. The detailed thought chain prompting design is as follows.

[0090] In the preliminary stage, basic fusion of multi-modal information is performed, the overall layout is quickly scanned, the distribution pattern of key components is identified, and the mapping relationship between the visual elements in the component hierarchical view and the component coordinate data in the visualization mark set is constructed: the overall layout and distribution of the visual elements in the UI screenshot are observed, the key components in the screenshot are identified: TextView, Button, ImageView, and other controls, the visual elements in the screenshot are matched with the input coordinate information, and several areas that may have visual overlap and need to be checked are noted.

[0091] The model detects the nine types of defects one by one in a predefined order, and each type of defect detection strictly follows a three-step verification process: "visual inspection → coordinate analysis → conclusion judgment". This formatting ensures the consistency and completeness of the reasoning process. First, the judgment conditions for each type of defect are clearly defined:

[0092] Text overlap defect judgment condition: The judgment of text overlap defect needs to meet the conditions of both visual and coordinate dimensions, that is, two or more text elements are observed to overlap or cross display in the screenshot, and the corresponding TextView components are found to have overlapping areas in spatial position through coordinate analysis, which is specifically determined by calculating whether the intersection area of the component bounding box is greater than zero.

[0093] Picture missing defect judgment condition: The identification of picture missing defect mainly depends on whether the ImageView component is normally loaded and displayed. When a blank placeholder, error icon or obvious picture loading failure identifier appears in the screenshot, it is judged that the defect is picture missing.

[0094] Null display defect judgment condition: Null display defect needs to detect whether technical null identifier that should not be displayed to users appears in the interface, including but not limited to "null", "NULL", "undefined", "N / A" and other strings. Such defects usually indicate that an exception occurs in the data binding or content filling process, which needs to be identified through pattern matching and semantic analysis of text content.

[0095] Text truncation defect judgment condition: Text truncation defect is a common problem in mobile device adaptation. The judgment standard is whether the text content is visually truncated by the container boundary. It mainly includes three categories: text top truncation, text bottom truncation and text side truncation. According to the component information, it is judged whether the text is completely displayed visually.

[0096] Component occlusion defect judgment condition: Component occlusion defect involves abnormal hierarchical relationship between UI elements. When the key interactive component is partially or completely covered by other elements, affecting the normal operation of users, it constitutes such defect. The judgment standard mainly analyzes from two angles: whether the coordinates and vision of text components and non-text components overlap, and whether the coordinates and vision of non-text components overlap.

[0097] Button and navigation bar overlap defect judgment condition: Button and navigation bar overlap defect pays special attention to the conflict problem between interactive elements and system UI. When the buttons, menus or other clickable elements in the application overlap with the status bar, navigation bar or gesture operation area of the device, it will seriously affect the user experience, which needs to be judged by detecting whether the component position invades the system reserved area.

[0098] Application crash defect judgment condition: Application crash defect is detected by identifying abnormal interface state, including white screen, black screen, system error dialog box, ANR (application unresponsive) prompt or other obvious application abnormal termination identifier.

[0099] Split screen does not adapt and folding does not adapt are similar. After the conversion form, the application cannot fill the device screen, which is mainly judged from the visual angle.

[0100] The detailed design example is as follows, taking text truncation as an example:

[0101] Text overlap detection:

[0102] Visual inspection: Observe whether the text in the screenshot is visually overlapped;

[0103] Coordinate analysis: Check the spatial relationship of TextView control coordinates;

[0104] Conclusion: No text overlap problem found.

[0105] Picture missing detection:

[0106] Visual inspection: Picture display status in screenshot;

[0107] Coordinate analysis: Picture control status;

[0108] Conclusion: No picture missing problem found.

[0109] Null display detection:

[0110] Visual inspection: Whether there is obvious Null, NULL, null, NA text in the screenshot;

[0111] Coordinate analysis: Whether the text content is inconsistent with the visual inspection;

[0112] Conclusion: No null display problem found;

[0113] Text overflow detection:

[0114] Visual inspection: Observe whether the text in the screenshot exceeds its container boundary;

[0115] Coordinate analysis: Compare the coordinates of the control containing the text with the text coordinates;

[0116] Conclusion: No text overflow problem found.

[0117] Text truncation detection:

[0118] Visual inspection: Obvious text truncation problem in multiple places in the screenshot;

[0119] Coordinate analysis: Check the text content and visual display;

[0120] -ID: 12, TextView [179, 74, 964, 179] text is Not connected, with text truncation below;

[0121] -ID: 57, TextView [0, 179, 1504, 363] text is No account configured, with text truncation below;

[0122] -ID: 64, TextView [118, 363, 2040, 481] text is Assistant, with text truncation below;

[0123] -ID: 68, TextView [118, 484, 2040, 602] text is Settings, appears truncated below text

[0124] -ID: 72, TextView [118, 605, 2040, 723] text is Recordings, appears truncated below text

[0125] -ID: 76, TextView [118, 726, 2040, 844] text is About, appears truncated below text

[0126] -ID: 80, TextView [185, 1625, 484, 1756] text is Quit, appears truncated below text

[0127] Conclusion: Found severe text truncation cases.

[0128] Component Occlusion Detection:

[0129] Visual Inspection: Check if there are any control overlapping issues in the screenshot;

[0130] Coordinate Analysis: Check the coordinate relationship between non-text components and text components, and between non-text components;

[0131] Conclusion: No component occlusion issues found.

[0132] Button and Navigation Bar Detection:

[0133] Visual Inspection: Check the visual relationship between buttons and navigation areas in the screenshot;

[0134] Coordinate Analysis: Compare the position of buttons with the navigation bar area;

[0135] Conclusion: No button and navigation bar overlapping issues found.

[0136] Application Crash Detection:

[0137] Visual Inspection: Observe if there are any abnormal interfaces in the screenshot;

[0138] Conclusion: No application crash issues found.

[0139] Fold Inadaptation Detection:

[0140] Visual Inspection: Analyze the adaptability of the interface in the screenshot in the folded state;

[0141] Conclusion: No fold inadaptation issues found.

[0142] Split Screen Inadaptation Detection:

[0143] Visual Inspection: Analyze the performance of the interface in the screenshot in the split screen state;

[0144] Conclusion: No split-screen incompatibility issues found.

[0145] In addition, by integrating and refining the results of the preliminary analysis, not only is the defect category number determined, but also accurate pixel-level positioning information is provided. This embodies expert-level diagnostic capabilities and can convert vague visual observations into precise technical descriptions. The design is as follows:

[0146] Defect classification and positioning: comprehensive judgment based on vision and coordinates: text truncation appears under the text;

[0147] Visual evidence: multiple text truncation issues are clearly visible in the screenshot;

[0148] Coordinate evidence:

[0149] -ID:12, TextView [179, 74, 964, 179] text is "Not connected", text truncation appears under the text;

[0150] -ID:57, TextView [0, 179, 1504, 363] text is "No account configured", text truncation appears under the text;

[0151] -ID:64, TextView [118, 363, 2040, 481] text is "Assistant", text truncation appears under the text;

[0152] -ID:68, TextView [118, 484, 2040, 602] text is "Settings", text truncation appears under the text;

[0153] -ID:72, TextView [118, 605, 2040, 723] text is "Recordings", text truncation appears under the text;

[0154] -ID:76, TextView [118, 726, 2040, 844] text is "About", text truncation appears under the text;

[0155] -ID:80, TextView [185, 1625, 484, 1756] text is "Quit", text truncation appears under the text.

[0156] Finally, the multi-modal large model will organize all the problems found into a structured output according to the standardized report format. The report format includes but is not limited to: basic information such as test application name, test device model, screen resolution, operating system version, detailed defect detection results, etc. Please refer to Figure 7The document presents a schematic diagram of a defect report result provided by an embodiment of this application. In the diagram, for each detected defect, the report clearly marks the defect category number, problem description, and precise coordinate range, providing the development team with direct and feasible fixes.

[0157] This application, by employing the aforementioned methods, introduces novel defect detection mechanisms specifically for foldable and split-screen scenarios, including split-screen incompatibility, folding incompatibility, and button / navigation bar overlap, constructing a comprehensive detection system encompassing nine categories of defects. This enables the application to effectively address the diverse forms of current mobile devices, providing support for quality assurance of next-generation mobile applications.

[0158] Please refer to Figure 8 This illustration shows a schematic diagram of a software interface defect detection device based on visual markers and thought chain prompts according to an embodiment of this application. The device includes an acquisition module 81 and a processing module 82, wherein...

[0159] The acquisition module 81 is used to acquire multi-state UI data of the target application under different interface display states. The multi-state UI data includes a set of UI screenshots under different interface display states, as well as a set of component information files corresponding to each screenshot.

[0160] Processing module 82 is used to preprocess the component information file set data using a component extraction algorithm and filter redundant component data in the component information file set data; construct an adjacency graph using the retained component information file set data, and traverse the adjacency graph based on a depth optimization algorithm to output a component hierarchy view sorted by proximity; draw a visual tag set based on the UI screenshot set data by traversing the component hierarchy view; construct a structured text description based on the component hierarchy view, and input the visual tag set and structured text description into a pre-constructed multimodal large model; perform software interface defect detection through the multimodal large model based on a thought chain prompt mechanism, and output the defect detection results.

[0161] In one possible implementation, the acquisition module 81 is used to preprocess the component information file set data using a component extraction algorithm and filter redundant component data in the component information file set data. Specifically, this includes: acquiring the basic attributes of the target component, where the target component is any component in the component information file set; inputting the basic attributes of the component into a component retention decision function and outputting the retention decision result; determining whether the retention decision result meets any of the filtering conditions, including a preset threshold for component size, the component being a decorative element, and the component being a non-clickable container component; if the retention decision result meets any of the filtering conditions, then the component information file data corresponding to the target component is used as redundant component data for filtering.

[0162] In a possible implementation, the processing module 82 is configured to construct the adjacency graph, specifically including: calculating the center point distance between any two different components in the reserved component information file set data; calculating the horizontal overlap length between the two different components by the left boundary x coordinate and the right boundary x coordinate of the two different components; calculating the vertical overlap length between the two different components by the upper boundary y coordinate and the lower boundary y coordinate of the two different components; determining the adjacency relationship between the two different components based on the center point distance, the vertical overlap length, and the horizontal overlap length; and constructing the adjacency graph based on the determination result of the adjacency relationship determination.

[0163] In a possible implementation, the processing module 82 is configured to traverse the adjacency graph based on a depth optimization algorithm to output the component hierarchical view sorted according to the proximity relationship, specifically including: selecting a component node corresponding to the upper left corner of the interface in the adjacency graph as a traversal starting point, and selecting a component node corresponding to the lower right corner of the interface as a traversal ending point to construct a natural browsing path traversal process; applying a depth-first traversal algorithm to sequentially access a plurality of neighbor component nodes in the adjacency graph that are spatially connected to a current component node, and preferentially accessing a component node with the smallest center point distance from the current component node when the plurality of neighbor component nodes are selectable; recording a sequence of component nodes generated in the natural browsing path traversal process according to the access order as an initial access sequence; appending isolated component nodes that are not accessed to the end of the initial access sequence to form a complete access sequence after the traversal is completed; and generating the component hierarchical view sorted according to the spatial proximity relationship according to the complete access sequence.

[0164] In a possible implementation, the processing module 82 is configured to draw the visualization marker set by traversing the component hierarchical view based on the UI screenshot set data, specifically including: obtaining the coordinate information and the category information of each component by traversing the component hierarchical view based on the UI screenshot set data; drawing a visualization marker layer on the UI screenshot set data based on the coordinate information; performing color coding on the visualization marker layer based on the category information by using a pre-defined color mapping function; and taking the set of the visualization marker layer after the color coding as the visualization marker set.

[0165] In a possible implementation, the processing module 82 is configured to construct the structured text description based on the component hierarchical view, specifically including: parsing metadata information based on the component hierarchical view, the metadata information including application name information, device model information, and operating system version information; generating the structured text description based on the metadata information, the structured text description including system instruction defect type descriptions and user input descriptions, the system instruction defect type descriptions including text overlap descriptions, picture missing descriptions, null value display descriptions, text truncation descriptions, component overlap descriptions, button and navigation bar overlap descriptions, application crash descriptions, folding screen inadaptation descriptions, and split screen inadaptation descriptions; and the user input descriptions including number descriptions, component type descriptions, category name descriptions, component coordinate descriptions, and component content descriptions.

[0166] In a possible implementation, the processing module 82 is configured to construct a labeling sample set based on historical multi-state UI data before constructing the structured text description based on the component hierarchical view and inputting the visual mark set and the structured text description into the pre-constructed multi-modal large model; and performing parameter adjustment on the multi-modal large model through a Few-shot adjustment strategy based on the labeling sample set.

[0167] In a possible implementation, the processing module 82 is configured to perform software interface defect detection based on a thought chain prompting mechanism through the multi-modal large model, and output a defect detection result, specifically including: constructing a mapping relationship between visual elements in the component hierarchical view and component coordinate data in the visual mark set; merging and encoding the visual mark set and the structured text description into a multi-modal input based on the mapping relationship, and generating step-by-step reasoning information through the multi-modal large model; and outputting the defect detection result based on the step-by-step reasoning information through a preset defect judgment condition, the preset defect judgment condition including a picture missing defect judgment condition, a null value display defect judgment condition, and a text truncation defect judgment condition.

[0168] It should be noted that the apparatus provided in the above embodiments is only used as an example for dividing the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0169] The present application also provides an electronic device. Referring to Figure 9 , Figure 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device can include at least one processor 901, at least one communication bus 902, a user interface 903, at least one network interface 904, and a memory 905.

[0170] The communication bus 902 is configured to realize the connection communication between the components.

[0171] The user interface 903 can include a display screen, a camera, and optionally a standard wired interface and a wireless interface.

[0172] The network interface 904 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0173] The processor 901 can include one or more processing cores. The processor 901 connects various parts of the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 905, and calling data stored in the memory 905. Optionally, the processor 901 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 901 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU is mainly used to process the operating system, user interface, and application programs; the GPU is used to render and draw the content to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 901, but can be realized by a separate chip.

[0174] The memory 905 may include random access memory (RAM) or read-only memory. Optionally, the memory 905 may include a non-transitory computer-readable storage medium. The memory 905 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 905 may also be at least one storage device located remotely from the aforementioned processor 901. (Refer to...) Figure 9 The memory 905, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a software interface defect detection application based on visual markers and thought chain prompts.

[0175] exist Figure 9 In the illustrated electronic device, the user interface 903 is primarily used to provide an input interface for the user and acquire user input data; while the processor 901 can be used to call the software interface defect detection application stored in the memory 905 based on visual tags and thought chain prompts. When executed by one or more processors 901, the electronic device performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0176] This application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, these instructions cause an electronic device to perform one or more of the methods described in the above embodiments.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0178] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments is merely illustrative, and the units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0180] In addition, the functional units in each embodiment of the present application can be integrated into a processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0181] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, and various program codes that can be stored.

[0182] The above is only exemplary embodiments of the present application, and cannot limit the scope of the present application. Any equivalent changes and modifications made in accordance with the teachings of the present application are still within the scope of the present application.

[0183] The present application is intended to cover any variations, uses or adaptive changes of the present application, which follow the general principles of the present application and include common knowledge or conventional technical means in the technical field of the present application not disclosed in the present application.

Claims

1. A software interface defect detection method based on visual markers and mind mapping cues, characterized in that, The method comprises: acquiring multi-state UI data of a target application in different interface display states, the multi-state UI data comprising a set of UI screenshot data in different interface display states and a set of component information file data corresponding to each screenshot; performing preprocessing on the set of component information file data by a component extraction algorithm and filtering redundant component data in the set of component information file data; constructing an adjacency graph based on the set of component information file data retained after the preprocessing, and traversing the adjacency graph based on a deep optimization algorithm to output a component hierarchical view sorted according to proximity relationships; based on the set of UI screenshot data, drawing a set of visual markers by traversing the component hierarchical view; constructing a structured text description based on the component hierarchical view, and inputting the set of visual markers and the structured text description into a pre-constructed multi-modal large model; performing software interface defect detection based on a thought chain prompting mechanism by the multi-modal large model, and outputting a defect detection result.

2. The method of claim 1, wherein, The preprocessing on the set of component information file data by the component extraction algorithm and the filtering of the redundant component data in the set of component information file data specifically comprise: acquiring component basic attributes corresponding to a target component, the target component being any one of the set of component information files; inputting the component basic attributes into a component retention decision function and outputting a retention decision result; determining whether the retention decision result satisfies any one of filtering conditions, the filtering conditions including a component size preset threshold, a component being a decorative element, and a component being a non-clickable container component; if the retention decision result satisfies any one of the filtering conditions, the component information file data corresponding to the target component is filtered as the redundant component data.

3. The method of claim 1, wherein, The construction of the adjacency graph specifically comprises: calculating the center point distance between any two different components in the set of component information file data retained after the preprocessing; calculating the horizontal overlap length between the two different components by the left boundary x coordinate and the right boundary x coordinate of the two different components; calculating the vertical overlap length between the two different components by the upper boundary y coordinate and the lower boundary y coordinate of the two different components; determining the adjacency relationship of the two different components based on the center point distance, the vertical overlap length, and the horizontal overlap length; constructing the adjacency graph based on the determination result of the adjacency relationship determination.

4. The method of claim 1, wherein, The traversal of the adjacency graph based on the deep optimization algorithm to output the component hierarchical view sorted according to proximity relationships specifically comprises: selecting a component node corresponding to the upper left corner of the interface in the adjacency graph as a traversal starting point and a component node corresponding to the lower right corner of the interface as a traversal ending point to construct a natural browsing path traversal process; applying a depth-first traversal algorithm to successively access a plurality of neighbor component nodes in the adjacency graph that are spatially connected to a current component node, and preferentially accessing a component node having the smallest center point distance from the current component node when a plurality of the neighbor component nodes are selectable; record a sequence of component nodes generated in the visiting order in the natural browsing path as an initial visiting sequence; append isolated component nodes not visited to the end of the initial visiting sequence to form a complete visiting sequence after the traversal is completed; generate the component hierarchical view ordered according to the spatial proximity relationship according to the complete visiting sequence.

5. The method of claim 1, wherein, the visual marker set is drawn by traversing the component hierarchical view based on the UI screenshot set data, specifically including: based on the UI screenshot set data, the coordinate information and the category information of each component are obtained by traversing the component hierarchical view; based on the coordinate information, a visual marker layer is drawn on the UI screenshot set data; based on the category information, color coding is performed on the visual marker layer through a predefined color mapping function; the set of visual marker layers after color coding is taken as the visual marker set.

6. The method of claim 1, wherein, the structured text description is constructed based on the component hierarchical view, specifically including: metadata information is parsed based on the component hierarchical view, the metadata information including application name information, device model information and operating system version information; the structured text description is generated based on the metadata information, the structured text description including system instruction defect type description and user input description, the system instruction defect type description including text overlap description, picture missing description, null value display description, text truncation description, component overlap description, button and navigation bar overlap description, application crash description, folding screen inadaptation description and split screen inadaptation description; the user input description including number description, component type description, category name description, component coordinate description and component content description.

7. The method of claim 1, wherein, before the structured text description is constructed based on the component hierarchical view and the visual marker set and the structured text description are input into the pre-constructed multi-modal large model, the method further includes: an annotation sample set is constructed based on historical multi-state UI data; based on the annotation sample set, the multi-modal large model is parameter adjusted through a Few-shot adjustment strategy.

8. The method of claim 1, wherein, the software interface defect detection is performed based on the thought chain prompt mechanism through the multi-modal large model, and the defect detection result is output, specifically including: a mapping relationship between visual elements in the component hierarchical view and component coordinate data in the visual marker set is constructed; based on the mapping relationship, the visual marker set and the structured text description are merged and coded as multi-modal input, and step-by-step reasoning information is generated through the multi-modal large model; based on the step-by-step reasoning information, the defect detection result is output through a preset defect judgment condition, the preset defect judgment condition including a picture missing defect judgment condition, a null value display defect judgment condition and a text truncation defect judgment condition.

9. A software interface defect detection apparatus based on visual markers and mind mapping cues, characterized by, the device includes an acquisition module and a processing module, wherein The acquisition module is configured to acquire multi-state UI data of a target application in different interface display states, wherein the multi-state UI data comprises a set of UI screenshot data in different interface display states and a set of component information file data corresponding to each screenshot; The processing module is configured to perform a preprocessing operation on the set of component information file data by using a component extraction algorithm, filter redundant component data in the set of component information file data, construct an adjacency graph based on the set of component information file data that is retained, traverse the adjacency graph based on a deep optimization algorithm to output a component hierarchical view sorted according to a proximity relationship, draw a set of visual markers based on the set of UI screenshot data by traversing the component hierarchical view, construct a structured text description based on the component hierarchical view, and input the set of visual markers and the structured text description into a pre-constructed multi-modal large model; and perform software interface defect detection based on a thought chain prompting mechanism by using the multi-modal large model, and output a defect detection result.

10. An electronic device, comprising: The electronic device comprises a processor, a communication bus, a user interface, a network interface, and a memory, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method according to any one of claims 1 to 8. The acquisition module is configured to acquire multi-state UI data of a target application in different interface display states, wherein the multi-state UI data comprises a set of UI screenshot data in different interface display states and a set of component information file data corresponding to each screenshot; The processing module is configured to perform a preprocessing operation on the set of component information file data by using a component extraction algorithm, filter redundant component data in the set of component information file data, construct an adjacency graph based on the set of component information file data that is retained, traverse the adjacency graph based on a deep optimization algorithm to output a component hierarchical view sorted according to a proximity relationship, draw a set of visual markers based on the set of UI screenshot data by traversing the component hierarchical view, construct a structured text description based on the component hierarchical view, and input the set of visual markers and the structured text description into a pre-constructed multi-modal large model; and perform software interface defect detection based on a thought chain prompting mechanism by using the multi-modal large model, and output a defect detection result. The electronic device comprises a processor, a communication bus, a user interface, a network interface, and a memory, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method according to any one of claims 1 to 8.

Images