UI anomaly detection method based on prior guided semantic segmentation and related device

Abstract

This application provides a UI anomaly detection method and related apparatus based on prior-guided semantic segmentation. The method includes: acquiring a screenshot of the current interface and underlying view tree data, and extracting the physical rendering and interaction logic attributes of each UI node; dynamically constructing an elastic semantic constraint prior tensor aligned with the screenshot space based on the physical rendering attributes; injecting this tensor as a spatial prior feature into a deep semantic segmentation network, performing spatial feature modulation during the inference phase, and outputting an actual semantic probability distribution map; constructing an asymmetric breach penalty mechanism in conjunction with interaction attributes, calculating the actual missing and overflow amounts distributed within the constraint range, and quantifying the structural breach score; and performing computational power gating scheduling based on the score, skipping multiple inferences when structural collapse is determined, and scheduling random deactivated inference to detect rendering distortion when no collapse occurs. This invention reduces semantic ambiguity in complex interfaces, can objectively distinguish between core interaction failures and visual deviations, and reduces system computational overhead.

Description

Technical Field

[0001] This invention relates to the fields of image processing and software automated testing technology, specifically to a UI anomaly detection method and related apparatus based on prior guided semantic segmentation. Background Technology

[0002] In mobile application and complex web system development, UI quality directly impacts user experience. Existing automated testing techniques primarily rely on scripts to parse view trees or pixel-based comparison for difference detection. Verifying only code logic cannot detect visual defects such as text overlap, blank screens, or abnormal compression; while pixel-based comparison methods are prone to generating numerous false positives when dealing with dynamic content, failing to accurately distinguish between fatal interaction failures and ordinary visual flaws.

[0003] Conventional depth vision models introduced to overcome the aforementioned shortcomings are prone to semantic misjudgments because they cannot perceive the structural context of the underlying UI. Furthermore, blind full-image inference on mobile devices or edge nodes may lead to computational overload and frame drops. In other words, existing methods sever the intrinsic connection between the underlying software physical structure and the front-end visual presentation, failing to establish a constraint mechanism for model inference. Consequently, they struggle to balance detection accuracy and system execution efficiency in complex testing scenarios. Summary of the Invention

[0004] In view of the aforementioned problems, this application is proposed to provide a UI anomaly detection method and related apparatus based on prior-guided semantic segmentation to overcome or at least partially solve the aforementioned problems, comprising: A UI anomaly detection method based on prior-guided semantic segmentation includes: Obtain the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data. Based on the physical rendering attributes, an elastic semantic constraint prior tensor that is spatially aligned with the original screenshot image is dynamically constructed. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate. The elastic semantic constraint prior tensor is injected as a spatial prior feature into the deep semantic segmentation network, and used to modulate the feature response in the forward inference stage of the deep semantic segmentation network, outputting an actual semantic probability distribution map that integrates structural priors. During the anomaly detection phase, the prediction results of the elastic semantic constraint prior tensor are used to determine the degree of matching of the actual semantic probability distribution map within the tolerance range defined by the elastic semantic constraint prior tensor, and the structural default score is calculated by combining the semantic missing amount and the overflow amount. The computational power gating scheduling is performed based on the prediction of structural state collapse. If the structural default score exceeds the preset collapse threshold, a structural rendering anomaly is determined, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

[0005] According to an embodiment of the present invention, the step of dynamically constructing a flexible semantic constraint prior tensor spatially aligned with the original screenshot image based on the physical rendering attributes includes: Extract the inner margin, outer margin, Z-axis height, and clipping rules from the physical rendering attributes; For text-type nodes, the boundary is expanded based on font rendering characteristics to determine the tolerance radius; For nodes with Z-axis height attributes, a spatial diffusion tolerance region is generated based on the depth attribute; The normalized constraint value of each pixel relative to the bounding box of the underlying view tree node is calculated based on the spatial distance mapping function to determine the prior constraint value at each pixel.

[0006] According to an embodiment of the present invention, the normalized constraint value of each pixel relative to the bounding box of the underlying view tree node is calculated based on the spatial distance mapping function to obtain the node. In pixel coordinates Prior constraint values ​​at the location The formula is:

[0007] in, For pixels to the DOM's original bounding box Physical spatial distance, The tolerance radius is dynamically determined based on the physical rendering attributes.

[0008] According to an embodiment of the present invention, the step of modulating the feature response and outputting an actual semantic probability distribution map that incorporates structural priors during the forward inference stage of the deep semantic segmentation network includes: Spatial attention feature maps are generated using the elastic semantic constraint prior tensor. ; The spatial attention feature map Injected into the feature extraction layer of the deep semantic segmentation network, visual features are processed. Pixel-by-pixel affine transformation modulation is performed to obtain the fused features. :

[0009] in, Represents the Hadama product. and These are the learnable modulation parameters in the deep semantic segmentation network, so that the feature response of the deep semantic segmentation network is constrained by the elastic semantic constraint prior tensor.

[0010] According to an embodiment of the present invention, the calculation of the structural default score by combining semantic missing amount and overflow amount includes: Identify the clickable, focusable, and swipeable attributes of each UI node from the interaction logic attributes; When a UI node with interactive attributes experiences semantic loss within the legal region defined by the elastic semantic constraint prior tensor, a first weighting coefficient is introduced when calculating the structural breach score. When a UI node without interactive attributes exceeds the tolerance boundary of the elastic semantic constraint prior tensor, a second weighting coefficient is introduced when calculating the structural default score; wherein the first weighting coefficient is greater than the second weighting coefficient.

[0011] According to an embodiment of the present invention, the method of scheduling a local region to perform a preset number of random deactivation inferences to detect local uncertainties includes: Determine whether there are locally connected components in the actual semantic probability distribution graph whose confidence level in a single inference is lower than the threshold value; If it exists, then Monte Carlo random deactivation inference is triggered for that local connected domain; Pixel-level information entropy is calculated based on the output results of multiple forward propagations, wherein the pixel-level information entropy is calculated based on the probability distribution of the multiple forward propagation results; if the pixel-level information entropy is greater than a preset entropy threshold, it is determined that the local connected component has experienced material rendering distortion.

[0012] According to an embodiment of the present invention, the method further includes: The deep semantic segmentation network is trained based on a rendering violation penalty loss function to optimize its performance in UI rendering anomaly detection tasks. Defined as:

[0013] in, For the indicator function generated based on the interaction logic attribute, when the coordinates The value is 1 if the node has an interactive attribute, and 0 otherwise. Characterizes the degree of missing semantic features. The degree of semantic feature overflow is represented, and the penalty coefficient satisfies the following conditions: .

[0014] A UI anomaly detection device based on prior-guided semantic segmentation includes: The context acquisition module is used to acquire the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data. The constraint tensor generation module is used to dynamically construct an elastic semantic constraint prior tensor that is aligned with the space of the original screenshot image based on the physical rendering attributes. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate. The cross-modal guided inference module is used to inject the elastic semantic constraint prior tensor as a spatial prior feature into the deep semantic segmentation network, and to modulate the feature response in the forward inference stage of the deep semantic segmentation network, outputting an actual semantic probability distribution map that integrates structural priors. The asymmetric default detection module is used to use the elastic semantic constraint prior tensor constraint prediction results during the anomaly detection stage to determine the degree of matching of the actual semantic probability distribution map within the tolerance range defined by the elastic semantic constraint prior tensor, and to calculate the structural default score by combining the semantic missing amount and the overflow amount. The computing power gating scheduling module is used to perform computing power gating scheduling based on structural state collapse prediction. If the structural default score exceeds a preset collapse threshold, a structural rendering anomaly is determined, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

[0015] A computer electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When executed by the processor, the computer program implements the UI anomaly detection method based on prior guided semantic segmentation as described above.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the UI anomaly detection method based on prior guided semantic segmentation as described above.

[0017] This application has the following advantages: In the embodiments of this application, a flexible semantic constraint prior tensor is constructed by extracting physical rendering attributes from the underlying view tree and injecting it into a deep semantic segmentation network for cross-modal spatial feature modulation. This allows the reasoning process of the visual model to be guided and constrained by the underlying software physical boundary, effectively reducing semantic ambiguity of similar controls in complex interfaces and improving the accuracy of segmentation results. At the same time, an asymmetric breach penalty mechanism is constructed by combining interaction logic attributes, which can objectively distinguish between functional structural failures and non-functional visual deviations, improving the rationality of anomaly judgment. In addition, a computing power gating scheduling mechanism based on structural state collapse prediction is introduced. When a serious structural breach is clearly identified, the high-energy-consuming local multiple random inactivation reasoning is actively skipped, and hardware and software computing resources are reasonably allocated, effectively reducing the computing power overhead and system latency of test nodes when performing high-frequency anomaly detection tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a UI anomaly detection method based on prior guided semantic segmentation provided in an embodiment of this application; Figure 2 This is a partial flowchart of step S120 in a UI anomaly detection method based on prior guided semantic segmentation provided in an embodiment of this application; Figure 3 This is a partial flowchart of step S140 in a UI anomaly detection method based on prior guided semantic segmentation provided in an embodiment of this application; Figure 4 This is a structural block diagram of a UI anomaly detection device based on prior guided semantic segmentation provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer electronic device provided in an embodiment of the present invention; 1. Computer electronic device; 2. External device; 3. Processing unit; 4. Bus; 5. Network adapter; 6. I / O interface; 7. Display; 8. Memory; 9. Random access memory; 10. Cache memory; 11. Storage system; 12. Program / utility; 13. Program module. Detailed Implementation

[0020] To make the objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, before providing a detailed description of the specific embodiments of this invention, the following explanations and definitions are provided for some key technical terms involved in this invention: Elastic semantic constraint prior tensor: refers to a multi-channel data structure that not only records the spatial coordinates of UI nodes, but also defines the physical boundaries and spatial tolerances of the allowed distribution of each semantic category at the pixel scale in combination with physical rendering attributes (such as anti-aliasing, shadow spread radius, etc.).

[0022] Cross-modal spatial modulation: refers to the process of injecting prior information from the non-image modality DOM space as an attention gating into the visual feature map during the forward inference process of a deep learning neural network, thereby shifting or enhancing the response intensity of the visual features pixel by pixel.

[0023] Computational power gating scheduling refers to a hardware resource allocation mechanism based on task status feedback. In this invention, it specifically refers to the scheduling behavior of automatically skipping the high-energy-consuming Monte Carlo random deactivation sampling calculation when the system predicts a severe structural collapse of the interface through low-energy residual calculation, thereby realizing the offloading of computing power.

[0024] Structural Default Score: This is a quantitative indicator used to characterize the degree of deviation between the actual rendered mask and the elastic semantic constraint field. The score is obtained by asymmetrically weighting the "semantic missing amount" (should be present but not) and the "semantic overflow amount" (should not be present but are present).

[0025] Monte Carlo random deactivation: refers to a technique that keeps the random deactivation layer of the network on during the inference phase, performs multiple random forward propagations on the same input image, and uses the probability distribution of the prediction results (such as information entropy) to measure the uncertainty of the model.

[0026] Physical rendering properties refer to the underlying properties that affect the final appearance of UI elements on the screen, including padding, margin, Z-axis height, view clipping, and font rendering anti-aliasing parameters.

[0027] Reference Figure 1 This illustration shows a UI anomaly detection method based on prior guided semantic segmentation according to an embodiment of this application, which includes the following steps: S110. Obtain the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data.

[0028] Specifically, during the operation of the application under test, original screenshots of the current application interface are captured synchronously. Simultaneously with image acquisition, corresponding underlying view tree data is extracted from the underlying framework of the application under test. This underlying view tree data reflects the hierarchical nesting relationship and attribute definitions of the UI nodes in the current interface. Based on this, for each UI node in the underlying view tree data, its physical rendering attributes and interaction logic attributes are extracted. The physical rendering attributes include, but are not limited to, attribute information used to define the node's spatial position, size, and visual presentation style; the interaction logic attributes are used to characterize the node's response to user operations. The extraction of the above attribute data is based on the parsing of the application's underlying data structure, ensuring the accuracy and completeness of the acquired information.

[0029] S120. Based on the physical rendering attributes, dynamically construct an elastic semantic constraint prior tensor that is spatially aligned with the original screenshot image. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate.

[0030] In this embodiment, based on the extracted physical rendering attributes of each UI node, the expected rendering area of ​​each node in two-dimensional space is calculated. Considering the physical characteristics of the rendering engine during the actual drawing process, an elastic expansion or contraction mechanism is applied to transform the expected rendering area into a constrained area containing a tolerance range. Then, the constrained areas of each UI node are mapped to a space completely consistent with the size and coordinate system of the original screenshot image, constructing the elastic semantic constraint prior tensor. This elastic semantic constraint prior tensor is organized using a multi-channel data structure, with different channels corresponding to different semantic categories. Its internal element values ​​quantify the rationality or allowable distribution range limit of a specific semantic category at the corresponding pixel coordinate position.

[0031] S130. The elastic semantic constraint prior tensor is injected into the deep semantic segmentation network as a spatial prior feature. It is used to modulate the feature response in the forward inference stage of the deep semantic segmentation network and outputs an actual semantic probability distribution map that integrates structural priors.

[0032] In the actual network inference process, the original screenshot image is input into a deep semantic segmentation network to extract visual features. For example, the deep semantic segmentation network can employ, but is not limited to, a UNet network based on a CNN architecture, a DeepLabV3 network, or a SegFormer network based on a Transformer architecture to extract visual features. Simultaneously, the constructed elastic semantic constraint prior tensor is injected as a spatial prior feature into the feature processing flow of the deep semantic segmentation network, for example, into the Layer 4 layer of the ResNet backbone network or the skip connection layer of the decoder. Specifically, in the forward inference stage of the network, the elastic semantic constraint prior tensor is used to perform pixel-by-pixel attention modulation or feature affine transformation on the visual feature map. This modulation process enables the deep semantic segmentation network to maintain or enhance feature responses within the prior constraint range and suppress feature responses outside the prior constraint range when extracting features. The modulated fused features continue to propagate forward in the network, ultimately outputting a probability distribution map of each pixel belonging to various semantic labels, i.e., the actual semantic probability distribution map.

[0033] S140. During the anomaly detection phase, the prediction results of the elastic semantic constraint prior tensor are used to determine the degree of matching of the actual semantic probability distribution map within the tolerance range defined by the elastic semantic constraint prior tensor, and the structural default score is calculated by combining the semantic missing amount and the overflow amount.

[0034] Specifically, after obtaining the actual semantic probability distribution map, it is spatially aligned and compared with the elastic semantic constraint prior tensor. The comparison process mainly considers whether the actual predicted semantic distribution falls within the legal tolerance range defined by the prior tensor. On the one hand, the area or number of pixels of the region where the prior tensor requires the existence of a certain semantic but the probability in the actual prediction is low is calculated, which is used as the semantic missing amount; on the other hand, the area or number of pixels of the region where the actual prediction contains a certain semantic but exceeds the limit range of the prior tensor is calculated, which is used as the semantic overflow amount. Based on the above semantic missing amount and overflow amount, and through a preset evaluation function or algorithm model, a weighted calculation is performed, and finally, the structural default score is quantitatively output. This score objectively reflects the severity of the current interface rendering result deviating from the underlying design expectation.

[0035] S150. Perform computational power gating scheduling based on structural state collapse prediction. If the structural default score exceeds a preset collapse threshold, a structural rendering anomaly is determined, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

[0036] This step makes branch scheduling decisions based on the calculated structural default score. When the structural default score is greater than or equal to a pre-set collapse threshold, it indicates that the current UI interface has experienced severe macroscopic structural deviation or control loss. In this case, it is directly determined as a structural rendering anomaly, and subsequent high-energy-consuming computational tasks for this area are blocked through internal scheduling instructions, i.e., random deactivation multiple forward propagation calculations based on Monte Carlo methods are no longer executed. Conversely, when the structural default score is less than the collapse threshold, it indicates that the macroscopic structure is within a reasonable tolerance, but there may be microscopic material or rendering defects. In this case, according to the scheduling strategy, the deep semantic segmentation network is controlled to keep the random deactivation layer on, and forward inference is repeated a preset number of times for the suspected abnormal local area. By collecting the probability distribution differences of multiple inferences, uncertainty metrics such as information entropy are calculated to complete fine-grained local uncertainty detection.

[0037] In the embodiments of this application, a flexible semantic constraint prior tensor is constructed by extracting physical rendering attributes from the underlying view tree and injecting it into a deep semantic segmentation network for cross-modal spatial feature modulation. This allows the reasoning process of the visual model to be guided and constrained by the underlying software physical boundary, effectively reducing semantic ambiguity of similar controls in complex interfaces and improving the accuracy of segmentation results. At the same time, an asymmetric breach penalty mechanism is constructed by combining interaction logic attributes, which can objectively distinguish between functional structural failures and non-functional visual deviations, improving the rationality of anomaly judgment. In addition, a computing power gating scheduling mechanism based on structural state collapse prediction is introduced. When a serious structural breach is clearly identified, the high-energy-consuming local multiple random inactivation reasoning is actively skipped, and hardware and software computing resources are reasonably allocated, effectively reducing the computing power overhead and system latency of test nodes when performing high-frequency anomaly detection tasks.

[0038] Reference Figure 2 As shown, in one embodiment of the present invention, step S120, dynamically constructing an elastic semantic constraint prior tensor aligned with the spatial dimensions of the original screenshot image based on the physical rendering attributes, includes: S210. Extract the inner margin, outer margin, Z-axis height, and clipping rules from the physical rendering attributes.

[0039] Specifically, the extraction process is implemented by accessing the attribute fields of the underlying view tree data structure. The inner and outer margins together define the effective logical response area and visual boundary offset of each UI node in the screen coordinate system. The Z-axis height reflects the stacking hierarchy of nodes in the direction perpendicular to the screen plane, used to determine the coverage relationship between different nodes. The clipping rules define the display constraints of the parent node on the rendering overflow of its child nodes. The extracted attribute data is converted into a unified format geometric description operator, providing a basic coordinate reference for subsequently constructing spatially aligned tensors.

[0040] S220. For text type nodes, the boundary is extended based on font rendering characteristics to determine the tolerance radius.

[0041] In this embodiment, for the UI node identified as text, the impact of font rendering characteristics on actual imaging is considered. These font rendering characteristics include, but are not limited to, pixel-level diffusion caused by anti-aliasing, font library differences, or typesetting engines. Based on these characteristics, the original geometric boundary of the text-type node is spatially expanded outwards, and the calculated expansion range is the tolerance radius. The process of determining the tolerance radius reduces the spatial distribution difference between the idealized logical boundary and the actual rendered pixel boundary, ensuring that the generated constraint field can encompass the actual text pixels.

[0042] S230. For nodes with Z-axis height attributes, generate a spatial diffusion tolerance region based on the depth attribute.

[0043] In this embodiment, for nodes with a non-zero Z-axis height attribute, the corresponding depth attribute is used to simulate physical projection or rendering drift during the rendering process. The depth attribute is directly related to the expansion range of the spatial diffusion tolerance region; that is, as the height value increases, the corresponding generated tolerance region increases accordingly. The spatial diffusion tolerance region corresponds to the visual impact range during node rendering in the three-dimensional logical space, and is used to compensate for the edge rendering uncertainty generated by the rendering engine when processing multi-level component stacking.

[0044] S240. Calculate the normalized constraint value of each pixel relative to the bounding box of the underlying view tree node based on the spatial distance mapping function, so as to determine the prior constraint value at each pixel.

[0045] Specifically, the spatial distance mapping function receives the spatial coordinates of each pixel in the original screenshot image and the geometric coordinates of the corresponding bounding box as input. By calculating the shortest Euclidean distance from the pixel to the edge of the bounding box and performing a nonlinear mapping with the tolerance radius and the spatial diffusion tolerance region, the distance dimension is transformed into a dimensionless value in the interval [0,1]. The normalized constraint value represents the degree of certainty that a pixel belongs to a specific semantic label; pixels closer to the boundary and within the tolerance range have a higher constraint response. Finally, the normalized constraint values ​​of all pixels are combined according to the pixel arrangement order of the original screenshot image to form the elastic semantic constraint prior tensor.

[0046] This embodiment establishes a mathematical relationship from the underlying logical structure to the top-level pixel distribution by integrating the physical geometric attributes and dynamic rendering characteristics of UI components. Utilizing a tolerance correction and distance mapping mechanism based on rendering characteristics, it achieves alignment between the logical design intent and the actual physical image, effectively absorbing pixel-level positional deviations caused by differences in rendering engines or cross-platform adaptation. This provides prior features with physical semantic support to improve the accuracy of rendering anomaly detection.

[0047] For example, the normalized constraint value of each pixel relative to the bounding box of the underlying view tree nodes is calculated based on the spatial distance mapping function to obtain the node. In pixel coordinates Prior constraint values ​​at the location The formula is:

[0048] in, For pixels to the DOM's original bounding box Physical spatial distance, The tolerance radius is dynamically determined based on the physical rendering attributes.

[0049] The above calculation process of the prior constraint values ​​constructs a constraint model based on distance decay. In this model, Characterizes the target pixel in the global pixel coordinate system With nodes DOM primitive bounding box The shortest straight-line distance between geometric boundaries. When pixels... Located in the enclosure When inside or exactly on the boundary, this distance It is zero. As the dynamic tolerance radius, it is defined by the bounding box. This serves as an effective buffer zone radiating outwards from the starting point. This tolerance radius is not a fixed value, but rather determined through the analysis of nodes. The physical rendering properties, such as inner margin, outer margin, and shadow diffusion parameters, are calculated in combination with preset rendering compensation rules.

[0050] Furthermore, the attenuation term in the formula A linear mapping relationship was established. As a pixel moves further away from the bounding box boundary, its corresponding constraint weight decreases linearly from 1. When the pixel is exactly within the tolerance radius... When the value reaches the edge, the decay term returns to zero. Furthermore, the outer wrapping of the formula... The function acts as a lower bound truncation. If the physical distance between a pixel and its bounding box exceeds the dynamically determined tolerance radius, the attenuation term will result in a negative value. The function forces the prior constraint values ​​to be Set to zero. The truncation operation ensures the non-negativity of the constraint value matrix, prevents the generation of negative feedback probability in invalid regions, and thus guarantees the stability of the subsequent tensor calculation process.

[0051] By combining distance scaling with lower bound truncation, prior constraint values ​​that conform to actual rendering tolerances are generated. This calculation method preserves the absolute constraint weights of the core region within the bounding box while establishing a smooth gradient distribution within the tolerance range and effectively shielding irrelevant pixel interference outside the tolerance region. This allows the generated prior tensor to accurately simulate the physical rendering boundary extension characteristics of UI components, providing structured local spatial constraint inputs for the deep segmentation network.

[0052] In one embodiment of the present invention, step S130, which modulates the feature response and outputs an actual semantic probability distribution map that incorporates structural priors during the forward inference stage of the deep semantic segmentation network, includes: 1) Generate spatial attention feature maps using the aforementioned elastic semantic constraint prior tensor. .

[0053] In this process, the number of channels in the elastic semantic constraint prior tensor is kept consistent with the number of feature channels in the layer to be modulated in the deep semantic segmentation network, or mapped to be consistent through a convolutional layer. The spatial attention feature map... The value of each pixel position represents the legality weight of that position in physical space, and the value distribution is in the interval [0, 1].

[0054] 2) The spatial attention feature map Injected into the feature extraction layer of the deep semantic segmentation network, visual features are processed. Pixel-by-pixel affine transformation modulation is performed to obtain the fused features. :

[0055] in, This represents the Hadamard product, which is the element-wise multiplication of corresponding features. and The modulation parameters that can be learned in the deep semantic segmentation network, the and All data dimensions are [size missing]. Channel-level scaling and offset feature vectors (where Input visual features The number of channels), so that the feature response of the deep semantic segmentation network is subject to the elastic semantic constraint prior tensor.

[0056] Specifically, when pixel coordinates When the prior constraint value is high, the scaling factor in the formula... The prior constraint value is increased, thereby enhancing the response strength of the visual feature at that location; if the prior constraint value approaches zero, the visual feature response is suppressed. This process ensures that the semantic probability distribution of the final output has higher confidence within the physically legal region, while the feature response of the illegal overflow region is forcibly weakened by the physical constraint.

[0057] In practical operation, the deep semantic segmentation network first performs preliminary visual semantic parsing on the screenshot image to generate an initial feature map. Then, it retrieves the elastic semantic constraint prior tensor that perfectly corresponds to the current spatial location of the screenshot and performs size alignment at a specific downsampling level of the feature map. By executing the aforementioned affine transformation instructions, the physical layout constraints of the lower-level view tree are injected into the higher-level semantic feature representation. Through this signal transmission path, the prediction behavior of the deep learning model no longer depends solely on the texture information of image pixels, but is subject to the logical intervention of the original design intent of the underlying software, thus forming a constraint mechanism on the model's output space.

[0058] This embodiment introduces pixel-wise affine transformation modulation based on prior tensors during the forward inference stage, enabling the feature maps generated by the deep semantic segmentation network to possess spatial awareness characteristics. This modulation mechanism directly correlates the visual feature response intensity with the underlying physical distribution probability, effectively suppressing aberrant activations outside the rendering tolerance boundary, thereby reducing semantic ambiguity in semantic segmentation tasks and improving the consistency between the output results and the underlying view tree logical structure.

[0059] Reference Figure 3 As shown, in one embodiment of the present invention, the calculation of the structural default score by combining semantic missing amount and overflow amount includes: S310. Identify the clickable, focusable, and swipeable attributes of each UI node from the interaction logic attributes.

[0060] The aforementioned identification process involves traversing the underlying view tree and extracting its tag attributes. For example, the event listener identifiers or state boolean values ​​bound to each UI node can be extracted by parsing the node declaration file or rendering tree node data encapsulated in the underlying view tree. For nodes marked as clickable, their physical entities typically correspond to buttons or hyperlink controls; focusable nodes often point to input boxes or state toggles; and scrollable nodes are associated with lists or scroll views.

[0061] S320. When a UI node with interactive attributes experiences semantic loss within the legal region defined by the elastic semantic constraint prior tensor, a first weighting coefficient is introduced when calculating the structural default score.

[0062] Semantic missing occurs when, within the legally defined pixel coordinate region of the semantic category, as specified by the elastic semantic constraint prior tensor, the actual semantic probability distribution output by the deep semantic segmentation network is lower than a set existence threshold, meaning the interactive control fails to render correctly at the visual level. Since UI nodes with interactive attributes carry the core business logic of the application, their semantic missing will directly hinder user operation. Therefore, when calculating the number of missing pixels or the integral of the probability deviation in this region, the error term is multiplied by a large first weighting coefficient. For example, the introduction of the first weighting coefficient can be achieved by applying a multiplier amplification or exponential enhancement to the basic penalty term, thereby giving it a dominant proportion in the total structural default score. This operator-level logical allocation ensures that rendering failures involving the interactive link can quickly boost the total score.

[0063] S330. When a UI node without interactive attributes exceeds the tolerance boundary of the elastic semantic constraint prior tensor, a second weighting coefficient is introduced when calculating the structural default score; wherein, the first weighting coefficient is greater than the second weighting coefficient.

[0064] Specifically, UI nodes without interactive attributes often refer to background color blocks, decorative borders, or static separator lines. Exceeding the tolerance boundary means that the actual rendering area of ​​this type of node overflows the legal physical extension and anti-aliasing tolerance range defined by the elastic semantic constraint prior tensor, i.e., semantic overflow occurs. For rendering overflow of such non-functional nodes, it is quantified as a visual defect rather than a functional blockage. Therefore, when calculating the error cost of its overflow pixels, a relatively small second weighting coefficient is used for weighted summation. Furthermore, the first weighting coefficient is greater than the second weighting coefficient, clarifying the difference in weight hierarchy among different UI nodes in the computational network. Specifically, the structural default score... The calculation formula is as follows:

[0065] in, Represents pixel coordinates The semantic missing value at a given point is obtained by processing the difference between the prior tensor constraint value and the actual predicted probability using an activation function. Represents pixel coordinates Semantic overflow at the location; This is the first weighting factor introduced for UI nodes with interactive attributes. This is a second weighting coefficient introduced for UI nodes that do not have interactive attributes, and satisfies... (For example, The possible value is 0.85. The value can be 0.15. Meanwhile, the preset collapse threshold serves as the benchmark for judging structural rendering anomalies, and its preferred value range is [0.75, 0.90]. In this embodiment, the exemplary value is 0.80.

[0066] The aforementioned asymmetric penalty mechanism directly links the underlying business interaction intent with the quantification process of visual rendering deviations. This changes the logic of traditional pixel-level difference comparisons, which equally weights all rendering errors. It allows anomaly detection to have appropriate tolerance for non-fatal visual deviations such as background overflow, while maintaining high sensitivity to structural defects that disrupt the interaction path, such as missing buttons or occluded input boxes. This effectively suppresses false alarms in static visuals while ensuring the detection effectiveness of core interactive functions.

[0067] In one embodiment of the present invention, the step of scheduling a local region to perform a preset number of random deactivation inferences to detect local uncertainties includes: 1) Determine whether there are any locally connected components in the actual semantic probability distribution graph where the confidence level of a single inference is lower than the threshold value.

[0068] This judgment process follows the single forward output stage of the deep semantic segmentation network and is used to delineate the physical regions in two-dimensional space suspected of experiencing rendering degradation. Specifically, after obtaining the actual semantic probability distribution map, each pixel node in the distribution map is traversed, and the maximum activation probability value representing each pixel belonging to the target semantic category is extracted. The extracted maximum activation probability value is compared pixel-by-pixel with a preset confidence threshold. Subsequently, for discrete pixels in the comparison result that are less than the threshold value, a morphological region growing algorithm or a connected component labeling algorithm is used to perform spatial aggregation. If the area span of the spatially closed region formed after aggregation exceeds the preset minimum pixel cluster size, it is defined as a local connected region with a single inference confidence value lower than the threshold value.

[0069] 2) If it exists, then trigger Monte Carlo random deactivation inference for that local connected domain.

[0070] When the aforementioned locally connected regions are identified, the scheduling logic generates a local interruption signal and initiates a resampling computation path. Specifically, only for the input feature tensor region corresponding to the marked locally connected region, the deep semantic segmentation network is restarted to perform multiple independent forward propagations. During these multiple forward propagations, the randomly deactivated layers within the network remain active. By introducing a preset deactivation rate parameter, the feature transmission paths of some neurons are randomly cut off during each forward derivation. This operation allows the same input features to generate sets of predicted probability tensors with numerical differences in consecutive local inferences, thereby reproducing the dynamic fluctuations of the prediction distribution within the model. This step strictly limits the recalculation range to the physical coordinate system of the locally connected region, blocking the matrix multiplication recalculation path across the entire global graph.

[0071] 3) Calculate pixel-level information entropy based on the output results of multiple forward propagations, wherein the pixel-level information entropy is calculated based on the probability distribution of the multiple forward propagation results; if the pixel-level information entropy is greater than a preset entropy threshold, it is determined that the local connected component has material rendering distortion.

[0072] Specifically, after obtaining the probability tensor set of the multiple forward propagation outputs, the predicted probabilities of each pixel in the local connected region for each semantic category are averaged along the time step or sampling batch dimension to generate a discrete probability distribution vector for that pixel. Subsequently, the pixel-level information entropy of this probability distribution vector is output based on the entropy calculation model in information theory. This pixel-level information entropy, as a data distribution metric, characterizes the dispersion of the model's semantic attribution determination for that pixel. When pixel features are aliased in a local area due to rendering degradation dynamics mechanisms such as component anti-aliasing failure, incomplete texture buffer loading, or jagged font edges, the prediction output under Monte Carlo sampling will exhibit high dispersion, manifested as a rise in the pixel-level information entropy value. Finally, the calculated pixel-level information entropy is compared with the preset entropy threshold. If the value exceeds the upper limit of the threshold, a detection signal is output to determine that the relevant pixel cluster has experienced material rendering distortion.

[0073] This embodiment employs a local uncertainty detection mechanism. By combining morphological connected component aggregation operations with confidence thresholds, the physical spatial boundary of the secondary verification is defined, and the range of Monte Carlo sampling is strictly compressed to this local region. This effectively prevents system computational overload caused by global repetitive inference. Simultaneously, by utilizing the uncertainty quantification distribution characteristics generated by random deactivation parameters, microscopic rendering distortion processes such as texture defects and blurred edges, which are difficult to locate directly in conventional single-pass visual inference, are made explicit, improving the detection resolution of deep visual defect features.

[0074] In some embodiments of the present invention, the method further includes: The deep semantic segmentation network is trained based on a rendering violation penalty loss function to optimize its performance in UI rendering anomaly detection tasks. Defined as:

[0075] in, For the indicator function generated based on the interaction logic attribute, when the coordinates The value is 1 if the node has an interactive attribute, and 0 otherwise. Characterizes the degree of missing semantic features. The degree of semantic feature overflow is represented, and the penalty coefficient satisfies the following conditions: .

[0076] For example, the step of training the deep semantic segmentation network based on the rendering default penalty loss function can be configured during the model parameter update phase. Specifically, the indicator function The mask is instantiated as a two-dimensional spatial mask matrix physically aligned with the resolution of the input image. By parsing the underlying view tree, the bounding box coordinates of interactive response nodes such as clicks and state transitions are extracted. The elements at the corresponding positions in the mask matrix are hard-set to 1, while the remaining static background or decorative areas are set to 0. In the loss calculation stage, the mask is aligned pixel-by-pixel with the prediction tensor output by the network.

[0077] Furthermore, the aforementioned and This is used to continuously quantify the one-way numerical deviation between the network's predicted output and the prior baseline. Among them, This represents the amount of missing data that physically exists but whose predicted probability by the network is below a threshold. This represents the overflow amount where the network predicts a probability higher than a threshold but the actual value does not exist. Specifically, the L1 distance norm can be used directly for calculation, or a one-sided truncation form of cross-entropy or a smoothed L1 loss function can be introduced to reduce the interference of extreme outlier pixels on the overall gradient update direction.

[0078] During the backpropagation of network errors, due to the constraint that the penalty coefficient must satisfy... This loss function constructs an asymmetric local gradient-weighted transport path. When an interactive node with a matrix element value of 1 experiences semantic loss, its error term is reduced by the coefficient. Multiplicative doubling generates a massive backpropagation gradient, forcing network weights to iterate preferentially in the direction of repairing the feature representation of that region. For semantic overflow occurring at non-interactive nodes with matrix element values ​​of 0, error propagation is affected by a smaller coefficient. Due to the attenuation limitation, the generated gradient response is relatively flat.

[0079] This embodiment transforms the underlying business interaction intent into mathematical constraints that directly intervene in the gradient descent of network parameters, thereby altering the convergence logic of conventional segmentation networks that calculate the prediction errors of all pixels by equal averaging. This configuration can guide the network to specifically reduce overfitting to static, non-interactive visual defects during the training process and improve the detection sensitivity for high-risk rendering faults such as missing core interactive controls.

[0080] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0081] Reference Figure 4 This illustration shows a UI anomaly detection device based on prior guided semantic segmentation according to an embodiment of this application, specifically including the following modules: The context acquisition module 401 is used to acquire the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data.

[0082] The constraint tensor generation module 402 is used to dynamically construct an elastic semantic constraint prior tensor that is aligned with the space of the original screenshot image based on the physical rendering attributes. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate.

[0083] The cross-modal guided inference module 403 is used to inject the elastic semantic constraint prior tensor as a spatial prior feature into the deep semantic segmentation network, and to modulate the feature response in the forward inference stage of the deep semantic segmentation network, outputting an actual semantic probability distribution map that integrates structural priors.

[0084] The asymmetric default detection module 404 is used to calculate the structural default score by using the elastic semantic constraint prior tensor constraint prediction results during the anomaly detection stage, judging the degree of matching of the actual semantic probability distribution map within the tolerance range defined by the elastic semantic constraint prior tensor, and combining the semantic missing amount and overflow amount.

[0085] The computing power gating scheduling module 405 is used to perform computing power gating scheduling based on structural state collapse prediction. If the structural default score exceeds a preset collapse threshold, a structural rendering anomaly is determined to have occurred, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

[0086] In one embodiment of the present invention, the constraint tensor generation module includes: The attribute extraction module is used to extract the inner margin, outer margin, Z-axis height, and clipping rules from the physical rendering attributes.

[0087] The tolerance radius determination module is used to expand the boundary of text type nodes based on font rendering characteristics to determine the tolerance radius.

[0088] The diffusion tolerance generation module is used to generate a spatial diffusion tolerance region based on the depth attribute for nodes with Z-axis height attributes.

[0089] The constraint value calculation module is used to calculate the normalized constraint value of each pixel relative to the bounding box of the underlying view tree node based on the spatial distance mapping function, so as to determine the prior constraint value at each pixel.

[0090] In one embodiment of the present invention, the constraint value calculation unit calculates the normalized constraint value of each pixel relative to the bounding box of the underlying view tree node based on the spatial distance mapping function, and obtains the node... In pixel coordinates Prior constraint values ​​at the location The formula is:

[0091] in, For pixels to the DOM's original bounding box Physical spatial distance, The tolerance radius is dynamically determined based on the physical rendering attributes.

[0092] In one embodiment of the present invention, the cross-modal guided inference module includes: The attention feature map generation module is used to generate spatial attention feature maps using the elastic semantic constraint prior tensor. .

[0093] Affine transform modulation module, used to convert the spatial attention feature map Injected into the feature extraction layer of the deep semantic segmentation network, visual features are processed. Pixel-by-pixel affine transformation modulation is performed to obtain the fused features. :

[0094] in, Represents the Hadama product. and These are the learnable modulation parameters in the deep semantic segmentation network, so that the feature response of the deep semantic segmentation network is constrained by the elastic semantic constraint prior tensor.

[0095] In one embodiment of the present invention, the asymmetric default detection module includes: The interaction attribute recognition module is used to identify the clickable, focusable, and swipeable attributes of each UI node from the interaction logic attributes.

[0096] The first weighted penalty module is used to introduce a first weighting coefficient when calculating the structural breach score when a UI node with interactive attributes has semantic missingness within the legal region defined by the elastic semantic constraint prior tensor.

[0097] The second weighted penalty module is used to introduce a second weighting coefficient when calculating the structural default score when a UI node without interactive attributes exceeds the tolerance boundary of the elastic semantic constraint prior tensor; wherein the first weighting coefficient is greater than the second weighting coefficient.

[0098] In one embodiment of the present invention, the computing power gating scheduling module includes: The confidence judgment module is used to determine whether there are local connected components in the actual semantic probability distribution graph whose confidence in a single inference is lower than the threshold value.

[0099] The deactivation inference triggering module is used to trigger Monte Carlo random deactivation inference for the local connected domain if the local connected domain exists.

[0100] The distortion determination module is used to calculate pixel-level information entropy based on the output results of multiple forward propagations, wherein the pixel-level information entropy is calculated based on the probability distribution of the multiple forward propagation results; if the pixel-level information entropy is greater than a preset entropy threshold, it is determined that the local connected component has material rendering distortion.

[0101] According to one embodiment of the present invention, the device further includes: The network training module is used to train the deep semantic segmentation network based on the rendering violation penalty loss function to optimize its performance in UI rendering anomaly detection tasks. The rendering violation penalty loss function... Defined as:

[0102] in, For the indicator function generated based on the interaction logic attribute, when the coordinates The value is 1 if the node has an interactive attribute, and 0 otherwise. Characterizes the degree of missing semantic features. The degree of semantic feature overflow is represented, and the penalty coefficient satisfies the following conditions: .

[0103] Reference Figure 5The illustration shows a computer electronic device for implementing a UI anomaly detection method based on prior guided semantic segmentation according to the present invention, which may specifically include the following: The aforementioned computer electronic device 1 is manifested in the form of a general-purpose computing device. The components of the computer electronic device 1 may include, but are not limited to: one or more processors or processing units 3, memory 8, and a bus 4 connecting different system components (including memory 8 and processing unit 3).

[0104] Bus 4 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Audio / Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0105] Computer electronic device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer electronic device 1, including volatile and non-volatile media, removable and non-removable media.

[0106] Memory 8 may include computer system readable media in the form of volatile memory, such as random access memory 9 and / or cache memory 10. Computer electronic device 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 11 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 4 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 13 configured to perform the functions of the embodiments of this application.

[0107] A program / utility 12 having a set (at least one) of program modules 13 may be stored, for example, in memory. Such program modules 13 include—but are not limited to—an operating system, one or more application programs, other program modules 13, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 13 typically perform the functions and / or methods described in the embodiments of this application.

[0108] The computer electronic device 1 can also communicate with one or more external devices 2 (e.g., keyboard, pointing device, display 7, camera, etc.), and with one or more devices that enable an operator to interact with the computer electronic device 1, and / or with any device that enables the computer electronic device 1 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through the I / O interface 6. Furthermore, the computer electronic device 1 can also communicate with one or more networks (e.g., local area network (LAN)), wide area network (WAN), and / or public networks (e.g., the Internet) through the network adapter 5. Figure 5 As shown, network adapter 5 communicates with other modules of computer electronic device 1 via bus 4. It should be understood that, although... Figure 5 Not shown, it may be combined with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing unit 3, external disk drive array, RAID system, tape drive and data backup storage system 11, etc.

[0109] The processing unit 3 executes various functional applications and data processing by running programs stored in memory 8, such as implementing a UI anomaly detection method based on prior guided semantic segmentation provided in the embodiments of this application.

[0110] In this application embodiment, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a UI anomaly detection method based on prior guided semantic segmentation as provided in all embodiments of this application.

[0111] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0112] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0113] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as a standalone software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the operator's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider). The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0114] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0115] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0116] The above provides a detailed description of the UI anomaly detection method and related apparatus based on prior guided semantic segmentation provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A UI anomaly detection method based on prior-guided semantic segmentation, characterized in that, include: Obtain the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data. Based on the physical rendering attributes, an elastic semantic constraint prior tensor that is spatially aligned with the original screenshot image is dynamically constructed. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate. The elastic semantic constraint prior tensor is injected as a spatial prior feature into the deep semantic segmentation network, and used to modulate the feature response in the forward inference stage of the deep semantic segmentation network, outputting an actual semantic probability distribution map that integrates structural priors. During the anomaly detection phase, the prediction results of the elastic semantic constraint prior tensor are used to determine the degree of matching of the actual semantic probability distribution map within the tolerance range defined by the elastic semantic constraint prior tensor, and the structural default score is calculated by combining the semantic missing amount and the overflow amount. The computational power gating scheduling is performed based on the prediction of structural state collapse. If the structural default score exceeds the preset collapse threshold, a structural rendering anomaly is determined, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

2. The method according to claim 1, characterized in that, The step of dynamically constructing a flexible semantic constraint prior tensor aligned with the spatial dimensions of the original screenshot image based on the physical rendering attributes includes: Extract the inner margin, outer margin, Z-axis height, and clipping rules from the physical rendering attributes; For text-type nodes, the boundary is expanded based on font rendering characteristics to determine the tolerance radius; For nodes with Z-axis height attributes, a spatial diffusion tolerance region is generated based on the depth attribute; The normalized constraint value of each pixel relative to the bounding box of the underlying view tree node is calculated based on the spatial distance mapping function to determine the prior constraint value at each pixel.

3. The method according to claim 1, characterized in that, The normalized constraint value of each pixel relative to the bounding box of the underlying view tree node is calculated based on the spatial distance mapping function to obtain the node. In pixel coordinates Prior constraint values ​​at the location The formula is: in, For pixels to the DOM's original bounding box Physical spatial distance, The tolerance radius is dynamically determined based on the physical rendering attributes.

4. The method according to claim 1, characterized in that, The modulated feature response used in the forward inference stage of the deep semantic segmentation network outputs an actual semantic probability distribution map that incorporates structural priors, including: Spatial attention feature maps are generated using the elastic semantic constraint prior tensor. ; The spatial attention feature map Injected into the feature extraction layer of the deep semantic segmentation network, visual features are processed. Pixel-by-pixel affine transformation modulation is performed to obtain the fused features. : in, Represents the Hadama product. and These are the learnable modulation parameters in the deep semantic segmentation network, so that the feature response of the deep semantic segmentation network is constrained by the elastic semantic constraint prior tensor.

5. The method according to claim 1, characterized in that, The calculation of the structural default score by combining semantic missing amount and overflow amount includes: Identify the clickable, focusable, and swipeable attributes of each UI node from the interaction logic attributes; When a UI node with interactive attributes experiences semantic loss within the legal region defined by the elastic semantic constraint prior tensor, a first weighting coefficient is introduced when calculating the structural breach score. When a UI node without interactive attributes exceeds the tolerance boundary of the elastic semantic constraint prior tensor, a second weighting coefficient is introduced when calculating the structural default score; wherein the first weighting coefficient is greater than the second weighting coefficient.

6. The method according to claim 1, characterized in that, The method of scheduling a local region to perform a preset number of random deactivation inferences to detect local uncertainties includes: Determine whether there are locally connected components in the actual semantic probability distribution graph whose confidence level in a single inference is lower than the threshold value; If it exists, then Monte Carlo random deactivation inference is triggered for that local connected domain; Pixel-level information entropy is calculated based on the output results of multiple forward propagations, wherein the pixel-level information entropy is calculated based on the probability distribution of the multiple forward propagation results; if the pixel-level information entropy is greater than a preset entropy threshold, it is determined that the local connected component has experienced material rendering distortion.

7. The method according to claim 1, characterized in that, The method further includes: The deep semantic segmentation network is trained based on a rendering violation penalty loss function to optimize its performance in UI rendering anomaly detection tasks. Defined as: in, For the indicator function generated based on the interaction logic attribute, when the coordinates The value is 1 if the node has an interactive attribute, and 0 otherwise. Characterizes the degree of missing semantic features. The degree of semantic feature overflow is represented, and the penalty coefficient satisfies the following conditions: .

8. A UI anomaly detection device based on prior-guided semantic segmentation, characterized in that, include: The context acquisition module is used to acquire the original screenshot image of the current interface of the application under test and the corresponding underlying view tree data, and extract the physical rendering attributes and interaction logic attributes of each UI node from the underlying view tree data. The constraint tensor generation module is used to dynamically construct an elastic semantic constraint prior tensor that is aligned with the space of the original screenshot image based on the physical rendering attributes. The elastic semantic constraint prior tensor is a multi-channel constraint data structure used to characterize the allowable distribution range of semantic categories at each pixel coordinate. The cross-modal guided inference module is used to inject the elastic semantic constraint prior tensor as a spatial prior feature into the deep semantic segmentation network, and to modulate the feature response in the forward inference stage of the deep semantic segmentation network, outputting an actual semantic probability distribution map that integrates structural priors. The asymmetric default detection module is used to use the elastic semantic constraint prior tensor constraint prediction results during the anomaly detection stage to determine the degree of matching of the actual semantic probability distribution within the tolerance range defined by the elastic semantic constraint prior tensor, and to calculate the structural default score by combining the semantic missing amount and the overflow amount. The computing power gating scheduling module is used to perform computing power gating scheduling based on structural state collapse prediction. If the structural default score exceeds a preset collapse threshold, a structural rendering anomaly is determined, and multiple forward propagation calculations based on random deactivation are skipped for the local area. If the structural default score does not exceed the collapse threshold, the local area is scheduled to perform a preset number of random deactivation inferences to detect local uncertainties.

9. A computer electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the UI anomaly detection method based on prior guided semantic segmentation as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the UI anomaly detection method based on prior guided semantic segmentation as described in any one of claims 1 to 7.

Images