Interface element positioning method and device based on deep learning, equipment and medium

Abstract

The embodiment of the invention provides an interface element positioning method and device based on deep learning, equipment and a medium, and relates to the technical field of computer vision. The method comprises the following steps: inputting an obtained to-be-detected interface image into a preset segmentation model, and analyzing to obtain types and positioning information of elements in the to-be-detected interface image; wherein the preset semantic segmentation model is a model which is obtained by training a first UNet convolutional neural network model and a second UNet convolutional neural network model and is used for performing feature extraction on the interface image to determine categories and position, the first UNet convolutional neural network model is used for performing global feature extraction on the interface image, and the second UNet convolutional neural network model is used for performing global feature extraction on the interface image. The second UNet convolutional neural network model is used for performing local feature extraction on the interface image. By means of the method, more accurate element positioning can be achieved in modern automatic testing of high-interaction and high-custom user interfaces, and the automation degree and reliability of interface analysis are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for locating interface elements based on deep learning. Background Art

[0002] With the rapid development of information technology, the scale and complexity of software systems are increasing day by day. Interface automation testing has become a key link in ensuring software quality. As the foundation and core of interface automation testing, element positioning is of self-evident importance.

[0003] Traditional interface element positioning methods, such as element positioning based on the Document Object Model (DOM), automated script parsing positioning expressions (XPath expressions, CSS selectors, Tag tag names), etc., are widely used, but often face challenges with dynamic content and complex layouts. Especially in modern highly interactive and highly customized user interface design, DOM-based element positioning methods and positioning expressions cannot effectively locate elements.

[0004] Therefore, providing a technical solution that can accurately achieve element positioning in the automated testing of modern highly interactive and highly customized user interfaces is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and medium for locating interface elements based on deep learning, so as to achieve the effect of accurately locating elements in the automated testing of modern highly interactive and highly customized user interfaces.

[0006] In a first aspect, an embodiment of the present application provides a method for locating interface elements based on deep learning, comprising:

[0007] Obtaining the interface image to be detected;

[0008] Inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected;

[0009] Among them, the semantic segmentation model is a model for extracting features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of the interface image, and the second UNet convolutional neural network model is used to extract local features of the interface image.

[0010] In a possible implementation, the inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected includes:

[0011] Performing normalization processing on the interface image to be detected to obtain a processed interface image to be detected;

[0012] Inputting the processed interface image to be detected into the semantic segmentation model, performing encoding and decoding processing on the processed interface image to be detected through the first UNet convolutional neural network model and the second UNet convolutional neural network model to obtain a first feature map and a second feature map of the processed interface image to be detected;

[0013] According to the first feature map and the second feature map, the category and positioning information of the elements in the interface image to be detected are analyzed.

[0014] In a possible implementation, the first UNet convolutional neural network model includes a global structure branch model and a hole convolution model;

[0015] The processed interface image to be detected is subjected to encoding and decoding processing by a first UNet convolutional neural network model, including:

[0016] Downsampling the processed interface image to be detected by using the global structure branch model to obtain a first coding feature map of the processed interface image to be detected;

[0017] Performing upsampling processing on the first encoding feature map through the dilated convolution model to obtain a first decoding feature map of the interface image to be detected after the processing;

[0018] The first encoding feature map and the first decoding feature map are cascaded to obtain the first feature map.

[0019] In a possible implementation, the second UNet convolutional neural network model includes a local refinement branch model and the dilated convolution model;

[0020] The processed interface image to be detected is subjected to encoding and decoding processing by a second UNet convolutional neural network model, including:

[0021] Downsampling the processed interface image to be detected by using the local refined branch model to obtain a second encoding feature map of the processed interface image to be detected;

[0022] Performing upsampling processing on the second encoding feature map through the dilated convolution model to obtain a second decoding feature map of the interface image to be detected after the processing;

[0023] Cascading the second encoding feature map and the second decoding feature map to obtain a cascade feature map;

[0024] The second feature map is obtained by performing feature extraction on the cascade feature map through height axial attention and width axial attention combined with a position offset term.

[0025] In a possible implementation, before inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected, the method further includes:

[0026] Element marking is performed on the interface image to be detected to obtain a label feature map corresponding to the interface image to be detected.

[0027] In a possible implementation, analyzing the category and location information of the element in the to-be-detected interface image based on the first feature map and the second feature map includes:

[0028] Adding the first feature map and the second feature map to obtain a fused feature map of the interface image to be detected;

[0029] Comparing the fused feature map with the label feature map to obtain the similarity between the fused feature map and the label feature map;

[0030] If the similarity is less than the preset similarity, continue to encode and decode the fused feature map through the first UNet convolutional neural network model and the second UNet convolutional neural network model until the similarity is greater than or equal to the preset similarity, thereby obtaining a feature map of the interface image to be detected;

[0031] According to the feature map, the category and location information of the elements in the interface image to be detected are obtained.

[0032] In a possible implementation, obtaining the category and location information of the element in the to-be-detected interface image according to the feature map includes:

[0033] The feature map is upsampled and convolved through the output layer of the semantic segmentation model, and the category and positioning information of the elements in the interface image to be detected are obtained through classifier prediction.

[0034] In a second aspect, an embodiment of the present application provides an interface element positioning device based on deep learning, comprising:

[0035] A first processing module is used to obtain an image of the interface to be detected;

[0036] A second processing module is used to input the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected;

[0037] Among them, the semantic segmentation model is a model for extracting features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of the interface image, and the second UNet convolutional neural network model is used to extract local features of the interface image.

[0038] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0039] The memory stores computer-executable instructions;

[0040] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0042] The embodiments of the present application provide a method, device, equipment and medium for locating interface elements based on deep learning, which inputs the acquired interface image to be detected into a preset segmentation model to analyze and obtain the category and positioning information of the elements in the interface image to be detected. Among them, the preset semantic segmentation model is a model for extracting features of the interface image to determine the category and positioning obtained by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of the interface image, and the second UNet convolutional neural network model is used to extract local features of the interface image. Through the above method, more efficient and accurate element positioning can be achieved in the automated testing of modern highly interactive and highly customized user interfaces, thereby improving the degree of automation and reliability of interface analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 1 ;

[0045] Figure 2 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 2 ;

[0046] Figure 3 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 3 ;

[0047] Figure 4 A schematic diagram of a specific implementation of a method for locating interface elements based on deep learning provided in this application;

[0048] Figure 5 A schematic diagram of the structure of an interface element positioning device based on deep learning provided in this application;

[0049] Figure 6 A schematic diagram of the structure of a computer device provided in this application.

[0050] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0052] First, let’s explain the terms involved in this application:

[0053] Interface automation testing: Interface automation testing is a method that automatically executes test cases and checks expected results by simulating the user's interaction with the software interface.

[0054] Element location: In web automation testing, element location refers to the process of identifying specific Hypertext Markup Language (HTML) elements on a web page. Elements include any user interface components such as buttons, links, text boxes, etc.

[0055] Computer vision: A branch of artificial intelligence that uses algorithms and deep learning techniques to extract and analyze image information.

[0056] Semantic segmentation: A computer vision technique that aims to classify each pixel in an image to identify and distinguish different objects and scenes.

[0057] Self-attention mechanism: A deep learning technique that allows the model to consider the relationship between each element in the sequence and all other elements when processing a sequence, thereby better understanding the contextual information in the sequence.

[0058] The application background of this application is then explained as follows:

[0059] With the rapid development of information technology and the widespread popularity of mobile devices, smart wearable devices, and IoT devices, the scale and complexity of software systems are increasing day by day. Interface automation testing has become a key link in ensuring software quality. Element positioning, as the foundation and core of interface automation testing, is of self-evident importance.

[0060] Traditional interface element positioning methods, such as element positioning based on the Document Object Model (DOM), automated script parsing positioning expressions, and interface element marking methods based on artificial intelligence (AI) to generate virtual code mapping, can perform related testing tasks of interface element text through artificial intelligence recognition of generated virtual code.

[0061] However, traditional interface element positioning methods often face challenges with dynamic content and complex layouts. Especially in modern highly interactive and highly customized user interface designs, DOM-based element positioning methods are not flexible enough or cannot effectively locate elements, and methods based on AI-generated virtual code mappings cannot be learned online. In terms of performance, when the element hierarchy is complex or there are multiple similar elements, the DOM-based element positioning method needs to traverse the entire DOM tree for element matching, which has low loading efficiency and consumes more system resources. In terms of positioning reliability, if some elements on the page are not loaded at the start of the test, the traditional interface element positioning method cannot correctly handle such asynchronously loaded elements, which may lead to positioning failure; if the modification is hard-coded in the script, if the positioning method needs to be modified, the automated tester needs to perform manual maintenance, which has a high maintenance cost.

[0062] Therefore, it is an urgent technical problem to provide a technical solution that can accurately realize element positioning in the automated testing of modern highly interactive and highly customized user interfaces, fill the gaps in the current element positioning under dynamic and complex layouts, meet the growing needs of software system interaction, improve the automation and reliability of interface analysis, and thus improve user experience and system operation efficiency.

[0063] Based on the above technical problems, the inventors discovered in the process of studying the interface element positioning method that the image information extraction technology is not restricted by the page DOM structure and can directly analyze the interface image data, thereby improving the efficiency and accuracy of interface element positioning by improving the accuracy of image information extraction. Specifically, by extracting feature information from the interface image with the help of semantic segmentation, locating elements with the help of feature information, and repeatedly iterating by comparing the feature map and label map of the semantic segmentation model, the accuracy of positioning of various complex custom elements in the interface is improved. Based on this, the present application provides an interface element positioning method, device, equipment and medium based on deep learning.

[0064] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0066] S101: Acquire an interface image to be detected.

[0067] Acquiring an image of the interface to be tested is the first step in deep learning-based interface element localization methods. This step provides input data for subsequent deep learning analysis. This typically involves capturing the current user interface screen from the target application or system. This can be accomplished by invoking a tool, automated testing framework, or dedicated image capture software. By capturing the interface image, visual information about the current interface can be obtained, enabling precise localization of user interface elements.

[0068] S102: Inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected.

[0069] Among them, the semantic segmentation model is a model that extracts features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of interface images, and the second UNet convolutional neural network model is used to extract local features of interface images.

[0070] In this step, the category and positioning information of the elements in the interface image are the basis for ensuring that the interface automation test can be executed correctly. Its categories include buttons, text boxes, drop-down menus, etc., and the positioning information is usually expressed in the form of coordinates. By identifying the category and positioning information of the elements, the automated testing tool can understand the expected behavior and purpose of each element, so as to formulate an appropriate testing strategy. For example, after identifying an element as a button, the testing tool can automatically perform click operations and verify the corresponding responses through its positioning coordinates. Combined with the category and positioning information of the elements, automated testing can cover the test scenarios more comprehensively, improve the accuracy and efficiency of the test, and provide valuable feedback for interface design and improvement.

[0071] The UNet convolutional neural network model is used for image segmentation and has a U-shaped topology, including an encoder and a decoder. The encoder is used to gradually extract the feature map of the input interface image to be detected and reduce the spatial resolution. The decoder is used to restore the feature map to the size of the original input interface image to be detected through upsampling operations, and gradually generate segmentation results. The preset semantic segmentation model includes two parallel UNet convolutional neural network models, a first UNet convolutional neural network model and a second UNet convolutional neural network model. The encoder of the first UNet convolutional neural network model is used to perform global feature extraction on the interface image to be detected, and the encoder of the second UNet convolutional neural network model is used to perform local feature extraction on the interface image to be detected.

[0072] Specifically, the first UNet convolutional neural network model and the second UNet convolutional neural network model fuse the feature information of multi-scale interface images by encoding first and then decoding, retaining both high-resolution and low-resolution features at different layers, which can greatly improve the accuracy of semantic segmentation and obtain more accurate interface element information.

[0073] The embodiment of the present application provides a method for locating interface elements based on deep learning, which obtains the category and positioning information of the elements in the interface image to be detected by inputting the acquired interface image to a preset semantic segmentation model for analysis. Among them, the preset semantic segmentation model is a model for extracting features of interface images to determine categories and positioning obtained by training two parallel UNet convolutional neural network models. The first UNet convolutional neural network model is used for global feature extraction of interface images, and the second UNet convolutional neural network model is used for local feature extraction of interface images. Through the above method, more efficient and accurate element positioning is achieved in the automated testing of modern highly interactive and highly customized user interfaces, thereby improving the degree of automation and reliability of interface analysis.

[0074] Figure 2 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 2 ,like Figure 2 As shown, this embodiment Figure 1 Based on the embodiment, in S102, the interface image to be detected is input into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected, specifically including:

[0075] S201: performing normalization processing on the interface image to be detected to obtain a processed interface image to be detected.

[0076] In this step, normalization aims to shrink the different features of the interface image data to a common scale, thereby improving stability in subsequent analysis or machine learning models. Normalization adjusts the range of image pixel values ​​to a standard range, such as [0, 1] or [-1, 1]. This reduces the impact of brightness and contrast differences between different regions of the interface image on semantic segmentation model training.

[0077] Specifically, for each 8-bit interface image, it is composed of pixels between [0, 255]. By subtracting the minimum pixel from the current pixel and dividing it by the maximum pixel minus the minimum pixel, the pixel is scaled from [0, 255] to [0, 1] to obtain the processed interface image to be detected. In some possible cases, the pixels of the interface image can be further centered by subtracting the mean pixel from the current pixel and dividing it by the standard deviation pixel, so that the data has zero mean and unit variance. Normalizing the interface image to be detected helps to reduce the dimensional differences between features, improve the convergence speed of the semantic segmentation model, and improve its robustness to different input interface images to be detected.

[0078] S202: Input the processed interface image to be detected into the semantic segmentation model, and perform encoding and decoding processing on the processed interface image to be detected through the first UNet convolutional neural network model and the second UNet convolutional neural network model to obtain the first feature map and the second feature map of the processed interface image to be detected.

[0079] like Figure 1 As described in the embodiment, the preset semantic segmentation model includes two parallel UNet convolutional neural network models, namely a first UNet convolutional neural network model and a second UNet convolutional neural network model. The encoder of the first UNet convolutional neural network model is used to perform global feature extraction on the interface image to be detected, and the encoder of the second UNet convolutional neural network model is used to perform local feature extraction on the interface image to be detected.

[0080] In one possible implementation, the first UNet convolutional neural network model includes a global structure branch model and a hole convolution model.

[0081] In the encoder of the first UNet convolutional neural network model, the global structure branch model is used to extract global features of the interface image, and in the decoder of the first UNet convolutional neural network model, the hole convolution model is used to integrate feature information from the encoder during decoding.

[0082] The processed interface image to be detected is encoded and decoded by the first UNet convolutional neural network model, including:

[0083] S20211: Down-sampling the processed interface image to be detected through the global structure branch model to obtain a first encoding feature map of the processed interface image to be detected.

[0084] In this step, the encoder of the global structure branch model used to encode the interface image to be detected contains multiple downsampling modules, which are composed of a series of convolutional layers and pooling layers, and are used to gradually reduce the size and number of channels of the encoded feature map, and gradually extract more advanced feature information.

[0085] Specifically, the global structure branch model downsamples the processed interface image to be detected twice through convolution operations. Each downsampling operation reduces the spatial size of the interface image. For example, if the spatial size of the processed interface image to be detected is 224×224, after two downsampling operations, two feature maps of different sizes, 112×112 and 56×56, are obtained, namely the first encoded feature map. This downsampling process reduces the spatial resolution of the interface image, reducing the computational effort while capturing the global information of the image.

[0086] S20212: Upsampling the first encoding feature map through a dilated convolution model to obtain a processed first decoding feature map of the interface image to be detected.

[0087] In this step, the decoder of the dilated convolutional model used to decode the first encoded feature map includes multiple upsampling modules, consisting of a series of deconvolutions or transposed convolutions, which gradually increase the size and number of channels of the feature map, gradually restoring the image resolution and retaining more detailed information. In addition, each layer of the decoder is typically skipped to the corresponding layer of the encoder to fuse low-resolution and high-resolution features and retain more detailed information.

[0088] Specifically, the dilated convolution model upsamples the first encoded feature map twice through a deconvolution operation. Each time upsampling is performed, the spatial size of the interface image will gradually recover to the original interface image size. For example, the spatial size of the first encoded feature map described in S20211 is 56×56. After two upsamplings, two feature maps of different sizes of 112×112 and 224×224 are obtained, namely the first decoding feature map. After the upsampling process, the low-resolution feature map is restored to the same spatial resolution as the original image size, which not only helps to improve the visual quality of the output image, but also ensures that the output result is spatially aligned with the input image, thereby achieving precise operations at the pixel level.

[0089] S20213: Cascading the first encoding feature map and the first decoding feature map to obtain a first feature map.

[0090] In this step, as described in S20212, each layer of the decoder is usually jump-connected to the corresponding layer of the encoder, so that the first encoding feature map and the first decoding feature map are cascaded. For example, the first decoding map of size 112×112 in S20212 is cascaded with the first encoding feature map of size 112×112 in S20211, and the obtained cascade feature map is input into the linear layer with a softmax activation function to obtain the processed first feature map of the interface image to be detected, where the softmax activation function can be formally expressed as:

[0091]

[0092] Among them, bias means that each element category output has a learnable bias, W i T Represents the linear transformation matrix, W n Indicates the width of the nth feature map, H n Indicates the height of the nth feature map, C n represents the channel of the nth feature map, B i represents the nth category, Fi The feature information of the first feature map of the interface image to be detected after processing.

[0093] In another possible implementation, the second UNet convolutional neural network model includes a local refinement branch model and a dilated convolution model.

[0094] In the encoder of the second UNet convolutional neural network model, the local refinement branch model is used to extract local features of the interface image, and in the decoder of the second UNet convolutional neural network model, the hole convolution model is used to integrate feature information from the encoder during decoding.

[0095] The processed interface image to be detected is encoded and decoded by the second UNet convolutional neural network model, including:

[0096] S20221: Down-sampling the processed interface image to be detected through the local refinement branch model to obtain a second encoding feature map of the processed interface image to be detected.

[0097] S20222: Upsampling the second encoding feature map through the dilated convolution model to obtain a processed second decoding feature map of the interface image to be detected.

[0098] S20223: Cascade the second encoding feature map and the second decoding feature map to obtain a cascade feature map.

[0099] The implementation principles and technical effects of encoding and decoding the processed interface image to be detected through the second UNet convolutional neural network in S20221-S20223 are similar to the implementation principles and technical effects of encoding and decoding the processed interface image to be detected through the first UNet convolutional neural network in S20211-S20213, and will not be repeated here.

[0100] S20224: Extract features from the cascaded feature map through height axial attention and width axial attention combined with a position offset term to obtain a second feature map.

[0101] In this step, the self-attention layer dynamically adjusts the representation of each element by calculating the correlation (or attention score) between each element and other elements in the input sequence. Height axial attention is a self-attention mechanism layer focused on the height axis of the feature map, which means calculating the attention weight in the height direction of the image, that is, performing feature aggregation and weighting along the vertical axis of the image to capture the vertical correlation in the image. Width attention is a self-attention mechanism layer focused on the width axis of the feature map, which means calculating the attention weight in the width direction of the image, that is, performing feature aggregation and weighting along the horizontal axis of the image to capture the horizontal correlation in the image. By applying the attention mechanism in the height and width directions respectively, the semantic segmentation model can pay more attention to the local features and structural information in the image. At the same time, combined with the position offset term, the sensitivity of the semantic segmentation model to spatial position information is further enhanced, thereby improving the accuracy and robustness of feature extraction.

[0102] First, the feature information of the feature map output by the self-attention layer can be formally expressed as formula (1), which is used to calculate the output feature y at position i ij , where the key vector (Key) and the value vector (Value) are obtained by linearly transforming the input features, and together with the query vector (Query) form a self-attention mechanism:

[0103]

[0104] Among them, y ij Represents the feature information of the feature map output from the self-attention layer, that is, the attention weighted result of the feature map at (i, j), where (i, j) represents a spatial position (such as a pixel or a local area); H represents the height of the input feature map, and W represents the width of the input feature map. k represents the query vector, which is the feature representation of the feature map at (i, j), and is used to calculate the similarity with the key vectors at other positions (T represents transposition, which converts the row vector into a column vector for easy calculation); hw Represents the key vector, which is the feature representation of the feature map at (h, w), and the query vector q ij Perform dot product operations to calculate attention weights, shapes, and q ij Same; v hw Represents a value vector, which is the actual feature content of the feature map at (h, w) and is used for weighted summation.

[0105] By using softmax to normalize the attention scores of all positions and convert them into probability distribution, the calculated global affinity will be v hwAggregation achieves deep insight into the global information of the feature map and the capture of non-local information. Therefore, based on the above-mentioned attention layer, this scheme subdivides the single self-attention layer into high-axis attention focusing on the height axis of the feature map and wide-axis attention focusing on the width axis of the feature map, and at the same time combines the position offset term to increase the position deviation, thereby improving the sensitivity of affinity to position information. Since the accuracy of relative position encoding directly affects the performance of the semantic segmentation model parameters, based on the high-axis attention and wide-axis attention combined with the position offset term, this scheme improves the model's ability to acquire specific areas by optimizing the attention beam focus of the second UNet convolutional model. Specifically, the wide-axis attention combined with the position offset term can be formally expressed as:

[0106]

[0107] Among them, y ij Represents the feature information of the feature map output by the width axis attention combined with the position offset term; W represents the width of the input feature map; L (q,k,m1,m2) is a learnable scalar weight used to adjust the influence of position encoding on attention, where L q is the learnable weight for the query vector q, L k is the learnable weight for the key vector k, L m1 and L m2 are all learnable weights for the value vector, It is a weighted combination of value vector and position encoding, which is used to replace the pure v in formula (1) hw ;z iw is a position encoding vector, which is a fixed sinusoidal code and is used to inject position information. The superscript q represents the encoding operation for the query vector, k represents the encoding operation for the key vector, and v represents the encoding operation for the value vector. iw represents the value vector for the width axis; k iw represents the key vector for the width axis, correspondingly, represents k iw The transpose of The meanings of the parameters are the same as those in formula (1).

[0108] High axial attention combined with position offset term can be formally expressed as:

[0109]

[0110] The meaning of each parameter is the same as that of the formal expression of width axis attention combined with position offset term (Formula (2)), and will not be repeated here. ih represents the value vector for the height axis; k ihrepresents the bond vector for the height axis.

[0111] As described above, the cascade features output by the second UNet convolutional neural network model are extracted through height axial attention and width axial attention respectively, combined with the position bias term, to obtain the second feature map information of the processed interface image to be detected.

[0112] S203: Analyze the category and location information of the element in the interface image to be detected according to the first feature map and the second feature map.

[0113] As described in S202, the first feature map is feature information obtained by performing global feature extraction on the interface image by the first UNet convolutional neural network, and the second feature map is feature information obtained by performing local feature extraction on the interface image by the second UNet convolutional neural network and combining height axial attention, width axial attention, and position offset terms. By combining the first feature map and the second feature map, the semantic segmentation model can simultaneously utilize global and local information to achieve more accurate and comprehensive interface image analysis, thereby more accurately analyzing the category and positioning information of elements in the interface image to be detected.

[0114] The embodiment of the present application provides a method for locating interface elements based on deep learning, which realizes accurate analysis of the interface image to be detected through two parallel UNet convolutional neural network models. First, the interface image to be detected is normalized to standardize the pixel value range and reduce the impact of brightness and contrast differences on model training. Then, the image is input into a preset semantic segmentation model, wherein the first UNet model is responsible for global feature extraction, and uses the global structure branch and the void convolution model for encoding and decoding to obtain the first feature map, capturing the overall structure and macro pattern of the image. The second UNet model focuses on local feature extraction, and obtains the second feature map through the local refinement branch and the void convolution model, combined with the height axial attention and the width axial attention and the position offset term, to enhance the ability to capture details and local features. By combining the first feature map and the second feature map, the semantic segmentation model can simultaneously utilize global and local information to more accurately analyze the category and positioning information of the elements in the interface image to be detected. Through the above method, the accuracy and robustness of the semantic segmentation model are improved, so that the model can better identify and locate each element when processing complex images.

[0115] Figure 3 A schematic diagram of the process of an interface element positioning method based on deep learning provided in this application Figure 3 , Figure 4 This is a schematic diagram of a specific implementation of a method for locating interface elements based on deep learning provided by this application, such as Figure 3 As shown, this embodiment Figure 2Based on the embodiment, in S203, the category and location information of the element in the interface image to be detected are analyzed according to the first feature map and the second feature map, which specifically includes:

[0116] S301: Marking elements of the interface image to be detected to obtain a label feature map corresponding to the interface image to be detected.

[0117] In this step, the label feature map is an image with the same spatial resolution as the original image, where the value of each pixel represents the category of the element at that location. The label feature map is primarily used to provide supervision information for semantic segmentation model training, helping the model learn the mapping from input image to desired output and allowing the model's output map to better fit the label feature map. Common annotation methods include bounding box annotation, pixel-level annotation, and instance segmentation annotation.

[0118] For example, the label feature diagram of the login interface image of the client logging into the QQ mailbox is as follows: Figure 4 As shown in the figure, the various functional components in the interface image are accurately distinguished through bounding box annotation. For example, the slogan "QQ Mailbox, Keep in Touch!" and its related text introduction and symbols at the top of the left area are marked as text areas. The "WeChat Login" and "QQ Login" buttons and the account and password input boxes at the top of the right area are marked as interactive buttons and text input boxes respectively. The "Retrieve Password", "Register Account" and "Feedback" in the middle are marked as hypertext link areas. The "Permission-related Options" and "Auto Login" at the bottom are also separately classified and labeled. Through refined annotation, this label feature map provides supervision signals for the semantic segmentation model, enabling it to accurately learn the semantic information of each element of the interface image, ultimately improving the ability to locate and understand the functions of components.

[0119] S302: Add the first feature map and the second feature map to obtain a fused feature map of the interface image to be detected.

[0120] In this step, the first feature map and the second feature map are added together, that is, the pixel positions and channels of each corresponding element of the two feature maps are added together. Each element in the resulting fused feature map of the interface image to be detected contains the global feature information from the first feature map and the local feature information from the second feature map. By integrating the feature information of different feature maps, the feature expression capability of the semantic segmentation model is enhanced. The principal element addition operation not only has the effect of smoothing the feature map, which helps reduce image noise and improve the robustness of the model, but also does not increase the computational complexity due to its relatively simple operation.

[0121] S303: Compare the fused feature map with the label feature map to obtain the similarity between the fused feature map and the label feature map.

[0122] In this step, the similarity between the fused feature map and the label feature map is used to quantify the degree of match between the output of the semantic segmentation model and the true label feature map, thereby evaluating the effectiveness of the semantic segmentation model. Similarity metrics commonly used include Intersection over Union (IOU), Dice coefficient, precision, F1-score, etc.

[0123] The calculation formula of Dice coefficient is:

[0124]

[0125] Where A represents the feature information of the fused feature map, B represents the feature information of the label feature map, |A∩B| represents the number of pixels in the intersection of the fused feature map and the label feature map, |A| represents the total number of feature information in the fused feature map, and |B| represents the total number of feature information in the label feature map. The Dice coefficient ranges from [0 to 1]. A larger value indicates a higher degree of overlap between the fused feature map and the label feature map, that is, a higher degree of similarity between the fused feature map and the label feature map.

[0126] Specifically, the fused feature map (query image Query), that is, the target to be retrieved or matched, is compared with the label feature map (annotated gallery). The semantic segmentation model pre-extracts all label feature maps in the gallery. When the query is input, the Dice coefficient is returned by calculating the number of pixels at the intersection of the fused feature map and the label feature map, and the similarity between the fused feature map and the label feature map is obtained. In addition, in the similarity loss calculation, since the result is between 0 and 1, and the attributes are labeled as 0 and 1, for attributes with a similarity greater than 0.5, it is set to 1 when calculating the loss, and for attributes with a similarity less than 0.5, it is set to 0 for calculation.

[0127] S304: Determine whether the similarity is less than a preset similarity.

[0128] S305: If the similarity is less than the preset similarity, continue to encode and decode the fused feature map through the first UNet convolutional neural network model and the second UNet convolutional neural network model until the similarity is greater than or equal to the preset similarity, thereby obtaining a feature map of the interface image to be detected.

[0129] In this step, the preset similarity is used as a criterion for the quality of the semantic segmentation model output. When the similarity is less than this value, it indicates that the semantic segmentation model output is not accurate enough and needs further optimization. This value cannot be set too high, otherwise it will lead to excessive iterative optimization processes, increasing computational cost and time consumption, and even causing the semantic segmentation model to overfit. Conversely, if this value is set too low, the model output quality will be suboptimal and unable to meet actual application requirements.

[0130] Specifically, if the preset similarity is 85%, that is, if the similarity between the fused feature map and the label feature map is less than 85%, the fused feature map is further encoded and decoded using the first UNet convolutional neural network model and the second UNet convolutional neural network model described in the aforementioned embodiment until the similarity is greater than or equal to 85%, thereby obtaining a feature map of the interface image to be detected. The preset similarity of 85% here is only for example and should be adjusted appropriately in actual applications according to the specific application scenario.

[0131] S306: Obtaining the category and location information of the element in the interface image to be detected according to the feature map.

[0132] In this step, the category information of the element includes button, text box, icon, and menu, and the location information of the element is represented by the coordinates of the upper left corner and the lower right corner of the circumscribed rectangle of the element.

[0133] In a possible implementation, the feature map is upsampled and convolved through the output layer of the semantic segmentation model, and the category and location information of the elements in the interface image to be detected are predicted by the classifier.

[0134] The output layer of the semantic segmentation model is a convolutional layer. Through upsampling convolution, the spatial size of the feature map is restored to the same size as the interface image to be detected, generating the final segmentation result. The number of channels in the output layer is equal to the number of categories of elements in the task, and an appropriate activation function, such as the softmax function or the Sigmoid function, is applied to generate the probability distribution of each pixel belonging to each category. The Sigmoid function can be formally expressed as:

[0135] F attr =sigmoid(RELU(BatchNorm(F i )))

[0136] Among them, F i Represents the feature map of the input and output layers, F attr Represents the feature map of the output layer.

[0137] After the feature map is restored to its original size, a classifier is used to predict the category of each pixel. This classifier can be a simple fully connected layer or a more complex support vector machine (SVM) classifier. SVM is a supervised learning model suitable for classification and regression tasks. It performs classification by finding the optimal hyperplane to separate data points of different categories.

[0138] For example, Figure 4Taking the login interface image of a client logging into QQ Mail as an example, the "WeChat Login" category is "Button" and its location information is [(120, 300)(280, 340)]. Similarly, the "Password Input Box" category is "Input Box" and its location information is [(100, 200)(300, 240)]; the "Retrieve Password" category is "Hypertext Link Area" and its coordinate range is [(50, 450)(150, 470)]. By accurately identifying and locating each element in the interface image, the automation and reliability of interface analysis are improved.

[0139] The embodiment of the present application provides a method for locating interface elements based on deep learning, which aims to accurately identify and locate each element in the interface image through a semantic segmentation model. By marking the elements of the interface image to be detected, a label feature map is generated to provide supervision information for the model. By adding the first feature map and the second feature map, a fused feature map is generated, and the global and local feature information are integrated to enhance the feature expression ability of the model. The fused feature map is compared with the label feature map, and the similarity is calculated to evaluate the accuracy of the model output. If the similarity is lower than the preset threshold, the feature map is further processed through an iterative optimization process until the desired similarity standard is reached. Finally, through upsampling convolution and classifiers, the spatial size of the feature map is restored and the category and positioning information of each pixel are predicted. Through the above method, buttons, text boxes, icons, menus and other elements in the interface image can be accurately identified, and their precise positioning information can be provided, which improves the automation and reliability of interface analysis and supports various application scenarios such as user interface optimization, automated testing and user behavior analysis.

[0140] Figure 5 This is a schematic diagram of the structure of an interface element positioning device based on deep learning provided by this application, such as Figure 5 As shown, the interface element positioning device 50 based on deep learning provided in this embodiment includes:

[0141] The first processing module 501 is used to obtain an image of an interface to be detected;

[0142] The second processing module 502 is used to input the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected;

[0143] Among them, the semantic segmentation model is a model that extracts features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of interface images, and the second UNet convolutional neural network model is used to extract local features of interface images.

[0144] In a possible implementation, the second processing module 502 is specifically configured to:

[0145] Performing normalization processing on the interface image to be detected to obtain a processed interface image to be detected;

[0146] Inputting the processed interface image to be detected into the semantic segmentation model, encoding and decoding the processed interface image to be detected through the first UNet convolutional neural network model and the second UNet convolutional neural network model to obtain the first feature map and the second feature map of the processed interface image to be detected;

[0147] The category and location information of the element in the interface image to be detected are analyzed based on the first feature map and the second feature map.

[0148] In one possible implementation, the first UNet convolutional neural network model includes a global structure branch model and a hole convolution model; when the processed interface image to be detected is encoded and decoded by the first UNet convolutional neural network model, the second processing module 502 is specifically used to:

[0149] Downsampling the processed interface image to be detected by using a global structure branch model to obtain a first encoding feature map of the processed interface image to be detected;

[0150] Upsampling the first encoding feature map through a dilated convolution model to obtain a processed first decoding feature map of the interface image to be detected;

[0151] The first encoding feature map and the first decoding feature map are cascaded to obtain a first feature map.

[0152] In one possible implementation, the second UNet convolutional neural network model includes a local refinement branch model and a dilated convolution model; when encoding and decoding the processed interface image to be detected by the second UNet convolutional neural network model, the second processing module 502 is specifically used to:

[0153] Downsampling the processed interface image to be detected by using the local refinement branch model to obtain a second encoding feature map of the processed interface image to be detected;

[0154] Upsampling the second encoding feature map through a dilated convolution model to obtain a processed second decoding feature map of the interface image to be detected;

[0155] Cascading the second encoding feature map and the second decoding feature map to obtain a cascade feature map;

[0156] The second feature map is obtained by extracting features from the cascade feature map through height axial attention and width axial attention combined with the position offset term.

[0157] In a possible implementation, the deep learning-based interface element positioning device 50 further includes a third processing module 503 for performing element marking on the interface image to be detected to obtain a label feature map corresponding to the interface image to be detected.

[0158] In a possible implementation, the second processing module 502 is specifically configured to:

[0159] Adding the first feature map and the second feature map to obtain a fused feature map of the interface image to be detected;

[0160] Compare the fusion feature map with the label feature map to obtain the similarity between the fusion feature map and the label feature map;

[0161] If the similarity is less than the preset similarity, the first UNet convolutional neural network model and the second UNet convolutional neural network model are continued to encode and decode the fused feature map until the similarity is greater than or equal to the preset similarity, thereby obtaining a feature map of the interface image to be detected;

[0162] According to the feature map, the category and location information of the elements in the interface image to be detected are obtained.

[0163] In a possible implementation, the second processing module 502 is further configured to:

[0164] The feature map is upsampled and convolved through the output layer of the semantic segmentation model, and the category and location information of the elements in the interface image to be detected are predicted by the classifier.

[0165] The deep learning-based interface element positioning device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0166] Figure 6 This is a schematic diagram of the structure of a computer device provided in this application. Figure 6 As shown, the computer device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0167] During the specific implementation process, at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 performs the above method.

[0168] The specific implementation process of the processor 601 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0169] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0170] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0171] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0172] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0173] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0174] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0175] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0176] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0178] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory ROM, random access memory (Random Access Memory, RAM), disk or optical disk, and other media that can store program code.

[0179] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0180] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A method for locating interface elements based on deep learning, characterized in that: include: Obtaining the interface image to be detected; Inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected; Among them, the semantic segmentation model is a model for extracting features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of the interface image, and the second UNet convolutional neural network model is used to extract local features of the interface image.

2. The method according to claim 1, characterized in that The step of inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected includes: Performing normalization processing on the interface image to be detected to obtain a processed interface image to be detected; Inputting the processed interface image to be detected into the semantic segmentation model, performing encoding and decoding processing on the processed interface image to be detected through the first UNet convolutional neural network model and the second UNet convolutional neural network model to obtain a first feature map and a second feature map of the processed interface image to be detected; According to the first feature map and the second feature map, the category and positioning information of the elements in the interface image to be detected are analyzed.

3. The method according to claim 2, characterized in that The first UNet convolutional neural network model includes a global structure branch model and a void convolution model; The processed interface image to be detected is subjected to encoding and decoding processing by a first UNet convolutional neural network model, including: Downsampling the processed interface image to be detected by using the global structure branch model to obtain a first coding feature map of the processed interface image to be detected; Performing upsampling processing on the first encoding feature map through the dilated convolution model to obtain a first decoding feature map of the interface image to be detected after the processing; The first encoding feature map and the first decoding feature map are cascaded to obtain the first feature map.

4. The method according to claim 3, characterized in that The second UNet convolutional neural network model includes a local refinement branch model and the hole convolution model; The processed interface image to be detected is subjected to encoding and decoding processing by a second UNet convolutional neural network model, including: Downsampling the processed interface image to be detected by using the local refined branch model to obtain a second encoding feature map of the processed interface image to be detected; Performing upsampling processing on the second encoding feature map through the dilated convolution model to obtain a second decoding feature map of the interface image to be detected after the processing; Cascading the second encoding feature map and the second decoding feature map to obtain a cascade feature map; The second feature map is obtained by performing feature extraction on the cascade feature map through height axial attention and width axial attention combined with a position offset term.

5. The method according to any one of claims 2 to 4, characterized in that Before inputting the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected, the method further includes: Element marking is performed on the interface image to be detected to obtain a label feature map corresponding to the interface image to be detected.

6. The method according to claim 5, characterized in that Analyzing the category and location information of the element in the to-be-detected interface image based on the first feature map and the second feature map includes: Adding the first feature map and the second feature map to obtain a fused feature map of the interface image to be detected; Comparing the fused feature map with the label feature map to obtain the similarity between the fused feature map and the label feature map; If the similarity is less than the preset similarity, continue to encode and decode the fused feature map through the first UNet convolutional neural network model and the second UNet convolutional neural network model until the similarity is greater than or equal to the preset similarity, thereby obtaining a feature map of the interface image to be detected; According to the feature map, the category and location information of the elements in the interface image to be detected are obtained.

7. The method according to claim 6, characterized in that The step of obtaining the category and location information of the element in the interface image to be detected according to the feature map includes: The feature map is upsampled and convolved through the output layer of the semantic segmentation model, and the category and positioning information of the elements in the interface image to be detected are obtained through classifier prediction.

8. An interface element positioning device based on deep learning, characterized in that: include: A first processing module is used to obtain an image of the interface to be detected; A second processing module is used to input the interface image to be detected into a preset semantic segmentation model for analysis to obtain the category and location information of the elements in the interface image to be detected; Among them, the semantic segmentation model is a model for extracting features of interface images to determine categories and positioning by training the first UNet convolutional neural network model and the second UNet convolutional neural network model. The first UNet convolutional neural network model is used to extract global features of the interface image, and the second UNet convolutional neural network model is used to extract local features of the interface image.

9. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images