Graphical interface similarity determination method and electronic device
By generating combined feature maps of graphical interface images and using a design feature extraction network, the similarity of graphical interface design feature vectors is calculated, which solves the problem of inaccurate similarity judgment of graphical interface design styles in the prior art and achieves accurate similarity judgment and display.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI HONGJI INFORMATION TECH CO LTD
- Filing Date
- 2022-12-23
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies, when determining the similarity of graphical interface design styles, are not accurate enough based on color and texture information, resulting in imprecise judgment results.
By generating a combined feature map of graphical interface images, design feature vectors are extracted using a trained design feature extraction network. The similarity between the design feature vectors of two graphical interface images is calculated, and cosine similarity is applied and offset processing is performed to obtain a similarity score.
It achieves accurate determination of the similarity of graphical interface design styles and provides an intuitive display of similarity.
Smart Images
Figure CN116363393B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method for judging the similarity of graphical interfaces, a training method for designing feature extraction networks, and electronic devices. Background Technology
[0002] A graphical user interface (GUI) is a computer user interface displayed graphically. Because GUIs directly face the user and significantly impact user experience, their design style is increasingly valued. Excellent design styles are widely referenced and even plagiarized. Therefore, determining the similarity between two GUIs is crucial in scenarios such as GUI design style infringement protection and resource searching. Current solutions often rely on general content or information like color and texture; however, this approach fails to accurately extract the design style of the GUI, resulting in inaccurate judgments. Summary of the Invention
[0003] The purpose of this application is to provide a method for judging the similarity of graphical interfaces, a method for training a feature extraction network, and an electronic device for accurately judging the design style similarity between two graphical interfaces.
[0004] On the one hand, this application provides a method for determining the similarity of graphical interfaces, including:
[0005] For each graphical interface image in the image pair to be compared, a combined feature map corresponding to each graphical interface image is generated; wherein, the image pair to be compared includes two graphical interface images, and the combined feature map is used to characterize the composite image features of the graphical interface images;
[0006] The combined feature map of each graphical interface image in the image pair to be compared is input into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image.
[0007] Determine the similarity between the design feature vectors of the two graphical interface images.
[0008] In one embodiment, before generating a combined feature map corresponding to each graphical interface image for each graphical interface image of the image pair to be compared, the method further includes:
[0009] Determine whether the two graphical interface images are of the same size;
[0010] If not, adjust the size of the two graphical interface images so that the adjusted two graphical interface images have the same size.
[0011] In one embodiment, generating a combined feature map corresponding to each graphical interface image of the image pair to be compared includes:
[0012] For each graphical interface image, generate its corresponding contour map;
[0013] For each graphical interface image, generate several corresponding element images; each element image corresponds to an element category.
[0014] For each graphical interface image, the graphical interface image, the corresponding contour map, and several element maps are stitched together along the channel dimension to obtain the combined feature map corresponding to the graphical interface image.
[0015] In one embodiment, generating a corresponding contour map for each graphical interface image includes:
[0016] Each graphical interface image is input into a trained contour map generation model to obtain the contour map corresponding to the graphical interface image.
[0017] In one embodiment, generating a plurality of element diagrams corresponding to each graphical interface image includes:
[0018] For each graphical interface image, target detection is performed on the graphical interface image to obtain target detection results corresponding to several element categories;
[0019] Based on the target detection results for each element category, an element map corresponding to each element category is generated, thereby obtaining several element maps corresponding to each graphical interface image.
[0020] In one embodiment, before inputting the combined feature map of each graphical interface image in the image pair to be compared into a trained design feature extraction network to obtain a design feature vector corresponding to each graphical interface image, the method further includes:
[0021] Generate corresponding combined feature maps for multiple sample interface images in the sample set; wherein, the sample set includes multiple sample interface images, and each sample interface image carries a design category label;
[0022] The combined feature map of the multiple sample interface images is input into a convolutional neural network to obtain the predicted category information output by the convolutional neural network for each combined feature map; wherein, the convolutional neural network includes a feature extraction network and a classifier;
[0023] The network parameters of the convolutional neural network are adjusted based on the difference between the predicted category information and the designed category label.
[0024] Repeat the above steps until the convolutional neural network converges, and use the feature extraction network of the trained convolutional neural network as the design feature extraction network.
[0025] In one embodiment, after determining the similarity between the design feature vectors of the two graphical interface images, the method further includes:
[0026] If the similarity is a cosine similarity, the cosine similarity is shifted to obtain the target similarity; wherein the target similarity is a non-negative number.
[0027] The target similarity is multiplied by a preset amplification factor to obtain a similarity score;
[0028] Output the similarity score corresponding to the image pair to be compared.
[0029] On the other hand, this application provides a method for training a feature extraction network, including:
[0030] Generate corresponding combined feature maps for multiple sample interface images in the sample set; wherein, the sample set includes multiple sample interface images, and each sample interface image carries a design category label;
[0031] The combined feature map of the multiple sample interface images is input into a convolutional neural network to obtain the predicted category information output by the convolutional neural network for each combined feature map; wherein, the convolutional neural network includes a feature extraction network and a classifier;
[0032] The network parameters of the convolutional neural network are adjusted based on the difference between the predicted category information and the designed category label.
[0033] Repeat the above steps until the convolutional neural network converges, and use the feature extraction network of the trained convolutional neural network as the design feature extraction network.
[0034] In one embodiment, generating corresponding combined feature maps for multiple sample interface images in the sample set includes:
[0035] For multiple sample interface images in the sample set, a corresponding contour map is generated for each sample interface image.
[0036] For multiple sample interface images in the sample set, several element images are generated for each sample interface image; wherein each element image corresponds to an element category.
[0037] For each sample interface image, the sample interface image, the corresponding contour map, and several element maps are stitched together along the channel dimension to obtain the combined feature map corresponding to the sample interface image.
[0038] On the other hand, this application provides a device for determining the similarity of graphical interfaces, including:
[0039] The generation module is used to generate a combined feature map corresponding to each graphical interface image in each graphical interface image of the image pair to be compared; wherein, the image pair to be compared includes two graphical interface images, and the combined feature map is used to characterize the composite image features of the graphical interface images.
[0040] The extraction module is used to input the combined feature map of each graphical interface image in the image pair to be compared into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image.
[0041] The determination module is used to determine the similarity between the design feature vectors of the two graphical interface images.
[0042] Furthermore, this application provides an electronic device, the electronic device comprising:
[0043] processor;
[0044] Memory used to store processor-executable instructions;
[0045] The processor is configured to execute the above-mentioned method for judging the similarity of graphical interfaces or the above-mentioned method for training the design feature extraction network.
[0046] In addition, this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to perform the above-described method for judging graphical interface similarity or the above-described method for training the design feature extraction network.
[0047] The proposed solution generates a combined feature map for each graphical interface image in the image pair to be compared. Then, a trained design feature extraction network is used to extract design feature vectors from the combined feature map. These design feature vectors can characterize the design style information of the graphical interface image, thereby determining the similarity between the two design feature vectors. This characterizes the similarity of the design styles between the two graphical interface images in the image pair to be compared, thus achieving accurate similarity determination. Attached Figure Description
[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below.
[0049] Figure 1 A schematic diagram illustrating an application scenario of the graphical interface similarity determination method provided in an embodiment of this application;
[0050] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0051] Figure 3 A flowchart illustrating a method for determining the similarity of graphical interfaces provided in an embodiment of this application;
[0052] Figure 4 Provided for an embodiment of this application Figure 3 A detailed flowchart of step 310 is shown below;
[0053] Figure 5 A graphical interface image provided for an embodiment of this application;
[0054] Figure 6 for Figure 5 Outline of the graphical user interface image;
[0055] Figure 7 for Figure 5 A schematic diagram of target detection in a graphical user interface image.
[0056] Figure 8 A schematic flowchart illustrating the training method for a design feature extraction network provided in an embodiment of this application;
[0057] Figure 9 A graphical interface image provided for an embodiment of this application;
[0058] Figure 10 A graphical interface image provided for an embodiment of this application;
[0059] Figure 11 A block diagram of a graphical interface similarity determination device provided in an embodiment of this application. Detailed Implementation
[0060] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0061] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0062] Robotic Process Automation (RPA) technology can simulate the operations employees perform on computers using a keyboard and mouse in their daily work. It can replace humans in tasks such as logging into systems, operating software, reading and writing data, downloading files, and reading emails. By using automated robots as virtual labor, enterprises can free employees from repetitive, low-value tasks, allowing them to focus their energy on high-value-added work. This enables enterprises to reduce costs and increase efficiency while undergoing digital and intelligent transformation.
[0063] RPA (Robotic Process Automation) is a software robot that replaces manual tasks in business processes and interacts with computer front-end systems like a human. Therefore, RPA can be seen as a software-based program robot running on a personal PC or server, mimicking user actions on a computer to automate repetitive tasks such as retrieving emails, downloading attachments, logging into systems, and data processing and analysis—faster, more accurate, and more reliable. While both RPA and traditional physical robots address the speed and accuracy issues in human work through specific rules, traditional physical robots are hardware-software hybrids requiring specific hardware support and software to perform tasks. RPA robots, on the other hand, are purely software-based; once the appropriate software is installed, they can be deployed to any PC or server to complete the assigned tasks.
[0064] In other words, RPA is a method and related technologies that utilize "digital employees" to perform business operations in place of humans. Essentially, RPA uses software automation technology to simulate human operation of computer systems, software, web pages, and documents, enabling unmanned processing of business information and execution of business actions, ultimately achieving automated process handling, cost savings, and improved efficiency. In graphical interface similarity judgment scenarios, RPA technology can replace the manual task of determining whether two graphical interfaces belong to the same application.
[0065] Figure 1 This is a schematic diagram illustrating an application scenario of the graphical interface similarity determination method provided in this application embodiment. For example... Figure 1 As shown, the application scenario includes a client 20 and a server 30. The client 20 can be a user terminal such as a host, mobile phone, or tablet computer, used to send at least two graphical interface images that need to be compared to the server 30. The server 30 can be a server, server cluster, or cloud computing center, which can perform similarity judgment on the at least two graphical interface images sent by the client 20.
[0066] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12. Figure 2 Taking a processor 11 as an example, the processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the process of the method in the following embodiments. In one embodiment, the electronic device 1 may be the aforementioned server 30, used to execute a method for judging the similarity of graphical interfaces or a method for training a feature extraction network.
[0067] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0068] This application also provides a computer-readable storage medium storing a computer program that can be executed by a processor 11 to perform the graphical interface similarity judgment method or the training method for designing a feature extraction network provided in this application.
[0069] See Figure 3 This is a flowchart illustrating a method for determining the similarity of graphical interfaces according to an embodiment of this application. Figure 3 As shown, the method may include steps 310 to 330.
[0070] Step 310: For each graphical interface image in the image pair to be compared, generate a combined feature map corresponding to each graphical interface image; wherein, the image pair to be compared includes two graphical interface images, and the combined feature map is used to characterize the composite image features of the graphical interface images.
[0071] The server can generate comparison image pairs by reading pre-stored pairs from memory, receiving them from the client, or cropping the two graphical interfaces. After obtaining the comparison image pairs, the server can generate combined feature maps corresponding to each graphical interface image. Here, the combined feature maps can represent the composite image features of the graphical interface images, including contour information features, element category features, etc. Element categories can include one or more combinations of text, icons, and images.
[0072] In one embodiment, before generating a combined feature map for the graphical user interface (GUI) images, the server can determine whether the two GUI images are of the same size. Here, size includes height, width, and number of channels. Generally, the GUI images in the image pair to be compared have the same color mode, so they have the same number of channels. For example, if both GUI images are in RGB (Red, Green, Blue) mode, then they both have 3 channels. Therefore, it is necessary to compare the width and height. Here, width refers to the number of pixels in the horizontal direction, and height refers to the number of pixels in the vertical direction.
[0073] On the one hand, if the number of channels, width, and height are all the same, it means that the two are the same size. In this case, the server can continue to execute the step of generating a combined feature map for each graphical interface image.
[0074] On the other hand, if the number of channels is the same, but at least one of the width and height is different, it indicates that the two graphical interface images are different sizes. The server can resize the two graphical interface images to make them the same size. For example, the server can unify the size of the two graphical interface images by re-cropping. Alternatively, the server can adjust the size by upsampling, downsampling, or other methods to make the two graphical interface images the same size. After unifying the size of the two graphical interface images, the server can perform the step of generating a combined feature map for each graphical interface image after unifying the size.
[0075] Step 320: Input the combined feature map of each graphical interface image in the image pair to be compared into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image.
[0076] The design feature extraction network is used to extract feature vectors from the combined feature maps. These feature vectors serve as design feature vectors, representing the design style of the graphical interface in the graphical interface image. This design feature extraction network can be trained using a convolutional neural network, as detailed below, and will not be elaborated further here.
[0077] After obtaining the two combined feature maps of the image pair to be compared, the server can input the combined feature maps into the trained design feature extraction network. The design feature extraction network processes the combined feature maps to obtain the design feature vector corresponding to each graphical interface image.
[0078] Step 330: Determine the similarity between the design feature vectors of the two graphical interface images.
[0079] After obtaining the design feature vectors of two graphical user interface images, the server can calculate the similarity between the two feature vectors. A higher similarity indicates a more similar design style between the two graphical user interface images. The similarity between two design feature vectors can be measured using methods such as cosine similarity, Euclidean distance, and cosine distance.
[0080] By using the above measures, after generating a combined feature map for each graphical interface image, design feature vectors representing the design style are extracted from the combined feature map. Thus, the similarity between the design feature vectors can accurately indicate the similarity of the design styles between two graphical interface images.
[0081] In one embodiment, after obtaining the similarity between the design feature vectors of two graphical interface images, if the similarity is a cosine similarity, the cosine similarity can be converted to obtain a similarity score that more intuitively represents the degree of similarity.
[0082] The server can offset the cosine similarity to obtain the target similarity. Here, the target similarity is a non-negative number. For example, the server can increment the cosine similarity by one to complete the offset. Since the value of cosine similarity ranges from -1 to 1, the target similarity is non-negative after the offset.
[0083] After obtaining the target similarity, the server can multiply the target similarity by a preset amplification factor to obtain a similarity score. Here, the amplification factor can be 50, which normalizes the similarity score to between 0 and 100. The higher the similarity score, the more similar the design styles of the two graphical interface images are. The server can visually demonstrate the similarity in design styles between the two graphical interface images by displaying the similarity score between the graphical interface output of the comparison results and the image to be compared.
[0084] In one embodiment, see Figure 4 This is provided as an embodiment of the present application. Figure 3 A detailed flowchart of step 310 is shown below. Figure 4 As shown, when generating a combined feature map for a graphical interface image, steps 311 to 313 can be performed as follows.
[0085] Step 311: For each graphical interface image, generate its corresponding contour map.
[0086] The contour map represents the contour information of the graphical interface image. The contour map is a single-channel image with the same width and height as the graphical interface image. The value of each pixel in the contour map is 0 or 1. Pixels with a value of 0 belong to the background, and pixels with a value of 1 belong to the contour.
[0087] See Figure 5 This is a graphical interface image provided in one embodiment of this application; see also Figure 6 ,for Figure 5 The server can detect the outlines of various elements (including text, icons, images, etc.) in the graphical interface image, set a pixel with a value of 1 at the outline position, and set a pixel with a value of 0 at the background position, thus obtaining the outline image.
[0088] In one embodiment, the server can input each graphical interface image into a trained contour map generation model, and process the graphical interface image through the contour map generation model to obtain a contour map. The contour map generation model can be trained using a neural network segmentation model.
[0089] By generating models using contour maps, complex graphical interface images can be processed to obtain contour maps more quickly and accurately.
[0090] Step 312: For each graphical interface image, generate several element images corresponding to it; where each element image corresponds to an element category.
[0091] An element map is used to represent element objects within a specific element category in a graphical user interface (GUI) image. An element map is a single-channel image with the same width and height as the GUI image. Pixels within an element map have values of 0 or 1; pixels with a value of 0 belong to the background, and pixels with a value of 1 belong to the interior of an element object. Each element map corresponds to one element category and includes all element objects within that category. For example, an element map corresponding to the element category "text" includes all text elements. Furthermore, if a GUI image does not contain any element objects corresponding to a certain element category, then all pixels in the element map corresponding to that element category will have a value of 0.
[0092] In one embodiment, for each graphical interface image, the server can perform object detection on the graphical interface image to obtain object detection results corresponding to several element categories. The server can input the graphical interface image into a trained object detection network to obtain the object detection results, which represent the position information of the element objects (usually represented in the form of position rectangular boxes) and the element categories of the element objects.
[0093] The server can generate element maps corresponding to each element category based on the object detection results of each element category, so as to obtain several element maps corresponding to each graphical interface image. After determining the position information of the element objects in the graphical interface image under any element category, the server can set the pixel points to 1 at the corresponding positions on the element map of this element category, and set the pixel points of the background part to 0.
[0094] See Figure 7 , for Figure 5 the schematic diagram of object detection of the graphical interface image in Figure 7 As shown, the detected icons and texts in the graphical interface image are represented by rectangular boxes for positions. In addition, there may be duplicate categories in the object detection results. For example, Figure 5 the texts "Baidu", "Bai", and "du" in
[0095] Step 313: For each graphical interface image, splice the graphical interface image, the contour map corresponding to the graphical interface image, and several element maps in the channel dimension to obtain a combined feature map corresponding to the graphical interface image.
[0096] For each graphical interface image, after obtaining the contour map and several element maps of this graphical interface image, the graphical interface image, the contour map, and several element maps can be spliced in the channel dimension. The splicing order is preset (for example: spliced in the order of the graphical interface image, the contour map, the element map corresponding to the icon, the element map corresponding to the text, and the element map corresponding to the picture). After splicing, a combined feature map corresponding to this graphical interface image is obtained.
[0097] For a graphical user interface image of size h*w*3, we can obtain an outline map of size h*w*1 and k element maps of size h*w*1. Here, h is the height, w is the width, and k is the total number of element categories. After concatenation, we can obtain a combined feature map of size h*w*(3+1+k).
[0098] Through the above measures, composite image features can be extracted from graphical interface images, and combined feature maps representing composite image features can be generated.
[0099] In one embodiment, before executing the graphical interface similarity determination method in this application, a feature extraction network can be trained, see [link to relevant documentation]. Figure 8 This is a flowchart illustrating a training method for a feature extraction network provided in an embodiment of this application. Figure 8 As shown, the method may include steps 810 to 840.
[0100] Step 810: Generate corresponding combined feature maps for multiple sample interface images in the sample set; wherein, the sample set includes multiple sample interface images, and each sample interface image carries a design category label.
[0101] The server can obtain a sample set, which includes sample interface images for m design categories, with p sample interface images for each design category. Here, m and p can be pre-configured as needed, and the sample interface images are graphical interface images used as training samples.
[0102] For multiple sample interface images in the sample set, the server can generate a combined feature map for each sample interface image. For each sample interface image, the server can generate a corresponding contour map and several element maps, each corresponding to a specific element category. For each sample interface image, the server can concatenate the sample interface image, its corresponding contour map, and the several element maps along the channel dimension to obtain the combined feature map corresponding to the sample interface image. Details on generating the combined feature map can be found in the relevant paragraphs above and will not be repeated here.
[0103] Step 820: Input the combined feature map of multiple sample interface images into the convolutional neural network to obtain the predicted category information output by the convolutional neural network for each combined feature map; wherein, the convolutional neural network includes a feature extraction network and a classifier.
[0104] Since feature extraction networks need to process combined feature maps, and the number of channels in combined feature maps is higher than that in general images, the convolutional kernels of the first layer of the feature extraction network can be set according to the number of channels in the combined feature maps. For example, if the element categories to be detected are 3, then the number of channels in the combined feature maps is 7, and the first layer of the feature extraction network can be set to have 7 convolutional kernels.
[0105] The server can input a combined feature map of multiple sample interface images into a convolutional neural network (CNN). The CNN's feature extraction network extracts feature vectors from the combined feature map, and a classifier processes these feature vectors to obtain the predicted category information corresponding to the combined feature map. Here, the predicted category information is the designed category information that the CNN predicts for the sample interface images during training.
[0106] Step 830: Adjust the network parameters of the convolutional neural network based on the difference between the predicted category information and the designed category labels.
[0107] Step 840: Repeat the above steps until the convolutional neural network converges, and use the feature extraction network of the trained convolutional neural network as the design feature extraction network.
[0108] For the same sample interface image, the server can calculate the difference between the predicted category information and the designed category label using a loss function, and then adjust the network parameters of the convolutional neural network based on this difference.
[0109] After adjusting the network parameters, you can return to step 820, input the combined feature map of the sample interface image into the adjusted convolutional neural network to obtain new predicted category information, and adjust the network parameters of the convolutional neural network again based on the difference between the new predicted category information and the designed category label.
[0110] This process can be repeated multiple times until the preset number of training iterations is reached, or until the value of the loss function tends to stabilize. At this point, the convolutional neural network can be considered to have converged, and a trained convolutional neural network can be obtained. In this case, the feature extraction network in the trained convolutional neural network can be used as the design feature extraction network.
[0111] The proposed solution can accurately determine the similarity of design styles among different graphical interface images.
[0112] See Figure 9 This is a graphical interface image provided in an embodiment of this application. Figure 9 and Figure 5 By constructing the image pair to be compared, the similarity score can be determined to be 50.
[0113] See Figure 10This is a graphical interface image provided in an embodiment of this application. Figure 10 and Figure 5 By constructing the image pair to be compared, the similarity score can be determined to be 90.
[0114] It is evident that, in comparison, Figure 10 Chinese graphical interface images and Figure 5 The graphical interface images are very similar.
[0115] Figure 11 This is a block diagram of a graphical interface similarity determination device according to an embodiment of the present invention, such as... Figure 11 As shown, the device may include:
[0116] The generation module 1110 is used to generate a combined feature map corresponding to each graphical interface image of the image pair to be compared; wherein, the image pair to be compared includes two graphical interface images, and the combined feature map is used to characterize the composite image features of the graphical interface images.
[0117] Extraction module 1120 is used to input the combined feature map of each graphical interface image in the image pair to be compared into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image.
[0118] The determination module 1130 is used to determine the similarity between the design feature vectors of the two graphical interface images.
[0119] The implementation process of the functions and roles of each module in the above-mentioned device is detailed in the implementation process of the corresponding steps in the above-mentioned method for judging the similarity of graphical interfaces, and will not be repeated here.
[0120] The apparatuses and methods disclosed in the several embodiments provided in this application can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0121] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0122] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
Claims
1. A method for determining the similarity of graphical interfaces, characterized in that, include: For each graphical interface image in the image pair to be compared, a combined feature map corresponding to each graphical interface image is generated; wherein, the image pair to be compared includes two graphical interface images, and the combined feature map is used to characterize the composite image features of the graphical interface images; The combined feature map of each graphical interface image in the image pair to be compared is input into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image. Determine the similarity between the design feature vectors of the two graphical interface images; For each graphical interface image in the image pair to be compared, a combined feature map corresponding to each graphical interface image is generated, including: For each graphical interface image, generate its corresponding contour map; For each graphical interface image, several corresponding element images are generated. Each element image represents the element objects within a specific element category of the graphical interface image. Each element image is a single-channel image with the same width and height as the graphical interface image. Pixels within an element image have values of 0 or 1; pixels with a value of 0 belong to the background, and pixels with a value of 1 belong to the interior of an element object. Each element image corresponds to an element category. Each element image includes all element objects within its corresponding element category. Element categories include text, icons, and images, or a combination of one or more of these. For each graphical interface image, the graphical interface image, the corresponding contour map, and several element maps are stitched together along the channel dimension to obtain the combined feature map corresponding to the graphical interface image.
2. The method according to claim 1, characterized in that, Before generating a combined feature map corresponding to each graphical interface image for each graphical interface image of the image pair to be compared, the method further includes: Determine whether the two graphical interface images are of the same size; If not, adjust the size of the two graphical interface images so that the adjusted two graphical interface images have the same size.
3. The method according to claim 1, characterized in that, The step of generating a corresponding contour map for each graphical interface image includes: Each graphical interface image is input into a trained contour map generation model to obtain the contour map corresponding to the graphical interface image.
4. The method according to claim 1, characterized in that, For each graphical interface image, generating a number of corresponding element images includes: For each graphical interface image, target detection is performed on the graphical interface image to obtain target detection results corresponding to several element categories; Based on the target detection results for each element category, an element map corresponding to each element category is generated, thereby obtaining several element maps corresponding to each graphical interface image.
5. The method according to claim 1, characterized in that, Before inputting the combined feature map of each graphical interface image in the image pair to be compared into the trained design feature extraction network to obtain the design feature vector corresponding to each graphical interface image, the method further includes: Generate corresponding combined feature maps for multiple sample interface images in the sample set; wherein, the sample set includes multiple sample interface images, and each sample interface image carries a design category label; The combined feature map of the multiple sample interface images is input into a convolutional neural network to obtain the predicted category information output by the convolutional neural network for each combined feature map; wherein, the convolutional neural network includes a feature extraction network and a classifier; The network parameters of the convolutional neural network are adjusted based on the difference between the predicted category information and the designed category label. Repeat the above method until the convolutional neural network converges, and use the feature extraction network of the trained convolutional neural network as the design feature extraction network.
6. The method according to claim 1, characterized in that, After determining the similarity between the design feature vectors of the two graphical interface images, the method further includes: If the similarity is a cosine similarity, the cosine similarity is shifted to obtain the target similarity; wherein the target similarity is a non-negative number. The target similarity is multiplied by a preset amplification factor to obtain a similarity score; Output the similarity score corresponding to the image pair to be compared.
7. A training method for a feature extraction network, characterized in that, include: Generate corresponding combined feature maps for multiple sample interface images in the sample set; wherein, the sample set includes multiple sample interface images, and each sample interface image carries a design category label; The combined feature map of the multiple sample interface images is input into a convolutional neural network to obtain the predicted category information output by the convolutional neural network for each combined feature map; wherein, the convolutional neural network includes a feature extraction network and a classifier; The network parameters of the convolutional neural network are adjusted based on the difference between the predicted category information and the designed category label. Repeat the above method until the convolutional neural network converges, and use the feature extraction network of the trained convolutional neural network as the design feature extraction network. The step of generating corresponding combined feature maps for multiple sample interface images in the sample set includes: For multiple sample interface images in the sample set, a corresponding contour map is generated for each sample interface image. For multiple sample interface images in the sample set, several corresponding element maps are generated for each sample interface image. Each element map represents the element objects within a certain element category of the graphical interface image. Each element map is a single-channel image with the same width and height as the graphical interface image. Pixel values within an element map are either 0 or 1; pixels with a value of 0 belong to the background, and pixels with a value of 1 belong to the interior of an element object. Each element map corresponds to an element category. Each element map includes all element objects under its corresponding element category. Element categories include text, icons, and images, or a combination of one or more of these. For each sample interface image, the sample interface image, the corresponding contour map, and several element maps are stitched together along the channel dimension to obtain the combined feature map corresponding to the sample interface image.
8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the graphical interface similarity judgment method according to any one of claims 1-6 or the design feature extraction network training method according to claim 7.