User interface sketch retrieval method based on modal fusion

Abstract

The invention discloses a sketch retrieval method based on a modal fusion network, belongs to the field of computer vision, and develops a sketch retrieval visual search framework based on modal fusion by taking a user interface sketch as an efficient information representation in a software development process in mobile application program development. Making a user interface sketch data set by using a data synthesis technology, and preprocessing the data set; a network of a feature extraction encoder and a modal fusion operator is utilized to perform feature extraction on a user interface sketch, and a feature encoder module is responsible for extracting meaningful visual feature representation from an original image and a sketch of a UI example so as to accurately capture information such as design elements and layout. After the modal fusion network extracts visual feature representations from natural images and sketches, the visual feature representations are mapped to a unified modal fusion embedding space. By learning a representation space fusing modal data points and learning and calculating cross-modal attention mapping between the modal data points, the research can reserve semantically related and modal specific features so as to unify information of two modals. Through the proposed multi-modal embedding framework, not only can the structure of the original image and the sketch of the UI example and the joint feature of the associated content be learned, but also the multi-modal embedding framework can be applied to guide the retrieval process of the UI example sketch.

Description

Technical Field

[0001] The present invention belongs to the technical field of interdisciplinary research on computer vision and human-computer interaction, and specifically relates to the construction of a sketch data generation and sketch retrieval method based on modal fusion. Background Art

[0002] With the rise of data-driven design and large-scale user interface datasets, designers have access to a vast amount of design examples containing rich information, such as popular visual illustrations, common folding patterns, and high-density layout implementations, which can greatly enhance various design tasks. However, in the context of big data, retrieving relevant design examples from large-scale UI images is a daunting task. Image-based UI retrieval techniques are generally not suitable for UI retrieval due to the inherent semantic complexity of images. While designers can easily search for general categories of examples through image retrieval, fine-grained search based on visual layout and content is much more difficult. Taking all of the above into consideration, UI image-based retrieval suffers from the following major issues: 1) It is difficult to accurately capture and understand users' complex requirements for UIs, especially when analyzing more detailed aspects of visual layout and content. 2) It is difficult to fully convey the details of design examples, reducing the adaptability of image-based techniques to ambiguous design requirements. 3) The semantic differences between images and text can make matching text queries with the raw pixel and code information of image-based design examples complex and time-consuming.

[0003] Sketches have proven to be an effective visual medium for conveying design concepts. Designers use sketches to conceive novel ideas, visualize abstract concepts, and quickly compare alternatives. Compared to using complex text-based and content-based query languages, sketches meet the needs of UI designers for specific design examples and provide a natural and intuitive retrieval method that allows users to easily express query conditions. Therefore, sketch-based user interface retrieval provides a new way for UI designers to express and convey their design intentions through hand-drawn sketches. However, given that UI images contain rich visual information and complex design elements, relying solely on traditional image retrieval technology may find it difficult to meet the precise retrieval needs of UI interfaces.

[0004] In terms of visual search, the method most related to the work of this patent is the aforementioned Swire (Reference: Forrest Huang, John F. Canny, and Jefrey Nichols. 2019. Swire: Sketch-based User Interface Retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI'19). Association for Computing Machinery, Glasgow w, Scotland Uk, Article 104, 10 pages. https: / / doi.org / 10.1145 / 3290605.3300334) system, which uses a deep neural network model to retrieve relevant UI examples from input sketches. Specifically, Swire trains two convolutional subnetworks on matching screenshots and their corresponding sketches. However, Swire only focuses on high-level layout information and has difficulty in performing fine-grained semantic understanding of sketches. Secondly, there is a gap in semantic and visual features between sketches and UI screenshots, which leads to inaccurate results for related example queries.

[0005] In order to better infer the content hierarchy of the UI in various UI queries and provide designers with examples that match the queries, this patent constructs a user interface sketch dataset and a sketch retrieval method based on modal fusion to effectively learn the fine-grained hierarchical structure and semantic relationship between UI sketches and original images to achieve accurate retrieval. Summary of the Invention

[0006] To help UI designers improve the efficiency of searching and creating design example libraries, this patent uses user interface sketches as a medium to achieve the goal of user interface sketch retrieval. To this end, taking advantage of the efficient information representation of sketch interactions, a sketch retrieval method based on modal fusion is proposed. This implements an efficient UI example search method for user interface sketch retrieval. This efficient UI example search method not only accelerates the design process, but also promotes the generation and exchange of ideas, further improving the quality and creativity of user interface design.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: 1. A sketch retrieval and description method based on a modality fusion network, the construction of the corresponding data set consists of the following steps:

[0008] (1) Create a user interface sketch dataset

[0009] Use crawler tools to crawl the VINS dataset from the mobile application database, directly call the interface to write data, and convert the intermediate files corresponding to the dataset into persistent storage data to complete the storage;

[0010] (2) Dataset preprocessing

[0011] Normalize the image size to maintain the aspect ratio, ensure that the pixel values ​​of the image are in the range of [-1, 1], and the size is regularized to 224×224;

[0012] (3) Constructing a sketch retrieval model for user interfaces based on modal fusion in the Torch9 framework

[0013] The sketch retrieval model has two main components: a feature extraction module and a modality fusion network. The feature extraction module is responsible for extracting meaningful visual feature representations from the original image and sketch of the UI example so as to accurately capture information such as design elements and layout. The modality fusion network extracts visual feature representations from natural images and sketches and maps them to a unified modality fusion embedding space. By learning a representation space of fused modality data points and learning and calculating cross-modal attention mappings between modality data points, this patent can retain semantically relevant and modality-specific features to unify the information of the two modalities. Finally, the cosine distance is used to calculate the similarity between the query sketch and the database user interface image.

[0014] (4) Use the training set to train the modality fusion network

[0015] (a) Setting the network’s hyperparameters

[0016] The model is trained end-to-end using the Adam optimizer. The initial learning rate is kept at 1×e throughout the training process. -6 , batch_size is 128, and the iterative training is 5000 times;

[0017] (b) Initialization of the sketch retrieval model

[0018] The weights and biases of the VGG16 network and ViT-base (ViT-B) trained on the ImageNet2012 dataset are used as the initial weights and biases of the feature encoder in the sketch retrieval model. The pre-trained ViT-base model is used as the backbone network of the modality fusion network as the representation space of the fused modality data points, and the cross-modal attention mapping between the modal data points is learned and calculated.

[0019] (c) Set the training attributes of the convolutional layers, pooling layers, and fully connected layers of the VGG16 network in the modality fusion network. A training attribute value of 1 indicates participation in training, and a training attribute value of 0 indicates non-participation in training. The ViT-base (ViT-B) network uses 12 layers of cross-modal attention to produce a 768-dimensional output embedding.

[0020] (d) A modality-fusion-based user interface sketch retrieval network was trained using the training set of the user interface sketch dataset. The preprocessed dataset was split into training, validation, and test sets in an 80:10:10 ratio. 3108 pairs of matching edge detection maps and sketch-screenshot images were created for training, and 406 pairs were created for testing. During training, a cross-entropy loss function was used to measure the difference between the model predictions and the true labels, thereby guiding the optimization of model parameters. Training batches were constructed by calculating the cross-entropy between the predicted probability distribution and the true labels to measure the uncertainty and error of the model predictions. A smaller cross-entropy loss value indicates a closer match between the model predictions and the true labels. The optimizer updates the model parameters based on this loss, allowing the model to gradually learn more accurate predictions. By minimizing the cross-entropy loss, the model adjusts its parameters to improve performance and generalization. Simultaneously, a stochastic gradient descent strategy was used for model training to provide final test results.

[0021] Preferably, the network model is replaced by any model to enhance the quality of the extracted features. The entire framework is programmed using the PyTorch deep learning framework, and the experimental program runs on a single Nvidia GeForce RTX 3090 GPU, an 8-core Intel Xeon processor and 32GB / RAM.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention deeply indexes the visual features, semantic information and structural information of the original images and sketches of UI examples. The sketch retrieval method based on modal fusion mainly consists of two parts: a feature extraction module and a modal fusion network. In the feature extraction module, by using advanced computer vision technology, representative visual feature representations are extracted from the original images and sketches of UI examples. These feature representations can accurately capture important information such as design elements and layout, providing strong support for subsequent retrieval and analysis. In the modal fusion network, by introducing the concept of modal fusion, the visual feature representations of natural images and sketches are mapped into a unified modal fusion embedding space. By learning the representation space of the fused modal data points and calculating the cross-modal attention mapping between the modal data points, the information of the two modalities is effectively unified. Through the proposed multimodal embedding framework, not only can the joint features of the structure and associated content of the original images and sketches of UI examples be learned, but they can also be applied to guide the UI example sketch retrieval process. In summary, the technical solution of the present invention has important innovative significance and practical value in improving the efficiency and accuracy of user interface sketch retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the sketch retrieval method based on modal fusion in Example 1 of this patent.

[0025] Figure 2 This is the workflow for making a user interface dataset.

[0026] Figure 3 Generate sketch steps

[0027] Figure 4 Result of user interface sketch with enhanced annotations.

[0028] Figure 5 This is a partial image display of the generated user interface dataset.

[0029] Figure 6 It is a structural diagram of sketch retrieval based on modality fusion.

[0030] Figure 7 It is the semantic layout representation graph in the sketch retrieval process.

[0031] Figure 8 It is the result of the user interface sketch retrieval. DETAILED DESCRIPTION

[0032] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0033] This embodiment proposes a user interface sketch retrieval method based on modal fusion, aiming to solve the problem of UI design example search. This method is a user interface sketch interaction technology solution proposed to address the common problems of low search efficiency and inaccurate search results in the current UI design field. By utilizing modal fusion technology to fuse the visual information and semantic information of the sketch, a technical solution for efficient retrieval of UI design examples is achieved. The overall solution architecture is shown in the figure below. Figure 1 As shown in the figure, it includes two main modules: the collection of user interface sketch datasets and the user interface sketch retrieval method based on modal fusion. This technical solution will provide UI designers with a more convenient and accurate way to search for design resources, promoting the progress and innovation of user interface design.

[0034] This embodiment collected 3,450 synthetically generated UI sketches that are statistically similar to real-life UI screens. To generate the dataset for this patent, the patent collected and enhanced annotation information from the VINS dataset. Data synthesis technology was used to randomly generate a high-quality, large-scale dataset from 17,979 hand-drawn UI element sketches from the UI Sketch dataset, covering 21 UI element categories. The image aspect ratio was maintained at 720×1280. The dataset was randomly divided into training and test sets with no overlap between the training and test sets. The training set contained 3,108 pairs of matched edge detection images and sketch-screenshots for training, and 406 pairs for testing.

[0035] like Figure 2 As shown in the figure, the acceptance flow chart of the user interface sketch dataset is explained in detail.

[0036] like Figure 3 As shown, the enhanced annotation process of the user interface sketch is demonstrated.

[0037] like Figure 4 As shown in Figure 2, the specific generation process of the user interface sketch is demonstrated.

[0038] like Figure 5 As shown, a specific example of the final generated user interface sketch is presented.

[0039] (2) Dataset preprocessing

[0040] In order to extract image features more accurately, the present invention normalizes the sizes of the original images and sketches of the user interface in the dataset and regularizes them to the same size. The specific steps are as follows: First, use Opencv to resize all images to the same size by performing proportional scaling according to the maximum size. Secondly, for smaller images, padding is performed around the image to make it reach the required size. Subsequently, the aspect ratio of the image is maintained and the image is standardized to ensure that the pixel value of the image is within a range of [-1,1]. Next, a certain proportional scaling method is used to adjust the image to the target size, i.e., 224×224 pixels. Finally, in order to expand the diversity of the original image dataset, a data augmentation method with random noise and the Canny edge contour detection algorithm are used to generate a total of 9230 images. This series of data processing steps ensures the consistency and richness of the dataset, providing a good data foundation for subsequent feature extraction and model training;

[0041] like Figure 6 . Original UI image with normalized size

[0042] like Figure 7 .Sketch of the user interface with normalized dimensions

[0043] like Figure 8 .Original user interface image after data enhancement

[0044] (3) A sketch retrieval model for user interfaces based on modal fusion is constructed under the Torch9 framework. The sketch retrieval model has two main components: a feature extraction module and a modal fusion network. The feature extraction module is responsible for extracting meaningful visual feature representations from the original image and sketch of the UI example so as to accurately capture information such as design elements and layout. The modal fusion network extracts visual feature representations from natural images and sketches and maps them to a unified modal fusion embedding space. By learning a representation space of fused modal data points and learning and calculating cross-modal attention mapping between modal data points, this study is able to retain semantically related and modality-specific features to unify the information of the two modalities. Finally, the cosine distance is used to calculate the similarity between the query sketch and the database user interface image. Sketch retrieval model based on modal fusion.

[0045] (4) Use the training set to train the modality fusion network. The specific steps are as follows:

[0046] (a) Setting the network’s hyperparameters

[0047] The model is trained end-to-end using the Adam optimizer. The initial learning rate is kept at 1×e throughout the training process. -6 , batch_size is 128, and the training iteration is 5000 times.

[0048] (b) Training process of sketch retrieval model based on modality fusion network:

[0049] The weights and biases of the VGG16 network trained on the ImageNet2012 dataset and ViT-base (ViT-B) published on the Caffe official website are used as the initial weights and biases of the feature encoder in the sketch retrieval model. At the same time, the pre-trained ViT-base model is used as the backbone network of the modality fusion network as the representation space of the fused modal data points, and the cross-modal attention mapping between modal data points is learned and calculated.

[0050] (c) Set the training attributes of the convolutional layers, pooling layers, and fully connected layers of the VGG16 network in the modal fusion network. A training attribute value of 1 indicates participation in training, and a training attribute value of 0 indicates non-participation in training. The ViT-base (ViT-B) network uses 12 layers of cross-modal attention to generate a 768-dimensional output embedding.

[0051] (d) Using the training set of the constructed user interface sketch dataset to train the user interface sketch retrieval network based on modality fusion

[0052] The preprocessed dataset was divided into training, validation, and test sets in an 80:10:10 ratio. 3,108 pairs of matching edge detection maps and sketch-screenshots were created for training, and 406 pairs were created for testing. During training, a cross-entropy loss function was used to measure the difference between the model predictions and the true labels, thereby guiding the optimization of model parameters. Training batches were constructed by calculating the cross-entropy between the predicted probability distribution and the true labels to measure the uncertainty and error of the model predictions. A smaller cross-entropy loss value indicates a closer match between the model predictions and the true labels. The optimizer updates the model parameters based on this loss, allowing the model to gradually learn more accurate predictions. By minimizing the cross-entropy loss, the model adjusts its parameters to improve performance and generalization. Simultaneously, a stochastic gradient descent strategy was used for model training until the set number of iterations was reached. Training then terminated, resulting in a trained modal fusion network to provide the final test results.

[0053] With the goal of fusing instance discriminative features of sketch and original image modalities, this patent adopts a ViT-based modal fusion network. The core idea of ​​this network is to use the feature representation of the input image obtained by the φ(·) and ψ(·) functions as the initial feature embedding representation of the modal fusion network, and propagate it to the cross-modal embedding layer through the linear layer. In order to achieve cross-modal feature fusion and interaction, this patent introduces a learning vector of the same size as the output representation of φ(·) and ψ(·), called the instance tag, which is attached to the cross-modal embedding. In general, this implementation defines the cross-modal fusion network as a binary function It takes as input an original image x and a sketch y (of the same instance) and returns two vectors V x and V y , which are the Cross-Attention representations from sketch to photo and photo to sketch, respectively.

[0054] In the modal fusion network, Cross-Modal-Attention is responsible for achieving cross-modal feature fusion and interaction, so that the features of sketches and photos can influence and complement each other. Let the instance labels of the original image and sketch branches be represented as X c and Y c , the patch label set of the sketch branch is denoted as Y patch This patent firstly c Through the cross-modal embedding layer T x→y ( Xc )S propagates and maps it from the original image mode to the sketch mode, obtaining X c and Y patch Splice and get S=[X c ,Y patch ]. Then use the learnable matrix W q and W k X c and Y c Project to the same dot product space C with D dimensions D In which X is calculated c The attention scores are then Softmaxed on their product to obtain the attention weights, which are then multiplied by the corresponding features to obtain the fused feature representation. The following formula can be used to calculate the cross-modal attention embedding from the original image to the sketch branch:

[0055]

[0056]

[0057] in, L is a fully connected projection head that maps the sketch-to-photo embeddings into the same representation space CD, which we call the cross-modal attention space. norm(·) Representation layer normalization operation

[34] . Since cross-modal attention follows the same operational semantics as self-attention, it can also be computed for multiple attention heads

[35] . For m heads, m cross-modal attention operations are performed simultaneously, and then their outputs are concatenated and projected, where D is set to D / m to keep the computation and number of parameters constant.

[0058] This implementation designs a method called Cross-Modal Attention Queue Contrast (CMAQC) to learn the cross-modal attention space CD. The goal of CMAQC is to represent the cross-modal attention (CMA) V x (CMA-photo) and V y The positive sketch-photo pairs (CMA-sketch) are brought closer together, while separating the CMA-sketch and CMA-photo representations of different instances. This alignment is achieved by learning useful feature representations of CMA-sketch and CMA-photo, while distinguishing them from negative pairs (CMA-sketch representations of other instances). The CMA-sketch representations of other instances are regarded as negative values ​​in this setting, by minimizing the binary cross entropy loss based on a (k+1)-way softmax, where k is the number of negative instances.

[0059] Consider a photo x and its corresponding sketches y1 and y2. The cross-attention-representations of the original image-sketch (x, y1) and (x, y2) are and The above loss function Loss is a cross entropy loss function, which aims to learn the cross-modal attention space C D ,Right now

[0060]

[0061] in κ is a fixed-size dynamic queue consisting of the CMA-sketch representations of the previous mini-batch, and τ is a hyperparameter that controls the density of the distribution (a higher value produces a softer distribution). For each new sample, after calculating the above formula, its photo is converted to the sketch representation Queue into κ. The weights of the network are frozen when . During training, sketch pairs are randomly sampled, so It will also be called at some point during training. Since the dot product is a symmetric similarity metric, minimizing the CMAQC loss can also satisfy the corresponding symmetric property that the sketch representation is closer to its photo representation than to other photo representations.

Claims

1. A sketch retrieval and description method based on a modality fusion network, the construction of the corresponding dataset consists of the following steps: (1) Create a user interface sketch dataset Use crawler tools to crawl the VINS dataset from the mobile application database, directly call the interface to write data, and convert the intermediate files corresponding to the dataset into persistent storage data to complete the storage; (2) Dataset preprocessing Normalize the image size to maintain the aspect ratio, ensure that the pixel values ​​of the image are in the range of [-1, 1], and the size is regularized to 224×224; (3) Constructing a sketch retrieval model for user interfaces based on modal fusion in the Torch9 framework The sketch retrieval model has two main components: a feature extraction module and a modality fusion network. The feature extraction module is responsible for extracting meaningful visual feature representations from the original image and sketch of the UI example so as to accurately capture information such as design elements and layout. The modality fusion network extracts visual feature representations from natural images and sketches and maps them to a unified modality fusion embedding space. By learning a representation space of fused modality data points and learning and calculating cross-modal attention mappings between modality data points, this patent can retain semantically relevant and modality-specific features to unify the information of the two modalities. Finally, the cosine distance is used to calculate the similarity between the query sketch and the database user interface image. (4) Use the training set to train the modality fusion network (a) Setting the network’s hyperparameters The model is trained end-to-end using the Adam optimizer. The initial learning rate is kept at 1×e throughout the training process. -6 , batch_size is 128, and the iterative training is 5000 times; (b) Initialization of the sketch retrieval model The weights and biases of the VGG16 network and ViT-base (ViT-B) trained on the ImageNet2012 dataset are used as the initial weights and biases of the feature encoder in the sketch retrieval model. The pre-trained ViT-base model is used as the backbone network of the modality fusion network as the representation space of the fused modality data points, and the cross-modal attention mapping between the modal data points is learned and calculated. (c) Set the training attributes of the convolutional layer, pooling layer, and fully connected layer of the VGG16 network in the modality fusion network. A training attribute value of 1 indicates participation in training, and a training attribute value of 0 indicates non-participation in training. The ViT-base (ViT-B) network uses 12 layers of cross-modal attention, producing a 768-dimensional output embedding; (d) A modality-fusion-based user interface sketch retrieval network was trained using the training set of the user interface sketch dataset. The preprocessed dataset was split into training, validation, and test sets in an 80:10:10 ratio. 3108 pairs of matching edge detection maps and sketch-screenshot images were created for training, and 406 pairs were created for testing. During training, a cross-entropy loss function was used to measure the difference between the model predictions and the true labels, thereby guiding the optimization of model parameters. Training batches were constructed by calculating the cross-entropy between the predicted probability distribution and the true labels to measure the uncertainty and error of the model predictions. A smaller cross-entropy loss value indicates a closer match between the model predictions and the true labels. The optimizer updates the model parameters based on this loss, allowing the model to gradually learn more accurate predictions. By minimizing the cross-entropy loss, the model adjusts its parameters to improve performance and generalization. Simultaneously, a stochastic gradient descent strategy was used for model training to provide final test results.

2. The sketch retrieval and description method based on modality fusion network according to claim 1 is characterized in that: The network model was replaced by any model to enhance the quality of the extracted features. The entire framework was written using the PyTorch deep learning framework, and the experimental program ran on a single Nvidia GeForce RTX 3090 GPU, an 8-core Intel Xeon processor and 32GB / RAM.

Images