Method for searching similar resources or generating prototype layer based on image and related device

Abstract

The invention discloses a method and a related device for searching similar resources or generating a prototype layer based on an image, and the method comprises the steps: carrying out the color space conversion and edge detection of an interface screenshot uploaded by a user, extracting a feature vector in the interface screenshot, and enabling the feature vector to comprise the shape, color, texture and layout of a design element; constructing a resource library containing predefined interface elements, wherein the resource library comprises design elements and feature vectors thereof; searching is conducted in a resource library, a predefined interface element list with the similarity with the feature vectors exceeding a preset threshold value is searched, and the predefined interface element list comprises previews, names and categories of the design elements; predefined interface elements selected by a user are added to the design canvas, fine adjustment is conducted on style parameters of the predefined interface elements, and the style parameters comprise the size, the color, the font, the margin and the shadow. Through intelligent recommendation and fine adjustment, the time of manual search and resource adjustment is shortened, and the prototype design efficiency and design consistency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software design, and particularly to a method and related device for searching for similar resources or generating prototype layers based on images. Background Art

[0002] In the traditional prototyping process, product managers and designers often need to manually search for and select interface component resources, which is not only time-consuming but also inefficient. In addition, manually adjusting the styles of components to maintain consistency is also a cumbersome task. To improve the efficiency and quality of prototyping, some AI prototyping tools such as Uizard and Framer can automatically generate design elements according to user input. However, these tools usually lack in-depth integration of existing resources and fine-tuning capabilities.

[0003] To solve the above problems, the present invention develops a method and related device for searching for similar resources or generating prototype layers based on images, so as to reduce the time for manually searching for and adjusting resources through intelligent recommendation and fine-tuning, and improve the efficiency, consistency, and accuracy of prototyping. Summary of the Invention

[0004] In view of the above problems, a method and device for searching for similar resources or generating prototype layers based on images are proposed, so as to reduce the time for manually searching for and adjusting resources, and improve the efficiency, consistency, and accuracy of prototyping.

[0005] According to one aspect of the present invention, a method for searching for similar resources or generating prototype layers based on images is provided, including:

[0006] Performing color space conversion and edge detection on the interface screenshot uploaded by the user, and extracting the feature vectors in the interface screenshot, where the feature vectors include the shape, color, texture, and layout of the design elements;

[0007] Constructing a resource library containing predefined interface elements, where the resource library includes design elements and their feature vectors;

[0008] Searching in the resource library to find a list of predefined interface elements whose similarity to the feature vectors exceeds a preset threshold, where the list of predefined interface elements includes preview images, names, and categories of design elements;

[0009] Adding the predefined interface elements selected by the user to the design canvas, and performing fine-tuning on the style parameters of the predefined interface elements, where the style parameters include size, color, font, margin, and shadow.

[0010] In an alternative approach, the color space conversion and edge detection of the interface screenshot uploaded by the user, and the extraction of the feature vector from the interface screenshot further include:

[0011] Divide the image of the interface screenshot in the HSV color space into multiple color regions, count the number of pixels, average hue, saturation, and brightness values in each color region, and form a color feature vector;

[0012] Perform a Hough transform on the Canny edge detection result of the interface screenshot, extract the length and angle of the straight line and the radius and center coordinates of the circle, and form a shape feature vector;

[0013] Calculate the texture feature of the interface screenshot through the gray-level co-occurrence matrix, and form a texture feature vector, where the texture feature includes contrast, energy, homogeneity, and correlation;

[0014] Divide the interface screenshot into grids, count the number and type of design elements in each grid, and calculate the element correlation between adjacent grids to form a layout feature vector;

[0015] Combine the color feature vector, the shape feature vector, the texture feature vector, and the layout feature vector to form the feature vector of the interface screenshot.

[0016] In an alternative approach, the calculation formula for the similarity of the feature vectors is:

[0017] Similarity(F u ,F r )=w s ×Similarity s (S u ,S r )+w c ×

[0018] Similarity c (C u ,C r )+w t ×Similarity t (T u ,T r )+w l ×Similarity l (L u ,L r )

[0019] where F u ,F r represent the feature vector of the interface screenshot uploaded by the user and the feature vector of the predefined interface element in the resource library respectively; Su , S r respectively represent the shape feature vectors of the user-uploaded interface screenshot and the shape feature vectors of the predefined interface elements in the resource library; C u , C r respectively represent the color feature vectors of the user-uploaded interface screenshot and the color feature vectors of the predefined interface elements in the resource library; T u , T r respectively represent the texture feature vectors of the user-uploaded interface screenshot and the texture feature vectors of the predefined interface elements in the resource library; L u , L r respectively represent the layout feature vectors of the user-uploaded interface screenshot and the layout feature vectors of the predefined interface elements in the resource library; w s , w c , w t , w l respectively represent the weights of shape similarity, color similarity, texture similarity, and layout similarity; Similarity s , Similarity c , Similarity t , Similarity l are the shape similarity, color similarity, texture similarity, and layout similarity functions respectively.

[0020] In an optional manner, the calculation formula for the color similarity is:

[0021]

[0022] where H ui , S ui , V ui respectively represent the average hue, saturation, and brightness of the i-th color region of the user-uploaded interface screenshot; H ri , S ri , V ri respectively represent the average hue, saturation, and brightness of the i-th color region of the predefined interface element in the resource library; P ui represents the number of pixels in the i-th color region of the user-uploaded interface screenshot.

[0023] In an optional manner, the calculation method for the shape similarity further includes:

[0024] Normalize the shape feature vector S u of the user-uploaded interface screenshot and the shape feature vector S r of the predefined interface element in the resource library respectively;

[0025] Calculate S u and Sr Obtain the shape similarity by calculating the similarity between them;

[0026] Smooth the shape similarity through a Gaussian function.

[0027] In an alternative approach, the calculation formula for the shape similarity is:

[0028] Similarity s (S u ,S r ) = ∑(w i *Hausdorff(L ui ,L ri )) / ∑w i

[0029] where S u ,S r are the shape feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; w i is the weight of the i-th shape feature, and the weight is set according to the significance and distinctiveness of the feature; L ui ,L ri are the i-th local shape features of the user interface screenshot and the resource library element respectively, and the local shape feature is a contour segment represented by Freeman chain code; Hausdorff(.) is the Hausdorff distance function.

[0030] In an alternative approach, the predefined interface elements include buttons, text boxes, icons, navigation bars, tab bars, lists, cards, and modal boxes, where each predefined interface element contains visual features and functional attributes;

[0031] The construction of the resource library containing predefined interface elements further includes:

[0032] Preprocess each predefined interface element, and the preprocessing includes size adjustment, format conversion, and background removal;

[0033] Extract the shape feature vector, color feature vector, texture feature vector, and layout feature vector of the preprocessed predefined interface element and add them to the resource library.

[0034] In an alternative approach, before searching in the resource library, the method further includes:

[0035] If there is perspective distortion in the interface screenshot uploaded by the user, perform perspective correction to restore the image to the front view;

[0036] Among them, perspective correction uses the Hough transform to detect straight lines in the image and calculate the intersection points between the straight lines, and uses the intersection points for perspective transformation.

[0037] According to another aspect of the present application, there is provided a device for searching for similar resources or generating a prototype layer based on an image, including:

[0038] A feature vector extraction module, configured to perform color space conversion and edge detection on the interface screenshot uploaded by the user, and extract the feature vector in the interface screenshot, wherein the feature vector includes the shape, color, texture, and layout of the design element;

[0039] A resource library construction module, configured to construct a resource library including predefined interface elements, wherein the resource library includes design elements and their feature vectors;

[0040] A resource search module, configured to search in the resource library to find a list of predefined interface elements whose similarity to the feature vector exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of the design elements;

[0041] An element addition module, configured to add the predefined interface elements selected by the user to the design canvas and perform fine-tuning of the style parameters of the predefined interface elements, and the style parameters include size, color, font, margin, and shadow.

[0042] According to still another aspect of the present application, there is provided a computing device, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0043] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above method for searching for similar resources or generating a prototype layer based on an image.

[0044] The solution provided in the above embodiments of the present invention performs color space conversion and edge detection on the interface screenshots uploaded by the user, extracts feature vectors in the interface screenshots, where the feature vectors include the shape, color, texture, and layout of design elements; constructs a resource library containing predefined interface elements, where the resource library includes design elements and their feature vectors; searches in the resource library to find a list of predefined interface elements whose similarity to the feature vectors exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of design elements; adds the predefined interface elements selected by the user to the design canvas and fine-tunes the style parameters of the predefined interface elements, and the style parameters include size, color, font, margin, and shadow. Through intelligent recommendation and fine-tuning, the present invention reduces the time for manual search and adjustment of resources, and improves the efficiency of prototype design and the consistency and accuracy of design.

[0045] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above description and other purposes, features, and advantages of the present invention more obvious and understandable, the following specifically illustrates the specific embodiments of the present invention. Brief Description of the Drawings

[0046] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0047] Figure 1 A flowchart showing the method for searching for similar resources or generating prototype layers based on an image in an embodiment of the present invention is shown;

[0048] Figure 2 A functional structure diagram of the device provided in an embodiment of the present application is shown;

[0049] Figure 3 A structural diagram of a computing device provided in an embodiment of the present application is shown. Detailed Description of the Embodiments

[0050] The exemplary embodiments of the present invention will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0051] The following uses specific embodiments to elaborate in detail on the method and related device for searching for similar resources or generating prototype layers based on images proposed by the present invention.

[0052] Embodiment 1:

[0053] Figure 1 The functional structure diagram of the method for searching for similar resources or generating prototype layers based on images according to the embodiments of the present invention is shown. Specifically, as Figure 1 shown, the following steps are included:

[0054] Step S101, perform color space conversion and edge detection on the interface screenshot uploaded by the user, and extract the feature vector in the interface screenshot, where the feature vector includes the shape, color, texture, and layout of the design elements.

[0055] In this embodiment, by combining color space conversion and edge detection, the sensitivity of feature extraction to factors such as illumination changes, noise, and image quality is reduced. Even if the quality of the uploaded interface screenshot is not high, features can be extracted more accurately. Different types of interface elements have significant differences in shape, color, texture, and layout. By extracting the above differential features, different elements can be distinguished (such as the rounded corners of buttons, the white background of text boxes, the vector shapes of icons, etc.). Since the interface screenshot does not depend on a specific UI framework or design language, it can be applied to various types of interface screenshots.

[0056] Specifically, for color space conversion, since the HSV space is more in line with visual perception and can separate color information (hue) from brightness information (value), it is more convenient to extract color features. Therefore, use an image processing library (such as OpenCV, Pillow) to convert the interface screenshot uploaded by the user from the RGB color space to the HSV color space (hue, saturation, value).

[0057] For edge detection, perform edge extraction on the interface screenshot through the Canny edge detection algorithm. Among them, for the shape feature vector, perform the Hough transform on the result of the Canny edge detection to extract the parameters of straight lines and circles. It is also possible to perform contour analysis on the edge contour to obtain information such as the length, area, perimeter, and convex hull of the contour. For the color feature vector, divide the HSV image into multiple color regions, for example, divide it into regions such as red, orange, yellow, green, cyan, blue, and purple according to the hue value, and count the number of pixels, average hue, saturation, and brightness values in each region. For the texture feature vector, calculate the texture features using the gray-level co-occurrence matrix. For the layout feature vector, divide the interface screenshot into grids (for example, divide it into a 3x3 grid), count the number and types of design elements in each grid (such as buttons, text boxes, icons, etc.), and calculate the element correlation between adjacent grids (such as the probability that both adjacent grids contain buttons).

[0058] Finally, combine the color feature vector, shape feature vector, texture feature vector, and layout feature vector to form the final feature vector. For example, the feature vector is represented as:

[0059] feature_vector =

[0060] / / Color features (button blue, text box white)

[0061] 0.2, 0.7, 0.9, / / Proportion of blue area pixels, average hue, average saturation, average brightness 0.8, 0.0, 1.0, / / Proportion of white area pixels, average hue, average saturation, average brightness

[0062] / / Shape features (button rounded rectangle, text box rectangle)

[0063] 10, 50, 30, / / Radius of rounded corners of rounded rectangle, length, width

[0064] 0, 80, 40, / / Radius of rounded corners of rectangle (0), length, width

[0065] / / Texture features

[0066] 0.1, 0.8, 0.9, 0.2, / / Contrast, energy, homogeneity, correlation of button

[0067] 0.05, 0.95, 0.99, 0.9, / / Contrast, energy, homogeneity, correlation of text box

[0068] / / Layout features

[0069] 1, 0, 0, 1 / / Number of buttons in grid 1, number of text boxes in grid 1, number of buttons in grid 2,

[0070] / / Number of text boxes in grid 2 (2x2 grid)

[0071] , after extracting the feature vector, similar interface elements can be searched in the resource library.

[0072] In an alternative approach, the color space conversion and edge detection of the interface screenshot uploaded by the user, and the extraction of the feature vector from the interface screenshot further includes:

[0073] Divide the image in the HSV color space of the interface screenshot into multiple color regions, and count the number of pixels, average hue, saturation, and brightness values in each color region to form a color feature vector;

[0074] Perform the Hough transform on the Canny edge detection result of the interface screenshot, extract the length and angle of the straight line and the radius and center coordinates of the circle, and form a shape feature vector;

[0075] Calculate the texture features of the interface screenshot through the gray-level co-occurrence matrix to form a texture feature vector, where the texture features include contrast, energy, homogeneity, and correlation;

[0076] Divide the interface screenshot into grids, count the number and type of design elements in each grid, and calculate the element correlation between adjacent grids to form a layout feature vector;

[0077] Combine the color feature vector, the shape feature vector, the texture feature vector, and the layout feature vector to form the feature vector of the interface screenshot.

[0078] In this embodiment, dividing the HSV color space into multiple color regions and counting the number of pixels and the average HSV values in each region can more finely describe the color distribution of the image and improve the accuracy of color similarity matching. For example, it can distinguish between a pure blue button and a button with a blue gradient. Using the Hough transform to extract straight lines and circles and directly obtain the basic shape parameters of the interface elements can quickly identify and match UI elements such as text boxes (rectangles), buttons (rounded rectangles or circles), dividing lines (straight lines), etc. By calculating the element correlation between adjacent grids, the spatial relationship between UI elements (such as a button below a text box, an icon to the left of a text label, etc.) can be better captured.

[0079] Specifically, divide the hue (H) value in the HSV space into multiple intervals (red, orange, yellow, green, cyan, blue, purple, etc.), and each interval represents a color region. Count the number of pixels (P i ) in each color region, and calculate the average hue (H i ), average saturation (S i ), and average brightness (V i ) of all pixels in this region. The color feature vector is expressed as: [P 1 , H 1 , S 1 , V 1 , P 2 , H 2 , S 2 , V 2 ,..., P n , H n , S n , V n , where n is the number of color regions.

[0080] Use the Hough transform to detect lines and circles in the results of Canny edge detection. Among them, the Hough transform returns the length and angle of the line, and the radius and center coordinates of the circle.

[0081] Calculate the gray-level co-occurrence matrix of the image by the gray-level co-occurrence matrix method, and count the frequency of simultaneous occurrence of pixel pairs with different gray levels in the image (such as the horizontal distance is 1). Calculate the contrast, energy, homogeneity, and correlation according to the gray-level co-occurrence matrix method. Among them, the energy reflects the uniformity of the gray distribution of the image and the thickness of the texture. The homogeneity reflects the amount of local gray change in the image. The correlation reflects the similarity degree of the texture in the image.

[0082] Divide the interface screenshot into an m×n grid, and count the number of different types of UI elements in each grid. The types of UI elements include buttons, text boxes, icons, labels, etc. Optionally, use a pre-trained UI element detector (such as a deep learning-based object detection model) to automatically identify UI elements. Calculate the relevance of UI elements between adjacent grids through conditional probability (such as calculating the probability that there is a button in grid B under the condition that there is a button in grid A). The layout feature vector can be expressed as: [Grid_1_Button,Grid_1_Textbox,...,Grid_2_Button,Grid_2_Textbox,...,P(Grid_B_Button|Grid_A_Button),...], where Grid_i_Element represents the number of a certain element in the i-th grid, and P(...|...) represents the conditional probability.

[0083] For example, the user uploads an interface screenshot containing a blue rounded button and a white text box. Divide the HSV color space into 7 color regions. Among them, the number of pixels in the blue region is relatively large, the average hue is close to blue, and the average saturation and average brightness are also relatively high. The number of pixels in the white region is also relatively large, the average hue is not obvious, the average saturation is relatively low, and the average brightness is relatively high. The number of pixels in other color regions is relatively small. For the shape features, the Hough transform detects a line (the edge of the text box) and a circle (the approximate shape of the rounded button), and records the length and angle of the line, as well as the radius and center coordinates of the circle. For the texture features, the texture features (contrast, energy, homogeneity, correlation) of the text box reflect that its surface is relatively smooth and uniform. The texture features of the button reflect the existence of gradient or shadow effects on its surface. For the layout features, divide the screenshot into a 2x2 grid, the button is located in the upper left grid, and the text box is located in the lower right grid. Therefore, the number of buttons in the upper left grid is 1, the number of text boxes in the lower right grid is 1, and the number of elements in other grids is 0. The relevance of elements between adjacent grids reflects the relative positions of the button and the text box.

[0084] In an alternative approach, the calculation formula for the feature vector similarity is as follows:

[0085] Similarity(F u ,F r ) = w s ×Similarity s (S u ,S r ) + w c ×

[0086] Similarity c (C u ,C r ) + w t ×Similarity t (T u ,T r ) + w l ×Similarity l (L u ,L r )

[0087] where F u ,F r represent the feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; S u ,S r represent the shape feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; C u ,C r represent the color feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; T u ,T r represent the texture feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; L u ,L r represent the layout feature vectors of the user-uploaded interface screenshot and the predefined interface elements in the resource library respectively; w s , w c , w t , w l represent the weights of shape similarity, color similarity, texture similarity, and layout similarity respectively; Similarity s , Similarity c , Similarity t , Similarity l are the shape similarity, color similarity, texture similarity, and layout similarity functions respectively.

[0088] In this embodiment, since different features play different roles in different scenarios (for example, color is used to distinguish button types, and layout is used to identify list structures, etc.), four different visual features, namely shape, color, texture, and layout, can capture the similarity of interface elements more accurately compared to a single feature.

[0089] In an alternative approach, the calculation formula for the color similarity is as follows:

[0090]

[0091] where H ui , S ui , and V ui respectively represent the average hue, saturation, and brightness of the i-th color region of the interface screenshot uploaded by the user; H ri , S ri , and V ri respectively represent the average hue, saturation, and brightness of the i-th color region of the predefined interface element in the resource library; P ui represents the number of pixels in the i-th color region of the interface screenshot uploaded by the user.

[0092] In this embodiment, dividing the interface screenshot into multiple color regions and calculating the color similarity of each region can better handle the situation of uneven color distribution (such as identifying the gradient color or shadow included in the button through region division). Using the number of pixels as the weight makes the larger color regions have a greater impact on the similarity calculation and can better represent the overall color style of the interface. By normalizing by 3×∑P ui , the value range of the similarity is between 0 and 1, which is convenient for comparing the color similarities between different interface elements.

[0093] Specifically, divide the interface screenshot into a fixed number of grid regions, and use the K-means clustering algorithm to cluster the pixels in the interface screenshot into several color regions according to the color values. Among them, the color of the clustering center is used as the representative color of the region. Alternatively, use an image segmentation algorithm to segment the interface screenshot into regions with similar color characteristics. For each color region, calculate the average values of its hue, saturation, and brightness, ensuring that the same color space (such as HSV or HSL) is used for color conversion. Calculate the color similarity between the interface screenshot uploaded by the user and the predefined interface elements in the resource library according to the above formula. It should be noted that the number of color regions of the interface screenshot uploaded by the user and the predefined interface elements in the resource library may be different. All interface screenshots can be forced to be divided into the same number of regions, and the Hungarian algorithm is used to find the regions with the most similar colors between the two interface screenshots for matching. Among them, if the number of regions is different, zeros can be filled, and the number of regions of the interface screenshot with the smaller number of regions is selected as the benchmark, ignoring the extra regions of the other interface screenshot.

[0094] In an alternative approach, the method for calculating the shape similarity further includes:

[0095] Normalize the shape feature vector S of the interface screenshot uploaded by the user u and the shape feature vector S of the predefined interface elements in the resource library r respectively;

[0096] Calculate the similarity between S u and S r to obtain the shape similarity;

[0097] Smooth the shape similarity through a Gaussian function.

[0098] In this embodiment, Gaussian smoothing allows a certain degree of shape deformation (such as slight rotation, scaling, or distortion) without causing a significant decrease in similarity. By Gaussian smoothing, the similarity values are aggregated in similar regions, thereby further improving the matching accuracy.

[0099] In an alternative approach, the formula for calculating the shape similarity is:

[0100] Similarity s (S u , S r ) = ∑(w i * Hausdorff(L ui , L ri )) / ∑w i

[0101] where S u , S rThey are the shape feature vectors of the user-uploaded interface screenshots and the shape feature vectors of the predefined interface elements in the resource library; w i is the weight of the i-th shape feature, and the weight is set according to the significance and distinctiveness of the feature; L ui ,L ri are the i-th local shape features of the user interface screenshot and the resource library element respectively. The local shape feature is a contour segment represented by Freeman chain code. Hausdorff(.) is the Hausdorff distance function.

[0102] In this embodiment, by using local shape features, compared with global shape features, even if part of the shape is occluded or deformed, other local features can still be used for matching. Therefore, the robustness to occlusion, deformation, and noise is improved, and complex shapes (since complex shapes can be decomposed into multiple simple local shape features) can be better processed. Freeman chain code describes the directional changes of the contour. Compared with directly storing the contour point coordinates, Freeman chain code can represent the contour more compactly. Hausdorff distance measures the maximum distance between two point sets and does not require establishing a point correspondence between two shapes. Therefore, it is sensitive to local differences and can effectively detect the subtle differences of shapes.

[0103] Specifically, use a contour extraction algorithm (such as findContours in OpenCV) to extract the contour of the image. Use a fixed-length segmentation or a segmentation method based on curvature change to divide the extracted contour into multiple segments. Use Freeman chain code to represent each contour segment. Among them, Freeman chain code is a coding method that uses numbers to represent the contour direction. For example, 0 represents to the right, 1 represents to the upper right, and 2 represents upward. For each local shape feature, calculate the Hausdorff distance between the user interface screenshot and the resource library element. Among them, for each point p in the local shape feature L ui of the user interface screenshot, calculate its shortest distance d(p, L ri ) to all points in the local shape feature L ri of the resource library element, calculate the maximum value h(L ui , L ri ) = max(d(p, L ri ))), where p belongs to L ui . Similarly, calculate h(L ri , L ui ), and the Hausdorff distance is Hausdorff(L ui , L ri ) = max(h(L ui , L ri ), h(L ri , Lui ))。Set weights according to the significance and distinctiveness of features. Among them, significance refers to the frequency and intensity of features appearing in the image (for example, edges that are clearer and longer contour segments are more significant). Distinctiveness refers to the ability of features to distinguish different shapes (for example, sharp corners are more distinctive than smooth curves).

[0104] Step S102: Construct a resource library containing predefined interface elements, where the resource library includes design elements and their feature vectors.

[0105] In this embodiment, the predefined interface elements are sourced from multiple channels, including existing design specification libraries (imported from existing design specification documents or component libraries), open-source UI libraries (open-source UI component libraries), and self-created UI elements. Classify and name the predefined interface elements.

[0106] In this embodiment, the predefined interface elements include buttons, text boxes, icons, navigation bars, tab bars, lists, cards, and modal boxes. Each predefined interface element contains visual features and functional attributes;

[0107] The construction of the resource library containing predefined interface elements further includes:

[0108] Preprocess each predefined interface element, and the preprocessing includes size adjustment, format conversion, and background removal;

[0109] Extract shape feature vectors, color feature vectors, texture feature vectors, and layout feature vectors from the preprocessed predefined interface elements and add them to the resource library.

[0110] Step S103: Search in the resource library to find a list of predefined interface elements whose similarity to the feature vector exceeds a preset threshold. The list of predefined interface elements includes preview images, names, and categories of design elements.

[0111] In this embodiment, similarity search is performed through feature vectors (including various features such as color, shape, texture, semantic information, etc.). Compared with keyword matching, it realizes a more intelligent interface search. For example, even if the element names are different, they can be found if they are visually similar.

[0112] In an alternative approach, before searching in the resource library, the method further includes:

[0113] If there is perspective distortion in the interface screenshot uploaded by the user, perform perspective correction to restore the image to a front view;

[0114] Among them, for perspective correction, use the Hough transform to detect lines in the image and calculate the intersections between the lines, and perform perspective transformation using the intersections.

[0115] In this embodiment, it is less sensitive to factors such as shooting angle and shooting distance. Users do not need to strictly shoot interface screenshots at a specific angle or distance, which reduces the usage threshold for users. The Hough transform converts straight lines in the image space into points in the parameter space, and then counts the points in the parameter space to find the most likely straight lines. According to the characteristics of the straight lines such as length and angle, the straight lines representing the interface edges are filtered. For example, straight lines that are too short or whose angles are too close to horizontal or vertical are filtered. The intersection points between the filtered straight lines are calculated by linear algebra methods, and the intersection points located within the image range are filtered. The goal is to find the intersection points representing the four corner points of the interface, and perspective transformation is performed using the intersection points.

[0116] Step S104: Add the pre-defined interface elements selected by the user to the design canvas, and finely adjust the style parameters of the pre-defined interface elements. The style parameters include size, color, font, margin, and shadow.

[0117] In this embodiment, an access entry to the interface element library is provided in the design tool. The pre-defined interface elements are added to the design canvas by dragging, and parameters such as the size, color, font, margin, and shadow of the elements are adjusted through the operation panel.

[0118] The solution provided in the above embodiments of the present invention performs color space conversion and edge detection on the interface screenshots uploaded by the user, extracts feature vectors in the interface screenshots, where the feature vectors include the shape, color, texture, and layout of the design elements; constructs a resource library containing pre-defined interface elements, where the resource library includes design elements and their feature vectors; searches in the resource library to find a list of pre-defined interface elements whose similarity to the feature vectors exceeds a preset threshold. The list of pre-defined interface elements includes preview images, names, and categories of the design elements; adds the pre-defined interface elements selected by the user to the design canvas, and finely adjusts the style parameters of the pre-defined interface elements. The style parameters include size, color, font, margin, and shadow. The present invention reduces the time for manually searching and adjusting resources through intelligent recommendation and fine-tuning, and improves the efficiency of prototype design and the consistency and accuracy of the design.

[0119] Embodiment 2:

[0120] Figure 2 Shows a functional structure diagram of the method and device for searching for similar resources or generating prototype layers based on images according to an embodiment of the present invention. As Figure 2 shown, the device includes:

[0121] The feature vector extraction module 201 is configured to perform color space conversion and edge detection on the interface screenshot uploaded by the user, and extract the feature vectors in the interface screenshot, where the feature vectors include the shape, color, texture, and layout of the design elements;

[0122] The resource library construction module 202 is configured to construct a resource library including predefined interface elements, where the resource library includes design elements and their feature vectors;

[0123] The resource search module 203 is configured to search in the resource library to find a list of predefined interface elements whose similarity to the feature vectors exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of the design elements;

[0124] The element addition module 204 is configured to add the predefined interface elements selected by the user to the design canvas and perform fine-tuning on the style parameters of the predefined interface elements, where the style parameters include size, color, font, margin, and shadow.

[0125] The solution provided in the above embodiments of the present invention performs color space conversion and edge detection on the interface screenshot uploaded by the user, extracts the feature vectors in the interface screenshot, where the feature vectors include the shape, color, texture, and layout of the design elements; constructs a resource library including predefined interface elements, where the resource library includes design elements and their feature vectors; searches in the resource library to find a list of predefined interface elements whose similarity to the feature vectors exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of the design elements; adds the predefined interface elements selected by the user to the design canvas and performs fine-tuning on the style parameters of the predefined interface elements, where the style parameters include size, color, font, margin, and shadow. The present invention reduces the time for manually searching and adjusting resources through intelligent recommendation and fine-tuning, and improves the efficiency of prototype design and the consistency and accuracy of design.

[0126] Embodiment Three:

[0127] Figure 3 Fig. shows a schematic structural diagram of an embodiment of the computing device of the present application, and the specific implementation of the computing device is not limited in the specific embodiments of the present application.

[0128] As Figure 3 shown, the computing device may include: a processor 302, a communication interface 304, a memory 306, and a communication bus 308.

[0129] Among them: The processor 302, the communication interface 304, and the memory 306 communicate with each other through the communication bus 308. The communication interface 304 is used to communicate with network elements of other devices such as clients or other servers. The processor 302 is used to execute the program 310, and specifically can execute the relevant steps in the above method embodiments for calculating devices to search for similar resources based on images or generate prototype layers.

[0130] Specifically, the program 310 may include program code, and the program code includes computer operation instructions.

[0131] The processor 302 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the computing device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0132] The memory 306 is used to store the program 310. The memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0133] The program 310 is specifically used to cause the processor 302 to perform the following operations:

[0134] Perform color space conversion and edge detection on the interface screenshot uploaded by the user, and extract the feature vector in the interface screenshot, where the feature vector includes the shape, color, texture, and layout of the design element;

[0135] Build a resource library containing predefined interface elements, where the resource library includes design elements and their feature vectors;

[0136] Search in the resource library to find a list of predefined interface elements whose similarity to the feature vector exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of design elements;

[0137] Add the predefined interface elements selected by the user to the design canvas, and fine-tune the style parameters of the predefined interface elements, where the style parameters include size, color, font, margin, and shadow.

[0138] Optionally, the program 310 causes the processor to perform the following operations:

[0139] Divide the image in the HSV color space of the interface screenshot into multiple color regions, count the number of pixels, average hue, saturation, and brightness values in each color region, and form a color feature vector;

[0140] Perform Hough transform on the Canny edge detection result of the interface screenshot, extract the length and angle of the straight line and the radius and center coordinates of the circle, and form a shape feature vector;

[0141] Calculate the texture feature of the interface screenshot through the gray-level co-occurrence matrix to form a texture feature vector, where the texture feature includes contrast, energy, homogeneity, and correlation;

[0142] Divide the interface screenshot into grids, count the number and type of design elements in each grid, and calculate the element correlation between adjacent grids to form a layout feature vector;

[0143] Combine the color feature vector, the shape feature vector, the texture feature vector, and the layout feature vector to form the feature vector of the interface screenshot.

[0144] Optionally, the calculation formula for the feature vector similarity is:

[0145] Similarity(F u ,F r )=w s ×Similarity s (S u ,S r )+w c ×

[0146] Similarity c (C u ,C r )+w t ×Similarity t (T u ,T r )+w l ×Similarity l (L u ,L r )

[0147] Where F u ,F r represent the feature vectors of the interface screenshot uploaded by the user and the feature vectors of the predefined interface elements in the resource library respectively; S u ,S r represent the shape feature vectors of the interface screenshot uploaded by the user and the shape feature vectors of the predefined interface elements in the resource library respectively; C u ,C rrespectively represent the color feature vectors of the user-uploaded interface screenshot and the color feature vectors of the predefined interface elements in the resource library; T u ,T r respectively represent the texture feature vectors of the user-uploaded interface screenshot and the texture feature vectors of the predefined interface elements in the resource library; L u ,L r respectively represent the layout feature vectors of the user-uploaded interface screenshot and the layout feature vectors of the predefined interface elements in the resource library; w s ,w c ,w t ,w l respectively represent the weights of shape similarity, color similarity, texture similarity, and layout similarity; Similarity s ,Similarity c ,Similarity t ,Similarity l are respectively the shape similarity, color similarity, texture similarity, and layout similarity functions.

[0148] Optionally, the calculation formula for the color similarity is:

[0149]

[0150] where H ui ,S ui ,V ui respectively represent the average hue, saturation, and brightness of the i-th color region of the user-uploaded interface screenshot; H ri ,S ri ,V ri respectively represent the average hue, saturation, and brightness of the i-th color region of the predefined interface element in the resource library; P ui represents the number of pixels in the i-th color region of the user-uploaded interface screenshot.

[0151] Optionally, the program 310 causes the processor to perform the following operations:

[0152] Normalize the shape feature vector S u of the user-uploaded interface screenshot and the shape feature vector S r of the predefined interface element in the resource library respectively;

[0153] Calculate the similarity between S u and S r to obtain the shape similarity;

[0154] Smooth the shape similarity through a Gaussian function.

[0155] Optionally, the calculation formula for the shape similarity is as follows:

[0156] Similarity s (S u ,S r )=∑(w i *Hausdorff(L ui ,L ri )) / ∑w i

[0157] where S u ,S r are the shape feature vectors of the user-uploaded interface screenshot and the pre-defined interface elements in the resource library respectively; w i is the weight of the i-th shape feature, and the weight is set according to the significance and distinctiveness of the feature; L ui ,L ri are the i-th local shape features of the user interface screenshot and the resource library element respectively, and the local shape feature is a contour segment represented by Freeman chain code; Hausdorff(.) is the Hausdorff distance function.

[0158] Optionally, the pre-defined interface elements include buttons, text boxes, icons, navigation bars, tab bars, lists, cards, and modal boxes, where each pre-defined interface element includes visual features and functional attributes.

[0159] Optionally, the program 310 causes the processor to perform the following operations:

[0160] The building of the resource library containing pre-defined interface elements further includes:

[0161] Preprocessing each pre-defined interface element, where the preprocessing includes size adjustment, format conversion, and background removal;

[0162] Extracting the shape feature vector, color feature vector, texture feature vector, and layout feature vector of the preprocessed pre-defined interface element and adding them to the resource library.

[0163] Optionally, before searching in the resource library, the program 310 causes the processor to perform the following operations:

[0164] If there is perspective distortion in the interface screenshot uploaded by the user, perform perspective correction to restore the image to a frontal view;

[0165] where perspective correction uses the Hough transform to detect the straight lines in the image and calculate the intersection points of the straight lines, and uses the intersection points for perspective transformation.

[0166] The solution provided in the above embodiments of the present invention performs color space conversion and edge detection on the interface screenshots uploaded by the user, extracts the feature vectors in the interface screenshots, where the feature vectors include the shapes, colors, textures, and layouts of design elements; constructs a resource library containing predefined interface elements, where the resource library includes design elements and their feature vectors; searches in the resource library to find a list of predefined interface elements whose similarity to the feature vectors exceeds a preset threshold, and the list of predefined interface elements includes preview images, names, and categories of design elements; adds the predefined interface elements selected by the user to the design canvas and fine-tunes the style parameters of the predefined interface elements, where the style parameters include size, color, font, margin, and shadow. Through intelligent recommendation and fine-tuning, the present invention reduces the time for manual search and adjustment of resources, and improves the efficiency of prototype design and the consistency and accuracy of design.

[0167] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. The structure required to construct such systems will be apparent from the above description. In addition, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the present invention described herein can be implemented using various programming languages, and the description of the specific language above is for the purpose of disclosing the best mode of the present invention.

[0168] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0169] Similarly, it should be understood that, in order to streamline the present invention and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all of the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0170] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0171] In addition, those skilled in the art can understand that although some of the embodiments herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means that it is within the scope of the present invention and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0172] Each component embodiment of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or device program (such as a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0173] It should be noted that the above embodiments are illustrative of the present invention and not restrictive thereof, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for searching similar resources or generating prototype layers based on an image, characterized in that: include: Perform color space conversion and edge detection on the interface screenshots uploaded by users, and extract feature vectors from the interface screenshots, wherein the feature vectors include the shape, color, texture, and layout of the design elements; Constructing a resource library containing predefined interface elements, wherein the resource library includes design elements and feature vectors thereof; Searching the resource library to find a list of predefined interface elements whose similarity to the feature vector exceeds a preset threshold, the list of predefined interface elements including a preview image, name and category of the design element; The predefined interface element selected by the user is added to the design canvas, and the style parameters of the predefined interface element are fine-tuned, and the style parameters include size, color, font, margin and shadow.

2. The method for searching similar resources or generating prototype layers based on images according to claim 1, characterized in that: The performing color space conversion and edge detection on the interface screenshot uploaded by the user and extracting the feature vector in the interface screenshot further includes: Divide the image of the HSV color space of the interface screenshot into multiple color regions, count the number of pixels, average hue, saturation and brightness value in each color region, and form a color feature vector; Performing Hough transform on the Canny edge detection result of the interface screenshot to extract the length and angle of the straight line and the radius and center coordinates of the circle to form a shape feature vector; Calculating the texture features of the interface screenshot by using a gray level co-occurrence matrix to form a texture feature vector, wherein the texture features include contrast, energy, homogeneity and correlation; Dividing the interface screenshot into grids, counting the number and type of design elements in each grid, calculating the element correlation between adjacent grids, and forming a layout feature vector; The color feature vector, the shape feature vector, the texture feature vector and the layout feature vector are combined to form a feature vector of the interface screenshot.

3. The method for searching similar resources or generating prototype layers based on images according to claim 1, characterized in that: The calculation formula of the feature vector similarity is: Similarity(F u ,F r )=w s ×Similarity s (S u ,S r )+w c × Similarity c (C u ,C r )+w t ×Similarity t (T u ,T r )+w l ×Similarity l (L u ,L r ) Among them, F u ,F r They represent the feature vectors of the user-uploaded interface screenshots and the feature vectors of the predefined interface elements in the resource library; S u ,S r They represent the shape feature vector of the user-uploaded interface screenshot and the shape feature vector of the predefined interface element in the resource library respectively; C u ,C r They represent the color feature vector of the user-uploaded interface screenshot and the color feature vector of the predefined interface elements in the resource library respectively; T u ,T r They represent the texture feature vector of the user-uploaded interface screenshot and the texture feature vector of the predefined interface elements in the resource library respectively; L u ,L r They represent the layout feature vector of the user-uploaded interface screenshot and the layout feature vector of the predefined interface elements in the resource library respectively; w s , w c , w t , w l Represents the weights of shape similarity, color similarity, texture similarity, and layout similarity respectively; Similarity s , Similarity c , Similarity t , Similarity l They are shape similarity, color similarity, texture similarity and layout similarity functions respectively.

4. The method for searching similar resources or generating prototype layers based on images according to claim 3, characterized in that: The calculation formula of the color similarity is: Among them, H ui , S ui , V ui Respectively represent the average values ​​of hue, saturation and brightness of the i-th color area of ​​the user upload interface screenshot; H ri , S ri , V ri Respectively represent the average values ​​of hue, saturation, and brightness of the i-th color area of ​​the predefined interface element in the resource library; P ui Indicates the number of pixels in the i-th color area of ​​the screenshot of the user upload interface.

5. The method for searching similar resources or generating prototype layers based on images according to claim 3, characterized in that: The method for calculating the shape similarity further comprises: The shape feature vector S of the screenshot of the user upload interface u and the shape feature vector S of the predefined interface elements in the resource library r Normalization is performed separately; Calculate S u and S r similarity between them, obtaining the shape similarity; The shape similarity is smoothed by a Gaussian function.

6. The method for searching similar resources or generating prototype layers based on images according to claim 5, characterized in that: The calculation formula of the shape similarity is: Similarity s (S u ,S r )=∑(w i *Hausdorff(L ui ,L ri )) / ∑w i Among them, S u ,S r are the shape feature vectors of the screenshots of the interface uploaded by the user and the shape feature vectors of the predefined interface elements in the resource library; i is the weight of the i-th shape feature, and the weight is set according to the significance and discrimination of the feature; L ui ,L ri are the i-th local shape features of the user interface screenshot and resource library element respectively. The local shape feature is a contour fragment represented by the Freeman chain code; Hausdorff(.) is the Hausdorff distance function.

7. The method for searching similar resources or generating prototype layers based on images according to claim 1, characterized in that: The predefined interface elements include buttons, text boxes, icons, navigation bars, tab bars, lists, cards, and modal boxes, wherein each predefined interface element includes visual features and functional attributes; The step of constructing a resource library containing predefined interface elements further includes: Preprocessing each predefined interface element, wherein the preprocessing includes resizing, format conversion, and background removal; The shape feature vector, color feature vector, texture feature vector and layout feature vector are extracted from the preprocessed predefined interface elements and added to the resource library.

8. The method for searching similar resources or generating prototype layers based on images according to claim 1, characterized in that: Before searching in the resource library, the method further includes: If the interface screenshot uploaded by the user has perspective distortion, perspective correction is performed to restore the image to the frontal perspective; The perspective correction uses Hough transform to detect straight lines in the image and calculate the intersection points between the straight lines, and uses the intersection points to perform perspective transformation.

9. A device for searching similar resources or generating prototype layers based on images, characterized in that: include: A feature vector extraction module is used to perform color space conversion and edge detection on the interface screenshots uploaded by users, and extract feature vectors from the interface screenshots, wherein the feature vectors include the shape, color, texture and layout of the design elements; A resource library construction module, used to construct a resource library containing predefined interface elements, wherein the resource library includes design elements and their feature vectors; A resource search module, used to search the resource library to find a list of predefined interface elements whose similarity with the feature vector exceeds a preset threshold, wherein the list of predefined interface elements includes a preview image, name and category of the design element; The element adding module is used to add the predefined interface element selected by the user to the design canvas and fine-tune the style parameters of the predefined interface element, wherein the style parameters include size, color, font, margin and shadow.

10. A computing device for searching for similar resources or generating a prototype layer based on an image, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for searching for similar resources based on an image or generating a prototype layer as described in any one of claims 1 to 8.

Images