An interactive UI design search method and system based on a convolutional auto-encoding network

Abstract

The application discloses an interactive UI design search method and system based on a convolutional auto-encoding network, which comprises the following steps: obtaining a Rico data set and performing convolutional neural network analysis to obtain reference feature data; collecting user demand and judging the user demand to obtain a user demand data type; if the user demand data is a frame line graph type, performing frame line graph UI search matching on the user demand; if the user demand data is a color type, performing color UI search matching on the user demand; and displaying the corresponding UI page obtained through the matching; the frame line graph UI search is matching the frame line graph of the UI page in the reference feature data according to the size and position relationship of the frame line graph drawn by the user; the color UI search is matching the color histogram in the reference feature data through the four-ary tree algorithm and the color, position area and size of the color graph drawn by the user. The application has the effects of improving search efficiency and matching accuracy.

Description

Technical Field

[0001] This application relates to the field of interactive search, and in particular to an interactive UI design search method and system based on convolutional autoencoder networks. Background Technology

[0002] Convolutional autoencoders (CAs) are neural networks that copy input to output. They compress input information into a hidden space representation, then reconstruct this representation during reconstruction and output the result. CAs can preserve image details while also repairing coded artifacts. UI design is a crucial part of software development, effectively enhancing user experience and product benefits. However, current UI search engines mostly rely on keywords and image similarity, making it difficult for UI designers to quickly find similar UIs when searching for a specific type. This limits the designer's experience and makes it difficult to describe complex visual semantics, indicating room for improvement. Summary of the Invention

[0003] To improve search efficiency and matching accuracy, this application provides an interactive UI design search method and system based on convolutional autoencoder networks.

[0004] Firstly, this application provides an interactive UI design search method based on convolutional autoencoder networks, employing the following technical solution:

[0005] An interactive UI design search method based on convolutional autoencoders, comprising:

[0006] Obtain the Rico dataset and analyze and process the UI data in the Rico dataset to obtain the wireframe dataset and color dataset;

[0007] Image feature extraction of wireframe datasets is performed using convolutional neural networks to obtain reference image features;

[0008] A color histogram is constructed from the color dataset, and color features are extracted from the color histogram to obtain reference histogram data.

[0009] The reference histogram data is weighted to obtain reference weighted data;

[0010] Collect user needs and output user need data; the user need data includes user need type, user weight data, and user need data.

[0011] The user requirement type is determined. If the user requirement data is a frame diagram type, then image feature extraction is performed on the user requirement data to obtain user image feature data.

[0012] A KNN search is performed on the reference image feature data and the user image feature data to obtain the first matching result;

[0013] Based on user weight data, the similarity between user image feature data and reference image feature data included in the first matching result is calculated to output first similarity data; and the first similarity data is compared with a preset first standard value to filter out the first matching result whose first similarity data is greater than the first standard value, and output the second matching result.

[0014] The second matching result is displayed, and user behavior is collected to obtain feedback signals from users regarding their preferred UI pages.

[0015] If a feedback signal exists, the reference image feature data is further filtered based on the feedback to obtain a third matching result. An interactive genetic algorithm is then used to process the third matching result to obtain a fourth matching result, which is then displayed until the user is satisfied with the displayed result.

[0016] If the user's demand type is color-based, then color features are extracted from the user demand data to obtain user histogram data, and the user histogram data is weighted to obtain user weighted data.

[0017] The similarity between user-weighted data and reference-weighted data is calculated to obtain the second similarity data;

[0018] The second similarity data is compared with the preset second standard value to filter out the first color matching results and display them. At the same time, user feedback data is collected and rematched based on the feedback data to filter out the second color matching results and display them, until the user is satisfied with the display results.

[0019] Preferably, the UI page color information in the color dataset is converted from RGB to HSV to obtain the HSV color space;

[0020] Non-uniform quantization is performed on the HSV color space to gradually change the hue H in the HSV color space from 0° to 360°, dividing it into 7 unequal parts, and outputting the color subspace.

[0021] The color subspace is computed to obtain a color histogram that represents the distribution of each color component, and color features are extracted based on the color histogram.

[0022] Preferably, the UI page in the color dataset is divided into quadtrees based on the quadtree algorithm to obtain page sub-regions;

[0023] Extract the color features from the sub-regions of the page to obtain the color histogram of the selected area;

[0024] Store the color histogram of the selected area into the corresponding node; at the same time, divide the sub-region of the page into quadtrees again, and build color histograms and extract and store the corresponding color features of the divided areas. Repeat the above steps until the UI page is indivisible or the divided area contains only one pixel.

[0025] A quadtree relationship is established for the color feature data of each node, so that a UI data index can be created for the UI data.

[0026] Preferably, the color data on the y-axis of each selected area's color histogram and the peak y-axis data of the color histogram of the entire UI page are obtained;

[0027] The ratio of the y-axis data and the y-axis peak data is calculated to obtain the color peak percentage;

[0028] The area of ​​each sub-region of the page is calculated and its ratio to the area of ​​the entire UI page is calculated to obtain the area percentage of the selected area.

[0029] The color parameters of each page sub-region are obtained by multiplying the color peak percentage and the selected area percentage of each sub-region.

[0030] Preferably, if the user's requirement is a color type, then the number and types of colors of the lines or graphics drawn by the user are obtained;

[0031] Generate a corresponding axisymmetric bounding box selection area based on the color type;

[0032] The process of generating a corresponding axisymmetric bounding box selection area based on color type specifically includes:

[0033] Obtain the area coordinates of the lines or shapes drawn by the user to get the maximum and minimum values ​​of the horizontal axis and the vertical axis, namely Xmax, Xmin, Ymax, and Ymin.

[0034] Group Xmax, Xmin, Ymax, and Ymin to obtain Pmin=[Xmin, Ymin] and Pmax=[Xmax, Ymax]. Here, Pmin is the set of minimum values ​​of the two coordinate axes, which is the coordinates of the top-left vertex of the bounding box selection area; Pmax is the set of maximum values ​​of the two coordinate axes, which is the coordinates of the bottom-right vertex.

[0035] The geometric center is calculated based on Xmax, Xmin, Ymax, and Ymin to obtain the geometric center data, i.e., c = (Pmin + Pmax) / 2.

[0036] Based on Pmin, Pmax, and c, determine the location and size of the bounding box selection area.

[0037] Preferably, a KNN search is performed on the first matching result to obtain a UI page with a similarity greater than the second standard value, which is then displayed as the second matching result, and the second matching result is used as the initial population.

[0038] Users evaluate the second matching results based on their own needs, select their preferred UI pages, and output information about their preferred UI pages.

[0039] Based on Euclidean distance, calculate the similarity between the desired UI page information and the UI data to obtain the individual fitness of each UI page in the UI data, and output the individual fitness set;

[0040] The individual fitness set is screened to obtain the fitness screening results;

[0041] The UI search results are filtered based on the fitness selection algorithm and the second matching results to obtain the fourth matching result, which is then displayed. The above steps are repeated until the maximum number of iterations is reached or the user is satisfied.

[0042] Preferably, the corresponding region data in the user weight data is obtained based on the user weight data, and the reference image feature data corresponding to the first matching result is matched according to the region data;

[0043] The similarity of the reference image feature data and the user image feature data is compared to determine the similarity data of each reference image feature data; the similarity comparison is to calculate the Euclidean distance between the reference image feature data of the first matching result and the user image feature data to obtain the Euclidean distance result and use the Euclidean distance result as the similarity between the wireframes.

[0044] Based on user weight data, the similarity data of each reference region is weighted and calculated to obtain the overall similarity; the weight calculation is to perform a dot product calculation on the user weight data and the similarity data for each region to obtain the overall similarity.

[0045] The overall similarity is compared with the first standard value to filter out the second matching results.

[0046] Preferably, based on the Rico dataset, statistical analysis is performed on the UI data to obtain a wireframe unit set;

[0047] The UI page is wireframed based on the wireframe unit set to obtain the wireframe dataset;

[0048] The wireframe rendering process involves obtaining the component data of each UI page based on the UI data. The component data includes component type and coordinates.

[0049] Create a blank page of the same size as the UI page;

[0050] Match the constituent data based on the wireframe unit set to obtain the corresponding wireframe component;

[0051] Based on the coordinate information in the constituent data, the wireframe components are placed in the same position on the blank page to obtain the UI wireframe diagram, i.e., the wireframe dataset.

[0052] Preferably, the user draws the user UI requirements based on the wireframe unit set to obtain the user wireframe diagram;

[0053] Regional data is obtained by dividing the user's wireframe into regions;

[0054] Weights are assigned to user wireframes based on regional data to obtain user weight data for different regions.

[0055] Secondly, this application provides an interactive UI design search system based on convolutional autoencoder networks, employing the following technical solution:

[0056] An interactive UI design search system based on convolutional autoencoder networks, comprising:

[0057] The Rico data module includes a Rico data acquisition unit and a Rico data analysis unit. The Rico data acquisition unit is used to collect and store designer tools and UI data to obtain a Rico dataset. The Rico data analysis unit is signal-connected to the Rico data acquisition unit and is used to receive the Rico dataset, perform wireframe analysis on the Rico dataset to obtain a wireframe dataset, extract color features from the Rico dataset to obtain a color histogram, and perform weighted calculations based on the color histogram to obtain reference color parameters.

[0058] The user requirement acquisition module includes a wireframe acquisition unit and a color acquisition unit. The wireframe acquisition unit is used to acquire wireframe diagrams drawn by users based on wireframe unit sets and user weight settings for different regions, and output requirement wireframe data. The requirement wireframe data includes wireframe diagram data and user weight data.

[0059] The color acquisition unit is used to acquire the colors drawn by the user, determine the region of the colors drawn by the user according to the axisymmetric bounding box, and output the required color data.

[0060] The data processing module includes a wireframe data processing unit and a color data processing unit. The wireframe data processing unit is signal-connected to the Rico data analysis unit and the wireframe acquisition unit, and is used to receive the wireframe dataset and the required wireframe data, and to extract image features from the wireframe dataset and the required wireframe data to obtain corresponding image feature data, namely reference image feature data and user image feature data. The user image data is then matched with the reference image feature data using KNN search to obtain matching data.

[0061] The color data processing unit is signal-connected to the color acquisition unit and is used to receive the required color data, perform color feature extraction on the required color data to obtain a required color histogram, and perform weighted calculation on the required color histogram to obtain the required color parameters.

[0062] The comparison feedback module includes a data comparison unit and a user feedback unit; the data comparison unit is signal-connected to the Rico data analysis unit, the color data processing unit and the wireframe data processing unit, and is used to receive the reference color parameters, the required color parameters and matching data and perform similarity comparison to filter out the display result data and display it;

[0063] The user feedback unit is signal-connected to the data comparison unit, and is used to receive the user's selected preferred UI page in the display result data, analyze the UI page to obtain the analysis result, output the analysis result to the data processing unit, and perform re-matching based on the analysis result.

[0064] In summary, this application includes at least one of the following beneficial technical effects:

[0065] By acquiring the Rico dataset to obtain a large amount of UI data, and analyzing and processing this UI data, the same UI page can be transformed into two datasets: wireframe data and color data. Therefore, corresponding wireframe and color datasets are generated from the Rico dataset. Image feature extraction is performed on the wireframe dataset using a convolutional neural network to obtain reference image features. Color feature extraction and color histogram construction are performed on the color dataset to obtain reference histogram data. Weighted calculations are then performed on the reference histogram data to obtain weighted reference data, which serves as reference data for color matching. User requirements are collected; if the user-drawn pattern is a wireframe drawing, a wireframe UI retrieval step is performed; if the user-drawn pattern is a color drawing... The image then undergoes a color UI retrieval step, providing users with multiple retrieval methods to meet various user needs. The frame line UI retrieval step involves extracting image features from the user's frame line drawing to obtain user image feature data. Based on this user image feature data, a KNN search is performed on reference image feature data to obtain all reference image feature data containing the user's image feature data, which serves as the first matching result. According to the user's weight settings for the drawn user frame line drawing, the first matching result is then re-matched with the user image feature data to obtain first similarity data. This first similarity data is compared with a first standard value to further filter out the display results, i.e., the second matching result. The second matching result is displayed. During the display process, the user can select a satisfactory UI page from the second matching result and provide feedback. The feedback UI page is analyzed, and the reference image feature data is matched and filtered to obtain the third matching result. The third matching result is then processed by an interactive genetic algorithm to obtain the fourth matching result, which is then displayed. The above steps are repeated until the user is satisfied. The color UI retrieval step involves extracting color features from the user's color map to obtain the corresponding user histogram data. The user histogram data is weighted to obtain user weighted data. The similarity between the user weighted data and the reference weighted data is calculated to obtain the second similarity. The system compares the second similarity data with the second standard value to filter the first color matching results, which are then displayed. Users can select and provide feedback on the displayed UI pages they find satisfactory. The feedback data is then analyzed and re-matched to filter and display the second color matching results. This process is repeated until the user is satisfied. Using both wireframes and color maps, the system provides a simpler and more direct understanding of the user's ideas regarding UI layout and color, resulting in high accuracy in interpreting user semantics. Furthermore, the interactive genetic algorithm ensures that each feedback-based re-match yields a UI page that more closely matches the user's desired UI, improving efficiency.

[0066] By converting the color information of the UI page from RGB to HSV to obtain the HSV color space, the displayed colors are more consistent with how the human visual system understands color. At the same time, the HSV color space is non-uniformly quantized to select a small number of representative colors, thereby reducing computational complexity and improving work efficiency.

[0067] By comprehensively utilizing the quadtree algorithm to divide and statistically analyze the UI page, corresponding sub-region color histograms are obtained. These sub-region color histograms are then stored on the corresponding response nodes, ensuring that each color pixel on the UI page corresponds to a color histogram. This allows for quick determination of whether a user-drawn color map contains the specified color feature by leveraging the color histogram information on the nodes, thus accelerating the matching process and improving efficiency. Attached Figure Description

[0068] Figure 1 This embodiment mainly illustrates the steps of the interactive UI design search method based on convolutional autoencoder networks;

[0069] Figure 2 This embodiment mainly illustrates the flowchart of the sub-steps of S300 in the interactive UI design search method based on convolutional autoencoder networks;

[0070] Figure 3 This embodiment mainly illustrates the weighted calculation steps in the interactive UI design search method based on convolutional autoencoders;

[0071] Figure 4 This embodiment mainly illustrates the sub-step flowchart of S500 in the interactive UI design search method based on convolutional autoencoder networks;

[0072] Figure 5 This embodiment mainly illustrates the sub-step flowchart of S1000 in the interactive UI design search method based on convolutional autoencoder networks;

[0073] Figure 6 This embodiment mainly illustrates the sub-step flowchart of S800 in the interactive UI design search method based on convolutional autoencoder networks;

[0074] Figure 7 This embodiment mainly illustrates the flowchart of the sub-steps of S100 in the interactive UI design search method based on convolutional autoencoder networks;

[0075] Figure 8 This embodiment mainly illustrates the flowchart of the steps for calculating user weight data in the interactive UI design search method based on convolutional autoencoder networks;

[0076] Figure 9 This embodiment mainly illustrates the module block diagram of an interactive UI design search system based on a convolutional autoencoder network.

[0077] Figure reference numerals: 1. Rico data module; 11. Rico data acquisition unit; 12. Rico data analysis unit; 2. User requirement acquisition module; 21. Wireframe acquisition unit; 22. Color acquisition unit; 3. Data processing module; 31. Wireframe data processing unit; 32. Color data processing unit; 4. Comparison and feedback module; 41. Data comparison unit; 42. User feedback unit. Detailed Implementation

[0078] The following is in conjunction with the appendix Figure 1-9 This application will be described in further detail.

[0079] This application discloses an interactive UI design search method and system based on convolutional autoencoder networks.

[0080] Example: Figure 1 As shown, the present invention provides an interactive UI design search method based on convolutional autoencoders, comprising:

[0081] S100: Obtain the Rico dataset and analyze and process the UI data in the Rico dataset to obtain the wireframe dataset and color dataset;

[0082] S200, based on a convolutional neural network, performs image feature extraction on the wireframe dataset to obtain reference image features;

[0083] S300: Construct a color histogram for the color dataset and extract color features from the color histogram to obtain reference histogram data.

[0084] S400, weighted calculation of the reference histogram data to obtain reference weighted data;

[0085] S500: Collect user needs and output user need data; the user need data includes user need type, user weight data, and user need data.

[0086] S600 performs type determination on the user requirement type. If the user requirement data is a frame diagram type, then the user requirement data is subjected to image feature extraction to obtain user image feature data.

[0087] S700 performs a KNN search on the reference image feature data and the user image feature data to obtain the first matching result;

[0088] S800: Based on user weight data, calculate the similarity between user image feature data and reference image feature data included in the first matching result to output first similarity data; compare the first similarity data with a preset first standard value to filter out the first matching result where the first similarity data is greater than the first standard value, and output the second matching result.

[0089] S900 displays the second matching result and collects user behavior data to obtain feedback signals from the user regarding their preferred UI page.

[0090] S1000, if there is a feedback signal, the reference image feature data is further filtered according to the feedback to obtain the third matching result, and the third matching result is processed by an interactive genetic algorithm to obtain the fourth matching result and the fourth matching result is displayed until the user is satisfied with the display result;

[0091] S1100, If the user requirement type is color-based, then color feature extraction is performed on the user requirement data to obtain user histogram data, and weighted calculation is performed on the user histogram data to obtain user weighted data.

[0092] S1200, calculate the similarity between the user-weighted data and the reference-weighted data to obtain the second similarity data;

[0093] S1300: The second similarity data is compared with the preset second standard value to filter out the first color matching result and display it. At the same time, the user's feedback data is collected and rematched based on the feedback data to filter out the second color matching result and display it, until the user is satisfied with the display result.

[0094] refer to Figure 2 In step S300, a color histogram is constructed from the color dataset, and color features are extracted from the color histogram to obtain reference histogram data, including the following steps:

[0095] S310 performs RGB to HSV conversion calculation on the UI page color information in the color dataset to obtain the HSV color space;

[0096] S320 performs non-uniform quantization on the HSV color space to gradually change the hue H in the HSV color space from 0° to 360°, dividing it into 7 unequal parts, and outputting the color subspace.

[0097] S330 calculates the color subspace to obtain a color histogram that represents the distribution of each color component, and extracts color features based on the color histogram.

[0098] In this embodiment, the RGB color information in the UI page stored in the computer system is converted into HSV color information to more intuitively reflect how the human visual system understands color. This is then converted into an HSV color space, which is non-uniformly quantized to divide it into seven unequal parts, corresponding to the psychologically perceptible colors: red, orange, yellow, green, cyan, blue, and violet. A small number of representative colors are selected to reduce computational complexity and improve efficiency. Simultaneously, color histograms are constructed for the divided color subspaces to obtain color histograms corresponding to different colors and their corresponding color characteristics.

[0099] refer to Figure 2 In step S300, a color histogram is constructed from the color dataset, and color features are extracted from the color histogram to obtain reference histogram data. This also includes the following steps:

[0100] S340, based on the quadtree algorithm, divides the UI page in the color dataset into quadtrees to obtain page sub-regions;

[0101] S350, extract the color features from the sub-regions of the page to obtain the color histogram of the selected area;

[0102] S360 stores the color histogram of the selected area into the corresponding node; at the same time, it divides the sub-region of the page into quadtrees again, and builds color histograms and extracts and stores the corresponding color features of the divided areas. The above steps are repeated until the UI page is indivisible or the divided area contains only one pixel.

[0103] S370 establishes a quadtree relationship for the color feature data of each node, enabling the creation of a UI data index for the UI data.

[0104] In this embodiment, the UI pages contained in the color dataset of the Rico dataset are divided into four sub-regions by quadtree partitioning. Color data within each sub-region is extracted to obtain its color histogram, which is then stored in the corresponding node. The sub-regions are further quadtree-partitioned to obtain the next level of sub-regions until the UI page can no longer be divided or the partitioned regions contain only a single pixel. Color features of each region are extracted to generate a corresponding selected color histogram, which is stored in the corresponding node. A quadtree relationship is established between nodes to facilitate indexing of UI data, accelerate the search for color similarity in user request data, and improve work efficiency.

[0105] refer to Figure 3 The weighted calculation includes the following steps:

[0106] S11, obtain the color data on the y-axis of each selected area's color histogram and the peak y-axis data of the color histogram of the entire UI page;

[0107] S12, calculate the ratio between the y-axis data and the y-axis peak data to obtain the color peak percentage;

[0108] S13, calculate the area size of each sub-region of the page and the area size of the entire UI page, and perform a ratio calculation to obtain the area ratio of the selected area;

[0109] S14, multiply the color peak percentage and the selected area percentage of each page sub-region to obtain the color parameters of each page sub-region.

[0110] In this embodiment, the proportion of the peak value of the selected color in the entire UI page's color histogram is determined by calculating the ratio of the y-axis peak value of the color in each selected color histogram to the peak value of the color in the entire UI page. Simultaneously, the area of ​​the page sub-region corresponding to each selected color histogram is obtained and its ratio to the area of ​​the entire UI page is calculated to obtain the selected area proportion of each color histogram's page sub-region. Based on the selected area proportion and the color peak value proportion of the corresponding selected area, the weight of that selected area in the entire UI page is determined. This provides reference data for similarity matching between the UI page and the user-drawn color information, reducing the impact of color differences in different areas on the similarity matching results when no weighting is applied.

[0111] refer to Figure 4 In step S500, collecting user requirements and outputting user requirement data includes the following steps:

[0112] S510, if the user's requirement is a color type, then obtain the number and types of colors of the lines or graphics drawn by the user.

[0113] S520 generates a corresponding axisymmetric bounding box selection area based on the color type;

[0114] The process of generating a corresponding axisymmetric bounding box selection area based on color type specifically includes:

[0115] S521, obtain the area coordinates of the lines or graphics drawn by the user to get the maximum and minimum values ​​of the horizontal axis and the maximum and minimum values ​​of the vertical axis, namely Xmax, Xmin, Ymax, Ymin;

[0116] S522, group Xmax, Xmin, Ymax, and Ymin to obtain Pmin=[Xmin, Ymin] and Pmax=[Xmax, Ymax], where Pmin is the set of minimum values ​​of the two coordinate axes, i.e., the coordinates of the vertex of the upper left corner of the bounding box selection area; Pmax is the set of maximum values ​​of the two coordinate axes, i.e., the coordinates of the vertex of the lower right corner.

[0117] S523, calculate the geometric center based on Xmax, Xmin, Ymax, Ymin to obtain the geometric center data, i.e., c=(Pmin+Pmax) / 2;

[0118] S524 determines the location and size of the bounding box selection based on Pmin, Pmax, and c.

[0119] In this embodiment, if the user's requirement is a color type, a corresponding number and type of axisymmetric bounding box selection areas are generated for the color types and numbers of the lines or patterns drawn by the user. These axisymmetric bounding box selection areas are used to confirm the position and size of the lines and patterns drawn by the user on the canvas, facilitating color similarity matching and improving matching efficiency. Specifically, the coordinates of the regions of lines or graphics of the same color drawn by the user are obtained, and the maximum and minimum values ​​of the horizontal and vertical axes are selected. The maximum values ​​are grouped together, and the minimum values ​​are grouped together to determine the coordinates of the top-left and bottom-right vertices of the bounding box. Based on the coordinates of the top-left and bottom-right vertices, the geometric center coordinates of the bounding box are determined, thereby determining the position and size of the bounding box. This allows for focused similarity matching of colors within the range of the bounding box selection area during color similarity matching, improving the accuracy of color similarity matching.

[0120] refer to Figure 5 In step S1000, if a feedback signal exists, the reference image feature data is further filtered based on the feedback to obtain a third matching result. An interactive genetic algorithm is then used to calculate the third matching result to obtain a fourth matching result, which is then displayed until the user is satisfied with the displayed result. This includes the following steps:

[0121] S1010, Perform KNN search on the first matching result to obtain a UI page with a similarity greater than the second standard value, and display it as the second matching result. At the same time, use the second matching result as the initial population.

[0122] S1020, the user evaluates the second matching result based on their own needs, selects the preferred UI page, and outputs the information of the preferred UI page;

[0123] S1030, calculate the similarity between the desired UI page information and the UI data based on Euclidean distance, so as to obtain the individual fitness of each UI page in the UI data and output the individual fitness set;

[0124] S1040, the individual fitness set is screened to obtain the fitness screening results;

[0125] S1050: Based on the roulette wheel selection algorithm, filter the fitness filtering results and the second matching results using UI search results to obtain the fourth matching result and display it; repeat the above steps until the maximum number of iterations is reached or the user is satisfied.

[0126] In this embodiment, a KNN search is performed on the first match to determine UI pages with high similarity. This UI page is used as the second matching structure and displayed. The second matching result is used as the initial population for the interactive genetic algorithm. The user filters and evaluates the displayed UI pages to select a preferred UI page. When the preferred UI page is matched again, the similarity relationship between the UI data of the UI page and the UI data in the Rico dataset is calculated based on Euclidean distance to obtain the individual fitness of each UI page in the Rico dataset. The UI pages are then filtered based on the individual fitness set to select UI pages similar to the preferred UI page, which are used as fitness filtering results. The fitness filtering result and the second matching result are selected using a roulette wheel selection algorithm to obtain a fourth matching result, which is then displayed. The above steps are repeated until the user is satisfied. By matching UI pages in the UI data with the user's preferred UI pages using an interactive genetic algorithm, the matching results are directed towards the user's needs, improving the accuracy and efficiency of UI page matching. The roulette wheel algorithm speeds up the iteration of UI pages that are similar to user needs, thereby improving retrieval efficiency.

[0127] refer to Figure 6 In step S800, based on user weight data, the similarity between user image feature data and reference image feature data included in the first matching result is calculated to output first similarity data; the first similarity data is then compared with a preset first standard value to filter out first matching results where the first similarity data is greater than the first standard value, and a second matching result is output, including the following steps:

[0128] S810: Obtain the corresponding region data in the user weight data based on the user weight data, and match the reference image feature data of the first matching result according to the region data.

[0129] S820, compare the similarity between the reference image feature data and the user image feature data to determine the similarity data of each reference image feature data; the similarity comparison is to calculate the Euclidean distance between the reference image feature data of the first matching result and the user image feature data to obtain the Euclidean distance result and use the Euclidean distance result as the similarity between the wireframes;

[0130] S830, based on user weight data, perform weight calculation on the similarity data of each reference region to obtain the overall similarity; the weight calculation is to perform a dot product calculation on the user weight data and the similarity data for each region to obtain the overall similarity.

[0131] S840, compare the overall similarity with the first standard value to filter out the second matching results.

[0132] In this embodiment, when performing similarity matching on user image feature data, the system obtains region data corresponding to different weights within the user weight data based on user weight data. It then matches the reference image feature data according to the corresponding region data, calculates the similarity data of the corresponding regions using the Euclidean distance algorithm, and performs a dot product operation between the similarity and the weights to obtain the overall similarity of the user image feature data, i.e., the overall similarity. For example, A represents the similarity, B represents the divided region data, A = [a1, a2, ..., an]T, B = [b1, b2, ..., bn]T, and the dot product of A and B is C = A.*B = [a1b1, a2b2, ..., anbn]T. By calculating the region similarity, calculation errors are reduced, matching accuracy is improved, and the second matching result selected during similarity filtering is more similar to the user's expected needs, thus increasing matching efficiency.

[0133] refer to Figure 7 In step S100, the Rico dataset is obtained, and the UI data in the Rico dataset is processed to obtain the wireframe dataset, including the following steps:

[0134] S110, based on the Rico dataset, performs statistical analysis on UI data to obtain a set of wireframe units;

[0135] S120, The UI page is wireframed based on the wireframe unit set to obtain the wireframe dataset;

[0136] S121, the wireframe processing involves obtaining the component data of each UI page based on the UI data, wherein the component data includes component type and coordinates;

[0137] S122, Create a blank page of the same size as the UI page;

[0138] S123, Match the constituent data according to the wireframe unit set to obtain the corresponding wireframe component;

[0139] S124. Based on the coordinate information in the constituent data, place the wireframe components in the blank page at the same position to obtain the UI wireframe diagram, i.e., the wireframe dataset.

[0140] In this embodiment, the UI data in the Rico dataset is analyzed to determine its composition structure. Statistical analysis of this structure is then performed to identify commonly used wireframe types, which are used as wireframe unit sets. Each UI data point is then wireframed, transforming the UI page into a UI wireframe diagram. Simultaneously, the position and size relationships of the wireframe unit sets within the UI wireframe diagram are recorded to obtain the wireframe dataset. By converting complex UI pages into simple UI wireframe diagrams, the computational process during data matching is reduced, improving matching efficiency.

[0141] refer to Figure 8 User weight data includes the following steps:

[0142] S21, the user draws the user UI requirements based on the wireframe unit set to obtain the user wireframe diagram;

[0143] S22, obtain region data by dividing the user wireframe into regions;

[0144] S23, weights are set on the user wireframe based on regional data to obtain user weight data for different regions.

[0145] In this embodiment, when a user draws a user outline diagram, the user divides the drawn user outline diagram into sections using a selection area on the interactive interface, and sets the weight of the outline units in the selected user outline diagram to determine the importance of the outline unit when matching. This allows the user to make directional settings according to their own needs, and also to perform a wide range or more precise search based on the weight set by the user.

[0146] Based on the description of the above embodiments of the interactive UI design search method based on convolutional autoencoders, this invention also discloses an interactive UI design search system based on convolutional autoencoders:

[0147] like Figure 9 As shown, an interactive UI design search system based on convolutional autoencoder networks includes: Rico data module 1, user requirement collection module 2, data processing module 3, and comparison feedback module 4.

[0148] Rico data module 1 includes Rico data acquisition unit 11 and Rico data analysis unit 12; Rico data acquisition unit 11 is used to collect and store designer tools and UI data to obtain Rico dataset;

[0149] The Rico data analysis unit 12 is signal-connected to the Rico data acquisition unit 11. It is used to receive the Rico dataset and perform wireframe analysis on the Rico dataset to obtain a wireframe dataset. At the same time, it extracts color features from the Rico dataset to obtain a color histogram.

[0150] User requirement acquisition module 2 includes wireframe acquisition unit 21 and color acquisition unit 22.

[0151] The wireframe acquisition unit 21 is used to acquire the wireframe diagram drawn by the user based on the wireframe unit set and the user weight settings of different regions, and output the requirement wireframe data; the requirement wireframe data includes wireframe diagram data and user weight data;

[0152] The color acquisition unit 22 is used to acquire the colors drawn by the user, determine the region of the colors drawn by the user according to the axisymmetric bounding box, and output the required color data.

[0153] The data processing module 3 includes a wireframe data processing unit 31 and a color data processing unit 32. The wireframe data processing unit 31 is signal-connected to the Rico data analysis unit 12 and the wireframe acquisition unit 21. It is used to receive wireframe datasets and required wireframe data, and to extract image features from the wireframe datasets and required wireframe data to obtain corresponding image feature data, namely reference image feature data and user image feature data. The user image data is matched with the reference image feature data through KNN search to obtain matching data.

[0154] The color data processing unit 32 is signal-connected to the Rico data analysis unit 12 and the color acquisition unit 22. It is used to receive the required color data and the reference color histogram, perform color feature extraction on the required color data to obtain the required color histogram, and perform weighted calculation on the reference color histogram and the required color histogram to obtain the reference color parameters and the required color parameters.

[0155] The comparison feedback module 4 includes a data comparison unit 41 and a user feedback unit 42. The data comparison unit 41 is signal-connected to the color data processing unit 32 and the wireframe data processing unit 31, and is used to receive reference color parameters, required color parameters and matching data and perform similarity comparison to filter out the display result data and display it.

[0156] The user feedback unit 42 is connected to the data comparison unit 41 by a signal. It is used to receive the user's selected UI page in the display result data, analyze the UI page to obtain the analysis result, output the analysis result to the data processing unit, and perform rematching based on the analysis result.

[0157] Compared with existing interactive UI design search methods and systems based on convolutional autoencoders, this invention improves search efficiency and matching accuracy.

[0158] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An interactive UI design search method based on convolutional autoencoder networks, characterized in that, include: Obtain the Rico dataset and analyze and process the UI data in the Rico dataset to obtain the wireframe dataset and color dataset; Image feature extraction of wireframe datasets is performed using convolutional neural networks to obtain reference image features; A color histogram is constructed from the color dataset, and color features are extracted from the color histogram to obtain reference histogram data. The reference histogram data is weighted to obtain reference weighted data; Collect user needs and output user need data; the user need data includes user need type, user weight data, and user need data. The user requirement type is determined. If the user requirement data is a frame diagram type, then image feature extraction is performed on the user requirement data to obtain user image feature data. A KNN search is performed on the reference image feature data and the user image feature data to obtain the first matching result; Based on user weight data, the similarity between user image feature data and reference image feature data included in the first matching result is calculated to output first similarity data; and the first similarity data is compared with a preset first standard value to filter out the first matching result whose first similarity data is greater than the first standard value, and output the second matching result. The second matching result is displayed, and user behavior is collected to obtain feedback signals from users regarding their preferred UI pages. If a feedback signal exists, the reference image feature data is further filtered based on the feedback to obtain a third matching result. An interactive genetic algorithm is then used to process the third matching result to obtain a fourth matching result, which is then displayed until the user is satisfied with the displayed result. If the user's demand type is color-based, then color features are extracted from the user demand data to obtain user histogram data, and the user histogram data is weighted to obtain user weighted data. The similarity between user-weighted data and reference-weighted data is calculated to obtain the second similarity data; The second similarity data is compared with the preset second standard value to filter out the first color matching results and display them. At the same time, user feedback data is collected and rematched based on the feedback data to filter out the second color matching results and display them, until the user is satisfied with the display results.

2. The interactive UI design search method based on convolutional autoencoder networks according to claim 1, characterized in that: The steps of constructing a color histogram of the color dataset and extracting color features from the color histogram to obtain reference histogram data further include: The color information of the UI page in the color dataset is converted from RGB to HSV to obtain the HSV color space; Non-uniform quantization is performed on the HSV color space to gradually change the hue H in the HSV color space from 0° to 360°, dividing it into 7 unequal parts, and outputting the color subspace. The color subspace is computed to obtain a color histogram that represents the distribution of each color component, and color features are extracted based on the color histogram.

3. The interactive UI design search method based on convolutional autoencoder networks according to claim 2, characterized in that: The steps of constructing a color histogram from the color dataset and extracting color features from the color histogram to obtain reference histogram data specifically include: Based on the quadtree algorithm, the UI page in the color dataset is divided into quadtrees to obtain the page sub-regions; Extract the color features from the sub-regions of the page to obtain the color histogram of the selected area; Store the color histogram of the selected area into the corresponding node; at the same time, divide the sub-region of the page into quadtrees again, and build color histograms and extract and store the corresponding color features of the divided areas. Repeat the above steps until the UI page is indivisible or the divided area contains only one pixel. A quadtree relationship is established for the color feature data of each node, so that a UI data index can be created for the UI data.

4. The interactive UI design search method based on convolutional autoencoder networks according to claim 3, characterized in that: The weighted calculation specifically includes: Obtain the y-axis data of the color in the color histogram of each selected area and the peak y-axis data of the color histogram of the entire UI page; The ratio of the y-axis data and the y-axis peak data is calculated to obtain the color peak percentage; The area of ​​each sub-region of the page is calculated and its ratio to the area of ​​the entire UI page is calculated to obtain the area percentage of the selected area. The color parameters of each page sub-region are obtained by multiplying the color peak percentage and the selected area percentage of each sub-region.

5. The interactive UI design search method based on convolutional autoencoder networks according to claim 1, characterized in that: The steps of collecting user needs and outputting user need data specifically include: If the user's requirement is a color type, then obtain the number and types of colors of the lines or graphics drawn by the user; Generate a corresponding axisymmetric bounding box selection area based on the color type; The process of generating a corresponding axisymmetric bounding box selection area based on color type specifically includes: Obtain the area coordinates of the lines or shapes drawn by the user to get the maximum and minimum values ​​of the horizontal axis and the vertical axis, namely Xmax, Xmin, Ymax, and Ymin. Group Xmax, Xmin, Ymax, and Ymin to obtain Pmin=[Xmin, Ymin] and Pmax=[Xmax, Ymax]. Here, Pmin is the set of minimum values ​​of the two coordinate axes, which is the coordinates of the top-left vertex of the bounding box selection area; Pmax is the set of maximum values ​​of the two coordinate axes, which is the coordinates of the bottom-right vertex. The geometric center is calculated based on Xmax, Xmin, Ymax, and Ymin to obtain the geometric center data, i.e., c = (Pmin + Pmax) / 2. Based on Pmin, Pmax, and c, determine the location and size of the bounding box selection area.

6. The interactive UI design search method based on convolutional autoencoder networks according to claim 1, characterized in that: The steps described above, namely, if a feedback signal exists, further filtering the reference image feature data based on the feedback to obtain a third matching result, performing an interactive genetic algorithm on the third matching result to obtain a fourth matching result, and displaying the fourth matching result until the user is satisfied with the displayed result, specifically include: Perform a KNN search on the first matching result to obtain UI pages with a similarity greater than the second standard value, and display them as the second matching result. At the same time, use the second matching result as the initial population. Users evaluate the second matching results based on their own needs, select their preferred UI pages, and output information about their preferred UI pages. Based on Euclidean distance, calculate the similarity between the desired UI page information and the UI data to obtain the individual fitness of each UI page in the UI data, and output the individual fitness set; The individual fitness set is screened to obtain the fitness screening results; The UI search results are filtered based on the fitness selection algorithm and the second matching results to obtain the fourth matching result, which is then displayed. The above steps are repeated until the maximum number of iterations is reached or the user is satisfied.

7. The interactive UI design search method based on convolutional autoencoder networks according to claim 1, characterized in that: The steps of calculating the similarity between user image feature data and reference image feature data included in the first matching result based on user weight data to output first similarity data, and comparing the first similarity data with a preset first standard value to filter out first matching results with first similarity data greater than the first standard value, and outputting a second matching result, specifically include: Based on the user weight data, obtain the corresponding region data in the user weight data, and match the reference image feature data of the first matching result according to the region data; The similarity of the reference image feature data and the user image feature data is compared to determine the similarity data of each reference image feature data; the similarity comparison is to calculate the Euclidean distance between the reference image feature data of the first matching result and the user image feature data to obtain the Euclidean distance result and use the Euclidean distance result as the similarity between the wireframes. Based on user weight data, the similarity data of each reference region is weighted and calculated to obtain the overall similarity; the weight calculation is to perform a dot product calculation on the user weight data and the similarity data for each region to obtain the overall similarity. The overall similarity is compared with the first standard value to filter out the second matching results.

8. The interactive UI design search method based on convolutional autoencoder networks according to claim 1, characterized in that: The steps of obtaining the Rico dataset and processing the UI data in the Rico dataset to obtain the wireframe dataset specifically include: Based on the Rico dataset, statistical analysis and processing are performed on UI data to obtain a set of wireframe units; The UI page is wireframed based on the wireframe unit set to obtain the wireframe dataset; The wireframe rendering process involves obtaining the component data of each UI page based on the UI data. The component data includes component type and coordinates. Create a blank page of the same size as the UI page; Match the constituent data based on the wireframe unit set to obtain the corresponding wireframe component; Based on the coordinate information in the constituent data, the wireframe components are placed in the same position on the blank page to obtain the UI wireframe diagram, i.e., the wireframe dataset.

9. The interactive UI design search method based on convolutional autoencoder networks according to claim 8, characterized in that: The user weight data is as follows: Users draw user UI requirements based on wireframe cell sets to obtain user wireframe diagrams; Regional data is obtained by dividing the user's wireframe into regions; Weights are assigned to user wireframes based on regional data to obtain user weight data for different regions.

10. An interactive UI design search system based on convolutional autoencoders, applied to the method described in any one of claims 1-9, characterized in that: include: The Rico data module (1) includes a Rico data acquisition unit (11) and a Rico data analysis unit (12); The Rico data acquisition unit (11) is used to acquire and store designer tools and UI data to obtain a Rico dataset; the Rico data analysis unit (12) is signal-connected to the Rico data acquisition unit (11) and is used to receive the Rico dataset and perform wireframe analysis on the Rico dataset to obtain a wireframe dataset; at the same time, color feature extraction is performed on the Rico dataset to obtain a reference color histogram. The user requirement acquisition module (2) includes a wireframe acquisition unit (21) and a color acquisition unit (22); the wireframe acquisition unit (21) is used to acquire the wireframe diagram drawn by the user based on the wireframe unit set and the user weight settings of different regions, and output the requirement wireframe data. The requirement wireframe data includes wireframe data and user weight data; The color acquisition unit (22) is used to acquire the colors drawn by the user, determine the region of the colors drawn by the user according to the axisymmetric bounding box, and output the required color data. The data processing module (3) includes a wireframe data processing unit (31) and a color data processing unit (32); the wireframe data processing unit (31) is signal-connected to the Rico data analysis unit (12) and the wireframe acquisition unit (21), and is used to receive the wireframe dataset and the required wireframe data, and to extract image features from the wireframe dataset and the required wireframe data to obtain corresponding image feature data, namely reference image feature data and user image feature data, and to match the user image data with the reference image feature data through KNN search to obtain matching data; The color data processing unit (32) is signal-connected to the Rico data analysis unit (12) and the color acquisition unit (22), and is used to receive the required color data, perform color feature extraction on the required color data to obtain the required color histogram; and perform weighted calculation based on the reference color histogram and the required color histogram to obtain the reference color parameters and the required color parameters. The comparison feedback module (4) includes a data comparison unit (41) and a user feedback unit (42); the data comparison unit (41) is signal-connected to the color data processing unit (32) and the wireframe data processing unit (31), and is used to receive the reference color parameters, the required color parameters and matching data and perform similarity comparison to filter out the display result data and display it; The user feedback unit (42) is signal-connected to the data comparison unit (41) and is used to receive the user's selected UI page in the display result data, analyze the UI page to obtain the analysis result, output the analysis result to the data processing unit, and perform rematching based on the analysis result.

Images