Interface collaborative evaluation system and method based on convolutional neural network
The UI collaborative evaluation system based on convolutional neural networks solves the problems of low efficiency and insufficient accuracy in existing UI evaluation technologies, and achieves efficient and accurate UI evaluation and optimization guidance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN HONGXIN TECH SERVICE CO LTD
- Filing Date
- 2023-09-13
- Publication Date
- 2026-06-19
AI Technical Summary
In the current technology, there is a lack of efficient and accurate evaluation systems for the UI interface of APP/WEB systems. Evaluation of a single image cannot accurately reflect the overall UI style, and the reliance on manual evaluation is inefficient.
An interface collaborative evaluation system based on convolutional neural networks is adopted. The system receives combined interface images through a data acquisition module, stitches them together through an image collaborative preprocessing module, scores them using a pre-trained image collaborative scoring model, and performs image processing and scoring using a convolutional neural network.
It improves the efficiency and accuracy of UI evaluation, enabling automated and precise evaluation of the interface and guiding development optimization.
Smart Images
Figure CN117315337B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an interface collaborative evaluation system based on convolutional neural networks. Background Technology
[0002] Currently, there is no standardized evaluation system for the UI of APP / WEB systems. Software developers and users have difficulty having a standard to evaluate the quality of the completed interface. Usually, they will find a dedicated UI designer to define and evaluate it, which is a lot of work and extremely inefficient.
[0003] In addition, existing UI interfaces are often evaluated based on a single image. However, this method of scoring based on a single image may have problems with the overall style accuracy in the evaluation and optimization process of APP / WEB applications. This is because the overall UI style and design of an APP or WEB system cannot be evaluated solely based on the quality of a single image. Summary of the Invention
[0004] This invention provides a collaborative evaluation system and method for interfaces based on convolutional neural networks, which solves the problem of difficulty in evaluating the UI of APP / WEB systems with high efficiency and high accuracy in the prior art.
[0005] In a first aspect, the present invention provides an interface collaborative evaluation system based on a convolutional neural network, comprising: a data acquisition module for receiving a combination of interface images uploaded by a user; the combination of interface images includes a preset number of interface images; wherein the interface images are images of the functional interfaces of the object to be evaluated; an image collaborative preprocessing module for stitching together all the interface images in the combination of interface images into a collaborative fusion image according to a preset method; and a collaborative scoring module for inputting the collaborative fusion image into a pre-trained image collaborative scoring model and outputting a scoring result; wherein the image collaborative scoring model is trained based on a collaborative fusion image sample set and a score label set corresponding to the collaborative fusion image sample set.
[0006] The interface collaborative evaluation system based on convolutional neural networks provided by this invention includes an image collaborative preprocessing module that stitches together all interface images in the interface image combination into a collaborative fusion image according to a preset method. Specifically, this includes: linearly stitching together all interface images along the horizontal or vertical direction, and using the stitched image as the collaborative fusion image.
[0007] The interface collaborative evaluation system based on convolutional neural networks provided by the present invention further includes a dataset acquisition module, which includes a first processing unit and a second processing unit. The first processing unit is used to randomly select a preset number of interface images from multiple interface images of the object to be evaluated and stitch them together to form collaborative fusion image samples, so as to construct a collaborative fusion image sample set. The second processing unit is also used to set the score labels corresponding to the collaborative fusion image samples, so as to construct a score label set.
[0008] The interface collaborative evaluation system based on convolutional neural networks provided by this invention is an image collaborative scoring model of convolutional neural networks; the convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected layer, a softmax layer and an output layer connected in sequence.
[0009] The interface collaborative evaluation system based on convolutional neural networks provided by this invention includes two interface images in the interface image combination.
[0010] The present invention provides an interface collaborative evaluation system based on convolutional neural networks, wherein the object to be evaluated is an application or a web page.
[0011] Secondly, the present invention provides a method for collaborative evaluation of interfaces based on convolutional neural networks, comprising: receiving a combination of interface images uploaded by a user; the combination of interface images includes a preset number of interface images; wherein the interface images are images of the functional interfaces of the object to be evaluated; stitching all the interface images in the combination of interface images into a collaborative fusion image according to a preset method; inputting the collaborative fusion image into a pre-trained image collaborative scoring model and outputting a scoring result; wherein the image collaborative scoring model is trained based on a collaborative fusion image sample set and a score label set corresponding to the collaborative fusion image sample set.
[0012] According to the interface collaborative evaluation method based on convolutional neural networks provided by the present invention, all interface images in the interface image combination are stitched together into a collaborative fusion image in a preset manner, including: linearly stitching all interface images along the horizontal or vertical direction, and using the stitched image as the collaborative fusion image.
[0013] The interface collaborative evaluation method based on convolutional neural networks provided by the present invention further includes: randomly selecting a preset number of interface images from multiple interface images of the object to be evaluated and stitching them together to form a collaborative fusion image sample, so as to construct a collaborative fusion image sample set; and setting score labels corresponding to the collaborative fusion image samples to construct a score label set.
[0014] According to the interface collaborative evaluation method based on convolutional neural networks provided by the present invention, the image collaborative scoring model is a convolutional neural network; the convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected layer, a softmax layer and an output layer connected in sequence.
[0015] Thirdly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the interface collaborative evaluation method based on convolutional neural networks as described above.
[0016] The interface collaborative evaluation system and method based on convolutional neural networks provided by this invention firstly performs splicing preprocessing on the interface images to be scored to generate a collaborative fusion image; then, the collaborative fusion image is input into a pre-trained image collaborative scoring model to obtain the interface image evaluation score. This technical solution improves the evaluation efficiency of UI interfaces and the overall evaluation accuracy.
[0017] This invention selects the CNN convolutional neural network model as the image collaborative scoring model to process interface images, thereby improving image processing efficiency and accuracy.
[0018] This invention is based on the training and learning of a CNN convolutional neural network model. After the model is trained, it can score and evaluate various UI screenshots and combinations of the object to be evaluated, guiding the development team to improve and optimize the interface. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the interface collaborative evaluation system based on convolutional neural networks provided by the present invention;
[0021] Figure 2 This is a schematic diagram of the processing flow of the image collaborative preprocessing module provided by the present invention;
[0022] Figure 3 This is a schematic diagram of the structure of the convolutional neural network provided by the present invention;
[0023] Figure 4 This is a flowchart illustrating the interface collaborative evaluation method based on convolutional neural networks provided by the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0026] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.
[0027] The following is combined Figures 1-4 This invention describes the interface collaborative evaluation system and method based on convolutional neural networks provided in the embodiments of the present invention.
[0028] Figure 1 This is a schematic diagram of the interface collaborative evaluation system based on convolutional neural networks provided by the present invention, as shown below. Figure 1As shown, the system includes: a data acquisition module 101, an image collaborative preprocessing module 102, and a collaborative scoring module 103.
[0029] The data acquisition module 101 is used to receive the combination of interface images uploaded by the user;
[0030] The interface image combination includes a preset number of interface images, each of which is an image of the functional interface of the object to be evaluated, and each interface image corresponds one-to-one with the functional interface.
[0031] The preset quantity can be set as needed, usually 2 or 3.
[0032] Optionally, the object to be evaluated can be an application or a web page. It is understood that different functional interfaces will be displayed during the use of an application or web page; for example, clicking on the current functional interface of the application may generate a new functional interface. That is, there may be multiple interface images for the object to be evaluated.
[0033] Optionally, the interface image is a screenshot of the functional interface.
[0034] The image collaborative preprocessing module 102 is used to stitch together all the interface images in the interface image combination into a collaborative fusion image according to a preset method.
[0035] The preset method can be linear stitching based on length and width, which is equivalent to stitching all the interface images together in the horizontal and vertical directions.
[0036] It should be noted that all interface images have the same length and width to facilitate image stitching and subsequent processing.
[0037] The collaborative scoring module 103 is used to input the collaboratively fused image into a pre-trained image collaborative scoring model and output the scoring result.
[0038] The image collaborative scoring model is trained based on a collaboratively fused image sample set and the score label set corresponding to the collaboratively fused image sample set.
[0039] It is understood that the collaborative fusion image sample set includes multiple collaborative fusion image samples, the score label set includes multiple score labels, and there is a one-to-one correspondence between the collaborative fusion image samples and the score labels. The collaborative fusion image samples and collaborative fusion images are constructed in the same way, and the score labels are used to characterize the scores of the collaborative fusion image samples.
[0040] The interface collaborative evaluation system based on convolutional neural networks provided by this invention performs splicing preprocessing on the interface images to be scored to generate a collaborative fusion image; the collaborative fusion image is then input into a pre-trained image collaborative scoring model to obtain the interface image evaluation score. This technical solution improves the evaluation efficiency of UI interfaces and the overall evaluation accuracy.
[0041] Based on the above embodiments, as an optional embodiment, the interface collaborative evaluation system based on convolutional neural networks provided by the present invention includes an image collaborative preprocessing module that stitches all interface images in the interface image combination into a collaborative fusion image according to a preset method. Specifically, this includes: linearly stitching all interface images along the horizontal or vertical direction, and using the stitched image as the collaborative fusion image.
[0042] Specifically, this embodiment will be explained using the example of stitching two interface images together in the horizontal direction.
[0043] The vectorization of the interface image is a three-dimensional vector representation of h*w*d, where h represents the length, w represents the width, and d represents the RGB color channels, which are 3 channels.
[0044] The collaborative scheme of this invention keeps the three RGB color channels unchanged and uses linear stitching based on length and width, which is equivalent to stitching two images together in the horizontal direction. Currently, the vector of a single interface image is (1080*1343*3), and the input vector is a combination of two interface image vectors. After collaborative fusion, the resulting image is (2160*1343*3).
[0045] Based on the above embodiments, as an optional embodiment, the interface collaborative evaluation system based on convolutional neural networks provided by the present invention further includes a dataset acquisition module, which includes a first processing unit and a second processing unit; the first processing unit is used to randomly select a preset number of interface images from multiple interface images of the object to be evaluated and stitch them together to form collaborative fusion image samples, so as to construct a collaborative fusion image sample set; the second processing unit is further used to set the score labels corresponding to the collaborative fusion image samples, so as to construct a score label set.
[0046] Specifically, multiple interface images of the object to be evaluated can be obtained by taking screenshots of the functional interfaces to construct a preliminary set of interface images. For example, the set of interface images includes: interface image A, interface image B, interface image C, interface image D, and interface image E.
[0047] Optionally, Figure 2 This is a schematic diagram of the processing flow of the image collaborative preprocessing module provided by the present invention, as shown below. Figure 2As shown, after obtaining the set of interface images, two interface images (interface image A and interface image B) are randomly selected from multiple interface images and stitched together to form a collaborative fusion image, which can be used as a sample image.
[0048] Taking the collaborative fusion image sample formed by the combination of the above interface image A and interface image B as an example, the UI designer manually scores the combination from the quality of individual images to the overall style quality. The score range is an integer between 1 and 5, which is the score label of the collaborative fusion image sample.
[0049] Similarly, based on the collaborative fusion image samples composed of combinations of different interface images, and the corresponding score labels of the collaborative fusion image samples, an image combination scoring dataset can be constructed; this image combination scoring dataset can be used to train an image collaborative scoring model.
[0050] Optionally, in order to train and validate the image collaborative scoring model, the image combination scoring dataset can be divided into training and test sets according to different proportions as needed. This is an existing technical approach and will not be elaborated on here.
[0051] Once the model training is complete and the image collaborative scoring model is obtained, users can upload a combination of interface images, and the collaborative score corresponding to that combination of interface images can be obtained, thus realizing the automatic evaluation function of APP / WEB interface UI.
[0052] Based on the above embodiments, the image collaborative scoring model is a convolutional neural network; Figure 3 This is a schematic diagram of the structure of the convolutional neural network provided by the present invention, as shown below. Figure 3 As shown, the convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected Softmax layer, and an output layer connected in sequence.
[0053] Unlike ordinary neural networks, the neurons in a CNN (Convolutional Neural Network) layer are arranged in three dimensions: width, height, and depth (the depth refers to the third dimension of the data in a given layer). Each layer of a CNN transforms the three-dimensional input data into three-dimensional activation data for the neurons and outputs it.
[0054] like Figure 3 As shown, the input to the convolutional neural network is a collaboratively fused image (2160*1343*3). For details on how to obtain this image, please refer to the above embodiment; it will not be repeated here. A brief description of each layer follows.
[0055] Convolutional Layer: Based on a CNN (Convolutional Neural Network), this layer uses three convolutional kernels to extract convolutional features from the co-processed image vector. The output vector of this layer has the same size as the input; the convolutional vector is still 2160*1343*3.
[0056] Layering: The combined vector of the image after convolution is passed through two 5*5 pooling layers, and the size of the pooled vector is transformed to 80*50*3.
[0057] Fully connected layer: After pooling, the layers are flattened and then transformed into 5*1 one-dimensional vectors through a fully connected (FC) network.
[0058] Softmax layer: After passing through the fully connected layer, the resulting one-dimensional vector is subjected to softmax processing. The processed result is in the form of [0,0,1,0,0] and is output to the output layer.
[0059] Output layer: Based on the vector result after softmax processing, determine the corresponding score. For example, [0,0,1,0,0] corresponds to a score of 3 points; [0,0,0,0,1] corresponds to a score of 5 points.
[0060] on the other hand, Figure 4 This is a flowchart illustrating the interface collaborative evaluation method based on convolutional neural networks provided by the present invention, as shown below. Figure 4 As shown, the steps include:
[0061] Step 401: Receive the combination of interface images uploaded by the user; the combination of interface images includes a preset number of interface images;
[0062] The interface image is an image of the functional interface of the object to be evaluated.
[0063] Step 402: Combine all the interface images in the interface image combination into a collaborative fusion image according to a preset method.
[0064] Step 403: Input the collaboratively fused image into the pre-trained image collaborative scoring model and output the scoring result;
[0065] The image collaborative scoring model is trained based on a collaboratively fused image sample set and the score label set corresponding to the collaboratively fused image sample set.
[0066] Optionally, according to the interface collaborative evaluation method based on convolutional neural networks provided by the present invention, all interface images in the interface image combination are stitched together into a collaborative fusion image in a preset manner, including: linearly stitching all interface images along the horizontal or vertical direction, and using the stitched image as the collaborative fusion image.
[0067] Optionally, the interface collaborative evaluation method based on convolutional neural networks provided by the present invention further includes: randomly selecting a preset number of interface images from multiple interface images of the object to be evaluated and stitching them together to form a collaborative fusion image sample, so as to construct a collaborative fusion image sample set; and setting score labels corresponding to the collaborative fusion image samples to construct a score label set.
[0068] Optionally, in the interface collaborative evaluation method based on convolutional neural networks provided by the present invention, the image collaborative scoring model is a convolutional neural network; the convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected layer, a softmax layer, and an output layer connected in sequence.
[0069] The technical solution of the method here is the same as that of the system, so it will not be described again.
[0070] In summary, the interface collaborative evaluation system and method based on convolutional neural networks provided by this invention firstly performs splicing preprocessing on the interface images to be scored to generate a collaborative fusion image; then, the collaborative fusion image is input into a pre-trained image collaborative scoring model to obtain the interface image evaluation score. This technical solution improves the evaluation efficiency of UI interfaces and the overall evaluation accuracy.
[0071] This invention selects the CNN convolutional neural network model as the image collaborative scoring model to process interface images, thereby improving image processing efficiency and accuracy.
[0072] This invention is based on the training and learning of a CNN convolutional neural network model. After the model is trained, it can score and evaluate various UI screenshots and combinations of the object to be evaluated, guiding the development team to improve and optimize the interface.
[0073] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the interface collaborative evaluation method based on convolutional neural networks provided in the above embodiments. The method includes: receiving a combination of interface images uploaded by a user; the combination of interface images includes a preset number of interface images; wherein the interface images are images of the functional interfaces of the object to be evaluated; stitching all the interface images in the combination of interface images into a collaborative fusion image according to a preset method; inputting the collaborative fusion image into a pre-trained image collaborative scoring model and outputting a scoring result; wherein the image collaborative scoring model is trained based on a collaborative fusion image sample set and a score label set corresponding to the collaborative fusion image sample set.
[0074] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An interface collaborative evaluation system based on a convolutional neural network, characterized by, include: The data acquisition module is used to receive a combination of interface images uploaded by the user; the combination of interface images includes a preset number of interface images; wherein, the interface images are images of the functional interface of the object to be evaluated, and all interface images have the same length and width; The image collaborative preprocessing module is used to stitch together all the interface images in the interface image combination into a collaborative fusion image according to a preset method. The collaborative scoring module is used to input the collaboratively fused image into a pre-trained image collaborative scoring model and output the scoring result. The image collaborative scoring model is trained based on a collaboratively fused image sample set and a score label set corresponding to the collaboratively fused image sample set. The image collaborative scoring model is a convolutional neural network. It also includes: a dataset acquisition module, which comprises a first processing unit and a second processing unit; The first processing unit is used to randomly select a preset number of interface images from multiple interface images of the object to be evaluated and stitch them together to form a collaborative fusion image sample, so as to construct a collaborative fusion image sample set; The second processing unit is further configured to set the score labels corresponding to the collaboratively fused image samples to construct a score label set. 2.The convolutional neural network-based interface co-evaluation system according to claim 1, wherein, The image collaborative preprocessing module stitches all the interface images in the interface image combination into a collaborative fusion image according to a preset method, specifically including: Linearly stitch all interface images along the horizontal or vertical direction, and use the stitched image as the collaborative fusion image.
3. The interface collaborative evaluation system based on convolutional neural networks according to claim 1, characterized in that, The image collaborative scoring model is a convolutional neural network; The convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected layer, a softmax layer, and an output layer connected in sequence.
4. The interface collaborative evaluation system based on convolutional neural networks according to claim 1, characterized in that, The interface image combination includes two interface images.
5. The interface collaborative evaluation system based on convolutional neural networks according to claim 1, characterized in that, The object to be evaluated is an application or a web page.
6. A collaborative evaluation method for interfaces based on convolutional neural networks, characterized in that, include: Receive a combination of interface images uploaded by the user; the combination of interface images includes a preset number of interface images; wherein, the interface images are images of the functional interface of the object to be evaluated, and all interface images have the same length and width; All interface images in the interface image combination are stitched together into a collaborative fusion image according to a preset method; The collaboratively fused image is input into a pre-trained image collaborative scoring model, and the scoring result is output; wherein, the image collaborative scoring model is trained based on a collaboratively fused image sample set and a score label set corresponding to the collaboratively fused image sample set, and the image collaborative scoring model is a convolutional neural network; Also includes: A predetermined number of interface images are randomly selected from multiple interface images of the object to be evaluated and stitched together to form a collaborative fusion image sample set; and, Set the score labels corresponding to the collaboratively fused image samples to construct a score label set.
7. The interface collaborative evaluation method based on convolutional neural networks according to claim 6, characterized in that, All interface images in the interface image combination are stitched together into a collaborative fusion image according to a preset method, including: Linearly stitch all interface images along the horizontal or vertical direction, and use the stitched image as the collaborative fusion image.
8. The interface collaborative evaluation method based on convolutional neural networks according to claim 6, characterized in that, The image collaborative scoring model is a convolutional neural network; The convolutional neural network includes a convolutional layer, multiple pooling layers, a fully connected layer, a softmax layer, and an output layer connected in sequence.