Deep learning-based user interface availability automatic scoring method and system
By constructing a deep learning-based dual-branch model that combines visual and textual features and dynamically adjusts the weights, the problems of low efficiency, low coverage, and poor adaptability in user interface usability assessment are solved. This enables efficient and comprehensive assessment and optimization suggestions, thereby improving user interface usability and user experience.
Patent Information
- Application Number
- CN202511810604.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-05-01
AI Technical Summary
Existing user interface usability assessment methods are inefficient, costly, subjective, have low coverage of assessment dimensions, and poor industry adaptability, making it difficult to meet the needs of rapid iteration of software products.
We employ a deep learning-based approach to construct a dual-branch model that integrates visual and textual features. We generate a dataset through image standardization and expert scoring, and combine an industry knowledge base and interpretable algorithms to generate an attention heatmap. We dynamically adjust the model weights, output a multi-dimensional usability score, and provide optimization suggestions.
It achieves objective, efficient, comprehensive, and dynamically adaptable usability assessment of the user interface, improving assessment efficiency, providing specific optimization suggestions, and enhancing the usability and user experience of the user interface.
Smart Images
Figure CN121957945A_ABST
Abstract
Description
A Deep Learning-Based Automatic User Interface Usability Scoring Method and System Technical Field
[0001] This application belongs to the field of software optimization technology, specifically relating to a method and system for automatically scoring the usability of user interfaces based on deep learning. Background Technology
[0002] User interface (UI) usability assessment is a core step in ensuring software product usability and improving user experience. Currently, existing UI usability assessments primarily rely on expert reviews or user testing, which suffers from inefficiency and high costs. For example, a complete assessment requires multiple experts to invest significant time, with costs reaching tens of thousands of yuan, making it difficult to meet the demands of daily iterations of numerous software solutions. Furthermore, existing assessment methods are highly subjective, with significant discrepancies (e.g., over 20%) in the scores given by different experts for the same UI design, and inconsistent assessment standards. Rule-based automated tools (such as the Figma plugin "UXCheck") can only examine pre-defined hardware specification issues and cannot identify software design features such as visual comfort and the rationality of interaction logic, resulting in low coverage of assessment dimensions. Moreover, because different industries have significantly different core UI usability requirements, existing assessment methods require customized frameworks for each industry, leading to high adaptation costs and difficulty in scalable reuse.
[0003] While related patents attempt to automate scoring methods, they suffer from issues such as insufficient feature extraction, poor model interpretability, and inadequate industry adaptability. Therefore, there is an urgent need for a novel evaluation method that can automatically replace manual scoring, standardize scoring, provide comprehensive coverage, and possess dynamic adaptability. Summary of the Invention
[0004] In a first aspect, embodiments of this application provide a method for automatically scoring the usability of user interfaces based on deep learning, comprising the following steps: S1. Collecting screenshots of user interfaces and performing image standardization and non-designed region cropping, then having experts score the processed images according to preset dimensions, and generating label data for each dimension based on the scores, and constructing a dataset containing a training set and a validation set using the processed images and the label data for each dimension; S2. Constructing a dual-branch deep learning model that integrates visual features and text features, and training it using the training set; the dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel; wherein, the visual branch is used to extract visual features from the processed images. Design features: The text branch extracts text from the processed image using OCR and generates semantic features. The fusion prediction channel fuses the visual design features and semantic features to jointly predict a multi-dimensional usability score; S3. The trained model is evaluated based on an industry knowledge base and validation set. An interpretable algorithm is used to generate an attention heatmap representing the design defect area, and a weight calibration algorithm is used to adjust the weight distribution of the trained model in each preset dimension, outputting the optimized model; S4. A screenshot of the new user interface to be evaluated is received. The optimized model is used to output a multi-dimensional usability score, and the attention heatmap is combined with a predefined defect rule base to locate design defects and generate visual optimization suggestions.
[0005] Further, step S1 is as follows: S11. Collect user interface screenshots covering various target industries from the application store, design platform, and internal design draft archives of the application software, and control the sample size to meet the minimum sample threshold; S12. Perform image standardization processing on the collected user interface screenshots, unify the resolution to the target pixel value and adopt the RGB color mode, and use the Mask R-CNN algorithm to crop the status bar and navigation bar as non-design areas to obtain the processed image; S13. Obtain the scores given by experts according to the set dimensions, and ensure the accuracy of the scores through cross-validation among experts, and use the verified scores as the labels of the corresponding dimensions; the set dimensions include general framework, common scenarios and components, interface style layout, ease of operation, ease of learning, and visibility; S14. Combine the processed image with the labels of the corresponding dimensions to obtain the dataset, and randomly divide the dataset into training set and validation set.
[0006] Further, in step S13, the specific steps of expert cross-validation are as follows: Calculate the Kappa value, the consistency coefficient, of the expert scores:
[0007] in, This refers to the Kappa coefficient value. It is the observational consistency ratio, which is the proportion of samples in which two experts give the same score to the same screenshot of the user interface out of the total number of samples. This refers to the expected consistency ratio, which is the proportion of evaluators expected to agree under random conditions. Between steps S12 and S13, a data augmentation step is included: the preprocessed image is expanded by randomly rotating it by a set angle, adjusting its brightness by a set range, and injecting Gaussian noise to increase the sample size to the target value. After step S13, a score normalization and weight fusion step is included: the StandardScaler function is used to normalize the scores of each dimension and map them to the [0,1] interval; the normalization formula is:
[0008] in, This is the original score. and These are the minimum and maximum scores for the corresponding dimensions; the initial weights of each dimension are determined using principal component analysis, and then weighted and fused into a comprehensive usability score.
[0009] Further, step S2 is as follows: S21. Constructing the visual branch: Based on the ResNet-50 pre-trained model, freeze the parameters of the first N layers, fine-tune the last M layers, and introduce convolutional blocks at the end to construct convolutional layers to output the visual design feature vector of the first preset dimension; S22. Constructing the text branch: Extract the interface text from the processed image using an OCR tool and input it into the BERT pre-trained model to generate the semantic feature vector of the second preset dimension; S23. Constructing the fusion prediction channel: Concatenate the visual design feature vector of the first preset dimension with the semantic feature vector of the second preset dimension, and output a multi-dimensional usability score through a fully connected layer to obtain the dual-branch deep learning model; S24. Use the Adam optimizer with mean squared error as the loss function to train the dual-branch deep learning model on the training set.
[0010] Further, step S24 is as follows: S241. Use the Adam optimizer and set the initial learning rate and the total number of training epochs; S242. In each training epoch, input the training set into the two-branch deep learning model for forward propagation, calculate the mean squared error loss function, and update the model weights through backpropagation; adopt a step-wise learning rate decay strategy, and after every N1 training epochs, multiply the learning rate by a preset decay factor, with the update formula as follows:
[0011] in, The current learning rate, The new learning rate after decay; S243. During training, the loss weights of the fully connected layer in the fusion prediction channel are dynamically adjusted when outputting scores for each preset dimension, based on the industry type of the samples in the current training batch; specifically, a set of dimension weight vectors is preset for each industry. Where D is the total number of rating dimensions, and When calculating the total loss, the loss for each sample is weighted, and the weighted loss function L is:
[0012] Where N is the number of samples and D is the total number of rating dimensions. For the i-th sample, the true score on the d-th dimension is... Predict a score for the corresponding model; It is the weight of the d-th rating dimension. The value is determined by the industry to which the sample belongs, from a preset dimension weight vector. The weights of each dimension are adjusted and controlled within a set range; S244. The training process is monitored using a validation set. When the validation set loss no longer decreases for N2 consecutive rounds, early stopping is triggered to terminate the training and the best-performing model is saved as the post-training model.
[0013] Further, step S3 is as follows: S31. Process the feature map output by the convolutional layer at the end of the visual branch using the Grad-CAM algorithm; specifically, calculate the gradient of the target dimension score relative to the feature map, and obtain the weight of each feature channel through global average pooling. Perform linear superposition and ReLU activation on the weighted feature map to generate an initial class activation map; S32. Upsample the initial class activation map to the same resolution as the original user interface screenshot to generate an attention heatmap, and extract continuous regions with activation values greater than a preset activation threshold from the attention heatmap as the key regions with the greatest impact on the scoring decision; S33. Construct an industry knowledge base containing a set number of rules, and use a multi-objective optimization algorithm to iteratively optimize the weights corresponding to each preset dimension in the fully connected layer of the fusion prediction channel with the goal of minimizing the scoring error of the model on the validation set of the set industry; S34. Based on the evaluation results on the validation set, perform hyperparameter tuning on the trained model, and output the final model integrating heatmap generation and industry adaptive weight configuration as the optimized model.
[0014] Furthermore, the fitness function used in step S33 to iteratively optimize the weights of each preset dimension using a multi-objective optimization algorithm is as follows:
[0015] in, Let be the root mean square error of the model on the validation set of industry k; The rule compliance score for industry k; and To balance the coefficients of the two objective weights, and Industry k rule compliance score The calculation method is as follows: apply the rules corresponding to industry k in the industry knowledge base to the validation set samples of industry k, and statistically analyze the proportion of samples whose output results are consistent with the rule requirements.
[0016] Further, step S4 specifically includes the following steps: S41. Receive a screenshot of the new user interface to be evaluated and perform image standardization and non-design area cropping processing as in step S12; S42. Input the preprocessed image into the optimized model and output usability scores and comprehensive scores for each preset dimension; S43. Predefine a defect rule base that stores the mapping relationship between UI element attributes and optimization suggestions, match the attention heatmap with the defect rule base, parse the UI element attributes in the key area, and trigger the corresponding optimization suggestions; S44. Display the scoring results in the form of a radar chart or bar chart, overlay the heatmap semi-transparently on the original user interface screenshot, and display the generated optimization suggestions in the form of text boxes in the key area.
[0017] Furthermore, in step S43, the rules included in the defect rule base include at least: when the button spacing is detected to be less than a preset first spacing threshold, generating an optimization suggestion to increase the button size to a preset target size; when the text contrast is detected to be less than a preset contrast threshold, generating an optimization suggestion to increase the contrast to a preset target contrast; step S4 also includes an iterative optimization step: S45. Modify the user interface according to the generated usability score and comprehensive score and optimization suggestions, take the screenshot of the modified user interface as new input, and re-execute step S4 until its comprehensive usability score reaches the preset qualified threshold.
[0018] Secondly, embodiments of this application also provide an automatic user interface usability scoring system based on deep learning, comprising: a data preprocessing module, used to collect user interface screenshots and perform image standardization processing and non-design region cropping processing, then have experts score the processed images according to preset dimensions, and generate label data for each dimension based on the scores, and construct a dataset containing a training set and a validation set using the processed images and the label data for each dimension; a model building and training module, used to build a dual-branch deep learning model that integrates visual features and text features, and train it using the training set; the dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel; wherein, the visual branch is used to extract visual design from the processed images. The system comprises several modules: a text branch extracts text from the processed image using OCR and generates semantic features; a fusion prediction channel integrates the visual design features with the semantic features to jointly predict a multi-dimensional usability score; a model optimization module evaluates the trained model based on an industry knowledge base and validation set, generates an attention heatmap representing design defect areas using an interpretable algorithm, and adjusts the weight distribution of the trained model in each preset dimension using a weight calibration algorithm, outputting the optimized model; and an automatic scoring and suggestion generation module receives screenshots of the new user interface to be evaluated, outputs a multi-dimensional usability score using the optimized model, and combines the attention heatmap with a predefined defect rule base to locate design defects and generate visual optimization suggestions.
[0019] As can be seen from the above technical solutions, this application has the following advantages: The user interface usability automatic scoring method and system based on deep learning provided in this application achieves objective, efficient, comprehensive and dynamically adaptable usability assessment of the user interface through automated data collection and preprocessing, deep learning model training and optimization, and automatic scoring and suggestion generation. It provides designers with specific optimization suggestions, improves the usability and user experience of the user interface, improves the efficiency of UI usability assessment, and fully covers and achieves dynamic industry adaptability. Attached Figure Description
[0020] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 is a flowchart illustrating the automatic user interface usability scoring method based on deep learning according to the present invention.
[0022] Figure 2 is a schematic diagram of the user interface usability automatic scoring system based on deep learning according to the present invention. Detailed Implementation
[0023] The various embodiments of this disclosure will be described more fully in the following detailed description of the specific steps of the deep learning-based automatic user interface usability scoring method. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.
[0024] This embodiment provides a deep learning-based automatic user interface usability scoring method, which effectively solves the problems of low efficiency, strong subjectivity, limited coverage dimensions, and poor industry adaptability of existing UI usability assessment methods.
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] Please refer to Figure 1, which is a flowchart of a deep learning-based automatic user interface usability scoring method in a specific embodiment. The method includes the following steps: S1. Collect user interface screenshots and perform image standardization and non-design area cropping. Then, experts score the processed images according to preset dimensions and generate label data for each dimension based on the scores. Use the processed images and the label data for each dimension to construct a dataset containing a training set and a validation set. It should be noted that by collecting and preprocessing UI screenshots from multiple channels, a standardized data foundation is provided for model training, ensuring data quality and diversity, and providing a basis for subsequent evaluation. S2. Construct a dual-branch deep learning model that integrates visual features and text features, and train it using the training set. The dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel. The visual branch is used to extract visual design features from the processed images, the text branch extracts text from the processed images using OCR and generates semantic features, and the fusion prediction channel fuses the visual design features and semantic features to jointly predict multi-dimensional usability scores. It should be noted that by constructing a fusion visual... A dual-branch deep learning model combining text features can comprehensively capture key features of UI design, enhancing the model's understanding and prediction capabilities regarding UI usability. S3. The trained model is evaluated based on an industry knowledge base and validation set. An interpretable algorithm is used to generate an attention heatmap representing design defect areas, and a weight calibration algorithm is used to adjust the weight distribution of the trained model in each preset dimension, outputting the optimized model. It should be noted that evaluating the model based on an industry knowledge base and validation set, and generating an attention heatmap using an interpretable algorithm, accurately locates design defects. Simultaneously, weight calibration optimizes the model, making it more suitable for different industry needs and enhancing its practicality and accuracy. S4. A screenshot of a new user interface to be evaluated is received. The optimized model outputs a multi-dimensional usability score, and, combined with the attention heatmap and a predefined defect rule base, design defects are located and visual optimization suggestions are generated. It should be noted that this step automatically scores and generates optimization suggestions for new UI screenshots, outputting the model evaluation results in an intuitive way, and providing targeted improvement measures based on the defect rule base, enabling rapid improvement of UI usability and achieving efficient iterative optimization of the design.
[0027] This embodiment achieves automated evaluation of UI usability, improving evaluation efficiency and objectivity, covering more evaluation dimensions, and can be dynamically adjusted according to different industries, making it highly adaptable to various industries.
[0028] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process of this embodiment, another user interface usability automatic scoring method based on deep learning is provided. This method includes the following steps: S1. Collect user interface screenshots and perform image standardization processing and non-design area cropping processing. Then, experts score the processed images according to preset dimensions, and generate label data for each dimension based on the scores. Use the processed images and the label data for each dimension to construct a dataset containing a training set and a validation set. The specific steps of step S1 are as follows: S11. Collect user interface screenshots covering various target industries from the application store, design platform, and internal design draft archives of the application software, and control the number of collected samples to meet the minimum sample threshold. For example, the target industries include the power, finance, and medical industries, and the minimum sample threshold is 100,000 images; S12. Perform image standardization processing on the collected user interface screenshots, unify the resolution to the target pixel value and adopt the RGB color mode, and use Mask. The R-CNN algorithm crops the status bar and navigation bar, which are considered non-design areas, to obtain the processed image; S13. Obtain the scores given by experts according to the set dimensions, and ensure the accuracy of the scores through cross-validation among experts, using the verified scores as labels for the corresponding dimensions; the set dimensions include general framework, common scenarios and components, interface style layout, operability, ease of learning, and visibility; in step S13, the specific steps of expert cross-validation are as follows: Calculate the Kappa value of the consistency coefficient of the expert scores:
[0029] in, This refers to the Kappa coefficient value. It is the observational consistency ratio, which is the proportion of samples in which two experts give the same score to the same screenshot of the user interface out of the total number of samples. S14. Combine the processed images with the corresponding dimension labels to obtain the dataset, and randomly divide the dataset into training and validation sets; S2. Construct a dual-branch deep learning model that integrates visual and textual features, and train it using the training set; The dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel; The visual branch is used to extract visual design features from the processed images, the text branch extracts text from the processed images using OCR and generates semantic features, and the fusion prediction channel fuses the visual design features and semantic features to jointly predict multi-dimensional usability scores; Step S2 is as follows: S21. Construct the visual branch: Based on the ResNet-50 pre-trained model, freeze the parameters of the first N layers, fine-tune the last M layers, and introduce convolutional blocks at the end to construct convolutional layers to output the visual design feature vector of the first preset dimension; S22. S23. Constructing the fusion prediction channel: The visual design feature vector of the first preset dimension is concatenated with the semantic feature vector of the second preset dimension, and a multi-dimensional usability score is output through a fully connected layer to obtain the dual-branch deep learning model; S24. Using the Adam optimizer with mean squared error as the loss function, the dual-branch deep learning model is trained on the training set; The specific steps of step S24 are as follows: S241. Use the Adam optimizer and set the initial learning rate and the total number of training rounds; S242. In each training round, the training set is input into the dual-branch deep learning model for forward propagation, the mean squared error loss function is calculated, and the model weights are updated through backpropagation; A step-wise learning rate decay strategy is adopted. After every N1 training rounds, the learning rate is multiplied by a preset decay factor, and the update formula is:
[0030] in, The current learning rate, The new learning rate after decay; S243. During training, the loss weights of the fully connected layer in the fusion prediction channel are dynamically adjusted when outputting scores for each preset dimension, based on the industry type of the samples in the current training batch; specifically, a set of dimension weight vectors is preset for each industry. Where D is the total number of rating dimensions, and When calculating the total loss, the loss for each sample is weighted, and the weighted loss function L is:
[0031] Where N is the number of samples and D is the total number of rating dimensions. For the i-th sample, the true score on the d-th dimension is... Predict a score for the corresponding model; It is the weight of the d-th rating dimension. The value is determined by the industry to which the sample belongs, from a preset dimension weight vector. The training process is monitored using a validation set. When the validation set loss no longer decreases for N2 consecutive rounds, early stopping is triggered to terminate training, and the best-performing model is saved as the post-trained model. The post-trained model is evaluated based on the industry knowledge base and validation set. An interpretable algorithm is used to generate an attention heatmap representing the design defect region, and a weight calibration algorithm is used to adjust the weight allocation of the post-trained model in each preset dimension, outputting the optimized model. The specific steps of step S3 are as follows: S31. The feature map output by the convolutional layer at the end of the visual branch is processed using the Grad-CAM algorithm. Specifically, the gradient of the target dimension score relative to the feature map is calculated, and global average pooling is used to obtain the gradient. S32. Obtain the weight of each feature channel, and perform linear superposition and ReLU activation on the weighted feature map to generate an initial class activation map; S33. Upsample the initial class activation map to the same resolution as the original user interface screenshot to generate an attention heatmap, and extract continuous regions with activation values greater than a preset activation threshold from the attention heatmap as the key regions with the greatest impact on the scoring decision; S34. Construct an industry knowledge base containing a set number of rules, and use a multi-objective optimization algorithm to iteratively optimize the weights corresponding to each preset dimension in the fully connected layer of the fusion prediction channel with the optimization objective of minimizing the scoring error of the model on the validation set of the set industry; The fitness function for iteratively optimizing the weights of each preset dimension using the multi-objective optimization algorithm in step S33 is as follows:
[0032] in, Let be the root mean square error of the model on the validation set of industry k; The rule compliance score for industry k; and To balance the coefficients of the two objective weights, and Industry k rule compliance score The calculation method is as follows: apply the rules corresponding to industry k in the industry knowledge base to the validation set samples of industry k, and statistically analyze the proportion of samples whose output results are consistent with the rule requirements; S34. Based on the evaluation results on the validation set, perform hyperparameter tuning on the trained model, and output the final model that integrates heatmap generation and industry adaptive weight configuration as the optimized model; S4. Receive the new user interface screenshot to be evaluated, use the optimized model to output a multi-dimensional usability score, and combine the attention heatmap with the predefined defect rule base to locate design defects and generate visual optimization suggestions; The specific steps of step S4 are as follows: S41. Receive the new user interface screenshot to be evaluated, and perform the same image standardization and non-design region cropping processing as in step S12; S42. Input the preprocessed image into the optimized model. S43. A defect rule base is predefined, storing the mapping relationship between UI element attributes and optimization suggestions. The attention heatmap is matched with the defect rule base, and the UI element attributes in the key area are parsed to trigger the corresponding optimization suggestions. In step S43, the rules included in the defect rule base at least include: when the button spacing is detected to be less than a preset first spacing threshold, an optimization suggestion to increase the button size to a preset target size is generated; when the text contrast is detected to be less than a preset contrast threshold, an optimization suggestion to increase the contrast to a preset target contrast is generated. S44. The scoring results are displayed in the form of a radar chart or bar chart, the heatmap is superimposed on the original user interface screenshot in a semi-transparent manner, and the generated optimization suggestions are displayed in the form of text boxes in the key area.
[0033] In some embodiments, unlike the embodiments described above, a data augmentation step is included between steps S12 and S13: the preprocessed image is expanded by randomly rotating it by a set angle, adjusting its brightness by a set magnitude, and injecting Gaussian noise to increase the sample size to the target value; after step S13, a score normalization and weight fusion step is included: the StandardScaler function is used to normalize the scores of each dimension and map them to the [0,1] interval; the normalization formula is:
[0034] in, This is the original score. and These are the minimum and maximum scores for the corresponding dimensions; the initial weights of each dimension are determined using principal component analysis, and then weighted and fused into a comprehensive usability score.
[0035] In some instances, unlike the above embodiments, step S4 also includes an iterative optimization step: S45. Modify the user interface based on the generated usability score, comprehensive score, and optimization suggestions, take the screenshot of the modified user interface as new input, and re-execute step S4 until its comprehensive usability score reaches the preset qualified threshold.
[0036] This application provides a specific example of data collection and preprocessing: Data Collection: UI screenshots were collected from app stores, design platform portfolios, and internal corporate design archives, covering industries such as power, finance, and healthcare, with a cumulative sample size of 150,000 images; Data Preprocessing: The collected UI screenshots were processed using Python's PIL library, with a uniform resolution of 1920×1080 pixels, RGB color mode, and Masking applied. The R-CNN algorithm was used to crop non-design areas such as the status bar and navigation bar; the StandardScaler function from the scikit-learn library was used to normalize the scores across six dimensions, mapping them to the [0,1] interval; initial weights were determined using the PCA algorithm, with weights of 0.15, 0.15, 0.2, 0.2, 0.15, and 0.05; data augmentation methods such as random_rotation, random_brightness, and GaussianNoise were used to expand the dataset to 450,000 samples; label generation: each UI interface was rated by 5 senior UX designers (e.g., with over 10 years of experience) and 30 crowdsourced users, with rating dimensions including general framework (e.g., weight 0.15), common scenarios and components (e.g., weight 0.15), interface style and layout (e.g., weight 0.15), ease of operation (e.g., weight 0.25), ease of learning (weight 0.2), and visibility (weight 0.1); the Kappa coefficient was used to verify the consistency of the expert ratings, and Kappa scores were removed. For samples with a ppa value < 0.8, the annotation accuracy is ensured to reach 97%. The deep learning model training architecture is constructed using a two-stage architecture of visual feature extraction and multi-task score prediction. The visual branch uses a ResNet-50 pre-trained model implemented in PyTorch, freezing the parameters of the first 10 stages and fine-tuning the last 7 stages to learn UI features, outputting a 2048-dimensional feature vector. A CBAM attention module is introduced, with a window size of 7×7, extracting features through 3×3 convolutional kernels. The text branch extracts text using the pytexbase OCR tool and generates 512-dimensional semantic features using the BERT-base-chinese model. Data input: The preprocessed data is randomly shuffled, with the first 300,000 samples used as the training set and the last 150,000 samples as the test set. Training process: The Adam optimizer is used, with an initial learning rate of 0.001 and a learning rate decay factor of 0.9, decaying every 5 epochs for a total of 30 epochs. The mean squared error (MSE) loss function is used, with the training objective being to control the deviation between the predicted score and the label score to 0.Within 3; intermediate results during training are recorded using TensorBoard; during training, dimension weights are dynamically adjusted based on industry type (e.g., power, finance, healthcare, e-commerce, etc.), with weight changes ranging from 10% to 30%; model evaluation: mean squared error (RMSE) and Spearman's rank correlation coefficient are calculated on the validation set, validating the model's scoring bias to within 0.25 and achieving a correlation coefficient of 0.82.
[0037] Model Evaluation and Optimization: Enhanced Interpretability: PyTorch's Grad-CAM algorithm was used to generate attention heatmaps with a threshold of 0.5, highlighting areas with significant impact on scores. Dynamic Weight Calibration: An industry knowledge base containing 800 rules was constructed, and dimensional weights were optimized using a genetic algorithm, adjusting weight configurations based on industry type (e.g., showroom, e-commerce) and user group (e.g., employees, seniors, children). For example, in the power scenario, the weight of the interaction fluency dimension was increased from 15% to 25%; in the medical scenario, the weight of the information clarity dimension was increased from 15% to 30%. Model Optimization: Based on the current model's evaluation results and feedback, the kernel size of the residual layer was adjusted to 3×3, the weight of the L2 regularization term was increased, and the number of training epochs was optimized using an early stopping strategy, thus improving model training performance. The mean squared error on the validation set was reduced to 0.18; Automatic scoring and optimization suggestions output: Automatic scoring: Input a new UI screenshot, and the system automatically preprocesses the data and outputs a score in 6 dimensions and a comprehensive score, with the time taken within 1 second; Defect location and suggestions: Combining heatmaps and defect rule bases, problem areas are identified and optimization suggestions are generated; For example, when the button spacing is found to be less than 8px, it is suggested to increase the button size to 48×48px; when the text contrast is less than 4.5:1, it is suggested to increase the contrast to 5:1; Visual output: The scoring results and optimization suggestions are displayed through bar charts and heatmaps; Design optimization: The design is adjusted according to the optimization suggestions generated by the system, and re-evaluated until the scoring requirements are met; After 3 iterations, the score of the design scheme improved from 3.2 to 4.3, and the time for each iteration was controlled within 30 minutes.
[0038] This application provides another specific example: dataset construction and preprocessing. Data collection: Screenshots of 100,000 mobile applications from sources such as the Google Play Store, Apple App Store, Behance, and Dribbble were collected using automated tools, covering fields such as electricity, finance, e-commerce, healthcare, and education. After data cleaning, 700,000 qualified samples were retained. Data preprocessing: Images were preprocessed using Python's OpenCV library, with a uniform size of 375×812 pixels. The Selective Search algorithm was used to segment the main regions, and the Word2Vec model was used to extract text features. The features were normalized using the StandardScaler function of the scikit-learn library, and 90% of the information was retained using the PCA algorithm to obtain initial weights. The RandomErasing data augmentation method was used to expand the dataset to 1.2 million samples. Label generation: Six senior designers and 50 crowdsourced users rated each interface across six dimensions, including usability (e.g., weight 0.25), visual appeal (e.g., weight 0.15), and linguistic feel (e.g., weight 0.05). 1) Information organization (e.g., weight 0.15), interaction (e.g., weight 0.25), and text clarity (e.g., weight 0.1); Kappa consistency coefficient is used to verify the consistency of the scoring, and samples with Kappa values <0.85 are removed to ensure that the annotation accuracy reaches 96%; Deep learning model training: Model architecture construction: A two-stage architecture of visual feature extraction + multi-task scoring prediction is adopted; The visual branch is based on the DenseNet-121 model implemented in PyTorch, the parameters of the first 5 stages are frozen, the last 8 stages are fine-tuned to learn UI features, and a 1024-dimensional feature vector is output. A CBAM attention module is introduced, with a window size of 5×5, and features are extracted using 3×3 convolutional kernels. The text branch uses the BERT-large model to generate 1024-dimensional semantic features. Data input: The preprocessed data is shuffled, and the first 800,000 data points are used as the training set, while the last 200,000 data points are used as the test set. Training process: The SGD optimizer is used, with an initial learning rate of 0.01 and a learning rate decay factor of 0.8, decaying once every 10 epochs, for a total of 40 epochs. The mean squared error (MSE) loss function is selected, and the training objective is to control the deviation between the predicted score and the label score to within 0.25.The training process was recorded using TensorBoard. During training, dimension weights were dynamically adjusted based on industry type (e.g., power, finance, healthcare, e-commerce), with weight changes ranging from 15% to 28%. Model evaluation: Mean squared error (RMSE) and Spearman's rank correlation coefficient (Srho) were calculated on the validation set, validating the model's scoring bias within 0.2 and achieving a correlation coefficient of 0.83. Model evaluation and optimization enhanced interpretability: PyTorch's Grad-CAM algorithm was used to generate attention heatmaps with a threshold of 0.6, showing areas with significant impact on scoring. Dynamic weight calibration: An industry knowledge base containing 1000 rules was constructed, and dimension weights were optimized using particle swarm optimization, adjusting weight configurations based on industry type (e.g., showrooms, e-commerce) and user groups (e.g., employees, seniors). For example, in e-commerce scenarios, the interaction fluency dimension weight was increased from 15% to 28%; in healthcare scenarios, the information clarity dimension weight was increased from 15% to 30%. Model Optimization: Based on the current model's evaluation results and feedback, the convolutional kernel size of the residual layer was adjusted to 5×5, the dropout layer probability was increased to 0.3, and the number of training epochs was optimized through an early stopping strategy. The model training effect was optimized until the mean squared error on the validation set decreased to 0.17. Automatic Scoring and Optimization Suggestion Output: Automatic Scoring: Input a new UI screenshot, and the system automatically performs data preprocessing and outputs a score across 6 dimensions and a comprehensive score, with the time taken within 0.8 seconds. Defect Localization and Suggestions: Combining heatmaps and defect rule bases, problem areas are identified, and optimization suggestions are generated. For example, when the interface density is found to be too high, it is suggested to adjust the layout and increase blank areas; when the text font size is less than 12pt, it is suggested to increase the font size to 12pt. Visualization Output: Scoring results and optimization suggestions are displayed through radar charts and heatmaps. Design Optimization: The design is adjusted based on the optimization suggestions generated by the system and re-evaluated until the scoring requirements are met. After 4 iterations, the design scheme's score improved from 3.5 to 4.6, and the time for each iteration was controlled within 45 minutes.
[0039] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0040] As shown in Figure 2, the following is an embodiment of the deep learning-based automatic user interface usability scoring system provided in this disclosure. This system and the deep learning-based automatic user interface usability scoring method of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the deep learning-based automatic user interface usability scoring system, please refer to the embodiments of the deep learning-based automatic user interface usability scoring method.
[0041] The system includes: a data preprocessing module, used to collect user interface screenshots and perform image standardization and non-design area cropping, then experts score the processed images according to preset dimensions, and generate label data for each dimension based on the scores, using the processed images and the label data for each dimension to construct a dataset containing training and validation sets; a model building and training module, used to build a dual-branch deep learning model that integrates visual and textual features, and train it using the training set; the dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel; wherein, the visual branch is used to extract visual design features from the processed images, and the text branch extracts visual design features from the processed images using OCR. The system extracts text from images and generates semantic features. A fusion prediction channel integrates the visual design features with the semantic features to jointly predict a multi-dimensional usability score. A model optimization module evaluates the trained model based on an industry knowledge base and validation set. It generates an attention heatmap representing design defect areas using an interpretable algorithm and adjusts the weight distribution of the trained model in each preset dimension using a weight calibration algorithm, outputting the optimized model. An automatic scoring and suggestion generation module receives screenshots of new user interfaces to be evaluated, outputs a multi-dimensional usability score using the optimized model, and combines the attention heatmap with a predefined defect rule base to locate design defects and generate visual optimization suggestions.
[0042] This embodiment achieves full automation of the entire process from data acquisition to model training, optimization, and final evaluation through the interactive collaboration of the data preprocessing module, model building and training module, model optimization module, and automatic scoring and suggestion generation module. This improves the efficiency and accuracy of usability assessment of the user interface and reduces the cost of manual assessment.
[0043] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A deep learning-based automatic user interface usability scoring method, characterized in that, The process includes the following steps: S1. Capture screenshots of the user interface and perform image standardization and cropping of non-design areas. Experts then score the processed images according to preset dimensions, generating label data for each dimension based on the scores. A dataset containing training and validation sets is constructed using the processed images and the label data for each dimension. S2. Construct a dual-branch deep learning model that integrates visual and textual features, and train it using the training set. The dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel. The visual branch extracts visual design features from the processed images, while the text branch extracts visual design features from the processed images using OCR. S3. Extract text from the image and generate semantic features. The fusion prediction channel integrates the visual design features with the semantic features to jointly predict a multi-dimensional usability score. S4. Evaluate the trained model based on an industry knowledge base and validation set. Use an interpretable algorithm to generate an attention heatmap representing the design defect area. Adjust the weight distribution of the trained model in each preset dimension using a weight calibration algorithm and output the optimized model. S5. Receive a screenshot of the new user interface to be evaluated. Use the optimized model to output a multi-dimensional usability score. Combine the attention heatmap with a predefined defect rule base to locate design defects and generate visual optimization suggestions.
2. The method for automatically scoring user interface usability based on deep learning according to claim 1, characterized in that, Step S1 is as follows: S11. Collect user interface screenshots covering various target industries from the application store, design platform, and internal design archives of the application software, and control the sample size to meet the minimum sample threshold; S12. Standardize the collected user interface screenshots, unify the resolution to the target pixel value and adopt the RGB color mode, and use the Mask R-CNN algorithm to crop the status bar and navigation bar as non-design areas to obtain the processed image; S13. Obtain the scores given by experts according to the set dimensions, and ensure the accuracy of the scores through cross-validation among experts, and use the verified scores as the labels of the corresponding dimensions; the set dimensions include general framework, common scenarios and components, interface style layout, operability, ease of learning, and visibility; S14. Combine the processed images with the labels of the corresponding dimensions to obtain the dataset, and randomly divide the dataset into training set and validation set.
3. The method for automatically scoring user interface usability based on deep learning according to claim 2, characterized in that, In step S13, the specific steps for expert cross-validation are as follows: Calculate the Kappa value, the consistency coefficient, among the expert scores: in, This refers to the Kappa coefficient value. It is the observational consistency ratio, which is the proportion of samples in which two experts give the same score to the same screenshot of the user interface out of the total number of samples. This refers to the expected consistency ratio, which is the proportion of evaluators expected to agree under random conditions. Between steps S12 and S13, a data augmentation step is included: the preprocessed image is expanded by randomly rotating it by a set angle, adjusting its brightness by a set range, and injecting Gaussian noise to increase the sample size to the target value. After step S13, a score normalization and weight fusion step is included: the StandardScaler function is used to normalize the scores of each dimension and map them to the [0,1] interval; the normalization formula is: in, This is the original score. and These are the minimum and maximum scores for the corresponding dimensions; the initial weights of each dimension are determined using principal component analysis, and then weighted and fused into a comprehensive usability score.
4. The method for automatically scoring user interface usability based on deep learning according to claim 2, characterized in that, The specific steps of step S2 are as follows: S21. Constructing the visual branch: Based on the ResNet-50 pre-trained model, freeze the parameters of the first N layers, fine-tune the last M layers, and introduce convolutional blocks at the end to construct convolutional layers to output the visual design feature vector of the first preset dimension. S22. Constructing the text branch: Extracting the interface text from the processed image using an OCR tool and inputting it into the BERT pre-trained model to generate a semantic feature vector of the second preset dimension; S23. Constructing the fusion prediction channel: Concatenating the visual design feature vector of the first preset dimension with the semantic feature vector of the second preset dimension, and outputting a multi-dimensional usability score through a fully connected layer to obtain a dual-branch deep learning model; S24. Using the Adam optimizer with mean squared error as the loss function, training the dual-branch deep learning model on the training set.
5. The deep learning-based automatic user interface usability scoring method according to claim 4, characterized in that, The specific steps of step S24 are as follows: S241. Use the Adam optimizer and set the initial learning rate and the total number of training epochs; S242. In each training epoch, input the training set into the two-branch deep learning model for forward propagation, calculate the mean squared error loss function, and update the model weights through backpropagation; A stepped learning rate decay strategy is adopted. Every N1 training epochs, the learning rate is multiplied by a preset decay factor, and the update formula is as follows: in, The current learning rate, The new learning rate after decay; S243. During training, the loss weights of the fully connected layer in the fusion prediction channel are dynamically adjusted when outputting scores for each preset dimension, based on the industry type of the samples in the current training batch; specifically, a set of dimension weight vectors is preset for each industry. Where D is the total number of rating dimensions, and When calculating the total loss, the loss for each sample is weighted, and the weighted loss function L is: Where N is the number of samples and D is the total number of rating dimensions. For the i-th sample, the true score on the d-th dimension is... Predict a score for the corresponding model; It is the weight of the d-th rating dimension. The value is determined by the industry to which the sample belongs, from a preset dimension weight vector. The weights of each dimension are adjusted and controlled within a set range; S244. The training process is monitored using a validation set. When the validation set loss no longer decreases for N2 consecutive rounds, early stopping is triggered to terminate the training and the best-performing model is saved as the post-training model.
6. The method for automatically scoring user interface usability based on deep learning according to claim 4, characterized in that, Step S3 is as follows: S31. Process the feature map output by the convolutional layer at the end of the visual branch using the Grad-CAM algorithm; specifically, calculate the gradient of the target dimension score relative to the feature map, and obtain the weight of each feature channel through global average pooling. Perform linear superposition and ReLU activation on the weighted feature map to generate an initial class activation map; S32. Upsample the initial class activation map to the same resolution as the original user interface screenshot to generate an attention heatmap, and extract continuous regions with activation values greater than a preset activation threshold from the attention heatmap as the key regions with the greatest impact on the scoring decision; S33. Construct an industry knowledge base containing a set number of rules, and use a multi-objective optimization algorithm to iteratively optimize the weights corresponding to each preset dimension in the fully connected layer of the fusion prediction channel with the goal of minimizing the scoring error of the model on the validation set of the set industry; S34. Based on the evaluation results on the validation set, perform hyperparameter tuning on the trained model, and output the final model that integrates heatmap generation and industry adaptive weight configuration as the optimized model.
7. The method for automatically scoring user interface usability based on deep learning according to claim 6, characterized in that, In step S33, the fitness function for iteratively optimizing the weights of each preset dimension using a multi-objective optimization algorithm is as follows: in, Let be the root mean square error of the model on the validation set of industry k; The rule compliance score for industry k; and To balance the coefficients of the two objective weights, and Industry k rule compliance score The calculation method is as follows: apply the rules corresponding to industry k in the industry knowledge base to the validation set samples of industry k, and statistically analyze the proportion of samples whose output results are consistent with the rule requirements.
8. The method for automatically scoring user interface usability based on deep learning according to claim 6, characterized in that, The specific steps of step S4 are as follows: S41. Receive the screenshot of the new user interface to be evaluated, and perform the same image standardization and non-design area cropping processing as in step S12; S42. Input the preprocessed image into the optimized model, and output the usability score of each preset dimension and the comprehensive score; S43. Predefine a defect rule base that stores the mapping relationship between UI element attributes and optimization suggestions, match the attention heatmap with the defect rule base, parse the UI element attributes in the key area, and trigger the corresponding optimization suggestions; S44. Display the scoring results in the form of a radar chart or bar chart, overlay the heat map on the original user interface screenshot in a semi-transparent manner, and display the generated optimization suggestions in the form of text boxes in the key areas.
9. The method for automatically scoring user interface usability based on deep learning according to claim 8, characterized in that, In step S43, the rules included in the defect rule base include at least: when the button spacing is detected to be less than a preset first spacing threshold, generating an optimization suggestion to increase the button size to a preset target size; when the text contrast is detected to be less than a preset contrast threshold, generating an optimization suggestion to increase the contrast to a preset target contrast; step S4 also includes an iterative optimization step: S45. Modify the user interface according to the generated usability score and comprehensive score and optimization suggestions, take the screenshot of the modified user interface as new input, and re-execute step S4 until its comprehensive usability score reaches the preset qualified threshold.
10. A deep learning-based automatic user interface usability scoring system, characterized in that, include: The data preprocessing module is used to collect screenshots of the user interface and perform image standardization and non-design area cropping. Experts then score the processed images according to preset dimensions and generate label data for each dimension based on the scores. A dataset containing training and validation sets is constructed using the processed images and the label data for each dimension. The model building and training module is used to construct a dual-branch deep learning model that integrates visual and textual features, and trains it using the training set. The dual-branch deep learning model includes a visual branch, a text branch, and a fusion prediction channel. The visual branch extracts visual design features from the processed images, the text branch extracts text from the processed images using OCR and generates semantic features, and the fusion prediction channel fuses the visual design features and semantic features to jointly predict multi-dimensional usability scores. The model optimization module evaluates the trained model based on an industry knowledge base and validation set. It uses an interpretable algorithm to generate an attention heatmap representing regions with design defects and adjusts the weight distribution of the trained model in each preset dimension using a weight calibration algorithm, outputting the optimized model. The automatic scoring and suggestion generation module receives screenshots of new user interfaces to be evaluated, uses the optimized model to output multi-dimensional usability scores, and combines the attention heatmap with a predefined defect rule base to locate design defects and generate visual optimization suggestions.