User interface detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a user interface detection method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-detected user interface image; extracting at least one item of to-be-detected dimension information in the user interface image; selecting a target analysis model corresponding to the to-be-detected dimension information from a plurality of packaged analysis models according to the to-be-detected dimension information; and analyzing the corresponding to-be-detected dimension information through the target analysis model to obtain detection data. According to the method, multiple analysis models are packaged in a unified manner, intelligent model selection and load balancing are realized, and comprehensive user interface detection can be completed through the analysis models without manual intervention.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically to a user interface detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of software application development, the quality of the user interface (UI) has become a key factor affecting user experience. Currently, UI testing for most applications still relies on manual inspection by testers, which is inefficient and prone to missing problems. Automated tools, on the other hand, are mainly based on predefined rules and can only detect a limited number of types of problems, failing to adapt to diverse UI designs. Summary of the Invention

[0003] This application provides a user interface detection method, apparatus, electronic device, and storage medium, which uniformly encapsulates multiple analysis models to achieve intelligent model selection and load balancing, enabling comprehensive user interface detection to be completed through analysis models without manual intervention.

[0004] In a first aspect, embodiments of this application provide a user interface detection method, the method comprising:

[0005] Acquire the user interface image to be inspected;

[0006] Extract at least one dimension information to be detected from the user interface image;

[0007] Based on the dimension information to be detected, a target analysis model corresponding to the dimension information to be detected is selected from several encapsulated analysis models;

[0008] The detection data is obtained by analyzing the corresponding dimension information to be detected using the target analysis model.

[0009] In some embodiments, selecting the target analysis model corresponding to the dimension information to be detected from a plurality of encapsulated analysis models based on the dimension information to be detected includes:

[0010] Analyze the dimension type of the dimension information to be detected;

[0011] Based on the dimension type and the preset detection objective, the target analysis model corresponding to the dimension information to be detected is determined from several encapsulated analysis models.

[0012] In some embodiments, the detection data is obtained by analyzing the corresponding dimension information to be detected through the target analysis model, including:

[0013] Based on the preset hierarchical detection strategy, the corresponding target analysis model is invoked to analyze the information of the dimension to be detected, and detection data is obtained.

[0014] In some embodiments, the plurality of analysis models are provided with data interfaces, and the corresponding detection dimension information is analyzed through the target analysis model to obtain detection data, including:

[0015] Determine the input data format of the data interface of the target analysis model;

[0016] The dimensional information to be detected is converted into data to be detected in the input data format;

[0017] The data to be detected is input into the target analysis model to obtain the detection data.

[0018] In some embodiments, after obtaining detection data by analyzing the corresponding dimension information to be detected through the target analysis model, the method further includes:

[0019] The detection data is filtered based on preset data filtering rules to obtain the detection data to be displayed;

[0020] The adjustment plan is determined based on the data to be displayed;

[0021] A test report in a preset format is obtained based on the data to be displayed and the adjustment scheme.

[0022] In some embodiments, after extracting at least one dimension information to be detected from the user interface image, the method further includes:

[0023] Determine the location information of the dimension information to be detected in the user interface image;

[0024] The detection report marks the detection data on the user interface image based on the location information.

[0025] In some embodiments, the preset data filtering rules include an importance level standard for the detection results. The filtering of the detection data based on the preset data filtering rules to obtain the detection data to be displayed includes:

[0026] The importance level of the test data is determined based on the importance level standard of the aforementioned importance level test results;

[0027] The detection data to be displayed is obtained based on the detection data that meets the preset standards for the importance level.

[0028] In some embodiments, after acquiring the user interface image to be detected, the method further includes:

[0029] The user interface image is preprocessed to obtain a standardized image;

[0030] Extracting at least one dimension information to be detected from the user interface image includes:

[0031] Extract at least one dimension information to be detected from the standardized image.

[0032] Secondly, embodiments of this application provide a user interface detection device, the device comprising:

[0033] The data acquisition module is used to acquire the user interface image to be detected;

[0034] An information extraction module, which is communicatively connected to the data acquisition module, is used to extract at least one dimension information to be detected from the user interface image;

[0035] The model determination module is communicatively connected to the information extraction module and is used to select the target analysis model corresponding to the dimension information to be detected from a number of encapsulated analysis models based on the dimension information to be detected.

[0036] The information analysis module is communicatively connected to the model determination module and is used to analyze the corresponding detection dimension information through the target analysis model to obtain detection data.

[0037] In some embodiments, the model determination module is further configured to analyze the dimension type of the dimension information to be detected; and determine the target analysis model corresponding to the dimension information to be detected from a plurality of encapsulated analysis models according to the dimension type and the preset detection purpose.

[0038] In some embodiments, the information analysis module is further configured to call the corresponding target analysis model based on a preset hierarchical detection strategy to analyze the information of the dimension to be detected and obtain detection data.

[0039] In some embodiments, the information analysis module is further configured to determine the input data format of the data interface of the target analysis model; convert the dimension information to be detected into data to be detected in the input data format; and input the data to be detected into the target analysis model to obtain detection data.

[0040] In some embodiments, the information analysis module is further configured to filter the detection data based on preset data filtering rules to obtain detection data to be displayed; determine an adjustment scheme based on the data to be displayed; and obtain a detection report in a preset format according to the data to be displayed and the adjustment scheme.

[0041] In some embodiments, the information analysis module is further configured to determine the location information of the dimension information to be detected in the user interface image; and to mark the detection data on the user interface image based on the location information in the detection report.

[0042] In some embodiments, the information analysis module is further configured to determine the importance level of the detection data based on the importance level standard of the importance level detection result; and to obtain the detection data to be displayed based on the detection data whose importance level meets the preset standard.

[0043] In some embodiments, the information extraction module is further configured to preprocess the user interface image to obtain a standardized image; and extract at least one dimension information to be detected from the standardized image.

[0044] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps in the user interface detection method described in any of the above claims.

[0045] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions for execution by a controller to implement the user interface detection method as described in any of the preceding claims.

[0046] The solution adopted in the application embodiment uniformly encapsulates multiple analysis models. A suitable target analysis model is selected based on the dimensional information to be detected for targeted detection, enabling user interface detection to be completed through the analysis models without manual intervention. Furthermore, dimensional information to be detected is extracted according to dimensions, allowing for more comprehensive multi-dimensional detection of the user interface. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the user interface detection method provided in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the user interface detection system provided in the embodiments of this application;

[0050] Figure 3 This is a schematic diagram of the structure of a user interface detection device according to an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0053] Currently, UI testing for most applications still relies on manual inspection by testers or is based on predefined rules, which leads to the following problems:

[0054] Manual inspection is the primary method: Currently, UI testing for most applications still relies on manual inspection by testers, which is inefficient, prone to missing problems, has high labor costs, is difficult to standardize and scale, and lacks intelligent problem analysis and repair suggestions.

[0055] Rule-driven automation tools: Existing automation tools are mainly based on predefined rules, which can only detect a limited number of types of problems. They cannot adapt to diverse UI designs, have a limited detection range, cannot fully cover UI / UX issues, and cannot adapt to the rapid iteration pace of development.

[0056] Limitations of image comparison methods: Traditional pixel-level comparison methods are too sensitive to UI changes and are prone to false alarms.

[0057] Lack of semantic understanding: Existing tools cannot understand the semantics of UI content, making it difficult to detect issues such as content truncation and copy errors.

[0058] Please refer to Figure 1 This application provides a user interface detection method, the specific process of which can be as follows: S110 to S140, the method includes:

[0059] S110. Obtain the user interface image to be detected.

[0060] Specifically, the user interface image to be detected is acquired, which includes, but is not limited to, text and icons. There is no limit to the number of user interface images processed in parallel simultaneously. Furthermore, the user interface image to be detected can be a directly imported user interface or a screenshot; this embodiment does not impose any specific limitations.

[0061] It should be noted that the solution of this application can be applied to user interface detection on multiple platforms, including but not limited to user interface detection of web applications (supporting screenshots of various browsers), user interface detection of desktop applications (Windows, macOS, Linux), user interface detection of mobile applications (iOS, Android), and user interface detection of different device forms such as tablets and smartwatches. In other words, the solution of this application has stronger platform compatibility.

[0062] Furthermore, the solution proposed in this application is also applicable to dynamic content detection, including but not limited to short video and GIF animation analysis, detecting user interface anomalies during animation, detecting experience issues such as flickering and jumping during page loading, interactive screen recording analysis, and evaluating the rationality of user operation processes.

[0063] In one embodiment, this step is followed by: S210, preprocessing the user interface image to obtain a standardized image.

[0064] Specifically, since the user interface images to be detected come from different sources and have different parameters such as format and resolution, in order to ensure that the information contained in the user interface images to be detected can be accurately identified, the user interface images are preprocessed to obtain standardized images, thereby reducing the effort and difficulty of calling the analysis model later.

[0065] The preprocessing method is based on the requirements for standardized images and can be configured according to needs. When acquiring the user interface image to be detected, relevant information about the user interface image is obtained, including but not limited to the file path and application it belongs to. Preprocessing methods include, but are not limited to, cropping parameters, compression ratios, and processing options. Examples include image format checking and standardization (supporting PNG, JPEG, WEBP, etc.), intelligent cropping of system UI elements (status bar, navigation bar, bottom indicator, etc.), image compression and size optimization (adaptively controlled within API limits), automatic blurring of sensitive information (configurable recognition and processing strategies), and image quality assessment and anomaly detection (blur, underexposure, overexposure, etc.). Figure 2 The diagram shows the structure of the user interface detection system applied in this application. The analysis controller of the business logic layer performs image preprocessing in the core processing layer. If there are too many images to process, a batch processing engine can be used.

[0066] S120. Extract at least one dimension information to be detected from the user interface image.

[0067] Specifically, the solution of this application performs multi-dimensional detection on the user interface image to be detected, extracting at least one dimension information to be detected from the user interface image according to dimension classification. The dimension classification depends on the detection purpose of the user interface image. Dimension classification includes, but is not limited to, text, title bar, font size, icons, etc.

[0068] Different dimensions of information to be detected can be extracted using different methods. For example, text-based information can be recognized using a general-purpose OCR (Optical Character Recognition) algorithm. For icon-based information, different software has its own customized icons; therefore, an icon database can be pre-built, and the user interface images can be compared with this database to extract the icon-based information from the user interface images. Similarly, appropriate tools can be selected to extract the dimensional information to be detected from the user interface images.

[0069] The dimensions of user interface detection include, but are not limited to, UI layout anomaly detection, design consistency detection, font size standard detection, interaction obstacle detection, content quality detection, and performance indicator detection. The corresponding information for the dimensions to be detected may be the same or different for different detection dimensions.

[0070] S130. Select the target analysis model corresponding to the dimension information to be detected from the encapsulated analysis models according to the dimension information to be detected.

[0071] Specifically, this application uniformly encapsulates the APIs (Application Program Interfaces) of multiple analysis models. These models include personalized models (e.g., those focusing on text analysis and recognition) and general-purpose models (e.g., AI). Personalized models are specifically configured for different problem types, dimension types, and detection objectives, such as... Figure 2 As shown, the AI ​​model layer contains multiple models. Based on the information of each dimension to be detected, the target analysis model corresponding to the information of each dimension to be detected is selected from the encapsulated analysis models.

[0072] Appropriate analysis models are selected for different dimensions of information to be detected, so as to obtain detection data under that dimension. Among them, a multi-dimensional trade-off algorithm based on cost, speed and accuracy is used to select appropriate analysis models. It supports model health checks and automatic degradation mechanisms. If the accuracy of the analysis results of a certain analysis model is low or the amount of valid detection data is small, the priority of the analysis model can be reduced to achieve intelligent load balancing and avoid overloading of a single model. That is, when processing multiple user interface images in parallel, the number of user interface images processed by each analysis model is limited, and model performance monitoring and dynamic adjustment are supported.

[0073] It integrates open-source multimodal models such as Llama-Vision, Qwen-VL, and InternVL, supports local model deployment, meets data security and privacy protection requirements, and provides model performance benchmarking and comparative analysis. It supports fully offline deployment in enterprise intranet environments, provides a Docker containerized deployment solution, supports elastic scaling of Kubernetes clusters, and enables private deployment. It supports multi-node distributed deployment, improving processing power, implementing task queues and load balancing, optimizing resource utilization, and supporting edge computing deployment to reduce network latency.

[0074] In one embodiment, this step includes: S310, analyzing the dimension type of the dimension information to be detected; S320, determining the target analysis model corresponding to the dimension information to be detected from a plurality of encapsulated analysis models according to the dimension type and the preset detection purpose.

[0075] Specifically, the analysis involves identifying the dimension type of the information to be detected. Dimension classifications include, but are not limited to, text, title bars, font sizes, and icons. Based on the dimension type and the preset detection objective, the target analysis model corresponding to the dimension information to be detected is determined from several encapsulated analysis models. The preset detection objective is a detection objective set based on detection requirements, including but not limited to detecting typos, detecting font sizes, and detecting interface consistency.

[0076] S140. Analyze the corresponding detection dimension information through the target analysis model to obtain detection data.

[0077] Specifically, such as Figure 2 As shown, the model scheduler calls different analysis models. Each target analysis model analyzes the corresponding dimension information to be detected to obtain detection data. Different analysis models have different focuses and detection objectives, resulting in different detection data obtained from analyzing the corresponding dimension information.

[0078] For example, UI layout anomaly detection includes layout issues such as element overlap, misalignment, overflow, and misalignment; uneven margins and spacing; adaptation issues of responsive layouts at different sizes; and detection of controls extending beyond screen boundaries.

[0079] Design consistency checks include detecting inconsistencies in fonts, colors, spacing, and component styles; assessing compliance with design specifications; checking brand consistency (logo, color scheme, etc.); and verifying consistency in icon style and size.

[0080] Font size standardization checks include verification of preset font size standards (LargeTitle:34, Title1:28, Title2:22, etc.), font weight usage standard checks (normal / bold forms), font size consistency checks for similar text, and verification of the logical rationality of font size hierarchy.

[0081] Interaction barrier detection includes button and clickable area size appropriateness detection (recommended minimum 44pt), detection of missing visual feedback for operable elements, identification of form input experience issues, navigation flow clarity assessment, and accessibility issue detection.

[0082] Content quality checks include detection of text truncation or semantic incompleteness, detection of placeholders not being replaced (such as "{{name}}", "Lorem ipsum", etc.), internationalization / localization issues (mixed languages, translation errors, etc.), detection of spelling and grammar errors, validation of terminology consistency, and content suitability checks (text length matching container).

[0083] Performance metrics detection includes detecting abnormal loading status, unclear or missing error messages, blank pages and white screen issues, image loading failures, and network anomaly display issues.

[0084] In addition, it supports intelligent comparison of design drafts, automatically comparing the actual user interface with design drafts from Figma, Sketch, etc., detecting the degree of deviation between the implementation and the design, generating a difference heatmap, and supporting pixel-level precision comparison and tolerance configuration.

[0085] In one embodiment, this step includes: S410, calling the corresponding target analysis model based on a preset hierarchical detection strategy to analyze the information of the dimension to be detected, and obtaining detection data.

[0086] Specifically, the layered detection strategy is set based on the detection purpose. The layered detection strategy includes, but is not limited to, the basic visual detection layer, the semantic understanding detection layer, the user experience detection layer, and the design specification detection layer. Each detection layer sets several detection conditions based on its detection purpose and business scenario.

[0087] For example, the basic visual inspection layer includes layout structure analysis and anomaly identification, visual element consistency checks, and basic interactive element identification and verification. The semantic understanding inspection layer includes text content semantic integrity analysis, contextual logic consistency checks, and multilingual and localization issue detection. The user experience inspection layer includes interaction flow rationality assessment, usability and accessibility checks, performance and anomaly detection. The design specification inspection layer includes design system consistency verification, brand specification compliance checks, and platform design guideline compliance assessment.

[0088] Based on the preset hierarchical detection strategy, the corresponding target analysis model is invoked to analyze the information of the dimension to be detected, and the detection data under each detection layer is obtained.

[0089] In one embodiment, the plurality of analysis models are provided with data interfaces. Step S140, analyzing the corresponding dimension information to be detected through the target analysis model to obtain detection data, includes: S510, determining the input data format of the data interface of the target analysis model; S520, converting the dimension information to be detected into data to be detected in the input data format; S530, inputting the data to be detected into the target analysis model to obtain detection data.

[0090] Specifically, the solution in this application integrates several analysis models, provides a command-line interface including a complete CLI tool supporting scripted calls and automated integration, and also provides a RESTful API interface. This standardized API interface supports third-party system integration, as well as CI / CD pipeline integration, Continuous Integration (CI) and Continuous Delivery (CD), such as seamless integration with Jenkins, GitLab CI, GitHub Actions, Azure DevOps, etc. It provides quality gate functionality, supporting custom blocking rules and whitelist mechanisms. It supports incremental detection, analyzing only the changes relative to the baseline version. That is, when the user interface image to be detected is an updated version, the updated parts in the user interface image are detected based on the previous version, reducing the data processing volume for detecting the user interface image. Furthermore, it integrates code coverage tools and correlates with UI test coverage analysis. Figure 2 As shown, the interface layer sets up various types of interfaces.

[0091] In addition, the testing framework is deeply integrated, seamlessly integrating with testing frameworks such as Appium, Selenium, Playwright, and Cypress. It supports automatic screenshot capture, test case association, and result tracking, provides test assertion API, supports automated UI quality verification, integrates performance testing tools, and associates UI rendering performance analysis.

[0092] It integrates with enterprise toolchains to automatically create detected issues as work items in systems such as JIRA, Azure DevOps, and Linear. It supports real-time notifications from instant messaging tools such as Slack, Teams, and DingTalk. It also integrates with document systems such as Confluence and Notion to automatically generate test reports. Furthermore, it integrates with code review tools (such as Gerrit and Phabricator) to provide UI suggestions during the code review phase.

[0093] The APIs of multiple analysis models are uniformly encapsulated, thus each analysis model has its own data interface. Furthermore, third-party plugins are supported to extend detection capabilities, and the open API interfaces are available for developer integration.

[0094] For any integrated monitoring tool, its similar analysis models all have data interfaces. When using the target analysis model for analysis, the input data format of the target analysis model's data interface is determined, the information of the dimension to be detected is converted into the corresponding input data format, and then the data to be detected is input into the target analysis model to obtain the detection data. Therefore, an interface is developed to integrate and encapsulate various analysis models.

[0095] In one embodiment, after step S140, which involves analyzing the corresponding dimension information to be detected through the target analysis model to obtain detection data, the method further includes: S610, filtering the detection data based on preset data filtering rules to obtain detection data to be displayed; S620, determining an adjustment scheme based on the data to be displayed; and S630, obtaining a detection report in a preset format based on the data to be displayed and the adjustment scheme.

[0096] Specifically, each target analysis model generates a large amount of detection data, but not all of it needs to be displayed in the report. Therefore, based on display requirements, pre-defined data filtering rules are used to filter the detection data to obtain the data to be displayed. Furthermore, while analyzing the detection dimensions to obtain the detection data, the target analysis model also provides corresponding adjustment schemes to address any shortcomings. Finally, the data to be displayed and the adjustment schemes are combined to generate a detection report in a pre-defined format.

[0097] The format of the inspection report is not limited and can be set and converted according to needs. For example, a JSON format inspection report contains structured data, which is easy to integrate with other systems and perform secondary processing. It includes complete issue metadata and statistical information and supports custom field expansion. Alternatively, an HTML visualization report is available, which is an intuitive visual report containing issue annotation images and detailed descriptions. It supports issue filtering, sorting, and search functions, making it suitable for manual review and team discussions. Another option is a batch summary report, which supports statistical analysis and trend reports of multiple images, provides issue distribution charts and quality score trends, and supports cross-version comparative analysis.

[0098] Furthermore, the analysis model is continuously optimized based on user feedback and labeled data, enabling fine-tuning and specialization of the model in specific application areas, supporting incremental learning without the need to retrain the entire model.

[0099] It should be noted that this application integrates with the defect management system to manage the detection data obtained from the target analysis model, automatically creating defect tickets for systems such as JIRA and Bugzilla from the detected detection data, supporting the synchronization of detection data status and automated workflow, and providing issue priority mapping and allocation strategies based on data filtering rules.

[0100] In one embodiment, after step S120, extracting at least one dimension information to be detected from the user interface image, the method further includes: S710, determining the location information of the dimension information to be detected in the user interface image; and S720, marking the detection data on the user interface image based on the location information in the detection report.

[0101] Specifically, when extracting the dimensional information to be detected from the user interface image, the positional information of this dimensional information within the user interface image is simultaneously acquired. Since new images may be generated after preprocessing or extracting the dimensional information, to avoid losing the positional information, the coordinates of various types of information in the user interface image can be standardized, enabling unified transformation of multi-resolution coordinates and conversion between relative and absolute coordinates. Simultaneously, the positional information of the dimensional information to be detected within the user interface image is input into the target analysis model. This allows the detection data of each dimensional information output by the target analysis model to determine its position within the user interface image, facilitating marking at the corresponding location in the user interface image when displaying the detection data in the detection report, thus enabling rapid localization.

[0102] This embodiment provides pixel-level coordinate positioning, supporting the marking of problem areas on the original user interface image, so that users can quickly and intuitively locate the problem areas. At the same time, comparing with the original user interface image makes it easier to understand the detected problems.

[0103] In one embodiment, the preset data filtering rules include a test result importance level standard. Step S610, filtering the test data based on the preset data filtering rules to obtain test data to be displayed, includes: S810, determining the importance level of the test data based on the test result importance level standard; S820, obtaining test data to be displayed based on the test data whose importance level meets the preset standard.

[0104] Specifically, each target analysis model yields a large amount of detection data, and some of this data may have a negligible impact on the user interface. Therefore, the preset data filtering rules based on the impact of various types of detection data on the user interface include a importance level standard for the detection results. This standard defines the importance level of various types of detection data, and detection data whose importance level meets the preset standard is selected as the detection data to be displayed. For example, detection data with a higher importance level is selected as the detection data to be displayed.

[0105] It can also score various types of test data based on the importance level standard of the test results, and then weight them to obtain the final score of each type of test data. The test data whose final score meets the preset standard is used as the test data to be displayed.

[0106] In addition, it can predict potential UI issues based on historical data and code changes, provide regression testing suggestions, focus on high-risk areas, establish a database of detection data problem patterns, and quickly identify common problem types.

[0107] In this embodiment, a monitoring method based on hierarchical wake-up and multi-sensor collaborative verification is adopted. A multi-level sensor category is set up, and the power consumption is woken up in a hierarchical manner from low to high. Low-power sensors are used for monitoring under normal conditions. The monitoring results of low-power sensors are used to wake up high-power sensors, thereby reducing the power consumption of the entire system, significantly reducing the false alarm rate, and improving the system reliability.

[0108] Furthermore, the solution proposed in this application achieves the following: Single-image in-depth analysis: Supports comprehensive in-depth analysis of a single screenshot, including multi-dimensional problem detection such as UI, UX, content, and performance. Batch intelligent processing: Supports batch analysis of all images in a directory, which can be used in conjunction with automated testing processes and supports concurrent processing to improve efficiency. Multi-model compatibility: Compatible with various AI vision models such as OpenAI GPT-4V / DALL-E, Google Gemini, and Claude 3, which can be flexibly switched according to needs and costs. Intelligent preprocessing: Automatically performs image cropping, standardization, and blurring of sensitive information, reducing API call costs. Historical record comparison: Supports comparison with historical version screenshots to track problem resolution progress and regression detection. And it has at least the following technical effects:

[0109] 1. Significantly improves detection efficiency and quality

[0110] Highly automated: Comprehensive UI quality inspection can be completed without human intervention, working 24 / 7.

[0111] Strong batch processing capability: Supports parallel processing of large-scale screenshots, capable of processing hundreds of images at a time, improving efficiency by 10-50 times;

[0112] Fast detection speed: the analysis time for a single image is controlled within 30 seconds, and batch processing supports concurrent acceleration;

[0113] High accuracy: The core issue detection accuracy is >95%, significantly surpassing traditional rule-driven detection tools;

[0114] Wide coverage: Supports detection of 6 major categories and 30+ subcategories of issues, comprehensively covering UI / UX quality dimensions.

[0115] 2. Deep intelligent analysis capabilities

[0116] Semantic understanding capability: Based on the natural language understanding capability of large models, it can discover international semantic problems that are difficult to detect by traditional tools;

[0117] Context analysis: Able to understand the contextual relationships between UI elements and identify logical inconsistencies;

[0118] Intelligent assessment: Provides quantitative quality scores and priority rankings to guide optimization directions;

[0119] Root cause analysis: It not only identifies problems, but also analyzes the possible root causes of those problems;

[0120] Intelligent repair suggestions: Provides specific and feasible repair suggestions and best practice references.

[0121] 3. Significant cost-effectiveness

[0122] Significantly reduced labor costs: Reduces manual UI testing workload by more than 80%, freeing up testers to handle higher-value tasks;

[0123] API cost optimization: Effectively control AI API call costs through intelligent preprocessing, model selection, and caching mechanisms;

[0124] Early problem detection: Identifying and fixing UI issues early in development avoids the high costs of fixing them later;

[0125] Reduce user complaints: Identify user experience issues in advance to reduce user complaints and negative reviews after the application is launched;

[0126] Improve development efficiency: A rapid feedback mechanism helps development teams iterate and optimize quickly.

[0127] 4. Excellent integration and scalability

[0128] Multiple interface support: Provides multiple integration methods such as CLI, RESTful API, and SDK to adapt to different use cases;

[0129] Full platform compatibility: Supports UI detection for multiple platforms including mobile applications (iOS, Android), web applications, and desktop applications;

[0130] High configurability: Supports custom detection rules, scoring criteria, and report formats;

[0131] Enterprise-level integration: Seamless integration with mainstream CI / CD, testing frameworks, and defect management systems;

[0132] Cloud-native architecture: supports containerized deployment, elastic scaling, and microservice architecture.

[0133] 5. Actual Business Value

[0134] Significantly improved user experience: Through systematic UI quality testing, the end-user experience and satisfaction have been significantly improved;

[0135] Product quality assurance: Establish a standardized UI quality assessment system to ensure consistent product quality;

[0136] Development process optimization: After integration into the CI / CD process, left-shift testing is implemented to identify problems early in the development process;

[0137] Improved team collaboration: Unified quality standards and visual reports facilitate collaboration among design, development, and testing teams;

[0138] Brand image protection: Avoid UI issues from affecting brand image and user trust.

[0139] 6. Technological Innovation and Industry Value

[0140] Filling a technological gap: For the first time, a general-purpose large model is systematically applied to the field of application UI quality inspection;

[0141] Methodological contributions: A systematic methodology for multi-model fusion, multi-dimensional detection, and intelligent scoring was proposed;

[0142] Standardization Promotion: Establishing standardized methods and best practices for UI quality inspection in the industry;

[0143] Ecosystem building: Provide an open plug-in architecture to promote the development of the industry ecosystem;

[0144] Technology popularization: Lowering the barriers to the application of AI technology in the field of software testing and promoting the digital transformation of the industry.

[0145] 7. Safety and Reliability Guarantee

[0146] Data security: Supports local deployment and private deployment solutions to ensure enterprise data security;

[0147] Privacy protection: Provides automatic blurring of sensitive information to protect user privacy;

[0148] System stability: Multiple fault-tolerance mechanisms and automatic degradation functions ensure stable and reliable service;

[0149] Audit trail: Complete operation logs and audit trails to meet enterprise compliance requirements;

[0150] Disaster recovery: Supports data backup and disaster recovery to ensure business continuity.

[0151] 8. Sustainable Development Value

[0152] Continuous learning capability: Continuously optimize detection capabilities based on user feedback and usage data;

[0153] Community-driven development: An open architecture supports community contributions and ecosystem building;

[0154] Adapting to technological evolution: Modular design facilitates adaptation to new AI models and technological developments;

[0155] Industry-adaptable expansion: Testing rules and standards can be customized according to the needs of different industries.

[0156] This embodiment also provides a user interface detection device, which can be integrated into a terminal device. For example, such as... Figure 3 As shown, the user interface detection device 900 includes:

[0157] The data acquisition module 910 is used to acquire the user interface image to be detected;

[0158] The information extraction module 920 is communicatively connected to the data acquisition module 910 and is used to extract at least one dimension information to be detected from the user interface image;

[0159] The model determination module 930 is communicatively connected to the information extraction module 920 and is used to select the target analysis model corresponding to the dimension information to be detected from a plurality of encapsulated analysis models based on the dimension information to be detected.

[0160] The information analysis module 940 is communicatively connected to the model determination module 930 and is used to analyze the corresponding detection dimension information through the target analysis model to obtain detection data.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0162] In some embodiments of this application, the user interface detection device 900 can be implemented as a computer program, which can be implemented in, for example... Figure 4 The computer device shown is running the program. The computer device's memory can store the various program modules that make up the user interface detection device 900, for example, Figure 3 The data acquisition module 910, information extraction module 920, model determination module 930, and information analysis module 940 are shown. The computer program comprised of these modules causes the processor to execute the steps of the image detection methods in the various embodiments of this application described in this specification.

[0163] The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements an image detection method.

[0164] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] Accordingly, this application also provides an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Alternatively, the electronic device can be a server.

[0166] An electronic device includes one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor as described in the user interface detection method. The steps of the user interface detection method described here can be steps from the user interface detection methods of the various embodiments described above.

[0167] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to execute the steps of the user interface detection method described above. The steps of the user interface detection method here can be the steps in the user interface detection methods of the various embodiments described above.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The foregoing has provided a detailed description of a user interface detection method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A user interface detection method, the method comprising: Acquire the user interface image to be inspected; Extract at least one dimension information to be detected from the user interface image; Based on the dimension information to be detected, a target analysis model corresponding to the dimension information to be detected is selected from several encapsulated analysis models; The detection data is obtained by analyzing the corresponding dimension information to be detected using the target analysis model.

2. The user interface detection method according to claim 1, characterized in that, The step of selecting the target analysis model corresponding to the dimension information to be detected from a plurality of encapsulated analysis models based on the dimension information to be detected includes: Analyze the dimension type of the dimension information to be detected; Based on the dimension type and the preset detection objective, the target analysis model corresponding to the dimension information to be detected is determined from several encapsulated analysis models.

3. The user interface detection method according to claim 1, characterized in that, The detection data is obtained by analyzing the corresponding dimension information to be detected using the target analysis model, including: Based on the preset hierarchical detection strategy, the corresponding target analysis model is invoked to analyze the information of the dimension to be detected, and detection data is obtained.

4. The user interface detection method according to claim 1, characterized in that, The aforementioned analysis models are equipped with data interfaces. By analyzing the corresponding detection dimension information through the target analysis model, detection data is obtained, including: Determine the input data format of the data interface of the target analysis model; The dimensional information to be detected is converted into data to be detected in the input data format; The data to be detected is input into the target analysis model to obtain the detection data.

5. The user interface detection method according to claim 1, characterized in that, After obtaining the detection data by analyzing the corresponding dimension information to be detected through the target analysis model, the method further includes: The detection data is filtered based on preset data filtering rules to obtain the detection data to be displayed; The adjustment plan is determined based on the data to be displayed; A test report in a preset format is obtained based on the data to be displayed and the adjustment scheme.

6. The user interface detection method according to claim 5, characterized in that, After extracting at least one dimension information to be detected from the user interface image, the method further includes: Determine the location information of the dimension information to be detected in the user interface image; The detection report marks the detection data on the user interface image based on the location information.

7. The user interface detection method according to claim 5, characterized in that, The preset data filtering rules include importance level standards for the detection results. The filtering of the detection data based on these preset rules to obtain the detection data to be displayed includes: The importance level of the test data is determined based on the importance level standard of the aforementioned importance level test results; The detection data to be displayed is obtained based on the detection data that meets the preset standards for the importance level.

8. The user interface detection method according to claim 1, characterized in that, After acquiring the user interface image to be detected, the process further includes: The user interface image is preprocessed to obtain a standardized image; Extracting at least one dimension information to be detected from the user interface image includes: Extract at least one dimension information to be detected from the standardized image.

9. A user interface detection device, characterized in that, The device includes: The data acquisition module is used to acquire the user interface image to be detected; An information extraction module, which is communicatively connected to the data acquisition module, is used to extract at least one dimension information to be detected from the user interface image; The model determination module is communicatively connected to the information extraction module and is used to select the target analysis model corresponding to the dimension information to be detected from a number of encapsulated analysis models based on the dimension information to be detected. The information analysis module is communicatively connected to the model determination module and is used to analyze the corresponding detection dimension information through the target analysis model to obtain detection data.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the user interface detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are for execution by the controller to implement the user interface detection method as described in any one of claims 1 to 8.

Images