Product UI function recognition and defect detection method based on multimodal large model

Through the multimodal large model to identify UI functions and generate test sequences, the limitations and maintenance complexity of test cases generation in existing UI function testing methods are solved, and efficient function-level defect detection and user experience improvement are achieved.

CN118467349BActive Publication Date: 2024-12-03ZHEJIANG UNIV
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410538197.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-30
Publication Date
2024-12-03
Estimated Expiration
2044-04-30

AI Technical Summary

Technical Problem

The existing UI function testing methods have problems such as high manual testing costs, limitations of automated testing tools, maintenance complexity, and BDD language ambiguity. It is difficult to generate comprehensive and accurate test cases with business meanings, and insufficient support for complex scenarios.

Method used

Using product UI function identification and defect detection methods based on multimodal large model, through page jump link request strategy and iterative feedback, a test function list and operation sequence with business meaning are generated, and a functional level defect evaluation is carried out to ensure that Test coverage and accuracy.

Benefits of technology

Improves testing efficiency and coverage, automates discovery and recording of functions and visual defects, reduces maintenance costs, and improves product quality and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118467349B_ABST
    Figure CN118467349B_ABST
Patent Text Reader

Abstract

The present invention discloses a product UI function identification and defect detection method based on a multimodal large model, comprising: (1) collecting application introductions and application page relationship diagrams, analyzing the above information through a multimodal large model and generating a product function list; (2) for each functional item requirement in the product function list, using a test script to perform recursive testing to generate a test action sequence; (3) when executing the test action sequence, recording the result of each test execution, and at the same time, using the multimodal large model to evaluate whether the result contains functional or visual defects. The present invention utilizes a multimodal large model to perform function identification, test sequence generation and defect assessment, and can intelligently generate a test function list and operation sequence with business meaning through a page jump link request strategy and iterative feedback method, thereby improving test efficiency and coverage; at the same time, it can help discover and record various functional and visual defects, which is helpful to improve product quality and user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a product UI function recognition and defect detection method based on a multimodal large model. Background Art

[0002] Generating an operation sequence with business meaning is an important and challenging task in UI functional testing. The goal of functional testing is to verify whether the software system correctly implements various functions according to the design requirements, and these functions are usually closely related to the user's business needs. Therefore, generating an operation sequence with business meaning is crucial to effectively cover business scenarios and discover potential functional defects. The following are several main current situations of traditional UI functional testing:

[0003] 1. Manual Testing

[0004] Testers will manually combine the business process of the software to generate test objectives and operation sequences. By simulating the user's real operation path, testers can better capture the software's functions and business logic, thereby generating test cases with business significance. However, with the continuous iteration and update of software functions, the testing cost and complexity of manual testing are increasing, and omissions and duplications are prone to occur.

[0005] 2. Use Behavior Driven Development (BDD)

[0006] BDD is a software development method based on business requirements, which emphasizes writing test cases by describing software behavior in natural language. In UI testing, testers can use BDD frameworks (such as Cucumber, SpecFlow, etc.) to write test cases with business meanings to better understand and verify the functions of the software. However, there are still some defects:

[0007] Over-focusing on behavior description and ignoring test implementation, BDD focuses on the expression of business requirements and the description of business behavior, but sometimes it may lead to over-focusing on behavior description and ignoring the details of test implementation. This may result in test cases that are not comprehensive or accurate enough.

[0008] Language ambiguity: Sometimes the natural language descriptions used in BDD may be ambiguous or vague, leading to biased or misunderstood understanding of the test cases. This may lead to a decline in the quality of the test cases.

[0009] High maintenance cost: The writing and maintenance of BDD test cases usually requires a lot of manpower and time. When the application's functions or business logic change, the test cases may need to be updated and adjusted frequently, increasing the complexity and cost of maintenance.

[0010] 3. Automated testing tools

[0011] Automated testing tools usually provide some advanced features, such as parameterized testing, data-driven testing, keyword-driven testing, etc., which can help testers more easily generate test objectives and operation sequences with business meaning. These tools can automatically generate corresponding test cases based on business logic and test objectives. Its disadvantages are:

[0012] Script fragility: Test cases in automated testing tools may be overly dependent on specific implementation details, which increases the fragility of the script. When the interface or functionality of the application changes, the test case may become invalid and need to be manually repaired or updated.

[0013] Limited to interface testing, automated testing tools are usually limited to interface-level testing and cannot cover all the functions and business logic of the application. This may result in some important functions or logic not being covered, thus affecting the comprehensiveness and quality of the test cases.

[0014] Insufficient support for complex scenarios: Automated testing tools may have limitations in handling complex test scenarios or business processes and cannot flexibly adapt to various complex situations. This may result in some boundary conditions or complex logic not being fully tested, thus affecting the coverage and accuracy of test cases. Summary of the invention

[0015] The present invention provides a product UI function identification and defect detection method based on a multimodal big model. The multimodal big model is used to perform function identification, test sequence generation and defect assessment. Through a page jump link request strategy and iterative feedback method, a test function list and operation sequence with business significance can be intelligently generated, thereby improving test efficiency and coverage. At the same time, automatic defect assessment can help discover and record various functional and visual defects, which is helpful to improve product quality and user experience.

[0016] A product UI function recognition and defect detection method based on a multimodal large model comprises the following steps:

[0017] (1) Collect application introductions and application page relationship diagrams, analyze the above information through a multimodal big model, and generate a list of product features;

[0018] (2) For each functional requirement in the product function list, use the test script to perform recursive testing to generate a test action sequence;

[0019] (3) When executing the test action sequence, the results of each test execution are recorded, and the multimodal large model evaluates whether the results contain functional or visual defects.

[0020] In step (1), the application page relationship diagram is a structured representation method used to show the overall structure and organization of an application, reflecting the relationship, jump path and layout between various pages and element modules;

[0021] The application page relationship diagram includes the following contents: a description of the page, a set of interactive elements on the page, a set of child pages to which the current page can jump directly, and a set of parent pages of the current page; the description of the page is a brief description of the page function, role, and layout information so that users can quickly understand the page content or function; the interactive elements on the page include the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and a description or document of the operation.

[0022] In step (1), the product function list includes the function name, function, priority and importance, execution process, test plan, and acceptance criteria.

[0023] In step (1), the above information is analyzed by a multimodal large model and a product function list is generated. The specific process is as follows:

[0024] (1-1) Design a loop script. The loop script first traverses the application page relationship graph and generates all possible page jump links based on the connection relationship between application pages.

[0025] (1-2) For each page jump link, the script sends a request to the multimodal big model, and the multimodal big model determines whether the link generates a functional item with business significance;

[0026] (1-3) The script processes the function items returned by the multimodal large model, adds the function items to the product function list, and records the number of functions that have been obtained for each link;

[0027] (1-4) After each loop iteration, the script checks whether the completed requests have covered all page jump links;

[0028] (1-5) When all page jump link navigation paths are processed, the loop terminates.

[0029] In step (1-2), the multimodal large model matches the functions with specific test scenarios and business meanings by identifying the jump relationships, information transmission and data flow between pages. Each function item generated has a complete business process and user experience path.

[0030] In step (2),

[0031] Use the test script to generate a test action sequence for each function item in the product function list, specifically:

[0032] The test script first calls the multimodal big model to generate initial single-step test actions and executes these actions in the actual test environment; after each test action is executed, the test script records the execution results and feeds the results back to the multimodal big model; the multimodal big model evaluates the current test status based on the feedback received and decides on the next test action to explore deeper functional implementations; this iterative process continues until the multimodal big model confirms that all key aspects of the functional item have been fully tested and covered.

[0033] Single-step test actions are defined as five standard types: click, enter, long press, slide, and return.

[0034] In order for the multimodal large model in iteration to understand the current test status and generate the next test action, it is necessary to provide execution result feedback and test progress tracking information;

[0035] Execution result feedback refers to the corresponding execution results generated by the automated testing tool after executing each test action, including screenshots and XML files; test progress tracking information is used to track the progress of the current test, that is, the set of test actions that have been executed.

[0036] In step (3), visual defects are defined as the following five types: overlapping interface elements, abnormal text display, disordered layout, image loading failure, and others; functional defects are defined as the following four types: function cannot be used normally, logical errors, data processing abnormalities, and performance issues;

[0037] For each defect found, record detailed information in the test report, including defect classification, defect description, specific steps, operating environment, screenshots or screen recordings.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. Systematic methodology: The present invention provides a systematic process, from function identification to test sequence generation to defect assessment. Each step has a specific implementation method and task decomposition combined with a multimodal large model, which helps to ensure operability and effectiveness.

[0040] 2. Function identification from the user's perspective: The multimodal large model of the present invention analyzes the correlation and process consistency between different pages from the user's perspective, and matches the functions with specific test scenarios and business implications to ensure that the product's functions meet user needs and provide a good user experience.

[0041] 3. Intelligent test action generation: Through iterative feedback in the recursive test process, the multimodal large model can intelligently generate the next single-step test action to adapt to the current test conditions, thereby improving the intelligence level of test sequence generation.

[0042] 4. Automatic defect discovery and analysis: When executing the functional level test sequence, the system records the execution results of each test, and uses a multi-modal large model to evaluate whether there are functional or visual defects. Through automated means, defects can be discovered and analyzed in a timely manner, providing strong support for problem repair and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a method for product UI function identification and defect detection based on a multimodal large model according to an embodiment of the present invention;

[0044] Figure 2 is a flowchart of step S10 in an embodiment of the present invention;

[0045] Figure 3 is a flow chart of step S20 in an embodiment of the present invention;

[0046] Figure 4 Flow chart of step S30 in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0048] like Figure 1 As shown, a product UI function recognition and defect detection method based on a multimodal large model includes the following steps:

[0049] S10: Collect application introductions and application page relationship diagrams. The multimodal big model analyzes the above information and generates a product feature list.

[0050] A loop script is used to manage the sending of requests and process the function list items returned by the multimodal large model. The script uses a depth-first traversal algorithm to obtain all page jump links in the application page relationship graph and submits them to the multimodal large model for analysis to ensure that all possible user paths and function scenarios are covered. The significance of this step is to analyze the overall structure and organization of the application from the user's perspective, reflect the relationship and layout between pages and element modules, and generate a function list with business significance.

[0051] S20: For each functional item, use the test script to perform recursive testing to generate a test action sequence. After the multimodal large model generates a single-step test action each time, the automated testing tool executes the action and obtains the execution result. Based on the execution result, the multimodal large model generates the next single-step test action and continues until it is confirmed that the test of the function is completed. The significance of this step is to use the iterative feedback mechanism to intelligently generate an operation sequence with business significance to improve test efficiency and coverage.

[0052] S30: When executing the functional-level test sequence, the results of each test execution are recorded, and the multimodal large model evaluates whether the results contain functional or visual defects. Visual defects include overlapping interface elements, abnormal text display, disordered layout, image loading failure, etc. Functional defects include inability to use functions normally, logical errors, data processing anomalies, performance problems, etc. For each defect found, the detailed information is recorded in the test report, including defect classification, defect description, specific steps, operating environment, screenshots or screen recordings, etc. The significance of this step is to help discover and record various functional and visual defects through automated defect assessment, thereby improving product quality and user experience.

[0053] In the embodiment of the present invention, the multimodal big model used may be an open source big model such as BLIP, MiniGPT, VisualGLM, or other commercial big models.

[0054] like Figure 2 As shown, the specific implementation of S10 is:

[0055] S101, collect application introductions and application page relationship diagrams.

[0056] Specifically, the application page relationship diagram is a structured representation method used to show the overall structure and organization of an application, reflecting the relationship between each page, element module, jump path and layout. The application page relationship diagram should include the following: a description of the page, a set of interactive elements on the page, a set of child pages that the current page can directly jump to, and a set of parent pages of the current page. Among them, the page description is a brief description of the page function, role, and layout information, so that users can quickly understand the page content or function. The interactive elements on the page contain four fields: id, action, target, and document, which respectively represent the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and the description or document of the operation.

[0057] S102, the loop script traverses the application page relationship graph to generate a page jump link.

[0058] Since applications usually have complex functional systems, a loop script is designed to manage the sending of requests and the processing of functional list items returned by the multimodal large model. The script uses a depth-first traversal algorithm to obtain all page jump links in the application page relationship graph and submits them to the multimodal large model for analysis to ensure that all possible user paths and functional scenarios are covered, so as not to miss any important functions.

[0059] The loop script first traverses the application page relationship graph and generates all possible page jump links based on the connection relationship between application pages.

[0060] The specific operations are as follows:

[0061] Parse the application page relationship graph: Build a navigation graph based on the application page relationship graph data. The navigation graph is a graph structure that represents the connection relationship between pages, where nodes represent pages and edges represent jump paths between pages.

[0062] Get page jump links: By traversing the navigation graph, you can get all possible page jump links. The process of traversing the navigation graph can use a graph traversal algorithm, such as depth-first search (DFS) or breadth-first search (BFS). The starting point of the traversal is the start page of the application, and the end point is all possible target pages.

[0063] Record page jump links: During the traversal process, record the page sequence that each page jump link passes through. These records will be saved in a list.

[0064] Process page jump links: For each record's page jump link, perform the next step of processing to verify the validity of the link.

[0065] S103, page jump link validity verification. The page jump link validity verification is completed by sending a request to the multimodal large model and determining whether a functional item with business meaning can be generated. The following specific steps are included:

[0066] Obtain page jump link information: In each loop iteration, obtain the next link to be verified from the predefined page jump link list.

[0067] Prepare request content: Build the request content, including key information on the current link, such as the start page, target page, operation elements, etc. This information can be obtained from the application page relationship diagram.

[0068] Send a request to the multimodal big model: Use a script to send a request to the multimodal big model, and send the constructed request content to the multimodal big model. The purpose of the request is to allow the multimodal big model to determine whether the current page jump link can generate a functional item with business significance. For each functional item identification, the multimodal big model should analyze the correlation and process consistency between different pages from the user's perspective. By identifying the jump relationship, information transmission and data flow between pages, the functions are matched with specific test scenarios and business meanings. Each functional item should show a complete business process and user experience path.

[0069] Receive the function items returned by the multimodal big model: Wait for the response of the multimodal big model and receive the function items returned by the multimodal big model. If the multimodal big model believes that the current page jump link can generate a function item with business meaning, it will be returned to the script. At this time, a structured data representation should be output to facilitate further functional testing, including: the name, function, priority and importance of the function, execution process, test plan, and acceptance criteria.

[0070] S104, record the verification result and the validity of the link. If the multimodal large model successfully generates a functional item with business significance, the page jump link is marked as valid; otherwise, it is marked as invalid.

[0071] S105, check the completion status. After each loop iteration, the script checks whether all page jump links have been covered.

[0072] S106, termination condition, the loop script should set a termination condition, for example, when all page jump links are processed, the loop terminates.

[0073] like Figure 3 As shown, the specific implementation of S20 is:

[0074] S201, Initialization: Get a valid function item from the product function list. Set the start page and initial state. Prepare test scripts and automated test tools.

[0075] S202, obtain the first single-step test action: provide the multimodal big model with the function item to be tested and the initial page information (screenshot and XML file), and ask the multimodal big model to return the single-step test action. The single-step test action is defined as five standard types: click, input, long press, slide, and return.

[0076] S203, execute the test action: the automated testing tool parses the action and automatically executes it, and obtains the execution results, including screenshots, XML files, etc.

[0077] S204, Feedback and Decision: Feedback the execution results to the multimodal big model, and the multimodal big model decides the next test action according to the current test status.

[0078] In order for the multimodal big model in iteration to understand the current test status and generate the next test action, it is necessary to provide execution result feedback and test progress tracking information. Execution result feedback is the corresponding execution result generated by the automated testing tool after executing each test action, including screenshots and XML files. Test progress tracking is used to track the progress of the current test, that is, the set of test actions that have been executed. This can help the multimodal big model determine whether the test has been completed or needs to continue to execute more test actions.

[0079] S205, recursive call: repeat the above steps until the termination condition is met.

[0080] S206, termination condition: check the progress and status of the current test. When the multimodal large model determines that the function item has been tested or the program crashes, the recursive process is terminated.

[0081] like Figure 4 As shown, the specific implementation of S30 is:

[0082] S301, when executing the functional level test sequence, the results of each test execution are recorded, and the multimodal large model evaluates whether the results contain functional or visual defects.

[0083] Among them, visual impairment is defined as the following five types:

[0084] 1. Overlapping interface elements: Improper interface layout design, or multiple views stacked together without properly handling the hierarchical relationship, resulting in overlapping elements.

[0085] 2. Abnormal text display (null or garbled characters): Abnormal text display due to data loss, damage or format error; garbled characters or failure to display normally due to the incompatibility of the font used in the application with the current device or system.

[0086] 3. Layout disorder: The interface is not correctly adapted to different devices or screen sizes, resulting in layout disorder.

[0087] 4. Image loading failed: The image cannot be loaded due to resource loss or network problems.

[0088] 5. Others.

[0089] Functional defects are defined into the following four types:

[0090] 1. Functions cannot be used normally: Due to logical errors in the front-end code, failure to call the service interface, or insufficient user permissions, the functions cannot be accessed or operated normally.

[0091] 2. Logical errors: There are logical design errors in the program, which cause the function execution results to be inconsistent with expectations or the handling of specific situations or boundary conditions to be imperfect or incorrect, resulting in logical errors.

[0092] 3. Data processing exception: Data format error or data source exception leads to data processing exception.

[0093] 4. Performance issues: The program code or algorithm is inefficient, or it occupies too many system resources, resulting in slow execution or system crashes.

[0094] S302, for each defect found, record detailed information in the test report, including defect classification, defect description, specific steps, operating environment, screenshots or screen recordings, etc.

[0095] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A product UI function identification and defect detection method based on a multimodal large model, characterized in that: The steps include: (1) Collect application introductions and application page relationship diagrams, analyze the collected information through a multimodal big model, and generate a product function list; the specific process is as follows: (1-1) Design a loop script. The loop script first traverses the application page relationship graph and generates all possible page jump links based on the connection relationship between application pages. (1-2) For each page jump link, the script sends a request to the multimodal big model, and the multimodal big model determines whether the link generates a functional item with business significance; (1-3) The script processes the function items returned by the multimodal large model, adds the function items to the product function list, and records the number of functions that have been obtained for each link; (1-4) After each loop iteration, the script checks whether the completed requests have covered all page jump links; (1-5) When all page jump link navigation paths are processed, the loop terminates; (2) For each functional requirement in the product function list, use the test script to perform recursive testing to generate a test action sequence; specifically: The test script first calls the multimodal big model to generate the initial single-step test actions and executes these actions in the actual test environment. After each test action is executed, the test script records the execution results and feeds the results back to the multimodal big model. The multimodal big model evaluates the current test status based on the feedback received and decides on the next test action to explore deeper functional implementation. This iterative process continues until the multimodal big model confirms that all key aspects of the functional item have been fully tested and covered. (3) When executing the test action sequence, the results of each test execution are recorded, and the multimodal large model evaluates whether the results contain functional or visual defects.

2. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: In step (1), the application page relationship diagram is a structured representation method used to show the overall structure and organization of an application, reflecting the relationship, jump path and layout between various pages and element modules; The application page relationship diagram includes the following contents: a description of the page, a set of interactive elements on the page, a set of child pages to which the current page can jump directly, and a set of parent pages of the current page; the description of the page is a brief description of the page function, role, and layout information so that users can quickly understand the page content or function; the interactive elements on the page include the unique identifier of the element, the type of operation performed on the element, the identifier of the target page, and a description or document of the operation.

3. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: In step (1), the product function list includes the function name, function, priority and importance, execution process, test plan, and acceptance criteria.

4. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: In step (1-2), the multimodal large model matches the functions with specific test scenarios and business meanings by identifying the jump relationships, information transmission and data flow between pages. Each function item generated has a complete business process and user experience path.

5. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: Single-step test actions are defined as five standard types: click, enter, long press, slide, and return.

6. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: In order for the multimodal large model in iteration to understand the current test status and generate the next test action, it is necessary to provide execution result feedback and test progress tracking information; Execution result feedback is that the automated testing tool generates corresponding execution results after executing each test action, including screenshots and XML files; Test progress tracking information is used to track the progress of the current test, that is, the set of test actions that have been executed.

7. The product UI function identification and defect detection method based on multimodal large model according to claim 1 is characterized in that: In step (3), visual defects are defined as the following five types: overlapping interface elements, abnormal text display, disordered layout, image loading failure, and others; functional defects are defined as the following four types: function cannot be used normally, logical errors, data processing abnormalities, and performance issues; For each defect found, record detailed information in the test report, including defect classification, defect description, specific steps, operating environment, screenshots or screen recordings.

Citation Information

Patent Citations

  • Reinforcement learning-based mobile terminal application test method, device, equipment and medium

    CN111538668A

  • User interface test method and related device

    CN112749081A

  • Abnormality test method and system applied to UI automation, terminal equipment and storage medium

    CN116431486A

  • Business process flexible reconstruction and collaborative optimization method

    CN116862211A

  • UI function automatic test tool and method

    CN117331831A