Automatic user interface assertion method and system based on multi-modal large model and program product

Abstract

The invention discloses an automatic user interface assertion method and system based on a multi-modal large model and a program product, and is applied to the field of UI automatic testing. In order to solve the problems that traditional assertion writing is complex and low in efficiency, and visual expectation is difficult to describe, the method comprises the steps that a screen image of a UI to be tested and a text instruction for describing an expectation state are obtained; inputting the image and the text instruction into a multi-modal large model (MLLM) configured to jointly process the image and text; analyzing whether the actual UI state in the image meets the expectation of a text instruction or not by utilizing the image-text joint understanding capability of the MLLM; and generating a final assertion result based on the model judgment result. According to the method, intelligent analysis of MLLM replaces complex assertion coding, natural language or structured text is allowed to be used for defining expectation, assertion creation is remarkably simplified, UI test development efficiency and maintainability are improved, and the UI state of the visual level can be verified more flexibly and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of user interface automation testing, and in particular, to a user interface automation assertion method based on a multimodal large model. Background Art

[0002] In the field of user interface (UI) automation testing, assertion is a key step in verifying whether the UI state conforms to the expected result. Traditional UI automation testing methods, such as testing solutions based on frameworks like AirTest and Appium, usually rely on testers to write detailed assertion codes to clearly verify the existence, attributes, and states of page elements. These assertions typically use technologies such as XPath and CSS selectors to locate UI elements and write corresponding logic for judgment.

[0003] However, this traditional assertion method has the following problems:

[0004] Low writing efficiency. Testers need to spend a lot of time writing and maintaining complex assertion codes. Especially when the UI interface changes frequently, the assertion codes also need to be modified accordingly, resulting in a relatively low overall efficiency of UI automation testing.

[0005] Poor readability. Complex assertion codes are difficult to understand and maintain, reducing the readability and maintainability of test cases.

[0006] Difficult to accurately describe visual expected states. Some visual expected states, such as the relative positions, colors, shapes of elements, etc., are difficult to accurately describe with traditional assertion codes, resulting in limited accuracy of assertions.

[0007] High maintenance cost. When the UI changes, a large number of assertion codes need to be modified, resulting in a relatively high maintenance cost.

[0008] Therefore, there is an urgent need for a user interface automation assertion method that can reduce the complexity of assertion writing, improve assertion efficiency and accuracy.

[0009] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present invention, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0010] In view of the technical problems existing in the above background art, the present invention aims to provide a user interface automation assertion method, system, and program product based on a multimodal large model to solve the technical problems of complex writing, low efficiency, and difficulty in accurately describing visual expected states of the traditional assertion method, thereby reducing the writing complexity of UI automation testing and improving the test efficiency and assertion accuracy.

[0011] An embodiment of the present invention provides a method for automated assertion of a user interface based on a multimodal large model, which is applied to the field of automated user interface testing, and includes the following steps:

[0012] Obtain the screen image information of the user interface to be evaluated;

[0013] Obtain a text assertion instruction used to describe the visual or content expected state that the user interface should achieve;

[0014] Input the screen image information and the text assertion instruction into a multimodal large model configured to jointly process image and text information;

[0015] Utilize the joint understanding ability of the multimodal large model for image content and text semantics, and instruct the model to perform the following determination operations: Based on the expected state description included in the text assertion instruction, analyze the actual user interface state presented in the screen image information to determine whether the actual state meets the expected state;

[0016] Generate an assertion result used to represent whether the automated assertion of the user interface is successful or not based on the determination result output after the multimodal large model performs the determination operation.

[0017] In some alternative embodiments, the obtained text assertion instruction is in the form of at least one of natural language text and / or structured text following predefined grammar rules and / or text containing predefined keywords indicating assertion targets or conditions, such that the multimodal large model can parse out specific expected state descriptions therefrom.

[0018] In some alternative embodiments, the method further includes:

[0019] When obtaining the screen image information, synchronously obtain the user interface hierarchical structure information corresponding to the screen image information;

[0020] Provide the user interface hierarchical structure information, together with the screen image information and the text assertion instruction, as inputs to the multimodal large model to utilize the structured information to assist and enhance the accuracy and robustness of the multimodal large model in performing the determination operation.

[0021] In some alternative embodiments, the user interface hierarchical structure information includes at least one of the identifier, text content, control type, screen boundary coordinates, visibility state, and interactivity state of the control elements in the user interface, providing structured context for the multimodal large model in addition to pixel information.

[0022] In some alternative embodiments, the multimodal large model is configured as a model that has been specifically domain-adapted or fine-tuned for the recognition of user interface elements, the understanding of user interface layouts, or image-based visual question answering tasks, in order to improve the accuracy of its determination of the user interface state.

[0023] In some alternative embodiments, the determined output result further includes a quantization metric, which is configured as a confidence score for the basic judgment made by the multimodal large model.

[0024] In some alternative embodiments, the process of generating an assertion result includes:

[0025] Comparing the confidence score with a preset confidence threshold;

[0026] When the confidence score reaches or exceeds the confidence threshold, generating a clear "assertion passed" or "assertion failed" result based on the basic judgment;

[0027] When the confidence score is lower than the confidence threshold, generating a result of "assertion uncertain" or triggering a mark that requires manual intervention, thereby using the confidence score to manage the reliability of the assertion result.

[0028] In some alternative embodiments, the determined output result further includes explanatory information, which explains the key visual regions or text elements based on which the model makes a judgment. The explanatory information is in the form of a text summary or a visual annotation on the screen image information, aiming to improve the transparency and debuggability of the assertion process.

[0029] In some alternative embodiments, the method is embedded in the execution process of the user interface automation test script and is triggered when it is necessary to verify the user interface state after the script has executed one or more interactive operation steps.

[0030] In some alternative embodiments, the method constitutes a continuously optimized closed loop, including:

[0031] Recording the generated assertion result and the corresponding input information;

[0032] Receiving feedback confirmation information from the operator or the verification system on whether the assertion result is correct;

[0033] Periodically or after collecting sufficient feedback, using the feedback confirmation information as a supervision signal to adaptively update or retrain the multimodal large model, so as to continuously improve its performance in this user interface automation assertion application scenario.

[0034] An embodiment of the present invention also provides a user interface automation assertion system based on a multimodal large model, which is applied to the field of user interface automation testing and is used to implement the above-mentioned user interface automation assertion method based on a multimodal large model, including:

[0035] An information acquisition module, configured to acquire the screen image information of the user interface to be evaluated, and acquire text assertion instructions for describing the visual or content expected state that the user interface should achieve;

[0036] A multimodal large model, which is configured to be able to jointly process image and text information, and is used to receive the screen image information and text assertion instructions provided by the information acquisition module, and utilize its joint understanding ability of image content and text semantics to perform a determination operation to determine whether the actual user interface state presented by the screen image information meets the expected state included in the text assertion instructions, and output a determination result;

[0037] A result generation module, connected to the multimodal large model, configured to generate an assertion result for characterizing whether the user interface automation assertion is successful or not based on the determination result output by the multimodal large model.

[0038] An embodiment of the present invention also provides a user interface automation assertion program product, which includes computer instructions that, when executed by a processor, implement the steps of the above-mentioned user interface automation assertion method based on a multimodal large model.

[0039] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

[0040] A user interface automation assertion method, system, and program product based on a multimodal large model of the present invention have the following beneficial effects:

[0041] The multimodal large model technology of the present invention can jointly encode and understand image and text information through deep learning methods, and establish the correlation between images and texts. In the field of user interface automation testing, the screen image contains the visual information of the user interface, while the text assertion instruction provides a semantic description of the expected state of the user interface. By comprehensively analyzing these two types of information using a multimodal large model, an accurate judgment of the user interface state can be achieved. The model analyzes the screen image to identify UI elements and their attributes, and combines text assertion instructions to understand the intentions of testers, thereby determining whether the current UI state meets the expectations. This method bypasses the complex element positioning and code writing processes in the traditional method, and simplifies the implementation method of assertions. Therefore, using a multimodal large model can reduce the difficulty of user interface automation testing and improve the testing efficiency. Description of the Drawings

[0042] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments read in conjunction with the accompanying drawings.

[0043] Figure 1 is a flowchart of a user interface automation assertion method based on a multimodal large model according to an embodiment of the present invention;

[0044] Figure 2 is a schematic structural diagram of a user interface automation assertion system based on a multimodal large model according to an embodiment of the present invention;

[0045] Figure 3 is a schematic application diagram of a user interface automation assertion method based on a multimodal large model according to an embodiment of the present invention. Detailed Embodiments

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0049] The present invention provides a method for automatically asserting a user interface based on a multimodal large model. By inputting the screen image information of the user interface to be evaluated and the text assertion instruction describing the visual or content expected state into the multimodal large model, the model is used to determine through its ability to jointly understand image content and text semantics, and generate an assertion result. The multimodal large model technology can jointly encode and understand image and text information through deep learning methods, and establish the correlation between images and texts. In the field of user interface automation testing, the screen image contains the visual information of the user interface, while the text assertion instruction provides the semantic description of the expected state of the user interface. By comprehensively analyzing these two types of information using the multimodal large model, an accurate judgment of the user interface state can be achieved. Specifically, the model identifies UI elements and their attributes by analyzing the screen image, and combines the text assertion instruction to understand the intention of the tester, so as to determine whether the current UI state meets the expectation. This method bypasses the complex element positioning and code writing processes in the traditional method, and simplifies the implementation method of assertion. Therefore, using the multimodal large model can reduce the difficulty of user interface automation testing and improve the testing efficiency. The present invention can reduce the complexity of writing assertions in UI automation testing. Testers can use natural language or structured text to describe the expected state without writing complex code, which improves the testing efficiency. At the same time, this method can more accurately describe the visual expected state and improve the accuracy of assertion.

[0050] As Figure 1 shown, the invention provides a method for automatically asserting a user interface based on a multimodal large model, which is applied to the field of user interface automation testing and aims to solve the technical problems that the traditional assertion method is complex to write and difficult to accurately describe the visual expected state. The method includes the following steps:

[0051] S100. Obtain the screen image information of the user interface to be evaluated. In this embodiment, it is also used to obtain the visual representation of the current state of the user interface under test. The screen image information here can be either a static image file or a real-time screenshot of the screen. The screen image information includes all visual elements such as controls, text, icons, and layouts in the user interface, providing a basis for the subsequent analysis and judgment of the multimodal large model. This step aims to convert the state of the user interface into a data form that can be processed by the model for subsequent assertion operations. The acquisition of screen image information can be achieved through various methods. For example, the screen image can be intercepted through the API interfaces provided by existing UI automation testing frameworks such as Appium and Airtest, or it can be achieved through the screenshot function provided by the operating system. In specific implementation, the appropriate screenshot method and image format can be selected according to actual needs. For example, the PNG format can be selected to retain the detailed information of the image, or the JPEG format can be selected to reduce the size of the image file. To further improve the quality of the image information, preprocessing can be performed on the obtained screen image, such as image denoising and image enhancement operations. Even without preprocessing, the multimodal large model can also achieve the technical purpose of the present invention. Combining this step with the subsequent steps and inputting the visual information into the multimodal large model is the premise for realizing automated assertion. Through this step, the complex visual information of the UI interface can be effectively converted into data input for model analysis, laying a foundation for the effective execution of the subsequent steps. Obtaining the screen image information enables the multimodal large model to understand the current state of the user interface, laying a foundation for accurately judging whether the interface meets the expectations.

[0052] S200. Obtain a text assertion instruction for describing the visual or content expected state that the user interface should achieve. In this embodiment, the text assertion instruction is an instruction for a tester to describe the expected state of the user interface in the form of natural language or structured text. Such a description can be about the existence, attributes, states, relative position relationships of UI elements, or even more complex visual effects. For example, the text assertion instruction can be "There should be a button with the title 'Login' displayed on the page", or "The username input box cannot be empty", etc. The text assertion instruction, combined with the screen image information, serves as the input to the multimodal large model together to guide the model to determine whether the user interface meets the expectations. The design of the text assertion instruction aims to solve the problems of complex writing and difficult accurate description of the visual expected state in traditional assertion methods. By using natural language or structured text, it reduces the writing burden of testers and improves the efficiency and accuracy of assertions. The form of the text assertion instruction can be various. For example: 1) Natural language text: Testers can directly describe the expected state using natural language, such as "There should be a red login button displayed on the page". 2) Structured text following predefined syntax rules: Testers can use predefined syntax rules to describe the expected state, such as "element[type=button,text=Login,color=red]". 3) Text containing predefined keywords indicating assertion targets or conditions: Testers can use predefined keywords to indicate the assertion targets or conditions, such as "EXIST button[Login]", where "EXIST" indicates that the assertion target is "existence", and "button[Login]" indicates that the target is "a button with the text 'Login'". The text assertion instruction provides a basis for the multimodal large model to understand the intentions of testers and can describe more complex expected states through natural language, etc. In specific implementation, the appropriate form of the text assertion instruction can be selected according to actual needs. For example, if complex visual effects need to be described, natural language text can be selected; if the accuracy of assertions needs to be improved, structured text following predefined syntax rules can be selected. The text assertion instruction enables testers to describe the expected state of the UI in a more natural and efficient manner, thereby reducing the complexity of testing work.

[0053] S300. Input the screen image information and the text assertion instruction into a multimodal large model configured to jointly process image and text information. In this embodiment, the multimodal large model is a deep learning model capable of simultaneously processing two different types of data, i.e., images and texts, and extracting useful information therefrom. In this embodiment, the multimodal large model receives the screen image information and the text assertion instruction as inputs, and by jointly analyzing these two types of information, understands the relationship between the current state and the expected state of the user interface, thereby determining whether the user interface meets the expectations. The configuration of the multimodal large model is the key to achieving automated assertion. It needs to possess powerful image recognition, semantic understanding, and reasoning capabilities in order to accurately determine whether the user interface meets the expectations. Besides the screen image information and the text assertion instruction, inputting only any one of the screen image information and the text assertion instruction can also achieve the technical purpose of the present invention. In one implementation, the multimodal large model can adopt the Transformer architecture. For example, the ViT (Vision Transformer) model can be used to process the image information, the BERT (Bidirectional Encoder Representations from Transformers) model can be used to process the text information, and the outputs of the two are fused through a certain fusion mechanism to achieve the joint understanding of image and text information. The ViT model divides the image into multiple image patches, and then takes these image patches as inputs, and learns the relationships between the image patches through the self-attention mechanism to achieve the recognition of the image. The BERT model, on the other hand, pre-trains on a large amount of text data to learn the semantic information of the text, thereby achieving the understanding of the text. The above models are only examples, and the multimodal large model can be any other model that can achieve the technical purpose of the present invention. The multimodal large model can more accurately understand the state of the user interface and achieve more intelligent automated assertion.

[0054] S400 uses the joint understanding ability of the multimodal large model for image content and text semantics to instruct the model to perform the following determination operations: Based on the expected state description included in the text assertion instruction, analyze the actual user interface state presented in the screen image information to determine whether the actual state meets the expected state. In this embodiment, the joint understanding ability of the multimodal large model for image content and text semantics is used to achieve the automatic judgment of the user interface state. The multimodal large model analyzes the screen image to identify UI elements and their attributes. For example, it can identify the position, size, color, text content, etc. of buttons. At the same time, the model parses the text assertion instruction to understand the intention of the tester. For example, it understands that the tester expects a certain button to display a specific text content. Then, the model fuses the image information and text information to determine whether the actual user interface state meets the expected state described in the text assertion instruction. This process simulates the judgment process of a human tester and realizes automated and intelligent assertion. There can be multiple specific implementation methods for the determination operation. For example: 1) Rule-based method: The model can pre-define some rules. For example, if the text assertion instruction is "There should be a button with the title 'Login' displayed on the page", the model will determine whether there is a button with the text content 'Login' in the screen image. If it exists, it is determined to meet the expected state; otherwise, it is determined not to meet the expected state. 2) Deep learning-based method: The model can learn the correlation between images and texts through deep learning algorithms, and then judge whether the actual user interface state meets the expected state described in the text assertion instruction according to the learned correlation. For example, the model can learn that a "red button" usually represents an "important operation". Therefore, if the text assertion instruction is "There should be a red login button displayed on the page", the model will determine whether there is a red button in the screen image. If it exists, it is determined to meet the expected state; otherwise, it is determined not to meet the expected state. In specific implementation, the appropriate implementation method of the determination operation can be selected according to actual needs. Using the understanding ability of the multimodal large model in this step can achieve more intelligent automated assertion, thereby improving the test efficiency and accuracy. Using the multimodal large model for user interface state determination can more accurately understand the intention of the tester and improve the intelligence level of automated assertion.

[0055] S500. Generate an assertion result used to represent whether the current user interface automation assertion is successful based on the determination result output after performing the determination operation by the multimodal large model. This step is responsible for converting the judgment result of the multimodal large model into a clear assertion result for use by testers or other automated systems. The form of the assertion result can be diverse. For example: 1) Boolean value: True indicates that the assertion is successful, and False indicates that the assertion fails. 2) Enumeration value: For example, Pass indicates that the assertion is successful, Fail indicates that the assertion fails, and Error indicates that an error occurs in the assertion. 3) Structured data containing a confidence score: For example, {"result": "Pass", "confidence": 0.95}, where result represents the assertion result and confidence represents the confidence score. The form of the assertion result can be selected according to actual needs, but no matter which form is adopted, it should be able to clearly represent the result of this assertion. During the process of generating the assertion result, the confidence score output by the multimodal large model can be combined to evaluate the reliability of the assertion result. For example, a confidence threshold can be preset in advance. Only when the confidence score reaches or exceeds this threshold, a clear "assertion passed" or "assertion failed" result is generated; if the confidence score is lower than this threshold, a "assertion uncertain" result can be generated, or a mark that requires manual intervention can be triggered. This method can effectively manage the reliability of the assertion result and avoid deviations in the test result caused by incorrect model judgments. The generation of the assertion result can provide a clear and definite assertion result, which is convenient for testers or other automated systems to use. Generating a clear assertion result can improve the test efficiency and reliability and support more advanced automated test processes.

[0056] In some embodiments, the user interface automation assertion method based on the multimodal large model further includes the following steps:

[0057] When obtaining the screen image information of the user interface to be evaluated, synchronously obtain the user interface hierarchical structure information corresponding to the screen image information;

[0058] Provide the user interface hierarchical structure information, together with the screen image information and the text assertion instruction, as inputs to the multimodal large model to utilize the structured information to assist and enhance the accuracy and robustness of the multimodal large model in performing the determination operation.

[0059] In this embodiment, the user interface hierarchical structure information refers to the description of the hierarchical relationship between user interface elements, usually represented in a tree structure. For example, a page may contain a root node, and multiple child nodes are contained under the root node. For example, buttons, text boxes, etc., and each child node can contain further lower-level child nodes. The user interface hierarchical structure information can provide attributes such as the type, position, size, text content, etc. of user interface elements, as well as the hierarchical relationship between them. Combining the user interface hierarchical structure information with the screen image information can provide more comprehensive user interface information for the multimodal large model, thereby improving the accuracy of assertions. For example, if the text assertion instruction is "There should be a button with the title 'Login' displayed on the page", the model can first find all UI elements of the button type according to the user interface hierarchical structure information, and then determine whether the text content of one of these buttons is "Login". This method can prevent the model from misjudging other UI elements, such as text labels, as buttons, thereby improving the accuracy of assertions. There are various ways to obtain the user interface hierarchical structure information. For example: 1) Through the API interfaces provided by the UI automation testing framework: For example, frameworks such as Appium and Airtest provide API interfaces that can obtain the hierarchical structure information of the user interface. 2) Through the accessibility API provided by the operating system: For example, the Windows operating system provides the UI Automation API that can obtain the hierarchical structure information of the user interface. In specific implementation, the appropriate acquisition method can be selected according to actual needs. Combining the user interface hierarchical structure information with the screen image information and the text assertion instruction can provide more comprehensive user interface information for the multimodal large model, significantly improving the accuracy and robustness of assertions. By introducing the user interface hierarchical structure information, the understanding ability of the multimodal large model for the user interface state can be effectively improved, thereby improving the quality of automated assertions.

[0060] In some embodiments, the user interface hierarchical structure information includes at least one of the identifier, text content, control type, screen boundary coordinates, visibility state, and interactivity state of the control elements in the user interface, providing structured context for the multi-modal large model in addition to pixel information. This information is a more fine-grained description of the user interface elements, which can further enhance the multi-modal large model's ability to understand the user interface state. 1) Identifier of the control element: Used to uniquely identify a UI element. For example, it can be the property values such as the ID or name of the element. 2) Text content: The text information displayed on the UI element. For example, the text of a button, the text in a text box, etc. 3) Control type: The type of the UI element. For example, button, text box, label, image, etc. 4) Screen boundary coordinates: The position and size of the UI element on the screen, usually represented by rectangular coordinates. 5) Visibility state: Whether the UI element is visible. For example, visible or invisible. 6) Interactivity state: Whether the UI element is interactive. For example, enabled or disabled. These structured information can provide richer context for the multi-modal large model, helping the model to more accurately identify UI elements and understand the UI layout, thereby improving the accuracy and robustness of assertions. For example, if the text assertion instruction is "click the login button", the model can first find the UI element of the button type according to the control type, then find the button with the text "login" according to the text content, and finally determine whether the button is visible and interactive. Only when all conditions are met is the assertion determined to be successful. Those skilled in the art understand that the implementation of the technical solution of the present invention is not affected by the absence of any one or more, or even all, of the above technical features. The multi-modal large model can not only understand the visual information in the screen image, but also understand the structured context included in the user interface hierarchical structure information, and thus can more accurately judge the user interface state. This structured context can further enhance the multi-modal large model's ability to understand the user interface state, thereby improving the reliability of automated assertions.

[0061] In some embodiments, the multimodal large model is configured to be a model that has been specifically domain-adaptively trained or fine-tuned for the recognition of user interface elements, the understanding of user interface layouts, or image-based visual question answering tasks, in order to improve the accuracy of its determination of the user interface state. This means that the multimodal large model does not directly use general image and text processing models, but is specifically trained or fine-tuned for the specific domain of user interface automation testing. This specific domain adaptation training or fine-tuning can significantly improve the performance of the model in this domain. For example, it can improve the accuracy of UI element recognition, the ability to understand UI layouts, the ability to understand user intentions, etc. Specifically, for the recognition of user interface elements, a dataset containing a large number of UI element annotations can be used to train the model so that it can accurately recognize various types of UI elements, such as buttons, text boxes, labels, images, etc. For the understanding of user interface layouts, a dataset containing a large number of UI layout annotations can be used to train the model so that it can understand the positional relationships, hierarchical relationships, etc. between UI elements. For image-based visual question answering tasks, a dataset containing a large number of UI images and corresponding questions can be used to train the model so that it can answer relevant questions based on the image content, such as "Is the 'Login' button displayed on the page?". Through the above specific domain adaptation training or fine-tuning, the multimodal large model can better meet the needs of user interface automation testing, thereby improving the accuracy and robustness of assertions. Conducting specific domain adaptation training or fine-tuning can enable the multimodal large model to better understand the user interface, thereby improving the accuracy of automated assertions.

[0062] In some embodiments, the determined result of the output further includes a quantization metric, which is configured to be a confidence score of the multimodal large model for its basic judgment. The confidence score is a quantitative representation of the confidence level of the multimodal large model for its judgment result, usually a value between 0 and 1. The higher the confidence score, the more confident the model is in its judgment result; the lower the confidence score, the less confident the model is in its judgment result. During the automated assertion process, the confidence score can be used to evaluate the reliability of the assertion result and take corresponding measures. For example, a confidence threshold can be preset in advance, and only when the confidence score reaches or exceeds this threshold is the assertion result considered reliable; if the confidence score is lower than this threshold, the assertion result can be considered unreliable and requires manual review or other processing. The confidence score can provide more abundant information to help testers evaluate the reliability of the assertion result and take corresponding measures. Combining the confidence score can effectively manage the reliability of the assertion result and avoid deviations in the test result caused by incorrect model judgments.

[0063] In some embodiments, the process of S500 generating the assertion result includes:

[0064] Compare the confidence score with a preset confidence threshold;

[0065] When the confidence score reaches or exceeds the confidence threshold, generate a clear "assertion passed" or "assertion failed" result based on the basic judgment;

[0066] When the confidence score is lower than the confidence threshold, generate a result of "assertion uncertain" or trigger a mark that requires manual intervention, so as to use the confidence score to manage the reliability of the assertion result.

[0067] In this embodiment, the method does not simply take the judgment result of the multimodal large model directly as the assertion result, but introduces a confidence threshold for filtering and managing the assertion result. The confidence threshold is a preset value used to judge whether the judgment result of the multimodal large model is reliable enough. For example, the confidence threshold can be set to 0.9, indicating that only when the confidence of the multimodal large model in its judgment result reaches more than 90% is the judgment result considered reliable. When the confidence score reaches or exceeds the confidence threshold, it means that the multimodal large model is relatively confident in its judgment result. At this time, a clear "assertion passed" or "assertion failed" result can be generated based on this judgment result. When the confidence score is lower than the confidence threshold, it means that the multimodal large model is not very confident in its judgment result. At this time, a result of "assertion uncertain" can be generated, or a mark that requires manual intervention can be triggered. For example, assign this assertion task to a manual tester for review. By introducing the confidence threshold, the reliability of the assertion result can be effectively managed, and the deviation of the test result caused by the model judgment error can be avoided. The implementation of the technical solution of the present invention is not affected even if any one or more, or even all, of the screen image information and the text assertion instruction are missing. By introducing the confidence threshold, the reliability of automated assertion can be effectively improved, and a more flexible assertion management strategy can be supported. Using the confidence score to manage the assertion result can improve the quality and efficiency of automated testing and reduce the testing risk.

[0068] In some embodiments, the determined output result further includes explanatory information that describes the key visual regions or text elements based on which the model makes a judgment. The form of the explanatory information is a text summary or a visual annotation on the screen image information, aiming to improve the transparency and debuggability of the assertion process. This explanatory information can help testers understand the reasons for the multi-modal large model to make a judgment, thereby better evaluating the reliability of the assertion result and conducting problem troubleshooting. 1) Text summary: The model can generate a piece of text briefly describing the key visual regions or text elements based on which it makes a judgment. For example, if the model determines that "a button with the title 'Login' is displayed on the page", it can generate a piece of text "The model recognizes a button with the text content 'Login'". 2) Visual annotation on the screen image information: The model can make annotations on the screen image to highlight the key visual regions based on which it makes a judgment. For example, if the model determines that "a button with the title 'Login' is displayed on the page", it can use a rectangular box to highlight the button on the screen image. The explanatory information can improve the transparency and debuggability of the assertion process, helping testers better understand the model behavior, thereby improving the test efficiency and quality.

[0069] In some embodiments, the user interface automation assertion method based on the multi-modal large model is embedded in the execution process of the user interface automation test script and is triggered when it is necessary to verify the user interface state after one or more interactive operation steps are completed in the script. This method can be used as an independent module and embedded into the existing user interface automation test script. During the execution of the test script, when it is necessary to verify the user interface state, this method can be called for assertion. In this way, this method can be conveniently integrated into the existing test process to improve the test efficiency. For example, test scripts can be written using UI automation test frameworks such as Airtest or Appium, and this method can be called in the script for assertion. This method is seamlessly integrated with the existing test process, which can conveniently improve the test efficiency. Embedding this method into the execution process of the user interface automation test script can facilitate automated assertion and improve the test efficiency.

[0070] In some embodiments, the user interface automation assertion method based on the multi-modal large model forms a continuously optimized closed loop, including:

[0071] Recording the generated assertion results and the corresponding input information;

[0072] Receiving feedback confirmation information from the operator or the verification system on whether the assertion result is correct;

[0073] Periodically or after sufficient feedback is collected, use the feedback confirmation information as a supervision signal to adaptively update or retrain the multimodal large model to continuously improve its performance in the application scenario of user interface automation assertion.

[0074] In this embodiment, the user interface automation assertion method based on the multimodal large model is not a one-time assertion process, but a continuously optimized closed loop. By recording the assertion results and the corresponding input information, a dataset for model training can be constructed; by receiving the feedback confirmation information of the operator or the verification system on the assertion results, the correctness label of the assertion results can be obtained; by periodically or after sufficient feedback is collected, using the feedback confirmation information as a supervision signal to adaptively update or retrain the multimodal large model, the performance of the model in the application scenario of user interface automation assertion can be continuously improved. Such a continuously optimized closed loop can enable the multimodal large model to continuously learn and adapt to new user interfaces and test requirements, thereby maintaining a high assertion accuracy and robustness. The implementation of the technical solution of the present invention is not affected by the absence of any one or more, or even all, of the screen image information and text assertion instructions. By constructing a continuously optimized closed loop, the performance of the multimodal large model in the application scenario of user interface automation assertion can be continuously improved, thereby improving the test efficiency and quality. Using the continuously optimized closed loop, the automated test system can continuously learn and evolve to adapt to the changing user interfaces and test requirements.

[0075] As Figure 2 shown is a user interface automation assertion system provided by another aspect of the present invention, which is applied to the field of user interface automation testing and is used to implement the above-mentioned user interface automation assertion method based on the multimodal large model, including:

[0076] An information acquisition module M100, configured to acquire the screen image information of the user interface to be evaluated, and acquire the text assertion instructions used to describe the visual or content expected state that the user interface should achieve;

[0077] A multimodal large model M200, which is configured to be able to jointly process image and text information, and is used to receive the screen image information and text assertion instructions provided by the information acquisition module M100, and use its joint understanding ability of image content and text semantics to perform a determination operation to determine whether the actual user interface state presented by the screen image information meets the expected state included in the text assertion instructions, and output a determination result;

[0078] The result generation module M300 is connected to the multimodal large model M200 and is configured to generate an assertion result for characterizing whether the current user interface automation assertion is successful based on the determination result output by the multimodal large model M200. Through the collaborative work of the information acquisition module M100, the multimodal large model M200, and the result generation module M300, user interface automation assertion can be achieved. Among them, the information acquisition module M100 is responsible for acquiring relevant information of the user interface (such as screen images, text assertion instructions, etc.); the multimodal large model M200 is responsible for analyzing and judging this information to generate a determination result; the result generation module is responsible for converting the determination result into a clear assertion result.

[0079] As Figure 3 shown, it is another implementation manner of a user interface automation assertion method based on a multimodal large model provided by an embodiment of the present invention. A user interface (UI) automation assertion system and method based on a multimodal large model (Multimodal Large Language Model, MLLM) include:

[0080] Building and configuring a user interface automation assertion system based on a multimodal large model:

[0081] UI automation scheduling system: First, build or configure a UI automation scheduling system. This system is developed and constructed by extending existing automation frameworks such as Appium and AirTest, and has the ability to manage and schedule multiple mobile devices under test, such as mobile phone A and mobile phone B, to execute automation test scripts. Testers write test cases through this system. The test cases contain conventional UI interaction operation steps such as clicking, inputting, and swiping, as well as text assertion instructions embedded at key nodes where the UI state needs to be verified. For example, in the verification of a successful login scenario, after the test script executes the login operation, the corresponding text assertion instruction may be written as: "Verify that the text 'Login Successful' is included at the top of the page and the user nickname 'Zhang San' is displayed."

[0082] Mobile device and interface operation: The UI automation scheduling system, according to the test task, instructs the target mobile device, such as mobile phone A, to execute the UI operation steps in the test script, so that it navigates to a specific user interface to be asserted, such as a specific page of an APP.

[0083] Information acquisition and sending: When the script executes to the assertion node, the UI automation scheduling system triggers the information acquisition process:

[0084] Instruct mobile phone A or capture the screen image information of the current interface through the proxy program on it.

[0085] Extract the corresponding text assertion instruction from the test script.

[0086] The UI automation scheduling system can also be configured to obtain the user interface structured information of the current interface simultaneously, which is not explicitly marked in the figure but can be used as additional data for image information and text instructions.

[0087] The UI automation scheduling system sends the packaged screen image information, text assertion instructions, and optional structured information to the assertion processing service through the network interface. In this embodiment, this service plays the role of the message system in the above embodiment and undertakes the subsequent processing logic.

[0088] Assertion processing service: Set up a backend assertion processing service. This service receives the assertion requests from the UI automation scheduling system, which contain image information and text assertion instructions. It is responsible for:

[0089] Verifying and formatting the received data.

[0090] Invoking the inference interface of the deployed multimodal large model for the formatted image and text data according to the predetermined protocol.

[0091] Multimodal large model: Deploy a multimodal large model. This model is preferably a model fine-tuned for UI understanding or visual question answering tasks and has strong joint image-text understanding capabilities. The MLLM receives the input from the assertion processing service and performs a determination operation: analyzing the screen image information and making a judgment based on the expected state described by the text assertion instructions. For example, "The top of the page contains the text 'Login Successful' and displays the user nickname 'Zhang San'", to determine whether the actual UI state in the image meets this expectation. The model outputs a determination result, which may include a boolean judgment such as satisfied / not satisfied and a confidence score.

[0092] Assertion execution and result feedback.

[0093] Result generation and return: The assertion processing service receives the determination result from the MLLM. Inside the service, according to the preset logic, for example, combining the confidence threshold judgment to generate the final assertion result, such as: Pass, Fail, Uncertain. The assertion processing service returns this assertion result to the UI automation scheduling system.

[0094] Test status update: After receiving the assertion result, the UI automation scheduling system updates the execution status of the assertion step in the corresponding test case and finally determines whether the entire test case passes or fails.

[0095] Through the implementation of this embodiment, when writing UI automated test cases, testers mainly focus on writing scripts for page interaction operations and writing text assertion instructions in a way close to natural language. Compared with the traditional method that requires writing complex element location codes such as XPath and status check logics, this embodiment significantly reduces the difficulty and workload of writing the assertion part and improves the development efficiency of test cases.

[0096] On the other hand, the present invention also provides a user interface automated assertion program product based on a multi-modal large model, including computer instructions, which implement the steps of the above-mentioned user interface automated assertion method based on a multi-modal large model when executed by a processor. By encapsulating the method of the present invention into a program product, it can be conveniently deployed and used on various computing devices, thereby improving the efficiency and quality of user interface automated testing. The program product can take various forms. For example, it can be an independent application program, a software library, a plug-in, etc. Those skilled in the art can select a suitable form of the program product according to actual needs.

[0097] The present invention effectively solves the technical problems existing in the prior art, such as the need for testers to write complex assertion codes, the difficulty in accurately describing the visual expected state, and the low efficiency of UI automated testing. Through the present invention, testers can describe the expected state of the user interface in natural language without writing complex codes, thereby reducing the testing difficulty, improving the testing efficiency, and more accurately describing the visual expected state, thus increasing the test coverage and quality, and providing a more efficient and intelligent solution for the field of UI automated testing.

[0098] The above content is a further detailed description of the present invention in combination with specific preferred implementation manners, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A user interface automation assertion method based on a multi-modal large model, applied to the field of user interface automation testing, characterized in that, It includes the following steps: Obtain the screen image information of the user interface to be evaluated; Obtain a text assertion instruction used to describe the visual or content expected state that the user interface should achieve; Input the screen image information and the text assertion instruction into a multimodal large model configured to jointly process image and text information; Utilize the joint understanding ability of the multimodal large model for image content and text semantics, and instruct the model to perform the following determination operations: Analyze the actual user interface state presented in the screen image information according to the expected state description included in the text assertion instruction to determine whether the actual state meets the expected state; Based on the determination result output after the multimodal large model performs the determination operation, generate an assertion result used to represent whether the automation assertion of the user interface is successful this time.

2. The method for automatically asserting a user interface based on a multi-modal large model according to claim 1, wherein The obtained text assertion instruction is in the form of at least one of natural language text and / or structured text following predefined grammar rules and / or text containing predefined keywords indicating assertion targets or conditions.

3. The method for automatically asserting a user interface based on a multimodal large model according to claim 1, wherein The method further includes: When obtaining the screen image information, synchronously obtain the user interface hierarchical structure information corresponding to the screen image information; Provide the user interface hierarchical structure information, together with the screen image information and the text assertion instruction, as inputs to the multimodal large model to utilize the structured information to assist and enhance the accuracy and robustness of the multimodal large model in performing the determination operation.

4. The method for automatically asserting a user interface based on a multimodal large model according to claim 3, wherein The user interface hierarchical structure information includes at least one of the identifier, text content, control type, screen boundary coordinates, visibility state, and interactivity state of the control elements in the user interface.

5. The method for automatically asserting a user interface based on a multi-modal large model according to claim 1, characterized in that, The multimodal large model is configured as a model that has been specifically domain-adaptively trained or fine-tuned for user interface element recognition, user interface layout understanding, or image-based visual question answering tasks.

6. The method for automatically asserting a user interface based on a multi-modal large model according to claim 1, characterized in that, The output determination result further includes a quantitative index, and the index is configured as a confidence score of the multimodal large model for its basic judgment.

7. The method for automatically asserting a user interface based on a multi-modal large model according to claim 6, wherein The process of generating the assertion result includes: Compare the confidence score with a preset confidence threshold; When the confidence score reaches or exceeds the confidence threshold, generate a clear assertion pass or assertion fail result based on the basic judgment; When the confidence score is lower than the confidence threshold, generate an assertion uncertain result or trigger a mark requiring manual intervention.

8. The method for automatically asserting a user interface based on a multi-modal large model according to claim 6, wherein, The output determination result further includes explanatory information, and the explanatory information explains the key visual regions or text elements based on which the model makes a judgment. The explanatory information is in the form of a text summary or a visual annotation on the screen image information.

9. A user interface automation assertion system based on a multi-modal large model, which is applied to the field of user interface automation testing and is used to implement the user interface automation assertion method based on a multi-modal large model described in any one of claims 1 to 8, characterized in that, It includes: An information acquisition module configured to obtain the screen image information of the user interface to be evaluated and obtain a text assertion instruction used to describe the visual or content expected state that the user interface should achieve; A multimodal large model, configured to jointly process image and text information, and used to receive the screen image information and the text assertion instruction provided by the information acquisition module, and execute a determination operation by using its joint understanding ability of image content and text semantics to determine whether the actual user interface state presented by the screen image information meets the expected state included in the text assertion instruction, and output a determination result; A result generation module, connected to the multimodal large model, configured to generate an assertion result for characterizing whether the user interface automation assertion is successful or not based on the determination result output by the multimodal large model.

10. A user interface automation assertion program product based on a multi-modal large model, characterized in that, The program product includes computer instructions, and when the computer instructions are executed by a processor, the steps of the user interface automation assertion method based on a multimodal large model according to any one of claims 1 to 8 are implemented.

Images