System for automated user interface testing and method thereof

Abstract

Disclosed is a system for automated user interface (UI) testing. The system has a test driver unit with multi-modal models 201, 203. The test driver unit creates an animated video of possible UI sta

Description

FIEED OF INVENTION The field of the present invention generally relates to a system for user interface (UI) testing. More specifically, the present invention pertains to automated UI testing, which is a crucial aspect of software development and application maintenance. BACKGROUND An effective user interface (UI) ensures that users can interact with an application in an intuitive and efficient manner. UI testing verifies that the UI functions as expected and provides a satisfactory user experience. Currently, the UI testing is primarily conducted through automated test drivers that operate on the Application Programming Interface (API) level. The existing test drivers interact with the application's UI through pre-determined click and keystroke patterns, simulating user interactions to test the functionality of the UI. However, the existing approach for the UI testing has several limitations. Firstly, the test drivers operate on the API level and not on the pixel level. This means that the existing test drivers may miss certain error patterns that occur at the pixel level, such as accidental overlays where one UI element covers another UI element, blocking interaction. Other potential issues that may go undetected include slight layout problems, such as misaligned text, and raw rendering bugs, such as color artifacts in the rendered image. Secondly, the test drivers rely on hard-coded click and keystroke patterns, which limits their ability to test the UI under a range of user interaction scenarios. As a result, the existing test drivers may not be able to detect issues that only occur under specific interaction patterns. Therefore, there is a need to overcome the problems discussed above. A more comprehensive approach to the automated UI testing is needed, one that can detect error patterns at the pixel level and accommodate a broader range of user interaction scenarios. This would ensure a more thorough and accurate evaluation of the UI, leading to improved functionality and user experience. OBJECT OF THE INVENTION An object of the present invention is to provide a system and a method for automated user interface (UI) testing to detect error patterns in current rendered UI that cannot be addressed by existing test drivers. Another object of the present invention is to provide an advanced system and a method for the automated UI testing on pixel level. Yet another object of the present invention is to provide a test driver unit that utilizes a first multi-modal model to detect perceived available actions by analyzing the current rendered UI. Yet another object of the present invention is to provide a test driver unit that utilizes a second multi-modal model to detect plurality of possible UI states and is pre-trained on a separate network with sample perceived available actions data and human-UI data comprising vehicle related interactions of user. Yet another object of the present invention is to provide a bug detector unit for detecting error patterns in the current rendered UI and is pre-trained on rendered UI images and corresponding videos for transitions. Yet another object of the present invention is to provide an efficient system and a method for detecting error patterns in the current rendered UI. Yet another object of the present invention is to provide a system that detects error patterns in temporal animations as well due to utilization of recurrent models. SUMMARY A system for automated User Interface (UI) testing in accordance with the present disclosure is disclosed. The system for automated User Interface (UI) testing is implemented by at least one processor. The system comprises a test driver unit and a bug detector unit. The test driver unit includes a first multi-modal model and a second multi-modal model. The test driver unit is configured to create a video of a plurality of possible UI states and their corresponding transitions for a current rendered UI. The bug detector unit is configured to detect error patterns in the current rendered UI using the video of the plurality of possible UI states and their corresponding transitions. The current rendered UI is provided to the test driver unit in a form of a plurality of input images. In an embodiment of the present invention, the first multi-modal model is configured to detect perceived available actions by analyzing the current rendered UI. The second multi-modal model is configured to determine the plurality of possible UI states by analyzing the perceived available actions received from the first multi-modal model. The second model is pre-trained on a first training data comprising sample perceived available actions data and human-UI interactions data to determine the plurality of possible UI states. The perceived available actions and the plurality of possible UI states are combined to create the video of the plurality of possible UI states and their corresponding transitions. In another embodiment of the present invention, the bug detector unit comprises a third multi-modal model configured to detect one or more undesirable UI states and a fourth multi-modal model configured to detect and process one or more unintentional UI interactions. The processing of the one or more unintentional UI interactions includes any one of either ignoring or correcting the one or more unintentional UI interactions. In an example embodiment, we may have a touchscreen in a vehicle, the touch screen can show a representation of available parking spots nearby, the user is supposed to select which parking spot to park into. In between the time when the user has decided to select one option and when the users tap is registered by the display, the options could have completely changed, leading to an erroneous user input. A method for automated User Interface (UI) testing in accordance with the present invention is disclosed. The method includes receiving a user input using the UI to generate a current rendered UI. The user input corresponds to a specific task to be performed on the UI. The method further comprises analyzing the current rendered UI by a test driver unit to generate a video of a plurality of possible UI states. Furthermore, the method involves detecting by a bug detector unit the error patterns in the current rendered UI using the video of the plurality of possible UI states received from the test driver unit. The foregoing paragraphs have been provided by way of general introduction and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF FIGURES The foregoing and other features of this disclosure will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the disclosure and are, therefore, not to be considered limiting of its scope, the disclosure will be described with additional specificity and detail through use of the accompanying drawings, in which: FIG. 1 illustrates an exemplary block diagram representing a system for automated user interface (UI) testing in accordance with the present disclosure; FIG. 2 illustrates an exemplary block diagram representing a processor of the system for automated UI testing in accordance with the present disclosure; FIG. 3a illustrates an exemplary block diagram representing functionality of a test driver unit in accordance with the present disclosure; FIG. 3b illustrates an exemplary block diagram representing functionality of a bug detector unit in accordance with the present disclosure; FIG. 4a illustrates an exemplary block diagram representing training of a second multi-modal model in accordance with the present disclosure; FIG. 4b illustrates an exemplary block diagram representing training of a third multi-modal model in accordance with the present disclosure; FIG. 4c illustrates an exemplary block diagram representing training of a fourth multi-modal model in accordance with the present disclosure; and FIG. 5 illustrates an exemplary flowchart representing a method for automated UI testing in accordance with the present disclosure. DETAILED DESCRIPTION Aspects of the present invention are best understood by reference to the description set forth herein. All the aspects described herein will be better appreciated and understood when considered in conjunction with the following descriptions. It should be understood, however, that the following descriptions, while indicating preferred aspects and numerous specific details thereof, are given by way of illustration only and should not be treated as limitations. Changes and modifications may be made within the scope herein without departing from the spirit and scope thereof, and the present invention herein includes all such modifications. As used herein, the term ‘exemplary’ or ‘illustrative’ means ‘serving as an example, instance, or illustration.’ Any implementation described herein as exemplary or illustrative is not necessarily to be construed as advantageous and / or preferred over other embodiments. Unless the context requires otherwise, throughout the description and the claims, the word ‘comprise’ and variations thereof, such as ‘comprises’ and ‘comprising’ are to be construed in an open, inclusive sense, i.e., as ‘including, but not limited to.’ Referring to FIG. 1, a block diagram represents an embodiment of a system 100 for automated user interface (UI) testing. The system 100 comprises at least one processor 101, a test driver unit 103, a bug detector unit 105, and a display unit 107. The processor 101 serves as the central processing unit of the system 100. It manages the overall operations and coordinates the activities of the other hardware components in the system 100. It takes instructions from the test driver unit 103 and the bug detector unit 105, executes these instructions, and manages the flow of information through the system 100. Further, the test driver unit 103 is equipped with a first multi-modal model 201 and a second multi-modal model 203. It is responsible for creating a video of a plurality of possible UI states 305 and their corresponding transitions for a current rendered UI 301. The first multi-modal model 201 within the test driver unit 103 is configured to detect perceived available actions 303 by analyzing the current rendered UI 301. The second multi-modal model 203 within the test driver unit 103, on the other hand, is configured to determine the plurality of possible UI states 305 by analyzing the perceived available actions 303 received from the first multi-modal model 201. Further, the current rendered UI 301 is provided to the test driver unit 103 in a form of a plurality of input images. The perceived available actions 303 and the plurality of possible UI states 305 are combined to create the video of the plurality of possible UI states 305 and their corresponding transitions. The bug detector unit 105 is configured to detect error patterns in the current rendered UI 301 using the video of the plurality of possible UI states 305 and their corresponding transitions. It is thus able to detect error patterns in the current rendered UI 301 at pixel level. The bug detector unit 105 comprises a third multi-modal model 205 configured to detect one or more undesirable UI states 307 and a fourth multi-modal model 207 configured to detect and process one or more unintentional UI interactions 309. The processing of these unintentional UI interactions includes either ignoring or correcting the unintentional UI interactions. Finally, the display unit 107 is an output device that presents the results of the UI testing. It shows the current rendered UI 301 and the detected error patterns. It also presents the video of the plurality of possible UI states 305 and their corresponding transitions, enabling users of the system 100 to visualize the potential states of the UI and understand the detected error patterns better. In an embodiment of the present invention, the system 100 may also include additional components, such as storage units for storing the training data sets used by the multi-modal models, and networking devices for enabling communication with other systems or devices. In an embodiment of the present invention, each of the multi-modal models within the test driver unit 103 and the bug detector unit 105 could be implemented using different machine learning algorithms, depending on the specific requirements of the UI testing. For instance, the first multi-modal model 201 could be implemented using a convolutional neural network, while the second multimodal model 203 could be implemented using a recurrent neural network. FIG. 2 is a block diagram illustrating an automated system 100 for User Interface (UI) testing. The system 100 is implemented by the at least one processor 101. The system 100 is organized into two units, namely the test driver unit 103 and the bug detector unit 105. The test driver unit 103 includes a first multi-modal model 201 and a second multi-modal model 203. The first multi-modal model 201, is programmed to detect perceived available actions 303 by analyzing the current rendered UI 301. In an embodiment, the first multi-modal model 201 used in the automated system 100 is a GPT4. In contrast, the second multi-modal model 203 is pre-trained on a first training data 401 comprising sample perceived available actions data and human-UI interactions data. This pre-training enables the second multi-modal model 203 to determine a plurality of possible UI states 305 by analyzing the perceived available actions 303 received from the first multi-modal model 201. The bug detector unit 105 is configured to detect error patterns in the current rendered UI 301 using the video of the plurality of possible UI states 305 and their corresponding transitions. To this end, the bug detector unit 105 includes a third multi-modal model 205 and a fourth multi-modal model 207. The third multi-modal model 205 is configured to detect one or more undesirable UI states 307 or renderings. The fourth multi-modal model 207, on the other hand, is programmed to detect and process one or more unintentional UI interactions 309. The processing of these unintentional UI interactions includes either ignoring or correcting the one or more unintentional UI interactions 309. An alternative embodiment of the current invention might include additional multi-modal models within the test driver unit 103 and the bug detector unit 105. These additional models could be pre-trained on different types of data to handle a broader range of UI states, interactions, and errors. In another embodiment, the test driver unit 103 and the bug detector unit 105 could be combined into a single unit. This combined unit could use a shared pool of multi-modal models to both generate the plurality of possible UI states 305 and detect error patterns. This could potentially reduce the computational resources required by the system 100. Furthermore, each of the multi-modal models 201, 203, 205, 207 used in the system 100 could be implemented using various machine learning algorithms, such as deep learning, reinforcement learning, or a combination thereof. The choice of machine learning algorithm could depend on the specific requirements of the UI testing task, such as the complexity of the UI, the variety of user interactions, and the desired accuracy of error detection. In another alternative embodiment, the system 100 could include additional hardware or software modules to handle tasks such as data preprocessing, model training, and model evaluation. These additional modules could improve the efficiency and effectiveness of the UI testing process. Referring to FIG. 3a, illustrating a block diagram representing functionality of the test driver unit 103 in accordance with some embodiments of the present invention. The test driver unit 103 primarily includes a first multi-modal model 201 and a second multi-modal model 203. Initially, a current rendered UI 301, is subject to analysis by the first multi-modal model 201. The first multi-modal model 201, which could be a generalized multi-modal model, is configured to detect the perceived available actions 303 by analyzing the current rendered UI 301. The outcome of this analysis is the perceived available actions 303. Further, the perceived available actions 303 are then inputted into the second multi-modal model 203. The second multi-modal model 203 is specifically pre-trained on the first training data 401 that includes the sample perceived available actions data and the human-UI interactions data. The second multi-modal model 203 is thus configured to determine the plurality of possible UI states 305 by analyzing the perceived available actions 303 received from the first multi-modal model 201. In an alternative embodiment, the first multi-modal model 201 could be another type of generalized multi-modal model, not limited to GPT4. Similarly, the second multi-modal model 203 could be trained on other types of data that include perceived available actions 303 and human-UI interactions related to different application areas, not limited to vehicle-related interactions. The plurality of possible UI states 305, are the result of the combined outputs from the first multi-modal model 201 and the second multi-modal model 203. The plurality of possible UI states 305 represent the various UI states that could be achieved from the current rendered UI 301. By this process, the video is created for the plurality of possible UI states 305 and their transitions, which provides a comprehensive visual representation of all the potential UI states that could be achieved from the current rendered UI 301. In some other embodiments, the test driver unit 103 could include additional multi-modal models, each trained on different types of data and configured to perform different types of analysis on the current rendered UI 301. This could provide a more comprehensive and in-depth analysis of the current rendered UI 301, leading to the detection of a wider range of the possible UI states 305. In conclusion, the detailed functionality of the test driver unit 103 as depicted in FIG. 3a provides a comprehensive mechanism for detecting and analyzing the plurality of possible UI states 305 from the current rendered UI 301. The test driver unit 103, through the combination of the first multi-modal model 201 and the second multi-modal model 203, ensures a thorough analysis of the current rendered UI 301, leading to the creation of the video that visually represents the plurality of possible UI states and their transitions. This effectively assists in the automated testing of the UI, thereby enhancing the efficiency and accuracy of the testing process. FIG. 3b illustrates an exemplary block diagram representing the functionality of the bug detector unit 105 in accordance with embodiments of the present invention. The bug detector unit 105, as represented above in FIG. 2, comprises a third multimodal model 205 and a fourth multimodal model 207, both of which are instrumental in the detection of the error patterns in the current rendered user interface (UI). The third multimodal model 205, as shown in FIG. 3b, is configured to detect one or more undesirable UI states 307. In an embodiment, the one or more undesirable UI states 307 include instances where critical information is not visible due to overlapping elements or missing elements, situations where critical actions are not inferable from the current UI state, layout inconsistencies which do not fulfill design requirements, and application impersonation protection issues. In addition to the third multimodal model 205, the bug detector unit 105 also comprises a fourth multimodal model 207. The fourth multi-modal model is configured to detect and process one or more unintentional UI interactions 309. The processing of the one or more unintentional UI interactions 309 includes either ignoring or correcting these interactions. The fourth multi-modal model 207 is trained on a third training data 405, which includes sample user inputs from previous user interactions and corresponding UI interactions. The third training data 405 is formed based on heuristics. For instance, if a user instantly reverts a UI interaction, it is likely that the interaction was unintentional and is probably an error. The system 100 is thus capable of recognizing such error patterns and taking appropriate actions. In addition to the third multi-modal model 205 and fourth multi-modal model 207, the bug detector unit 105 is trained to handle the plurality of possible UI states 305. These UI states could include various UI elements and their states, such as drop-down menus, pop-ups, selected text boxes, text boxes containing text, forms, tabs, and so on. The bug detector unit 105 leverages these varied UI states to facilitate comprehensive and effective testing of the UI. In an alternative embodiment, the bug detector unit 105 could also be configured to detect error patterns in temporal animations due to the utilization of recurrent model. This allows the system 100 to detect errors not just in static UI states, but also in dynamic UI states that involve motion or changes over time. This further enhances the comprehensiveness and effectiveness of the testing carried out by the system 100. In yet another alternative embodiment, the bug detector unit 105 could be configured to handle UI testing in different types of interfaces. For instance, the bug detector unit 105 could be trained to handle testing in graphical user interfaces, touch user interfaces, voice user interfaces, command-line interfaces, natural language user interfaces, gesture-based interfaces, augmented reality interfaces, virtual reality interfaces, and web user interfaces. This makes the system 100 highly versatile and adaptable to a wide range of UI testing requirements. In conclusion, FIG. 3b illustrates the core components of the bug detector unit 105 and their functionality in detecting and processing error patterns in the UI. The third multi-modal model 205 and the fourth multi-modal model 207, along with their capacity to handle the plurality of possible UI states 305, form the crux of the bug detector unit's 105 ability to carry out comprehensive and effective UI testing. FIG. 4a illustrates a block diagram showing the training of the second multimodal model 203 using the first training data 401, in accordance with the disclosed embodiments of the present invention. The second multi-modal model 203 is a significant component of the test driver unit 103, which is integral to the automated User Interface (UI) testing system 100. The second multi-modal model 203 is preconfigured to analyze the perceived available actions 303, which are derived from the current rendered UI 301. These perceived actions are obtained from the first multi-modal model 201, which is a generalized model. Further, the first training data 401 is instrumental in the pre-training of the second multi-modal model 203. The first training data includes the sample perceived available actions data and the human-UI interactions data. Particularly, the human-UI interactions data pertains to the interactions of a user with one or more user interfaces on an infotainment screen of a vehicle. The goal of the training is to enable the second multi-modal model 203 to determine the plurality of possible UI states 305 by analyzing the perceived available actions 303. The second multi-modal model 203 is trained to understand the exhaustive exploration of UI states. The first training data 401 may include various UI state examples such as the status of a drop-down menu, whether a popup is in front, whether a text box is selected, whether a text box has some text in it, the forms that are active, the tabs that are active, and so on. The first training data 401 also includes a loss function that is configured to reward models that get as close as possible to the known number of interesting UI states, covering all interesting states without exceeding or falling short of this number. Further, the comprehensive understanding of the training process of the second multi-modal model 203, reveals the vital role it plays in the automated UI testing system 100. In an alternative embodiment, the first training data 401 can also include other examples of the human-UI interactions, such as the interaction of the user with a navigation interface, a media interface, or a communication interface of a vehicle. These additional interaction examples can further enhance the training of the second multi-modal model 203, enabling it to analyze a broader range of perceived available actions 303. Referring to FIG. 4b, a block diagram for training the third multi-modal model 205 is presented. The block diagram illustrates the process of training the third multi-modal model 205 using the second training data 403. The third multimodal model 205, an integral part of the bug detector unit 105, and responsible for the detection of one or more undesirable user interface states or renderings. The third multi-modal model is trained on a specific set of data, referred to as the second training data 403. The second training data 403 comprises manually labeled undesirable UI states. Examples of these undesirable UI states include scenarios where a "next" button is not visible due to an unusually large font size chosen by the user, resulting in the "next" button overflowing. Additionally, the second training data 403 includes instances where the text is illegible because of a color mismatch, such as white font on a bright yellow background. Another example is a situation where all form fields are filled with text, no error message is displayed, but the "next" button remains greyed out. Yet another instance includes a scenario where tapping the "Mute" button has no effect because a popup reminding the user to pay for a SiriusXM subscription is displayed on the top right comer. Training the third multi-modal model 205 with the second training data 403 enables it to identify and recognize the one or more undesirable UI states when they occur during the automated UI testing process. It's important to note that the third multi-modal model 205 is not limited to the examples provided in the second training data 403. The model identifies a wide range of undesirable UI states, making it a versatile tool in the bug detector unit 105. In an example embodiment, the third multi-modal model described above, could include model which reinterprets actual user inputs in a vehicle based on what the user meant to do as compared to what the user interface actually made him do at that time. It could also take into account that the driver’s hand is covering parts of the display or that the honk of the firetruck right next to the car has likely been so loud that the most recent voice output was not heard. Or also that a video playing on the car with the audio “hey Mercedes” probably should not trigger the “hey Mercedes” voice assistant. As an alternative embodiment, the second training data 403 could also include instances of undesirable UI states caused by other UI design elements, such as icons or graphics, in addition to text-based elements. This would further enhance the capability of the third multi-modal model 205 to detect a broader range of undesirable UI states. In conclusion, the training of the third multi-modal model 205 using the second training data 403 is a crucial step in the automation of UI testing. It equips the system 100 with the ability to detect and identify undesirable UI states, which is a significant step towards ensuring a user-friendly and efficient UI. Referencing FIG. 4c, illustrates the block diagram representing training process of the fourth multi-modal model 207, within the automated user interface (UI) testing system 100. This training process involves the use of the third training data 405. The fourth multi-modal model 207 is a component of the bug detector unit 105 within the system 100. The fourth multi-modal model 207 is configured to detect and process one or more unintentional UI interactions 309. The processing of these unintentional UI interactions comprises either ignoring or correcting these interactions. This feature of the model has a direct impact on the overall efficiency of the testing system 100, ensuring that only relevant UI interactions are considered in the detection of errors and bugs. The training of the fourth multi-modal model 207 is carried out using the third training data 405. The third training data comprises user inputs from previous user interactions with the UI and is based on heuristics. One example of this can be if a user instantly reverts to a UI interaction, it is likely due to a user error. This heuristic-based approach allows the model to learn from past interactions, enhancing its ability to accurately detect and process unintentional UI interactions in future instances. An alternative embodiment for the fourth multimodal model 207 could involve training the fourth multi-modal model 207 using a combination of heuristic-based data and pattern recognition techniques. This could potentially improve the model's ability to distinguish between intentional and unintentional UI interactions, thereby improving the overall performance of the bug detector unit 105 within the system 100. Moreover, the third training data 405, while primarily comprising user inputs from previous interactions, could also be enriched with additional data sources for more comprehensive training. These could include simulated UI interaction data or crowd-sourced user interaction data, which would provide a wider range of interaction scenarios for the model to learn from. In conclusion, FIG. 4c provides a detailed view of the training process for the fourth multi-modal model 207 using the third training data 405. This training process is crucial for the model's ability to accurately detect and process unintentional UI interactions, which is a key aspect of the automated UI testing system's 100 functionality. Referring now to FIG. 5 illustrates a flowchart of a method 500 for the automated User Interface (UI) testing in accordance with the present invention. The method 500 is configured to be performed on the system 100 for automated UI testing. The method 500 comprises the following steps: In step 501, the method 500 commences with receiving a user input. The user input corresponds to a specific task to be performed on the UI and helps generate the current rendered UI 301. This current rendered UI 301 holds the perceived available actions 303 that are detected by the first multi-modal model 201 of the test driver unit 103. In an embodiment, the first multi-modal model 201 is a generalized multi-modal model such as GPT4. Also, the current rendered UI 301 is provided to the test driver unit 103 in the form of the plurality of input images. In step 503, the method 500 comprises analyzing by the test driver unit 103 the current rendered UI 301. The analysis involves utilizing a second multi-modal model 203 that is pre-trained on vehicle-related examples, such as interaction of a user with one or more user interfaces on the infotainment screen of the vehicle. The second multi-modal model 203 helps in generating the plurality of possible UI states 305 corresponding to the current rendered UI 301. The perceived available actions 303 detected by the first multi-modal model 201 and the plurality of possible UI states 305 generated by the second multi-modal model 203 are then combined to create the video. The video depicts a plurality of possible UI states 305 and their corresponding transitions for the current rendered UI 301. Subsequently, the method 500 proceeds to step 505 where the bug detector unit 105 detects the error patterns in the current rendered UI 301. The bug detector unit 105 utilizes the video of the plurality of possible UI states 305 and their corresponding transitions received from the test driver unit 103. The bug detector unit 105 comprises two multi-modal models. The third multi-modal model 205 is configured to detect undesirable UI states or renderings, while the fourth multimodal model 207 is configured to detect and process unintentional UI interactions. The processing of these unintentional UI interactions may include either correcting or ignoring the unintentional UI interactions. Additionally, the bug detector unit 105 identifies critical information that should be present but is not visible due to overlapping or missing elements. It also identifies critical actions that should be inferable from the current UI state but are not, for example, a power off button hidden behind some menu item. The unit also checks if the layout fulfills design requirements, like font and alignment of text fields, and provides protection against application impersonation. The training data for these models include the first training data 401, the second training data 403, and the third training data 405. The first training data 401 trains the second multi-modal model 203 and comprises the sample perceived available actions data and the human-UI interactions data. The second training data 403 trains the third multi-modal model 205 and comprises the manually labeled plurality of undesirable UI states. The third training data 405 trains the fourth multi-modal model 207 and comprises the user inputs from previous user interactions, based on heuristics. The method 500 thus provides an automated way to detect UI issues that cannot be addressed by basic test drivers, offering an advanced version of monkey testing. The present invention provides an advanced system 100 for automated user interface (UI) testing, which utilizes the test driver unit 103 and the bug detector unit 105. The test driver unit 103 is equipped with two multi-modal models, while the bug detector unit 105 uses two additional multi-modal models. These models are instrumental in analyzing the current rendered UI 301, generating a plurality of possible UI states 305, detecting error patterns, and processing unintentional UI interactions. The first multi-modal model 201 in the test driver unit 103 is utilized to detect the perceived available actions 303 by analyzing the current rendered UI 301. This model is a generalized multi-modal model, such as GPT4. The second multi-modal model 203 in the test driver unit 103 is pre-trained on the specific network with the description of the perceived available actions 303 corresponding to the rendered UI. This model is trained on the first training data 401 that includes the sample perceived available actions data and the human-UI interactions data. The human-UI interactions data is specifically related to the interactions of a user with the one or more user interfaces on the infotainment screen of the vehicle. The outputs obtained from the first multi-modal model 201 and second multi-modal model 203 are combined to produce the video. This video represents the plurality of possible UI states 305 and their corresponding transitions for the current rendered UI 301. This method 500 enables a thorough and comprehensive exploration of all potential UI states, providing an exhaustive analysis of the current rendered UI 301. The model's ability to analyze perceived available actions 303 and determine the plurality of possible UI states 305 is key to the system's 100 functionality. This capability, coupled with the specialized training provided by the first training data 401, underscores the innovative approach of the present invention in the automating UI testing. The bug detector unit 105 in the system 100 uses two multi-modal models to detect error patterns in the current rendered UI 301. The third multimodal model 205 is used to detect the one or more undesirable UI states 307 or renderings, while the fourth multi-modal model 207 is used to detect and process the one or more unintentional UI interactions 309. The processing of the one or more unintentional UI interactions 309 includes either ignoring or correcting the one or more unintentional UI interactions 309. The third multi-modal model 205 and the fourth multi-modal model 207 are trained on two separate data sets. The second training data 403, used for the third multi-modal model 205, includes the manually labeled undesirable UI states. The third training data 405, used for the fourth multi-modal model 207, includes user inputs from the previous user interactions, based on the heuristics. Overall, the present invention provides an advanced system 100 for automated UI testing that uses multiple multi-modal models to generate a comprehensive range of possible UI states and detect a wide variety of UI errors. This system 100 can be applied to any UI, making it a versatile tool for ensuring the quality and usability of Uis across a broad range of applications. The present system 100 is capable of detecting the error patterns in the temporal animations due to the utilization of the recurrent model. This functionality provides the advanced version of monkey testing. The system 100 operates at the pixel level, enabling it to uncover bugs in all layers of the UI. This enhances the efficiency of the UI testing process by providing the comprehensive and accurate evaluation of the UI. Although the present invention has been described in terms of certain preferred embodiments, various features of separate embodiments can be combined to form additional embodiments not expressly described. Moreover, other embodiments apparent to those of ordinary skill in the art after reading this disclosure are also within the scope of this invention. Furthermore, not all the features, aspects and advantages are necessarily required to practice the present invention. Thus, while the above detailed description has shown, described, and pointed out novel features of the invention as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the apparatus or process illustrated may be made by those of ordinary skill in the technology without departing from the spirit of the invention. The inventions may be embodied in other specific forms not explicitly described herein. 5 The embodiments described above are to be considered in all respects as illustrative only and not restrictive in any manner.

Claims

1. A system (100) for automated User Interface (UI) testing implemented by at least one processor (101), the system (100) comprising:a test driver unit (103) comprising a first multi-modal model (201) and a second multi-modal model (203), the test driver unit is configured to create a video of a plurality of possible UI states (305) and their corresponding transitions for a current rendered UI (301); anda bug detector unit (105) configured to detect error patterns in the current rendered UI (301) using the video of the plurality of possible UI states (305) and their corresponding transitions, thereby detecting error patterns in the current rendered UI (301) at pixel level, wherein the current rendered UI (301) is provided to the test driver unit (103) in a form of a plurality of input images.

2. The system (100) as claimed in claim 1, wherein the first multi-modal model (201) is configured to detect perceived available actions (303) by analyzing the current rendered UI (301), and the second multi-modal model (203) is configured to determine the plurality of possible UI states (305) by analyzing the perceived available actions (303) received from the first multi-modal model (201), the second multi-modal model (203) is pretrained on a first training data (401) comprising sample perceived available actions data and human-UI interactions data to determine the plurality of possible UI states (305), wherein the perceived available actions (303) andthe plurality of possible UI states (305) are combined to create the video of the plurality of possible UI states (305) and their corresponding transitions.

3. The system (100) as claimed in claim 2, wherein the human-UI interactions data is derived from vehicle related interactions comprising interaction of a user with one or more user interfaces on an infotainment screen of a vehicle.

4. The system (100) as claimed in claim 3, wherein the one or more user interfaces comprise graphical user interface, touch user interface, voice user interface, command-line interface, natural language user interface, gesturebased interface, augmented reality (AR) interface, virtual reality interface and web user interface.

5. The system (100) as claimed in claim 1, wherein the bug detector unit (105) comprises a third multi-modal model (205) configured to detect one or more undesirable UI states (307) and a fourth multi-modal model (207) configured to detect and process one or more unintentional UI interactions (309), wherein the processing of the one or more unintentional UI interactions (309) includes any one of either ignoring or correcting the one or more unintentional UI interactions (309).

6. The system (100) as claimed in claim 1, wherein the plurality of possible UI states (305) generated by the test driver unit (103) comprise any one of a drop-down menu, popup, selected text box, text box having text, forms, tabs, vehicle menu, climate menu, temperature slider at a particular temperature, temperature display, selected language, font size, theme, and a combination thereof.

7. The system (100) as claimed in claim 5, wherein the one or more undesirable UI states (307) comprise any one of missing critical information, missing inferable critical actions, mismatched layout design requirements, impersonation by 3rd party malicious application, and a combination thereof.

8. The system (100) as claimed in claim 5, wherein the fourth multi-modal model (207) is pre-trained on a third training data (403) comprising sample user inputs and corresponding UI interactions.

9. The system (100) as claimed in claim 1, wherein each of the third multimodal model (205) and the fourth multi-modal model (207) are recurrent models configured to detect error patterns in temporal animations.

10. A method (500) for automated User Interface (UI) testing, the method (500)5 comprises:receiving a user input using the UI to generate a current rendered UI (301), wherein the user input corresponds to a specific task to be performed on the UI;analyzing by a test driver unit (103) the current rendered UI (301) to 10 generate a video of a plurality of possible UI states (305) for the currentrendered UI (301); anddetecting by a bug detector unit (105), error patterns in the current rendered UI (301) using the video of the plurality of possible UI states (305) received from the test driver unit (103).

Images