Method of inspecting user interface, electronic device and recording medium

An AI-powered UI inspection method analyzes user interface images to detect and diagnose errors, addressing false positives in traditional code-based evaluations and enhancing UI evaluation efficiency.

KR103046382B1Active Publication Date: 2026-11-11COUPANG CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020240138714
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-10-11
Publication Date
2026-11-11
Estimated Expiration
2044-10-11

AI Technical Summary

Technical Problem

Existing UI automation inspection methods incorrectly evaluate user interfaces as failures when there are no actual errors, requiring manual developer intervention to correct these false positives.

Method used

An electronic device uses a trained artificial intelligence model to analyze snapshot images of user interfaces, determining the presence and cause of errors without inspecting code, employing a multi-modal large language model to process images and identify issues such as login failures, component loss, or obscuration.

Benefits of technology

Accurately identifies UI errors and their causes in images, reducing the need for manual code inspections and improving the reliability of UI evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024110723071-PAT00001_ABST
    Figure 112024110723071-PAT00001_ABST
Patent Text Reader

Abstract

A method for inspecting a user interface performed by an electronic device according to various embodiments of the present disclosure may include: acquiring an image containing information of a user interface provided through an application; determining whether an error has occurred in the user interface based on the image; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present disclosure relates to a method for inspecting a user interface, an electronic device, and a recording medium. Background Technology

[0003] User terminals display various user interfaces (UI) through applications. Applications may display various user interfaces to provide specific services to users. UI automation is performed to inspect (or verify) the user interfaces provided to users. UI automation inspection is carried out using programming code corresponding to the user interface. For example, it checks and verifies whether errors exist in the code. When the aforementioned UI automation inspection is performed, the inspection result may be evaluated as a failure even if there are errors in the code but no errors in the user interface actually displayed through the application. In this case, since there are no errors in the actual displayed user interface, it should be evaluated as a success rather than a failure, as it is a user interface that poses no problem when provided to the application. There is a problem in that it is cumbersome for developers to manually check and correct such errors. (Patent Document 1) US 2020 / 0257587 A1 (August 13, 2020) The problem to be solved

[0005] According to various embodiments of the present disclosure, the technical problem is to determine whether an error has occurred using an image including a user interface without inspecting code related to the user interface.

[0006] According to various embodiments of the present disclosure, the technical problem is to analyze the cause of an error by inspecting an image including a user interface.

[0007] According to various embodiments of the present disclosure, the technical problem is to determine whether an error has occurred in a user interface by examining a snapshot image of a user interface captured through a trained artificial intelligence model. means of solving the problem

[0009] A method for inspecting a user interface performed by an electronic device according to various embodiments of the present disclosure may include: acquiring an image containing information of a user interface provided through an application; determining whether an error has occurred in the user interface based on the image; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error.

[0010] According to one embodiment, the step of determining whether an error has occurred in the user interface may include: inputting the image into a trained artificial intelligence model; and obtaining information regarding whether an error has occurred in the user interface output from the artificial intelligence model.

[0011] According to one embodiment, the step of identifying the cause of the error may include the step of obtaining information regarding the cause of the error that occurred in the user interface output from the artificial intelligence model.

[0012] According to one embodiment, the artificial intelligence model may be a multi-modal large language model constructed by using a deep learning algorithm to model the correlation between the training input data set and the training output data set, wherein the model includes a plurality of images containing information of a plurality of user interfaces labeled as whether an error has occurred and the cause of the error that occurred, and sets the information regarding whether an error has occurred in the plurality of user interfaces and the cause of the error that occurred in the plurality of user interfaces as the training output data set.

[0013] According to one embodiment, the image may be a snapshot image of the user interface provided to the user's terminal through the application.

[0014] A method for inspecting a user interface according to one embodiment may further include: a step of requesting an inspection of the user interface to an administrator's terminal when it is determined that no error has occurred in the user interface; a step of receiving an inspection result regarding the user interface from the administrator's terminal; and a step of determining that the inspection result is false when the inspection result includes information that an error has occurred in the user interface.

[0015] A method for inspecting the user interface according to one embodiment may further include the step of transmitting information to the administrator's terminal that the user interface is in a state where it can be provided to the application when the inspection result is determined to be false.

[0016] A method for inspecting the user interface according to one embodiment may further include the step of transmitting the cause of the error to the administrator's terminal when it is determined that an error has occurred in the user interface; and the step of transmitting information to the administrator's terminal that the user interface is in a state where it cannot be provided to the application.

[0017] According to one embodiment, the cause of the error may include at least one of a first error cause in which access to a page including the user interface is not possible due to a login failure, a second error cause in which at least one component constituting the user interface is lost, and a third error cause in which at least one component constituting the user interface is obscured.

[0018] A method for inspecting the user interface according to one embodiment may further include a step of preprocessing the format of the image into a predetermined format.

[0019] An electronic device according to one embodiment of the present disclosure comprises: a communication circuit; one or more processors; and one or more memories for storing one or more instructions. The one or more processors may be configured to obtain an image containing information of a user interface provided through an application by executing the one or more instructions, determine whether an error has occurred in the user interface based on the image, and, if it is determined that an error has occurred in the user interface, identify the cause of the error.

[0020] According to one embodiment of the present disclosure, in a computer-readable non-transient recording medium having a program for executing on a computer a method for inspecting a user interface performed by an electronic device, the method for inspecting the user interface may include: a step of acquiring an image containing information of a user interface provided through an application; a step of determining whether an error has occurred in the user interface based on the image; and a step of identifying the cause of the error when it is determined that an error has occurred in the user interface. Effects of the invention

[0022] According to various embodiments of the present disclosure, it is possible to determine whether an error has occurred by using an image including a user interface without examining code related to the user interface.

[0023] According to various embodiments of the present disclosure, the cause of an error can be analyzed by examining an image including a user interface.

[0024] According to various embodiments of the present disclosure, it is possible to determine whether an error has occurred in a user interface by examining a snapshot image of a user interface captured through a trained artificial intelligence model. Brief explanation of the drawing

[0026] FIG. 1 is a diagram showing a system for performing a method of processing data according to one embodiment of the present disclosure. FIG. 2 is a block diagram of an electronic device according to one embodiment of the present disclosure. FIG. 3 is a flowchart of the operation of an electronic device according to one embodiment of the present disclosure. FIG. 4 is a flowchart of the operation of an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0027] The terms used in the embodiments have been selected to be as widely used as possible, taking into account their functions in the present disclosure; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in the present disclosure should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0028] When a part of a specification is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as “...part” or “...module” as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0029] The expression “at least one of a, b, and c” described throughout the specification may include ‘a alone’, ‘b alone’, ‘c alone’, ‘a and b’, ‘a and c’, ‘b and c’, or ‘a, b, and c all’.

[0030] The "terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all types of handheld-based wireless communication devices such as communication-based terminals like IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), and LTE (Long Term Evolution), smartphones, tablet PCs, etc.

[0031] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0033] In describing the embodiments, technical details that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is intended to convey the essence of the present invention more clearly without obscuring it by omitting unnecessary explanations.

[0034] For the same reason, some components in the attached drawings have been exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual dimensions. Identical or corresponding components in each drawing have been assigned the same reference numbers.

[0035] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0036] At this time, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing the means of instruction to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0037] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0038] FIG. 1 is a drawing showing a system (10) that performs a method of processing data according to one embodiment of the present disclosure.

[0039] Referring to FIG. 1, a system (10) according to one embodiment may include an electronic device (100), a user terminal (210), and a manager terminal (220). A system (10) according to one embodiment may be a system (10) for inspecting a user interface provided through an application. The user terminal (210) and the manager terminal (220) may be connected to the electronic device (100) via a network.

[0040] According to one embodiment, the electronic device (100) can provide various user interfaces to a user terminal through an application. The electronic device (100) can provide various user interfaces corresponding to various pages to a user terminal through an application related to e-commerce. For example, the electronic device (100) can provide one of a user interface corresponding to a search result page, a user interface corresponding to a product detail page, and a user interface corresponding to a product payment page to a user terminal through an application.

[0041] An electronic device (100) according to one embodiment can inspect a user interface provided through an application. The electronic device (100) can acquire an image containing information of the user interface provided through the application. The image containing information of the user interface may be a snapshot image capturing the user interface provided to the user's terminal (210) through the application. The electronic device (100) can inspect whether an error exists on the user interface through the image containing information of the user interface. A specific inspection method will be described later.

[0042] According to one embodiment, the user's terminal (210) is a terminal used by a customer, and the customer can use an application through the user's terminal (210). The user's terminal (210) can display various user interfaces received from the electronic device (100).

[0043] According to one embodiment, the administrator's terminal (220) may be an administrator's terminal (220) that manages a user interface. The administrator's terminal (220) can check whether an error has occurred in the user interface by checking programming code corresponding to the user interface provided through an application. The administrator's terminal (220) can transmit the inspection result to an electronic device (100).

[0044] FIG. 2 is a block diagram of an electronic device (100) according to one embodiment of the present disclosure.

[0045] An electronic device (100) according to one embodiment may include a processor (110), memory (120), and a communication circuit (130). At least one of the components included in the electronic device (100) may be omitted, or another component may be added to the electronic device (100). Additionally, some components may be implemented as an integrated unit or as a single or multiple unit. At least some components within the electronic device (100) may be implemented as an integrated unit or as a single or multiple unit. At least some components within the electronic device (100) may be connected to each other via a bus, GPIO (general purpose input / output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface), etc., to exchange data and / or signals.

[0046] A processor (110) of an electronic device (100) according to one embodiment is configured to perform operations or data processing regarding the control and / or communication of each component of the electronic device (100) and can be operatively connected to the components of the electronic device (100). The processor (110) can load commands or data received from other components of the electronic device (100) into memory (120), process commands or data stored in memory (120), and store result data. Unless there are special circumstances, in this disclosure, the processor (110) may mean a set of one or more processors (110).

[0047] A memory (120) of an electronic device (100) according to one embodiment may store various data used by at least one component (e.g., processor (110)). The memory (120) may store instructions for the operation of the processor (110) described above. A program may be stored in the memory (120) as software and may include, for example, an operating system, middleware, or an application. Unless otherwise specified, in this disclosure, memory (120) may mean a set of one or more memories (120).

[0048] A communication circuit (130) of an electronic device (100) according to one embodiment may establish a wired or wireless communication channel with an external device (e.g., user terminal (210), administrator terminal (220)) and transmit and receive various data with the external device. The communication circuit (130) of the electronic device (100) may include at least one port for connecting to the external device via a wired cable in order to communicate with the external device via a wired connection. The communication circuit (130) of the electronic device (100) may include a cellular communication module and be configured to be connected to a cellular network (e.g., 3G, LTE, 5G, Wibro, or Wimax). According to one embodiment, the communication circuit (130) of the electronic device (100) may include a short-range communication module and transmit and receive data with the external device using short-range communication (e.g., Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), UWB), but is not limited thereto.

[0049] FIG. 3 is a flowchart of the operation of an electronic device (100) according to one embodiment of the present disclosure.

[0050] Referring to the operation flowchart 300, the processor (110) of the electronic device (100) according to one embodiment may, in step 310, acquire an image containing information of a user interface provided through an application. The image containing information of a user interface may be a snapshot image capturing a user interface provided to a user's terminal (210) through an application. For example, the image may be a snapshot capturing a user interface corresponding to a detailed description page regarding a product.

[0051] A processor (110) according to one embodiment can determine, at step 320, whether an error has occurred in the user interface based on the image. The processor (110) can determine whether an error has occurred in the user interface by analyzing an image containing information about the user interface. If the processor (110) according to one embodiment determines at step 330 that an error has occurred in the user interface, it can identify the cause of the error.

[0052] The processor (110) can determine whether an error has occurred in the user interface using an artificial intelligence model. The processor (110) can input an image containing information about the user interface into the trained artificial intelligence model. In this case, the trained artificial intelligence model can output information regarding whether an error has occurred in the user interface. The processor (110) can obtain information regarding whether an error has occurred in the user interface output from the artificial intelligence model. In addition, in the above case, the trained artificial intelligence model can output information regarding the cause of the error that occurred in the user interface. The processor (110) can obtain information regarding the cause of the error that occurred in the user interface output from the artificial intelligence model. That is, the artificial intelligence model may be a model trained to output information regarding whether an error has occurred in the user interface and information regarding the cause of the error if an error has occurred, when an image containing information about the user interface is input.

[0053] Meanwhile, a predefined prompt can be input into the artificial intelligence model along with an image containing information about the user interface. The prompt may be, for example, text with the intent of "Please check if an error has occurred in the input image."

[0054] The artificial intelligence model may be a multi-modal large language model constructed by using a deep learning algorithm to model the correlation between the training input data set and the training output data set, wherein the model uses a training input data set comprising information on multiple user interfaces labeled as whether an error has occurred and the cause of the error, and uses information on whether an error has occurred on the multiple user interfaces and information on the cause of the error on the multiple user interfaces as the training output data set. Multi-modal means various formats and refers to the ability to input data in various formats such as text, images, audio, and video. That is, the multi-modal large language model may accept not only text but also images as input.

[0055] When an image containing information about a user interface that has an error is input into a trained artificial intelligence model, the artificial intelligence model can output that an error has occurred in the user interface and the cause of the error. For example, the artificial intelligence model can output information that an error has occurred in which a specific component (e.g., a widget) constituting the user interface is lost. Through this, the processor (110) can determine whether an error has occurred in the user interface, and if so, the cause of the error.

[0056] When an image containing information about a normal user interface without errors is input into a trained artificial intelligence model, the artificial intelligence model can output information that no errors occurred in the user interface.

[0057] The cause of an error occurring in the user interface may include at least one of a first error cause in which access to a page including the user interface is not possible due to a login failure, a second error cause in which at least one component constituting the user interface is lost, and a third error cause in which at least one component constituting the user interface is obscured. The processor (110) can identify the above-described error causes based on an image containing information of the user interface.

[0058] FIG. 4 is a flowchart of the operation of an electronic device (100) according to one embodiment of the present disclosure. Descriptions of content that overlap with those described in FIG. 3 will be omitted.

[0059] Referring to operation flowchart 400, the processor (110) of the electronic device (100) according to one embodiment can acquire an image containing information of a user interface at step 410. The processor (110) according to one embodiment can determine whether an error has occurred in the user interface at step 420. As described in FIG. 3, the processor (110) can determine whether an error has occurred in the user interface by inputting an image containing information of the user interface into a trained artificial intelligence model.

[0060] Meanwhile, the processor (110) may perform pre-processing of the image format into a predetermined format and input the pre-processed image into a trained artificial intelligence model. That is, the processor (110) may unify the image format into a predetermined format to improve the processing speed of the artificial intelligence model.

[0061] If it is determined that an error has occurred in the user interface, the processor (110) branches to step 430 (step 420 -> yes) and can identify the cause of the error that occurred in the user interface. In one embodiment, the processor (110) can transmit the cause of the error to the administrator's terminal (220) in step 440. In one embodiment, the processor (110) can transmit information to the administrator's terminal (220) in step 450 that the user interface is in a state where it cannot be provided to the application. Through this, the administrator can confirm that there is an error in the user interface and perform necessary tasks to correct the error that occurred.

[0062] If it is determined that no error has occurred in the user interface, branch to step 460 (step 420 -> No), and the processor (110) may request the administrator's terminal (220) to inspect the user interface. The administrator's terminal (220) may perform UI automation inspection on the code corresponding to the user interface. That is, the code corresponding to the user interface can be inspected through the administrator's terminal (220).

[0063] A processor (110) according to one embodiment may, at step 470, receive an inspection result regarding a user interface from a manager's terminal (220). The received inspection result may include information regarding whether an error has occurred (i.e., exists) in the code corresponding to the user interface. For example, the inspection result may include information that an error has occurred in a specific function of the code corresponding to the user interface.

[0064] A processor (110) according to one embodiment may determine at step 480 that the inspection result is false if the inspection result includes information that an error has occurred in the user interface. That is, the processor (110) may determine that the inspection result is false (false fail) because an error exists in the code corresponding to the user interface, but no error exists in the image containing information of the actual user interface displayed through the application. A processor (110) according to one embodiment may transmit information to the administrator's terminal (220) at step 490 that the user interface is in a state where it can be provided to the application.

[0065] Meanwhile, if the inspection result includes information that no error occurred in the user interface, the processor (110) can transmit information to the administrator's terminal (220) that the inspection result is true and the user interface is in a state where it can be provided to the application.

[0066] In the present drawing, an embodiment is described in which a request for inspection regarding the user interface is made to the administrator's terminal (220) when it is determined that no error has occurred in the user interface based on an image containing information about the user interface, but this order may be changed. That is, according to another embodiment, the processor (110) of the electronic device (100) may request an inspection regarding the user interface from the administrator's terminal (220) and receive an inspection result regarding the user interface from the administrator's terminal (220). If the inspection result contains information that an error has occurred in the user interface, the processor (110) may determine whether an error has occurred in the user interface based on an image containing information about the user interface. If the processor (110) determines that no error has occurred in the user interface based on an image containing information about the user interface, it may determine that the inspection result received from the administrator's terminal (220) is false.

[0067] According to the embodiments described above, by inputting a UI snapshot image into a trained artificial intelligence model, it is possible to determine whether a case in which an error occurred in the UI (also known as a success case) or a case in which no error occurred in the UI (also known as a failure case). Furthermore, even if a case is determined to be a failure case as a result of performing an automated UI inspection on the code corresponding to the UI, if an inspection performed via the UI snapshot image reveals that no error occurred in the user interface, such a case can be determined as a false fail case.

[0068] The aforementioned embodiments can be implemented as artificial intelligence (AI) through the processor and memory of an electronic device. The processor may consist of one or more processors, and the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (digital signal processors), graphics-dedicated processors such as GPUs and VPUs (vision processing units), or AI-dedicated processors such as NPUs. The one or more processors may be controlled to process input data according to predefined operation rules or AI models stored in memory. Alternatively, if the one or more processors are AI-dedicated processors, the AI-dedicated processors may be designed with a hardware structure specialized for processing a specific AI model.

[0069] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of learning data by a learning algorithm. Such learning may be performed on the electronic device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0070] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and can perform neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include, but is not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks.

[0071] Meanwhile, the present specification and drawings disclose preferred embodiments of the present invention. Although specific terms have been used, they are used merely in a general sense to facilitate the explanation of the technical content of the present invention and to aid in understanding the invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that, in addition to the embodiments disclosed herein, other variations based on the technical concept of the present invention are possible.

[0072] A device or terminal according to the embodiments described above may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and user interface devices such as a touch panel, a key, a button, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD: Digital Versatile Disc). Computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium may be readable by a computer, stored in memory, and executed by a processor.

[0073] The present embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the embodiment may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Similar to how components may be implemented as software programming or software elements, the present embodiment may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, Python, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present embodiment may employ prior art for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0074] The aforementioned embodiments are merely examples, and other embodiments may be implemented within the scope of the claims set forth below.

Claims

Claim 1 A method for inspecting a user interface performed by an electronic device, comprising: acquiring an image containing information of a user interface provided through an application; determining whether an error has occurred in the user interface based on the image; identifying the cause of the error if it is determined that an error has occurred in the user interface; requesting a manager's terminal to inspect a code corresponding to the user interface if it is determined that no error has occurred in the user interface; receiving an inspection result for the code corresponding to the user interface from the manager's terminal; determining that the inspection result is false if the inspection result for the code contains information that an error has occurred in the user interface; and transmitting information to the manager's terminal that the user interface is in a state where it can be provided to the application if it is determined that the inspection result is false. Claim 2 A method for inspecting a user interface according to claim 1, wherein the step of determining whether an error has occurred in the user interface comprises: inputting the image into a trained artificial intelligence model; and obtaining information regarding whether an error has occurred in the user interface output from the artificial intelligence model. Claim 3 A method for inspecting a user interface according to paragraph 2, wherein the step of identifying the cause of the error includes the step of obtaining information regarding the cause of the error that occurred in the user interface output from the artificial intelligence model. Claim 4 In paragraph 3, the artificial intelligence model is a multi-modal large language model constructed by using a deep learning algorithm to model the correlation between the training input data set and the training output data set, wherein the artificial intelligence model is a multi-modal large language model constructed by using a deep learning algorithm, wherein the artificial intelligence model is a training input data set comprising information on a plurality of user interfaces labeled as whether an error has occurred and the cause of the error that occurred, information on whether an error has occurred in the plurality of user interfaces and information on the cause of the error that occurred in the plurality of user interfaces are a training output data set. Claim 5 A method for inspecting a user interface according to claim 1, wherein the image is a snapshot image of the user interface provided to a user's terminal through the application. Claim 6 delete Claim 7 delete Claim 8 A method for inspecting a user interface according to claim 1, further comprising: a step of transmitting the cause of the error to an administrator's terminal when it is determined that an error has occurred in the user interface; and a step of transmitting information to the administrator's terminal that the user interface is in a state where it cannot be provided to the application. Claim 9 A method for inspecting a user interface according to claim 1, wherein the cause of the error comprises at least one of a first error cause of failure to access a page including the user interface due to a login failure, a second error cause of loss of at least one component constituting the user interface, and a third error cause of obscuring at least one component constituting the user interface. Claim 10 A method for inspecting a user interface according to claim 1, further comprising the step of preprocessing the format of the image into a predetermined format. Claim 11 An electronic device comprising: a communication circuit; one or more processors; and one or more memories for storing one or more instructions, wherein the one or more processors, by executing the one or more instructions, acquire an image containing information of a user interface provided through an application, determine whether an error has occurred in the user interface based on the image, and if it is determined that an error has occurred in the user interface, identify the cause of the error, and if it is determined that no error has occurred in the user interface, request a manager's terminal to inspect a code corresponding to the user interface, receive an inspection result for the code corresponding to the user interface from the manager's terminal, and if the inspection result for the code contains information that an error has occurred in the user interface, determine that the inspection result is false, and if it is determined that the inspection result is false, transmit information to the manager's terminal that the user interface is in a state where it can be provided to the application. Claim 12 A computer-readable non-transient recording medium having a program for executing on a computer a method for inspecting a user interface performed by an electronic device, wherein the method for inspecting the user interface comprises: a step of acquiring an image containing information of a user interface provided through an application; a step of determining whether an error has occurred in the user interface based on the image; a step of identifying the cause of the error if it is determined that an error has occurred in the user interface; a step of requesting a manager's terminal to inspect a code corresponding to the user interface if it is determined that no error has occurred in the user interface; a step of receiving an inspection result for the code corresponding to the user interface from the manager's terminal; a step of determining that the inspection result is false if the inspection result for the code contains information that an error has occurred in the user interface; and a step of transmitting information to the manager's terminal that the user interface is in a state where it can be provided to the application if it is determined that the inspection result is false.

Citation Information

Patent Citations

  • UI function automatic test tool and method

    CN117331831A

  • Intelligent responding to error screen associated errors

    US20200257587A1