Method of inspecting user interface, electronic device and recording medium

Abstract

Various embodiments of the present invention may include a method for inspecting a user interface via an electronic device, comprising the following steps: obtaining an image including information about the user interface provided by an application; determining, based on the image, whether an error has occurred in the user interface; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error.

Description

[Technical Field]

[0001] This invention relates to a method, electronic device, and recording medium for inspecting a user interface. [Previous Technology]

[0002] User terminals display various user interfaces (UIs) through applications. Applications can display various user interfaces to provide specific services to users. To check (or verify) the user interfaces provided to users, the industry is implementing UI automation. UI automation checks are performed using the code corresponding to the user interface. For example, checking and confirming whether there are errors in the code. If the above UI automation checks are performed, the check result may also be evaluated as a failure in the following situation: although there are errors in the code, there are actually no errors in the user interface displayed by the application. In this case, since there are no errors in the actually displayed user interface, it is a user interface that has no problems even when provided to the application, so the check result should be evaluated as a success rather than a failure. For such errors, there is a tedious problem that developers need to manually confirm and correct them. [Summary of the Invention]

[0003] [Problem to be Solved by the Invention] The technical problem to be solved by the various embodiments of the present invention is: to use an image including a user interface to determine whether an error has occurred, without having to check the code related to the user interface.

[0004] The technical problem that the various embodiments of the present invention are intended to solve is: to examine images including the user interface to analyze the cause of errors.

[0005] The technical problem to be solved by the various embodiments of the present invention is: to use a learned artificial intelligence model to examine the snapshot image obtained from the user interface, thereby determining whether an error has occurred in the user interface. [Technical means to solve the problem]

[0006] Various embodiments of the present invention may include the following steps for inspecting a user interface by means of an electronic device: obtaining an image including information about the user interface provided by an application; determining, based on the image, whether an error has occurred in the user interface; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error.

[0007] According to one embodiment, the step of determining whether an error has occurred in the user interface may include the following steps: inputting the image into a trained artificial intelligence model; and obtaining information output from the artificial intelligence model related to whether an error has occurred in the user interface.

[0008] According to one embodiment, the step of identifying the cause of the above-mentioned error may include the following steps: obtaining information related to the cause of the error generated in the above-mentioned user interface from the output of the above-mentioned artificial intelligence model.

[0009] According to one embodiment, the above-mentioned artificial intelligence model can be a multi-modal large language model constructed in the following manner: a plurality of images including information from a plurality of user interfaces are set as training input data sets, wherein the information from the plurality of user interfaces is marked with whether an error has occurred and the cause of the error; information related to whether an error has occurred in the plurality of user interfaces and information related to the cause of the error has occurred in the plurality of user interfaces are set as training output data sets; and a deep learning algorithm is used to model the correlation between the training input data sets and the training output data sets.

[0010] According to one embodiment, the above-mentioned image may be a snapshot image obtained by capturing the above-mentioned user interface of the user's terminal provided by the above-mentioned application.

[0011] According to one embodiment, the above method for checking the user interface may further include the following steps: when it is determined that no error has occurred in the user interface, requesting the administrator's terminal to check the user interface; receiving the check result of the user interface from the administrator's terminal; and when the check result includes information related to the occurrence of an error in the user interface, determining that the check result is false.

[0012] According to one embodiment, the above method for checking the user interface may further include the following steps: when it is determined that the above check result is false, transmitting information to the terminal of the administrator indicating that the user interface is in a state that can be provided to the application.

[0013] According to one embodiment, the above method for checking the user interface may further include the following steps: when it is determined that an error has occurred in the user interface, transmitting the cause of the error to the terminal of the administrator; and transmitting information to the terminal of the administrator indicating that the user interface is in a state that cannot be provided to the application.

[0014] According to one embodiment, the cause of the above error may include at least one of the following causes: a first error cause, which is that the page including the above user interface cannot be accessed due to login failure; a second error cause, which is that at least one component of the above user interface is missing; and a third error cause, which is that at least one component of the above user interface is covered.

[0015] According to one embodiment, the above method for inspecting the user interface may further include the following steps: preprocessing the above image into a predetermined format.

[0016] According to one embodiment of the present invention, an electronic device may include: a communication circuit; one or more processors; and one or more memory, which store one or more instructions; and the electronic device is configured such that the one or more processors execute the one or more instructions to obtain an image including information of a user interface provided by an application; based on the image, determine whether an error has occurred in the user interface; and when it is determined that an error has occurred in the user interface, identify the cause of the error.

[0017] According to one embodiment of the present invention, in a non-transitory computer-readable recording medium recording a program for executing a method for inspecting a user interface via an electronic device on a computer, the method for inspecting a user interface may include the following steps: obtaining an image including information about a user interface provided by an application; determining, based on the image, whether an error has occurred in the user interface; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error. [Effects of the Invention]

[0018] According to various embodiments of the present invention, an image including a user interface can be used to determine whether an error has occurred without checking the code associated with the user interface.

[0019] According to various embodiments of the present invention, images including the user interface can be examined to analyze the cause of errors.

[0020] According to various embodiments of the present invention, a learned artificial intelligence model can be used to examine the snapshot image obtained from the user interface, thereby determining whether an error has occurred in the user interface.

Implementation Method

[0022] The terminology used in the embodiments has been selected as widely used general terms as possible while considering the functions of the present invention, but may vary depending on the intent or precedent of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain situations, there may be terms arbitrarily selected by the applicant; in such cases, their meanings will be described in detail in the corresponding description section. Therefore, the terminology used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, and not simply based on the name of the term.

[0023] Throughout the specification, when a part is described as "including" a certain component, unless otherwise stated otherwise, it means that other components may be included, rather than excluding other components. Furthermore, the terms "...part" and "...module" used in the specification indicate a unit that processes at least one function or action, which may be implemented by hardware or software, or by a combination of hardware and software.

[0024] The expression “at least one of a, b and c” as recorded throughout the specification may include “a alone”, “b alone”, “c alone”, “a and b”, “a and c”, “b and c” or “a, b and c”.

[0025] The "terminal" mentioned below can be implemented by a computer or a portable terminal, which can connect to a server or other terminal via a network. Here, a computer includes, for example, a laptop, desktop computer, or laptop computer equipped with a web browser. A portable terminal is a wireless communication device that ensures portability and mobility, and may include, for example, IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution) and other communication terminals, as well as all types of handheld wireless communication devices such as smartphones and tablets.

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0028] In the description of the embodiments, descriptions of technical content well-known in the technical field to which this invention pertains and not directly related to this invention are omitted. The reason for this is that by omitting redundant descriptions, the main idea of ​​this invention is conveyed more clearly, so as to avoid confusion regarding the main idea of ​​this invention.

[0029] For the same reason, some constituent elements are exaggerated, omitted, or represented in a generalized manner in the accompanying drawings. Furthermore, the dimensions of each constituent element do not perfectly reflect the actual dimensions. In each drawing, the same or corresponding constituent elements are assigned the same reference number.

[0030] The advantages and features of the present invention, and the methods for achieving such advantages and features, will become apparent upon reference to the accompanying drawings and the embodiments described in detail below. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in many different forms. These embodiments are provided only to fully disclose the present invention and to enable those skilled in the art to fully understand the scope of the present invention. The present invention is defined only by the scope of the technical solution. Throughout this specification, the same reference numerals denote the same constituent elements.

[0031] At this time, it is understood that the combination of each block of the flowchart and the flowchart itself can be executed by computer program instructions. These computer program instructions can be loaded onto the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment. Therefore, these instructions, executed by the processor of the computer or other programmable data processing equipment, produce methods for performing the functions described in the blocks of the flowchart. In order to implement the functions in a specific manner, these computer program instructions can be stored in computer-usable or computer-readable memory that is oriented towards a computer or other programmable data processing equipment. Therefore, the instructions stored in the computer-usable or computer-readable memory can also produce an article of manufacture containing instructions for performing the functions described in the blocks of the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing equipment, thereby performing a series of action steps on the computer or other programmable data processing equipment to produce a process executed by the computer. Thus, the instructions executed by the computer or other programmable data processing equipment can also provide steps for performing the functions described in the blocks of the flowchart.

[0032] Furthermore, each block may represent a portion of a module, fragment, or code comprising one or more executable instructions for performing a specific logical function. Also, in some alternative embodiments, the functions mentioned in the blocks may be generated out of order. For example, two blocks represented successively may be executed substantially simultaneously, or occasionally in reverse order depending on their corresponding functions.

[0033] FIG1 is a diagram of a system 10 for implementing a method for processing data according to one embodiment of the present invention.

[0034] Referring to FIG1, according to one embodiment, system 10 may include electronic device 100, user terminal 210, and administrator terminal 220. According to one embodiment, system 10 may be a system 10 for inspecting a user interface provided by an application. User terminal 210 and administrator terminal 220 may communicate with electronic device 100 via a network.

[0035] According to one embodiment, the electronic device 100 can provide various user interfaces to a user terminal via an application. The electronic device 100 can provide various user interfaces corresponding to various pages to the user terminal via an e-commerce related application. For example, the electronic device 100 can provide one of the following interfaces to the user terminal via an application: a user interface corresponding to a search results page, a user interface corresponding to a product details page, and a user interface corresponding to a product checkout page.

[0036] According to one embodiment, the electronic device 100 can inspect a user interface provided by an application. The electronic device 100 can obtain an image including information about the user interface provided by the application. The image including user interface information can be a snapshot image obtained by capturing the user interface provided to the user's terminal 210 by the application. The electronic device 100 can use the image including user interface information to check whether there are errors in the user interface. Hereinafter, a specific inspection method will be described.

[0037] According to one embodiment, the user's terminal 210 is a terminal used by a customer, who can use applications through the user's terminal 210. The user's terminal 210 can display various user interfaces received from the electronic device 100.

[0038] According to one embodiment, the administrator's terminal 220 may be an administrator's terminal 220 that manages the user interface. The administrator's terminal 220 may check the code corresponding to the user interface provided by the application, thereby checking whether errors have occurred in the user interface. The administrator's terminal 220 may transmit the check results to the electronic device 100.

[0039] Figure 2 is a block diagram of an electronic device 100 according to one embodiment of the present invention.

[0040] An electronic device 100 according to one embodiment may include a processor 110, a memory 120, and a communication circuit 130. At least one of the components included in the electronic device 100 may be omitted, or other components may be added to the electronic device 100. A portion of the components may be integrated additionally or alternatively, or implemented in the form of a single or multiple individual components. At least some of the components within the electronic device 100 may be interconnected via buses, GPIO (General Purpose Input / Output), SPI (Serial Peripheral Interface), or MIPI (Mobile Industry Processor Interface) to transmit and receive data and / or signals.

[0041] In one embodiment, the processor 110 of the electronic device 100 is configured to perform calculations or data processing related to the control and / or communication of the various components of the electronic device 100, and can be operatively connected to the components of the electronic device 100. The processor 110 can load commands or data received from other components of the electronic device 100 into the memory 120, process the commands or data stored in the memory 120, and store the result data. Unless otherwise specified, in this invention, the processor 110 may refer to a collection of more than one processor 110.

[0042] In one embodiment, the memory 120 of the electronic device 100 may store various data used by at least one component (e.g., processor 110). The memory 120 may store instructions for the operation of the processor 110 described above. Programs may be stored in the memory 120 as software, such as operating systems, middleware, or applications. Unless otherwise specified, in this invention, memory 120 may refer to a collection of more than one memory 120.

[0043] In one embodiment, the communication circuit 130 of the electronic device 100 can establish a wired or wireless communication channel with an external device (e.g., a user's terminal 210, an administrator's terminal 220) and send and receive various data with the external device. For wired communication with the external device, the communication circuit 130 of the electronic device 100 may include at least one port for connecting to the external device via a wired cable. The communication circuit 130 of the electronic device 100 can be configured to include a cellular communication module and be connected to a cellular network (e.g., 3G, LTE, 5G, Wibro, or WiMAX). In one embodiment, the communication circuit 130 of the electronic device 100 may include a short-range communication module to send and receive data with the external device using short-range communication (e.g., Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), UWB), but is not limited thereto.

[0044] Figure 3 is a flowchart of the operation of an electronic device 100 according to one embodiment of the present invention.

[0045] Referring to the flowchart 300, in step 310, the processor 110 of an electronic device 100 in one embodiment can obtain an image including information about the user interface provided by an application. The image including the user interface information can be a snapshot image obtained by capturing the user interface of the user's terminal 210 provided by the application. For example, the image can be a snapshot obtained by capturing the user interface corresponding to the product's detailed description page.

[0046] According to one embodiment, in step 320, the processor 110 can determine whether an error has occurred in the user interface based on the aforementioned image. The processor 110 can determine whether an error has occurred in the user interface by analyzing the image including information about the user interface. According to one embodiment, in step 330, when it is determined that an error has occurred in the user interface, the processor 110 can identify the cause of the error.

[0047] The processor 110 can use an artificial intelligence model to determine whether an error has occurred in the user interface. The processor 110 can input an image including information about the user interface into a trained artificial intelligence model. In this case, the trained artificial intelligence model can output information related to whether an error has occurred in the user interface. The processor 110 can obtain the information related to whether an error has occurred in the user interface output by the artificial intelligence model. Furthermore, in the above case, the trained artificial intelligence model can output information related to the cause of the error that has occurred in the user interface. The processor 110 can obtain the information related to the cause of the error that has occurred in the user interface output by the artificial intelligence model. That is, the artificial intelligence model can be a model that learns in such a way that if an image including information about the user interface is input, it outputs information related to whether an error has occurred in the user interface, and information related to the cause of the error when an error occurs.

[0048] On the other hand, images including user interface information and predetermined prompts can be input into the artificial intelligence model. Prompts may be text such as "Please check if there are any errors in the input image".

[0049] The artificial intelligence model can be a multi-modal large language model constructed as follows: a plurality of images, including information from a plurality of user interfaces, are set as the training input dataset, wherein the information from the plurality of user interfaces is labeled with whether an error has occurred and the cause of the error; information related to whether an error has occurred in the plurality of user interfaces and information related to the cause of the error in the plurality of user interfaces are set as the training output dataset; and a deep learning algorithm is used to model the correlation between the training input dataset and the training output dataset. "Multimodal" means various forms, referring to the ability to input various forms of data such as text, images, audio, and video. That is, the multi-modal large language model can input not only text but also images.

[0050] When an image containing information about a user interface that has generated an error is input to a trained artificial intelligence model, the artificial intelligence model can output the cause of the error since an error has occurred in the user interface. For example, the artificial intelligence model can output information indicating that a specific component of the user interface (e.g., an interface toolset) is missing. In this way, the processor 110 can determine whether an error has occurred in the user interface, and if so, determine the cause of the error.

[0051] When an image containing information about a normal user interface that does not produce errors is input into a trained artificial intelligence model, the artificial intelligence model can output information about the user interface that does not produce errors.

[0052] The causes of errors generated in the user interface may include at least one of the following: a first error cause, which is that the page including the user interface cannot be accessed due to login failure; a second error cause, which is that at least one component constituting the user interface is missing; and a third error cause, which is that at least one component constituting the user interface is covered. The processor 110 may identify the above error causes based on an image including information about the user interface.

[0053] Figure 4 is an operation flowchart of an electronic device 100 according to one embodiment of the present invention. Descriptions that are repeated in the description in Figure 3 will be omitted.

[0054] Referring to the flowchart 400, in step 410, the processor 110 of the electronic device 100 of one embodiment can obtain an image including information about the user interface. In step 420, the processor 110 of one embodiment can determine whether an error has occurred in the user interface. As shown in FIG3, the processor 110 can input the image including information about the user interface into a trained artificial intelligence model, thereby determining whether an error has occurred in the user interface.

[0055] On the other hand, the processor 110 can also pre-process the image to a predetermined format and input the pre-processed image into the trained artificial intelligence model. That is, the processor 110 can unify the image format to a predetermined format to improve the processing speed of the artificial intelligence model.

[0056] If an error is determined to have occurred in the user interface, the process proceeds to step 430 (step 420 -> Yes), where the processor 110 identifies the cause of the error in the user interface. In step 440, the processor 110 of one embodiment can transmit the cause of the error to the administrator's terminal 220. In step 450, the processor 110 of one embodiment can transmit information to the administrator's terminal 220 indicating that the user interface is in a state where it cannot be provided to the application. In this way, the administrator can confirm the existence of an error in the user interface and perform necessary operations to correct the error.

[0057] If it is determined that no error has occurred in the user interface, the process proceeds to step 460 (step 420 -> No), where the processor 110 may request the administrator's terminal 220 to inspect the user interface. The administrator's terminal 220 may perform automated UI inspection on the code corresponding to the user interface. That is, the administrator's terminal 220 may be used to inspect the code corresponding to the user interface.

[0058] In step 470, the processor 110 of one embodiment may receive the user interface inspection result from the administrator's terminal 220. The received inspection result may include information related to whether an error has occurred (i.e., exists) in the code corresponding to the user interface. For example, the inspection result may include information about an error occurring in a specific function of the code corresponding to the user interface.

[0059] In step 480, when the check result includes information indicating an error in the user interface, the processor 110 of one embodiment can determine that the check result is false. That is, although there is an error in the code corresponding to the user interface, there is no error in the image including the information of the actual user interface displayed by the application, so the processor 110 can determine that the check result is false (false fail). In step 490, the processor 110 of one embodiment can transmit information indicating that the user interface is in a state that can be provided to the application to the administrator's terminal 220.

[0060] On the other hand, when the check result includes information that no error has been generated in the user interface, the processor 110 may transmit information to the administrator's terminal 220 indicating that the check result is true and the user interface is in a state that can be provided to the application.

[0061] Although this drawing illustrates an embodiment in which the administrator's terminal 220 is requested to check the user interface when it is determined, based on an image including information about the user interface, that no error has occurred in the user interface, the order of these actions can be changed. That is, according to another embodiment, the processor 110 of the electronic device 100 can request the administrator's terminal 220 to check the user interface and receive the check result from the administrator's terminal 220. When the check result includes information indicating an error has occurred in the user interface, the processor 110 can determine whether an error has occurred in the user interface based on the image including information about the user interface. When it is determined, based on the image including information about the user interface, that no error has occurred in the user interface, the processor 110 can determine that the check result received from the administrator's terminal 220 is false.

[0062] According to the above embodiment, a UI snapshot image can be input into a trained artificial intelligence model to determine whether it is a case where an error occurred in the UI (also known as a successful case) or a case where no error occurred in the UI (also known as a failed case). Furthermore, even if the result of the UI automation check performed on the code corresponding to the UI is judged as a failed case, but the result of the check performed using the UI snapshot image is that no error occurred in the user interface, this case can be judged as a false fail case.

[0063] The above embodiments can be implemented as artificial intelligence (AI) using the processor and memory of an electronic device. The processor can consist of one or more processors, which can be general-purpose processors such as CPUs, APs, and DSPs (digital signal processors), graphics-specific processors such as GPUs and VPUs (vision processing units), or AI-specific processors such as NPUs. One or more processors can be controlled to process input data according to predefined action rules or AI models stored in memory. Alternatively, when one or more processors are AI-specific processors, the AI-specific processors can be designed with hardware structures specifically for processing specific AI models.

[0064] The predefined action rules or artificial intelligence models are characterized by being created through learning. Here, "created through learning" means creating predefined action rules or artificial intelligence models in a manner set to perform desired characteristics (or, objectives) by learning an algorithm and using multiple learning data to learn a basic artificial intelligence model. This learning can be implemented within the electronic device implementing artificial intelligence of the present invention, or by a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples mentioned above.

[0065] An artificial intelligence model can consist of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network operations can be performed by operations between the results of the previous layer and the plurality of weight values. The plurality of weight values ​​of the plurality of neural network layers can be optimized by the learning results of the artificial intelligence model. For example, during the learning process, the plurality of weight values ​​can be updated by reducing or minimizing the loss or cost values ​​obtained by the artificial intelligence model. Artificial neural networks may include deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks, but are not limited to the examples mentioned above.

[0066] On the other hand, preferred embodiments of the present invention are disclosed in this specification and drawings. Although specific terminology is used, it is only for the purpose of readily illustrating the technical content of the present invention and helping to understand the general meaning of the invention, and is not intended to limit the scope of the present invention. Obviously, those skilled in the art to which this invention pertains can implement other variations based on the technical concept of the present invention in addition to the embodiments disclosed herein.

[0067] The apparatus or terminal in the above embodiments may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with external devices, and a user interface device such as a touch panel, key, or button. The method implemented by software modules or algorithms can be stored on a computer-readable recording medium as computer-readable code or program commands executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD). The computer-readable recording medium is distributed across a networked computer system, thereby enabling the distributed storage and execution of computer-readable code. The medium can be read by a computer, stored in memory, and executed in a processor.

[0068] This embodiment can be represented by functional blocks and various processing steps. These functional blocks can be implemented by different numbers of hardware and / or software components performing specific functions. For example, the embodiment can employ integrated circuits capable of performing various functions via the control of one or more microprocessors or other control devices, such as memory, processing, logic, lookup tables, etc. The constituent elements can be executed by software programs or software components. Similarly, this embodiment includes algorithms implemented by combinations of data structures, programs, conventions, or other programs, and therefore can be implemented using programming or scripting languages ​​such as C, C++, Java, assembler, Python, etc. Functionally, it can be implemented by algorithms executed in one or more processors. Furthermore, this embodiment can employ prior art for electronic environment setup, signal processing, and / or data processing. Terms such as "mechanism," "component," "mechanism," and "construction" are used broadly and are not limited to mechanical and physical structures. The above terms may be associated with processors and include the meaning of a series of software processes.

[0069] The above embodiment is only an example, and other embodiments can be implemented within the scope of the invention application described below. [Simplified Explanation of the Diagram]

[0021] Figure 1 is a diagram illustrating a system for implementing a method for processing data according to an embodiment of the present invention. Figure 2 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 3 is a flowchart illustrating the operation of an electronic device according to an embodiment of the present invention. Figure 4 is a flowchart illustrating the operation of an electronic device according to an embodiment of the present invention.

Claims

1. A method for inspecting a user interface, implemented via an electronic device, comprising the following steps: obtaining an image including information about the user interface provided by an application; determining, based on the image, whether an error has occurred in the user interface; if an error has occurred in the user interface, identifying the cause of the error; if, based on the image, it is determined that no error has occurred in the user interface, further inspecting the code corresponding to the user interface; if the inspection of the code indicates that an error exists in the code, determining that the inspection result for the user interface is false; and if the inspection result is false, transmitting information to the administrator's terminal indicating that the user interface is in a state that can be provided to the application.

2. The method for checking the user interface as described in Request 1, wherein the step of determining whether an error has occurred in the user interface includes the following steps: inputting the image into a trained artificial intelligence model; and obtaining information output from the artificial intelligence model related to whether an error has occurred in the user interface.

3. The method for inspecting the user interface as described in request item 2, wherein the step of identifying the cause of the above-mentioned error includes the following steps: obtaining information related to the cause of the error generated in the above-mentioned user interface from the output of the above-mentioned artificial intelligence model.

4. The method for inspecting the user interface as described in Request 3, wherein the aforementioned artificial intelligence model is a multimodal large-scale language model constructed as follows: A plurality of images including information from a plurality of user interfaces are set as the training input dataset, wherein the information from the plurality of user interfaces is labeled with whether an error has occurred and the cause of the error; Information related to whether an error has occurred in the plurality of user interfaces and information related to the cause of the error in the plurality of user interfaces are set as the training output dataset; A deep learning algorithm is used to model the correlation between the aforementioned training input dataset and the aforementioned training output dataset.

5. The method for inspecting the user interface as described in Request 1, wherein the image is a snapshot image of the user interface provided to the user's terminal by the application.

6. The method for checking the user interface as described in Request 1 further includes the following steps: when it is determined that an error has occurred in the user interface, transmitting the cause of the error to the administrator's terminal; and transmitting information to the administrator's terminal indicating that the user interface is in a state where it cannot be provided to the application.

7. The method for checking the user interface as described in Request 1, wherein the cause of the above error includes at least one of the following: a first error cause, which is that the page including the above user interface cannot be accessed due to login failure; a second error cause, which is that at least one component of the above user interface is missing; and a third error cause, which is that at least one component of the above user interface is covered.

8. The method for checking the user interface as described in Request 1 further includes the following steps: preprocessing the format of the above image to a predetermined format.

9. An electronic device comprising: The electronic device comprises: a communication circuit; one or more processors; and one or more memory units storing one or more instructions; wherein the one or more processors are configured such that: they execute the one or more instructions to obtain an image including information about a user interface provided by an application; based on the image, they determine whether an error has occurred in the user interface; if an error has occurred in the user interface, they identify the cause of the error; if no error has occurred in the user interface based on the image, they further check the code corresponding to the user interface; if the check of the code indicates that an error exists in the code, they determine that the check result for the user interface is false; and if the check result is false, they transmit information to the administrator's terminal indicating that the user interface is in a state that can be provided to the application.

10. A non-transitory recording medium that records a program for executing on a computer a method for inspecting a user interface via an electronic device, the method for inspecting the user interface comprising the following steps: obtaining an image including information about a user interface provided by an application; determining, based on the image, whether an error has occurred in the user interface; and, if it is determined that an error has occurred in the user interface, identifying the cause of the error; and, if it is determined based on the image that no error has occurred in the user interface, further inspecting the code corresponding to the user interface; if the inspection of the code indicates that an error exists in the code, determining that the inspection result for the user interface is false; and, if the inspection result is false, transmitting information to the administrator's terminal indicating that the user interface is in a state that can be provided to the application.

Images