Method for automatic recovery of object exploration information in RPA scenarios using generative ai, and apparatus therefor

Abstract

The present invention is for, in applying a target object exploration according to a RPA scenario based on a previous UI screen to the current UI screen, if a specific target object cannot be searched or an error occurs due to a change on a target app / web at the current time point, automatically extracting information on the most suitable corresponding object on the current UI screen, thereby automatically recovering from the target object exploration execution failure of the previous scenario. A method of the present invention may comprise: an error recognition step for detecting the occurrence of an error resulting from the application, to the current UI screen, of a target object exploration based on object exploration information of a RPA scenario created on the basis of a previous UI screen; an image information LLM inquiry step for generating a first prompt including a first image including at least a portion of the previous UI screen including a target object, and a second image including at least a portion of the current UI screen, and instructing a generative AI to return information of an object corresponding to the target object on the current UI screen; and an object exploration step for performing target object exploration on the current UI screen.

Description

Method for automatically recovering object search information in RPA scenarios using generative AI and device therefor

[0001] The present invention relates to Robotic Process Automation (RPA) based on generative AI (or 'Gen AI'), and more specifically, to a method for automatically recovering object search information of an RPA scenario by utilizing generative AI, and a device therefor.

[0002] Robotic Process Automation (RPA) technology, which allows software robots (or bots) to mimic and execute tasks performed by users on their PCs according to predetermined rules, is widely used. RPA typically involves a recording process where users pre-specify on-screen target objects and actions to be performed using scripts (or scenarios), followed by a bot executing the scripts.

[0003] However, in reality, one reason that can hinder the utilization of RPA technology is that information related to object exploration (or object exploration information) in a scenario that has already been created in the past may change for various reasons in the current target app / web, making it often impossible to explore a specified target object based on the object exploration information in the already created scenario. For example, various maintenance and repair work due to various reasons such as regular or irregular system UI upgrades, next-generation system construction, screen text changes, and button location changes may occur at a time when the user is not aware of the target object in a specific app / web UI screen.

[0004] FIGS. 1A to 1C illustrate cases where it is difficult to search for an intended target object on the current UI screen based on object search information of a scenario already created based on the previous UI screen due to such changes. For example, as in FIG. 1A, when the text is changed to a similar word, such as when an object displayed with the text "Find ID" (5) on the previous UI screen (2) is changed to "Search ID" (15) on the current UI screen (12), or when the text is changed to a conventional abbreviation, such as when an object displayed with the text "Frequently Asked Questions" (25) on the previous UI screen (22) is changed to "FAQ" (35) on the current UI screen (32), or when the design of a target object (45) on the previous UI screen (40) and the design of a target object (55) on the current UI screen (50) are different, such that a computer cannot recognize them as the same image based on image data, such situations can frequently occur.

[0005] In such cases, information (Name, Value, Control Type, etc.) stored in a scenario previously written by the user may no longer be valid for searching for the target object intended by the user on the current UI screen due to changes in the target app / web. In the prior art, when an error or failure in object search occurs during script execution, the user must manually recognize the reason, re-designate an object suitable for the target app / web environment, re-write or modify the scenario, and then manually repeat the process of registering or distributing it on the bot again.

[0006] However, this method requires human users to closely monitor errors or failures in object exploration and directly correct them. Therefore, for the scenario to run normally, a long RPA execution downtime (several hours to several days) is required until the user's work is completed. This significantly reduces operational efficiency, which fundamentally undermines the purpose of introducing RPA as an automation solution. In particular, in actual fields, the number of object exploration targets can range from tens to hundreds. Therefore, having humans manually correct the automation scenarios of RPA that have already been implemented in the field can be a cumbersome task that requires a great deal of time, effort, and money.

[0007] The present invention is to automatically extract the most appropriate corresponding object information on the current UI screen to automatically recover from failure in executing the target object search of the previous scenario when, in applying the target object search based on the object search information of the RPA scenario written based on the previous UI screen to the current UI screen, changes occur on the target app / web at the current point in time for various reasons, and the specified target object can no longer be searched based on the information on the written scenario or an error occurs.

[0008] In addition, the present invention is to exclude or minimize user intervention or manual work in the above-described object search failure situation, and to automatically extract the most appropriate corresponding object information on the current UI screen through linkage with a generative AI such as a multi-modal LLM, thereby automatically recovering from failure in target object search execution according to the previous scenario.

[0009] In addition, the present invention proposes a method that, when the change of the object search target on the current UI screen is so extensive that it is difficult to find similarities between them when compared with the previous UI screen, scenario rewriting may be inevitable, but when such change is minor, such as a simple change in the position / size of a UI object, a change in a similar word, or a change in the position in the screen structure, scenario rewriting can be avoided, thereby improving the usability of an already-written automation scenario, reducing the cost of monitoring and maintenance of the automation solution, and greatly improving the effect of introducing the automation solution.

[0010] In addition, the present invention proposes a method for automatically extracting the most similar optimal object search information from the current UI screen without user intervention in response to a minor change in object search information in the above-described object search failure situation, and re-searching the object based on this to perform the RPA scenario without interruption (the inventors call this 'automatic recovery').

[0011] In addition, the present invention proposes a method for analyzing the similarity between objects on the current UI screen and target objects based on object search information of a previous RPA scenario in the above-described object search failure situation, in cases where a human user can easily recognize similarity intuitively or conventionally, such as synonyms / abbreviations / multilingual changes / word combination changes, but the similarity cannot be properly extracted using a rigid simple text matching technology based on an algorithm, by utilizing the context-based reasoning ability of LLM similar to human judgment, to more efficiently and accurately derive corresponding objects, and to eliminate inconveniences such as having to refer to synonyms / abbreviations / multilingual databases, taking into account changes in the UI when writing an automation scenario.

[0012] In addition, the present invention proposes a method for extracting suitable search information through image analysis utilizing multi-modal LLM when automatically extracting search information for a corresponding object on a changed UI screen in the above-described object search failure situation.

[0013] In addition, the present invention proposes a method for extracting optimal corresponding object information through image analysis utilizing multi-modal LLM for information that does not have 'screen display text' such as images / icons or is difficult to extract using existing automated technology.

[0014] In addition, the present invention proposes a method for increasing the flexibility of similarity judgment by applying LLM to the analysis of 'screen display text' that is most directly and sensorily conveyed to human users, along with image analysis utilizing the above-described multi-modal LLM.

[0015] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the contents described in this specification.

[0016] In a first aspect of the present invention, a method for automatically recovering object search information of an RPA (Robotic Process Automation) scenario using a generative AI is provided, the method including: an error recognition step for detecting the occurrence of an error due to applying target object search based on object search information of an RPA scenario created based on a previous UI screen to a current UI screen; an image information LLM lookup step for generating a first prompt including a first image including at least a part of the previous UI screen and a second image including at least a part of the current UI screen, including the target object, to instruct the generative AI to return information on an object corresponding to the target object on the current UI screen; and an object search step for performing target object search targeting the current UI screen.

[0017] Here, in the image information LLM search step, the information returned by the generative AI in response to the first prompt may be information including display information and location information of an object corresponding to the target object.

[0018] Additionally, the information returned by the generative AI in response to the first prompt may further include similarity information between the target object and an object corresponding to the target object.

[0019] Additionally, the first prompt may further include an image of the target object.

[0020] Additionally, in the image information LLM retrieval step, the first image may further include a marker that displays the image of the target object on the previous UI screen.

[0021] Additionally, the first prompt may further include information as a few-shot prompt that enables the generative AI to recognize the target object within the context of the previous UI screen.

[0022] Here, the few-shot prompt may include one or more of the location, type, display text, and information about nearby objects of the target object on the previous UI screen.

[0023] In addition, the method may further include, after the error recognition step, a current UI object information extraction step of extracting information of a plurality of objects on the current UI screen.

[0024] Here, the information of the plurality of objects may include one or more of Control Type, Text, Meta-data, Bounds, Object Path, and Capture Image.

[0025] In addition, the method may further include a step of calculating a similarity by comparing information about each of the plurality of objects on the extracted current UI screen with information about the target object.

[0026] Here, the step of calculating the similarity may be to calculate the similarity based on cosine similarity or longest common sequence (LCS) according to each item of the contrasting information.

[0027] In addition, the step of calculating the similarity may be to generate a candidate object similarity list including similarity information of a plurality of objects on the current UI screen based on each of the similarities.

[0028] In addition, in the image information LLM lookup step of the method, the information returned by the generative AI in response to the first prompt may further include similarity information between the target object and an object corresponding to the target object, and the image information LLM lookup step may further include updating similarity information for an object corresponding to the target object in the candidate object similarity list.

[0029] In addition, the method may further include a text information LLM lookup step of generating a second prompt including text information of objects on the current UI screen and text information of the target object of the previous UI screen based on objects on the current UI screen corresponding to the top K similarities with the target object of the previous UI screen, and instructing the generative AI to select and return an object on the current UI screen that is determined to have the highest similarity to the target object based on the text information comparison.

[0030] In a second aspect of the present invention, a device is provided, including a processor; and a memory, wherein the memory includes instructions configured to cause the device to implement a specific operation for automatically recovering object search information of an RPA (Robotic Process Automation) scenario utilizing a generative AI when executed by the processor, wherein the specific operation may include: detecting the occurrence of an error due to applying target object search based on object search information of the RPA scenario created based on a previous UI screen to a current UI screen; generating a first prompt including a first image including at least a portion of the previous UI screen and a second image including at least a portion of the current UI screen, including the target object, thereby instructing the generative AI to return information of an object corresponding to the target object on the current UI screen; and performing target object search targeting the current UI screen.

[0031] According to the present invention, when applying target object search based on object search information of an RPA scenario created based on a previous UI screen to a current UI screen, if a change occurs for various reasons on the current target app / web, and a specified target object cannot be searched based on information on an already created scenario or an error occurs, the most appropriate corresponding object information on the current UI screen can be automatically extracted to automatically recover from failure in executing the target object search of the previous scenario.

[0032] In addition, according to the present invention, in the above-described object search failure situation, user intervention or manual work is excluded or minimized, and through linkage with a generative AI such as a multi-modal LLM, the most appropriate corresponding object information on the current UI screen is automatically extracted, thereby automatically recovering from failure in target object search execution according to the previous scenario.

[0033] In addition, according to the present invention, if the change of the object search target on the current UI screen is so extensive that it is difficult to find similarities between them when compared to the previous UI screen, scenario rewriting may be inevitable, but if such change is minor, such as a simple change in the position / size of a UI object, a change in a similar word, or a change in the position in the screen structure, scenario rewriting can be avoided, thereby improving the usability of an already created automation scenario, reducing the monitoring and maintenance operation costs of the automation solution, and greatly improving the introduction effect of the automation solution.

[0034] In addition, according to the present invention, in the above-described object search failure situation, the most similar optimal object search information is automatically extracted from the current UI screen without user intervention for minor object search information changes, and the object is searched again based on this to perform the RPA scenario without interruption (the inventors call this 'automatic recovery').

[0035] In addition, according to the present invention, in the above-described object search failure situation, when analyzing the similarity between the objects on the current UI screen and the target object based on the object search information of the previous RPA scenario, in the case where a human user can easily recognize similarity intuitively or conventionally, such as synonyms / abbreviations / multilingual changes / word combination changes, but the similarity cannot be properly extracted using a rigid simple text matching technology based on an algorithm, by utilizing the context-based reasoning ability of LLM similar to human judgment, the corresponding object can be derived more efficiently and accurately, and the inconvenience of having to refer to a synonyms / abbreviations / multilingual database, etc., taking into account the change of the UI when writing an automation scenario can be eliminated.

[0036] In addition, according to the present invention, when it is desired to automatically extract search information for a corresponding object on a changed UI screen in the above-described object search failure situation, suitable search information can be extracted through image analysis utilizing multi-modal LLM.

[0037] In addition, according to the present invention, even information that does not have a 'screen display text' such as an image / icon or that is difficult to extract using existing automated technology can be utilized to extract optimal target object information through image analysis using multi-modal LLM.

[0038] In addition, according to the present invention, in addition to image analysis utilizing the multi-modal LLM described above, the flexibility of similarity judgment can be further increased by applying LLM to the analysis of 'screen display text' that is most directly and sensorily conveyed to human users.

[0039] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the contents described in this specification.

[0040] The accompanying drawings are incorporated into and constitute a part of the detailed description to aid in the understanding of the present invention and together with the detailed description, illustrate embodiments and technical features of the present invention.

[0041] Figures 1a to 1c illustrate a case where an intended target object cannot be searched on the current UI screen based on object search information in a scenario already created based on the previous UI screen.

[0042] FIG. 2 is a flowchart illustrating a method for automatically recovering object search information in an RPA (Robotic Process Automation) scenario utilizing generative AI according to one embodiment of the present invention.

[0043] Figure 3 illustrates a table (135) defining items of search-related information for each object.

[0044] FIG. 4 is a drawing for explaining in detail the object information collection process (S20) during the automatic recovery process of the embodiment shown in FIG. 2.

[0045] Figure 5 is a drawing illustrating a hierarchical structure (140) between objects on the screen.

[0046] Figure 6 is a drawing illustrating an object search information list (110).

[0047] Figure 7 is a drawing for explaining in detail the process of calculating dictionary similarity based on weights for each search information shown in Figure 2 (S30).

[0048] Figure 8 illustrates a table (145) showing the weight for each search information.

[0049] Figure 9 illustrates a candidate object similarity list (125) reflecting the similarity of each search information item and the dictionary comprehensive similarity of multiple objects on the current UI screen.

[0050] Fig. 10 is a flowchart for explaining the process of querying and result processing (S40) to a multi-modal LLM (200) based on a candidate object similarity list (125).

[0051] Figure 11 illustrates a table (150) defining information values ​​for each object added to the first prompt and the format of a string for expressing these information values ​​on the prompt.

[0052] Figure 12 is an example of a first prompt (160) transmitted to LLM (Image).

[0053] Figure 13 illustrates a response (170) returned by LLM (Image) (220) in response to the first prompt (160).

[0054] Figure 14 is a table (180) illustrating each information value (182, 184) of objects to be queried in LLM (Text) (210) and the string format (186, 188) on the second prompt for each information value.

[0055] Figure 15 is an example of a second prompt (190) generated by a Bot as defined in the table (180) of Figure 14.

[0056] FIG. 16 is a diagram illustrating an answer (230) returned by LLM (Text) (210) in response to a second prompt (190).

[0057] Figure 17 illustrates a computing device capable of performing the proposed method of the present invention.

[0058] The present invention is susceptible to various modifications and embodiments. Specific embodiments will be described in detail below, with reference to the accompanying drawings. The following examples are provided to facilitate a comprehensive understanding of the methods, devices, systems, and / or storage media described herein. However, these examples are merely illustrative and the scope of the present invention is not limited thereto.

[0059] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terms used in the detailed description are only for the purpose of describing embodiments of the present invention and should never be construed as limiting. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0060] Additionally, although terms such as first, second, etc. may be used to describe various components, the components are not limited by the terms, and the terms are used only for the purpose of distinguishing one component from another.

[0061] Detecting errors that occur when applying the current UI screen to a pre-written RPA scenario.

[0062] FIG. 2 is a flowchart illustrating a method for automatically recovering object search information in an RPA (Robotic Process Automation) scenario utilizing generative AI according to one embodiment of the present invention. The automatic recovery process of the embodiment begins when a change, such as that exemplified in FIGS. 1a to 1c above, occurs, resulting in an error or failure in the target object search process that the Bot is attempting to perform on the current UI screen based on the search information defined in the existing RPA scenario.

[0063] First, the Bot detects the occurrence of an error in applying the target object search based on the object search information of the RPA scenario written based on the previous UI screen to the current UI screen (100), and receives the current UI screen (100), which is the current RPA target screen, and the existing search information (105 of FIG. 7), that is, the object search information of the RPA scenario written based on the previous UI screen (S10), and accordingly, the Bot invokes the subsequent process.

[0064] Generate a list of object navigation information on the current UI screen

[0065] Accordingly, the Bot collects or extracts information on multiple objects that appear on the current UI screen (100) according to the execution of the app / web (S20) and generates an object search information list (110). Specifically, the Bot can collect search-related information (or search information) for all objects that appear on the current UI screen (100) according to the execution of the app / web. Each object may have all of this search information or may have only some of it. In order to speed up processing, it is possible to utilize only some of the information that has been specified in advance, but for accuracy, all possible information is extracted so that it can be used for similarity calculation for automatic recovery.

[0066] FIG. 3 illustrates a table (135) defining items of search-related information for each object. Search information may include, for example, Control Type, Text, Meta-data, Bounds, Object Path, and Capture Image. In an embodiment, the Bot may extract or collect one or more of the above information for each of a plurality of objects on the current UI screen (100). Here, Control Type refers to the type of control of the corresponding object UI and indicates the type of action that the user can perform (e.g., Button, Hyperlink, Edit, Image, etc.). In addition, Text refers to a string contained in an object or displayed on the screen, which often discloses or suggests the content of the task that the user is trying to perform (e.g., Find ID, FAQ, etc.). In addition, Meta-data refers to information defined in an internal program related to an object, which often includes specific identification information provided by the developer (e.g., ClassName, AutomationID, FrameworkID, etc.). Also, Bounds refers to visual information (i.e. Position + Size) such as the location / size of an object on the app / web screen space. In addition, Object Path indicates the path of an object on the app / web screen logical structure, and this is often hierarchical information (e.g. Root / 2nd-Child / 4th-Child / 11th-Child / 㪋, etc.). In addition, Capture Image refers to image information such as a captured image of the object and the surrounding screen when the target object is specified.

[0067] FIG. 4 is a diagram for explaining in detail the object information collection process (S20) during the automatic recovery process of the embodiment illustrated in FIG. 2. The bot can collect information on all objects existing on the current UI screen, which is the target app / web screen (100) (S20), and generate an object search information list (110). At this time, for example, all information except the Capture Image can be extracted. In some cases, the Capture Image may be used for image analysis through multi-modal LLM in the future, so the Capture Image for each object on the current app / web screen may be unnecessary at this stage.

[0068] Fig. 5 is a diagram illustrating a hierarchical structure (140) between objects on the screen. Objects on the screen have a parent-child relationship with each other, and can be traversed in a tree format based on this during the Object Path extraction process. Since it may take a considerable amount of time to individually extract information on thousands of objects existing on a typical work system screen, the hierarchical tree node is traversed only once for each object to construct an Object Path, so that information extraction can be completed as quickly as possible. Fig. 6 is a diagram illustrating an object search information list (110) generated through the above-described process.

[0069] Generate a list of candidate object similarities

[0070] Automatic recovery of object search information according to an embodiment of the present invention may include a process of finding an object (corresponding object) that has the most similar information to the target object based on previous object search information among all objects on the app / web screen currently displayed on the UI screen (100). This can be viewed as an approach that indirectly selects an object determined to have the most similar information as the object corresponding to the target object based on previous object search information, since it may be difficult to find an object with exactly matching information. In this process, a prior similarity calculation (S30) based on a weight (described later) for each search information may be performed. Here, the reason why the expression 'prior' similarity calculation is used is because it is a similarity calculation performed for filtering in a previous step of the final similarity calculation to be described later through linkage with Gen AI or LLM.

[0071] According to one embodiment of the present invention, the bot may perform a process of calculating a similarity by comparing information about a plurality of objects on the current UI screen that have been extracted with information about a target object based on object search information of an RPA scenario created based on a previous UI screen. Here, the calculation of the similarity may be based on various methods suitable for each type of search information, such as cosine similarity or longest common sequence (LCS), depending on each item of the contrasted information, as will be described below. In addition, if necessary, a comprehensive similarity for the corresponding object may be calculated by a method such as a weighted summation of the similarities for each search information item.

[0072] FIG. 7 is a diagram for explaining in detail the weight-based dictionary similarity calculation process (S30) for each search information shown in FIG. 2. In this process, a score is calculated indicating how similar each item in the derived object search information list (110) is to the search information (105) of the previous search target based on the search information defined in the existing RPA scenario. At this time, the similarity can be calculated by applying an appropriate algorithm suited to the characteristics of each search information. Through this process, a candidate object similarity list (125) can be generated.

[0073] 1) Calculating similarity for Control Type:

[0074] As explained above through Figure 1, even in the changed target app / web screen, if there is an object corresponding to the target object of the previous screen, the content of the original intended operation through the target object before the change should be maintained as much as possible. Therefore, a comparison of Control Types is performed first, and this can be used as filtering information for subsequent comparison tasks. To this end, a cosine similarity comparison is performed between the Control Type string of the target object in the previous object search information and the Control Type string of the comparison target object on the current UI screen. At this time, since the Control Type is usually fixed to a predefined number of types (e.g., 40), the Cosine Similarity for each type can be calculated in advance, a score table can be created in advance, and the speed of the similarity calculation can be improved by utilizing this. Through this string cosine similarity search, it can be expected that a high similarity score will be assigned to controls with similar content. For example, Button / SplitButton / RadioButton, Tab / TabItem, ToolBar / MenuBar / Menu, etc. can be judged to have a higher degree of similarity in terms of control content.

[0075] 2) Text similarity calculation:

[0076] In general, since text information can be seen as the most important information that literally discloses or suggests the content of the task that the user is trying to perform, text similarity calculation is performed between the target object of the previous search information and the objects of the object search information list (110). However, if necessary, text similarity calculation can be performed on the compared target objects whose Control Type similarity score described above is above a certain level (e.g., 80 points). When applying such filtering, it may be desirable to keep the filtering score as high as possible, because empirically, it can lead to more accurate results when the type of action that the user is trying to perform during automatic recovery does not change as much as possible. Since text similarity is also important information for semantic comparison, a similarity calculation based on Cosine Similarity can be applied.

[0077] 3) Meta-data similarity calculation:

[0078] Meta-data information is not important information for general users, but it can contain identifier information that has been given meaning by the developer of the UI screen, such as ClassName and AutomationID, so it can be considered in calculating similarity. At this time, the target objects for comparison with the aforementioned Text comparison score above a certain level (for example, 60 points) can be filtered out, and when applying this filtering, it is desirable to keep the filtering score as low as possible. This is to reduce the possibility of a type 1 statistical error (true but discarded), in which a candidate that will ultimately be selected as the corresponding object is filtered out at this stage. Although Meta-data is composed of strings, it has the characteristics of a kind of non-specific identifier similar to a programming language or code rather than a general language, so the comparison score of the string itself can be used as a similarity measure by applying the LCS (Longest Common Sequence) algorithm.

[0079] 4) Calculating Bounds Similarity:

[0080] Bounds information is important information that includes the location / size shown to the user on the previous UI screen, but it is information that can easily change when the layout is changed. As in the case of calculating the text similarity described above, the filtering score is kept as low as possible to avoid the Type 1 error. In more detail, Bounds similarity can be calculated by comparing information such as a) width / height aspect ratio, b) width / height size, and c) location (x, y). In this case, in the case of a) aspect ratio, since the width / height itself can be vectorized and expressed in two dimensions, cosine similarity calculation can be applied directly. In addition, in the case of b) size and c) location, the similarity can be calculated through a linear function such as whether there is a difference of more than n times when compared to b) size and c) location of the previous search information. At this time, filtering based on a predetermined limit can be applied to each of size and location. For example, filtering can be set to occur if the size differs by more than twice, and filtering can be set to occur if the location differs by more than five times.

[0081] 5) Object Path Similarity Calculation:

[0082] Object Path information is important information that contains hierarchical information about the target object, so comparisons can be performed on all compared objects. Object Path information is also not a subject of comparison based on meaning, but rather a string representation of information about the hierarchical relationship between identified nodes. Therefore, the Longest Common Sequence (LCS) algorithm can be applied to calculate similarity. Using LCS, it is possible to derive the degree of matching between substrings, thereby scoring the presence or absence of hierarchical segments around the basic search information.

[0083] Through the above-described process, the similarity of each object can be calculated by comparing the information of each object on the current UI screen with the information on the target object on the previous UI screen. The dictionary comprehensive similarity of each object on the current UI screen can be calculated by a method such as weighting and adding up the detailed similarities described above. The default value of the weight can be set to a predetermined appropriate value, but the user can also adjust this value as needed. Fig. 8 illustrates a table (145) showing the weight for each search information.

[0084] As shown in FIGS. 2 and 7, a candidate object similarity list (125) is generated through the weight-based dictionary similarity calculation process (S30) for each search information described above. FIG. 9 illustrates a candidate object similarity list (125) that reflects the similarity for each search information item and the dictionary comprehensive similarity for multiple objects on the current UI screen. Since similarity is a relative value, it can be expressed in various quantification methods, and in FIG. 9, each similarity is expressed as a percentage.

[0085] Multimodal LLM (Image) query and processing via image-based first prompts

[0086] Fig. 10 is a flowchart illustrating a process of querying and processing results (S40) to a multi-modal LLM (200) based on a candidate object similarity list (125). As shown in Figs. 2 and 10, a query is made to the multi-modal LLM to select a final target from among the top K candidate lists on the candidate object similarity list (125), and the result is processed (S40), thereby obtaining automatically recovered object search information (130) through a process described below.

[0087] To this end, the top K candidates can be selected by first sorting based on the aforementioned comprehensive dictionary similarity, etc. The inventors of the present invention observed that simply selecting the highest-scoring object based on comprehensive dictionary similarity may be prone to error, and that correcting the comprehensive dictionary similarity described above by utilizing the context-based semantic inference function of the multi-modal LLM, which is similar to human judgment, as will be described in detail in this embodiment, can lead to more accurate results.

[0088] The reason is that the similarity calculation method based on the above-mentioned algorithm lacks functional flexibility because it is based on a fixed rigid algorithm, and has the limitation of being a mechanical comparison based on the syntax and the value of the data itself, whereas the comparison based on generative AI such as multi-modal LLM can reflect semantic similarity similar to the process in which humans recognize commonalities between objects, and enables comparison of context and inferred meaning beyond simple comparison of phrases and unit information, and furthermore, by utilizing image-based multi-modal LLM together with text-based LLM using captured images on the screen where the target object of the existing search information existed, it is understood that more diverse multi-sensory information can be applied to similarity calculation.

[0089] Below, the process of querying and processing results (S40) for a multi-modal LLM (200) is described in detail. This process can be broadly divided into an image information LLM retrieval process and a text information LLM retrieval process.

[0090] In the image information LLM lookup process, a process can be performed by the Bot to generate a first prompt including a first image including at least a part of a previous UI screen and a second image including at least a part of a current UI screen, and instruct the generative AI to return information about an object that is determined to correspond to a target object on the current UI screen.

[0091] First, the bot requests a similarity analysis using the multi-modal LLM(Image)(220) for the image information among the information of the object to be searched through the first prompt based on the image. More specifically, the image of the previous UI screen to which the target object of the existing object search information belonged and the marker information for the area of ​​the target object within the image are attached as context information to the first prompt and transmitted to the LLM(Image)(220), thereby causing the LLM(Image)(220) to perform few-shot prompting, and at the same time, the current UI screen image is transmitted to request the analysis result from the LLM(Image)(220).

[0092] Few-shot prompting allows LLM (Image) to more broadly and precisely understand the contextual categories the user is attempting to grasp within the image. The prompts for the few-shot prompts delivered with the image via the first prompt can include information that can only be visually judged or inferred from the screen image, such as:

[0093] 1) Location / size of the previous search target object

[0094] 2) Type of object to be searched

[0095] 3) Display text for the previous search target object (if available)

[0096] 4) Information on nearby objects around the previous search target object (location / size / type / display text, etc.)

[0097] This information must be converted into a sentence format that LLM can understand and applied to the prompt. These values ​​are either object search information for the target object in the previous scenario or fixed-format data handled within the relevant program, and thus can be extracted. Therefore, this information can be extracted, converted into sentences, and added to the prompt. Figure 11 illustrates a table (150) defining information values ​​for each object added to the first prompt and the format of the string for expressing these information values ​​on the prompt.

[0098] Fig. 12 is an example of a first prompt (160) transmitted to an LLM (Image). The illustrated first prompt (160) may include instructions (162) for an LLM (Image), a prompt (Few-shot) for temporary learning (Few-shot prompting) (163), a format of a response (RESPONSE FORMAT) (164), a first image (Template Image) (165) including at least a portion of a previous UI screen including a search target object, and a second image (User Image) (166) including at least a portion of a current UI screen that is predicted to include an object corresponding to the search target object corresponding to at least a portion of the previous UI screen.

[0099] Here, the first image (165) may further include a marker (168, indicated by a dotted line in FIG. 12) that indicates or displays an image of a target object on the previous UI screen. In addition, the first prompt (160) may further include an image (Find Image) (167) of the target object on the previous UI screen.

[0100] In addition, the prompt (Few-shot) (163) for temporary learning (Few-shot prompting) or the Few-shot prompt may be supplementary information or context information that enables the LLM (Image) (220) to recognize the target object within the context of the previous UI screen. As illustrated in Fig. 12, the Few-shot prompt may include one or more of the location, type, display text, and information about nearby objects of the target object on the previous UI screen.

[0101] Additionally, the response format (RESPONSE FORMAT) (164) included in the first prompt may indicate the format of the response that LLM (Image) (220) will return.

[0102] The information returned by the LLM (Image) (220) in response to the first prompt may include display information and location information of an object (corresponding object) on the current UI screen corresponding to the target object. If necessary, the information returned by the LLM (Image) (220) in response to the first prompt may further include the similarity between the target object and an object on the current UI screen corresponding to the target object. Fig. 13 illustrates an example of a response (170) returned by the LLM (Image) (220) in response to the first prompt (160). Here, "bounds" represents the current UI screen location ("335, 340") and size ("100, 35") of the object corresponding to the target object recognized by LLM (Image) (220), "display_text" represents text information displayed on the object recognized by LLM (Image) (220), and "score" represents the similarity score estimated by LLM (Image) (220) as a percentage.

[0103] The similarity analysis using the multi-modal LLM (Image) (220) through the first prompt based on the image described above can be performed based on the candidate object similarity list (125) (see FIGS. 2 and 7, etc.) targeting the top K candidate lists obtained through the above-described preliminary comprehensive similarity evaluation among the multiple objects on the current UI screen, but is not necessarily limited to this method, and can be performed on all of the multiple objects searched on the current UI screen, or can be applied to objects selected according to other predetermined criteria for limiting the scope.

[0104] In addition, if necessary, if the information returned by LLM (Image) (220) in response to the first prompt includes similarity information between the target object and an object on the current UI screen corresponding to the target object, the object corresponding to the target object may be found in the candidate object similarity list (125) generated in the previous process and the similarity information for the object may be updated or corrected. The updating or correction of the similarity information may be applied in an appropriate manner, such as by weighting or adding the similarity score returned by LLM (Image) (220) and the previously calculated similarity score.

[0105] Multimodal LLM (Text) querying and processing via text-based secondary prompts

[0106] Thereafter, the Bot may perform a text information LLM lookup process in which a second prompt is generated that includes text information of objects on the current UI screen and text information of the target object on the previous UI screen based on the objects on the current UI screen that correspond to the top K similarities with the target object on the previous UI screen, and instructs LLM (Text) (210) to select and return an object on the current UI screen that is determined to have the highest similarity to the target object based on the text information comparison.

[0107] This process is to enable LLM(Text)(210) to select the most semantically similar object based on the context by transferring the list of Top K information with the highest scores and the previous search target information to LLM(Text)(210) based on the candidate object similarity list (125) derived from the previous process, the similarity score extracted through LLM(Image)(220), etc.

[0108] At this time, information other than the text information currently displayed on the UI screen may act as noise in the semantic inference of LLM(Text)(210), so they can be removed, and only the text information can be configured as a second prompt to be transmitted to LLM(Text)(210). For practical convenience, the second prompt can be configured to set the transmitted information as an internal argument in the JSON format, which is easy to configure from a programming perspective, and the response of LLM(Text)(210) can also be induced to adhere to the JSON format as much as possible.

[0109] Figure 14 is a table (180) illustrating each information value (182, 184) of objects to be queried in LLM (Text) (210) and the string format (186, 188) on the second prompt for each information value.

[0110] In addition, FIG. 15 is an example of a second prompt (190) generated by a Bot according to the definition in the table (180) of FIG. 14. The illustrated second prompt (190) may include instructions (192) for LLM (Text) (210), a response format (RESPONSE FORMAT) (193), a response example (RESPONSE EXAMPLE) (194), a text information list (LIST) (195) of objects on the current UI screen, and text information (USER TASK) (196) of a target object on the previous UI screen. The "index" described in the text information list (LIST) (195) may indicate a ranking in the Top K information list on the candidate object similarity list (125) described above.

[0111] The second prompt passed to LLM(Text)(210) may require optimization depending on the type of LLM service being utilized. However, it is preferable to request that the transmission and response information be exchanged in a format that facilitates pre- and post-processing within the program, such as JSON. LLM(Text)(210) can return information about the object it determines to be most similar through its own context-based semantic inference, rather than the object with the highest similarity score on the transmitted text and similarity list.

[0112] FIG. 16 is a diagram illustrating a response (230) returned by LLM (Text) (210) in response to a second prompt (190).

[0113] The prior similarity calculation process (S30) shown in FIGS. 2 and 7 can be understood as a filtering operation that removes information of a kind of noise nature that is judged to be completely impossible in order to facilitate subsequent processing through the multi-modal LLM (200), and it was confirmed that a more accurate automatic recovery process can be performed by using the selected items of the LLM (Text) (210) returned from the LLM (Text) (210) as automatic recovery information for the object information that is the previous search target.

[0114] Device to which the proposed method of the present invention can be applied

[0115] Fig. 17 illustrates a device (120) capable of performing the proposed method of the present invention.

[0116] Referring to FIG. 17, a device (120) may be configured to implement the proposed method of the present invention. For example, the device (120) may be a computing device, a server device, a terminal device, a network device, etc. for performing the process of the present invention.

[0117] For example, the device (120) to which the proposed method of the present invention can be applied may include network devices such as repeaters, hubs, bridges, switches, routers, gateways, etc., computer devices such as desktop computers, workstations, etc., mobile terminals such as smartphones, portable devices such as laptop computers, etc., home appliances such as digital TVs, etc., and transportation means such as automobiles, etc. As another example, the device (120) to which the present invention can be applied may be included as a part of an ASIC (Application Specific Integrated Circuit) implemented in the form of a SoC (System On Chip).

[0118] The memory (20) can be connected to the processor (10) when it operates, and can store programs and / or commands for processing and controlling the processor (10), and can store data and information used in the present invention, control information required for data and information processing according to the present invention, temporary data generated during data and information processing, etc. The memory (20) can be implemented as a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, an SRAM (Static RAM), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0119] The processor (10) can be operatively connected to the memory (20) and the network interface (30), and controls the operation of each module in the device (120). In particular, the processor (120) can perform various control functions for performing the proposed method of the present invention. The processor (120) can also be called a controller, a microcontroller, a microprocessor, a microcomputer, a GPU (Graphics Processing Unit), etc. The proposed method of the present invention can be implemented by hardware, firmware, software, or a combination thereof. When the present invention is implemented using hardware, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processing Device), a DSPD (Digital Signal Processing Device), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), etc. configured to perform the present invention can be provided in the processor (10). Meanwhile, when implementing the proposed method of the present invention using firmware or software, the firmware or software may include instructions related to modules, procedures, or functions that perform functions or operations necessary to implement the proposed method of the present invention, and the instructions may be stored in the memory (20) or stored in a computer-readable recording medium (not shown) separate from the memory (20) and, when executed by the processor (10), the device (120) may be configured to implement the proposed method of the present invention.

[0120] In addition, the device (120) may include a network interface device (30). The network interface device (30) is connected to the processor (10) when in operation, and the processor (10) may control the network interface device (30) to transmit or receive wireless / wired signals carrying information and / or data, signals, messages, etc. through a wireless / wired network. The network interface device (30) supports various communication standards, such as, for example, IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and may transmit and receive control information and / or data signals according to the communication standards. The network interface device (30) may also be implemented outside the device (120) as needed.

[0121] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that do not have an explicit citation relationship in the patent claims may be combined to form an embodiment or incorporated as a new claim through a post-application amendment.

[0122] The present invention can be applied to various devices such as computing devices, server devices, terminal devices, and network devices capable of executing RPA scenarios utilizing generative AI.

Claims

1. A method for automatically recovering object search information in an RPA (Robotic Process Automation) scenario using generative AI. An error recognition step for detecting the occurrence of an error in applying target object search based on object search information of the RPA scenario created based on the previous UI screen to the current UI screen; An image information LLM lookup step that generates a first prompt including a first image including at least a part of the previous UI screen including the target object and a second image including at least a part of the current UI screen, thereby instructing the generative AI to return information of an object corresponding to the target object on the current UI screen; and A method comprising an object search step for performing target object search targeting the current UI screen.

2. In claim 1, A method wherein, in the image information LLM retrieval step, the information returned by the generative AI in response to the first prompt is information including display information and location information of an object corresponding to the target object.

3. In claim 2, A method wherein the information returned by the generative AI in response to the first prompt further includes similarity information between the target object and an object corresponding to the target object.

4. In claim 1, A method wherein the first prompt further includes an image of the target object.

5. In claim 1, A method, wherein the first image further includes a marker that displays an image of the target object on the previous UI screen.

6. In claim 1, A method wherein the first prompt further includes information as a few-shot prompt that enables the generative AI to recognize the target object within the context of the previous UI screen.

7. In claim 6, A method wherein the above few-shot prompt includes one or more of information about the location, type, display text, and surrounding nearby objects of the target object on the previous UI screen.

8. In claim 1, A method further comprising, after the above error recognition step, a current UI object information extraction step of extracting information of a plurality of objects on the current UI screen.

9. In claim 8, A method wherein the information of the plurality of objects includes one or more of Control Type, Text, Meta-data, Bounds, Object Path, and Capture Image.

10. In claim 8, A method further comprising a step of calculating a similarity by comparing information about each of a plurality of objects on the extracted current UI screen with information about the target object.

11. In claim 10, A method in which the step of calculating the above similarity performs similarity calculation based on cosine similarity or longest common sequence (LCS) according to each item of the above contrasting information.

12. In claim 10, A method in which the step of calculating the above similarity generates a candidate object similarity list including similarity information of a plurality of objects on the current UI screen based on each of the above similarities.

13. In claim 12, In the image information LLM retrieval step, the information returned by the generative AI in response to the first prompt further includes similarity information between the target object and the object corresponding to the target object. A method wherein the image information LLM lookup step further includes updating similarity information for an object corresponding to the target object in the candidate object similarity list.

14. In claim 1, A method further comprising a text information LLM lookup step in which a second prompt including text information of objects on the current UI screen and text information of the target object on the previous UI screen is generated based on objects on the current UI screen corresponding to the top K similarities with the target object on the previous UI screen, and the generative AI is instructed to select and return an object on the current UI screen that is determined to have the highest similarity to the target object based on the text information comparison.

15. A device including a processor and a memory, The above memory includes instructions configured to cause the device to implement a specific operation for automatic recovery of object search information in an RPA (Robotic Process Automation) scenario utilizing generative AI when executed by the processor, the specific operation being: Detecting the occurrence of an error in applying target object search based on object search information of the above RPA scenario written based on the previous UI screen to the current UI screen; Generating a first prompt including a first image including at least a portion of the previous UI screen including the target object and a second image including at least a portion of the current UI screen, thereby instructing the generative AI to return information of an object corresponding to the target object on the current UI screen; and A device comprising: performing target object search targeting the current UI screen.

Images