Artificial intelligence meta agent system based on language processing using computer input and output

Abstract

The present document relates to an artificial intelligence meta agent system based on language processing using computer input and output. The proposed meta agent system comprises: an intention interpretation unit for identifying an entity of a specific type from input data collected from an input device of a computer and allocating the entity to a predefined category to classify the entity; an intention work unit for activating an application installed in the computer and performing work intended by a user in the activated application according to the entity classified by the intention interpretation unit; and a result generation unit for providing the result of the work performed by the intention work unit to an output device of the computer in a preconfigured form.

Description

An artificial intelligence meta-agent system based on language processing using computer input and output.

[0001] The following description is about an artificial intelligence meta-agent system based on language processing using computer input / output.

[0002] Conversational AI is an AI system that can engage in human-like conversations with users through natural language processing and generation technologies. These systems can be utilized in a variety of applications, such as customer service chatbots, personal assistants, or voice-activated assistant services. The goal of conversational AI is to automate and improve human-machine communication, making it more intuitive and accessible. For example, conversational AI services are available to users through OpenAI's ChatGPT and Microsoft's Bing.

[0003] Conversational AI utilizes large-scale language models to extract and provide information through online internet searches, web crawling, and web scraping. However, utilizing web crawling or web scraping techniques has technical limitations, as the extraction of information is limited to information publicly available online. Therefore, technologies that can expand the foundational technology of conversational AI are urgently needed in this field.

[0004] Meanwhile, the rapid development of digital technologies, including the rapid rise of generative artificial intelligence, has provided opportunities for innovation, but it has also led to serious social problems, such as new discrimination and alienation, such as loss of economic opportunities for digitally disadvantaged groups such as the disabled and the elderly who lack the ability to use and utilize digital technologies.

[0005] A prime example is the difficulty experienced by people with disabilities and the elderly when ordering at kiosks in restaurants and other food and beverage establishments. The Ministry of Health and Welfare reported in a 2020 survey that 64.2% of seniors aged 65 and older experienced difficulty ordering at kiosks. The Korea Consumer Agency found that the reasons for seniors' difficulties using kiosks included: 1) the complex operation of the kiosks, 2) concern about the eyes of those behind them in line, and 3) the small size of the images and text.

[0006] As the enforcement decree of the Act on Prohibition of Discrimination against Persons with Disabilities and Remedies for their Rights, which includes the introduction of voice guidance kiosks to improve accessibility for the visually impaired as a policy for the disabled, is due to be implemented in January 2025, McDonald's is currently implementing voice guidance kiosk services for the visually impaired, and Lotteria, Burger King, and others are developing voice guidance kiosks. As a policy for the elderly, a partial amendment to the Elderly Welfare Act has been proposed to address the digital divide, and the National Human Rights Commission of Korea has recommended to the Minister of Science and ICT to improve the human rights violations of the elderly caused by the digital divide. In addition, the Digital Inclusion Act, which aims to ensure that all members of society can equally enjoy the benefits of digital technology without discrimination or exclusion, is currently under legislative notice.

[0007] However, even if the voice guidance kiosks currently provided to the visually impaired were to be provided to the elderly in the future, the difficulties experienced by the elderly in using the kiosks, which are 1) the complex operation of the kiosks and 2) the need to be mindful of the people behind them in line, would not be resolved. Furthermore, small business owners using voice guidance kiosks would have to pay for the voice guidance equipment and service in addition to the cost of installing the kiosk. Furthermore, from an environmental perspective, there are installation and maintenance costs associated with installing or maintaining new kiosks or voice guidance kiosk equipment, and electronic waste generated when replacing old equipment causes environmental problems.

[0008] In one aspect of the present invention, in order to solve the above-mentioned problem, it is an object of the present invention to obtain an artificial intelligence meta-agent system based on language processing using computer input / output, which operates a computer by identifying a user's intention from input collected from an input device of a computer unit, and provides the result of a task performed according to the user's intention to an output device of the computer, so as to expand the level of background technology of conversational artificial intelligence to computer input / output.

[0009] In another aspect of the present invention, the present invention provides a method for digitally excluded people, such as the elderly, to easily order by using a computer, including a smartphone, without checking the screen, without touching the screen, without searching the menu, without confusion, following voice guidance, and sitting at a seat through a conversation with a language processing-based artificial intelligence meta-agent provided according to embodiments of the present invention, which can replace the ordering methods using kiosks, tablet menus, and QR codes currently used in stores, thereby enabling more elderly people to enjoy the benefits of digital technology without difficulty and without discrimination or exclusion, thereby solving social problems caused by the digital divide.

[0010] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the art from the description of the present invention.

[0011] In one aspect of the present invention for solving the above-described problem, a meta-agent system based on language processing using computer input / output is proposed, comprising: an intention interpretation unit for identifying a specific type of entity from input data collected from an input device of the computer and classifying the entity by assigning the entity to a predefined category; an intention task unit for activating an application installed on the computer according to the entity classified by the intention interpretation unit and performing a task intended by the user in the activated application; and a result generation unit for providing a task result performed by the intention task unit to an output device of the computer in a preset format.

[0012] In one embodiment of the present invention, a voice processing unit may be additionally included to convert input data collected from an input device of the computer into text data, and the intent interpretation unit may be configured to obtain the text data from the voice processing unit.

[0013] In addition, a visual question answering (VQA) unit configured to detect objects from visual data among the above input data may be additionally included.

[0014] In addition, the system may further include an input data collection unit for collecting input data from an input device of the computer; and an output data transmission unit for transmitting the output data to an output device of the computer. Preferably, the meta-agent system can be installed and operated on the computer.

[0015] In addition, the intention interpretation unit is a meta-agent system characterized in that it classifies an entity based on natural language processing (NLP) including named entity recognition (NER) or a large language model (LLM) from the input data.

[0016] In addition, the VQA unit is a meta-agent system characterized in that it detects an object based on a multimodal large language model (multimodal LLM) including a visual question answering (VQA) model from multimodal data among the input data.

[0017] In addition, the intention task unit can be configured to perform the task intended by the user in the application, regardless of whether a module including a library or an API (Application Programming Interface) of an application corresponding to the object classified by the intention interpretation unit is provided.

[0018] The above-mentioned intention task unit may be configured to perform the task intended by the user in the application based on (2) a package including an accessibility service provided by the operating system of the computer, or (3) object detection using a multimodal large language model (multimodal LLM) including a visual question answering (VQA) model, if the module of the application is not provided.

[0019] In addition, the intention task unit may be configured to perform the task intended by the user in the application based on (1) a module including an application library or API (Application Programming Interface) corresponding to the object classified by the intention interpretation unit, (2) a package including an accessibility service provided by the operating system of the computer, or (3) object detection using a multimodal large language model (Multimodal LLM) including a visual question answering (VQA) model.

[0020] The above-mentioned intention task unit can operate by obtaining target UI (User Interface) element node information of the application when performing the task intended by the user in the application (1) based on the module of the application or (2) based on the package provided by the operating system of the computer.

[0021] In addition, the intention task unit can obtain target object detection information of the application and operate when the user performs the task intended by the application based on object detection using a multimodal LLM including (3) the VQA model.

[0022] Additionally, at least one of the intention interpretation unit or the intention task unit may additionally include a simplification processing unit for simplifying information required for performing the task intended by the user in the application.

[0023] Additionally, if additional input is collected through the input device while the meta agent system is performing a computer operation on behalf of the user, the system can generate the work result by reflecting the additional input.

[0024] As described above, according to the present invention, a virtual agent or secretary can act between a user and a computer.

[0025] In addition, the present invention has the effect of enabling a computer unit to search for and provide information required by a user on an Internet network using a conversational command based on voice language or text language.

[0026] In addition, the present invention has a remarkable effect of being able to replace a user's computer operation according to the user's intention by converting a conversational command based on a voice or text language containing the user's intention into a computer control command, thereby activating / deactivating applications of various operating systems, inputting photos and keywords into applications, and enabling touch, click, scroll, tap, swipe, etc.

[0027] As described above, the meta agent according to embodiments of the present invention sequentially provides the user with necessary information through a conversation between the user and the meta agent, identifies the user's intention, and performs an action that obtains the same effect as a physical action such as clicking or typing on an actual system according to the user's intention, thereby enabling the entire series of tasks requested by the user to be performed, thereby enabling the digitally disadvantaged, such as the elderly, to enjoy the benefits of digital technology without discrimination or exclusion.

[0028] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the detailed description and the description of the claims.

[0029] Figure 1 is a configuration diagram of an artificial intelligence meta-agent system based on language processing using computer input / output according to one embodiment of the present invention.

[0030] FIG. 2 is a diagram of a language processing-based artificial intelligence meta-agent system configuration using computer input / output in which input and output devices are separated from a computer unit according to one embodiment of the present invention.

[0031] Figure 3 is a detailed configuration diagram of an intention processing unit according to one embodiment of the present invention.

[0032] FIG. 4 is a configuration diagram of an artificial intelligence meta-agent system based on language processing using computer input / output including a detailed configuration according to one embodiment of the present invention.

[0033] FIG. 5 is a diagram for explaining a concept of performing a task through a target application according to one embodiment of the present invention.

[0034] FIGS. 6 to 9 are drawings for explaining a scenario of ordering food of a specific brand using a meta agent according to one embodiment of the present invention.

[0035] FIG. 10 is a diagram for explaining a concept of utilizing accessibility services according to one embodiment of the present invention.

[0036] FIG. 11 is a diagram illustrating a meta-agent system that additionally includes a simplification processing function according to one embodiment of the present invention.

[0037] The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the present invention.

[0038] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0039] Hereinafter, an embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0040] Figure 1 is a configuration diagram of a language processing-based artificial intelligence meta-agent system utilizing computer input / output according to one embodiment of the present invention. Figure 2 is a configuration diagram of a language processing-based artificial intelligence meta-agent system utilizing computer input / output, in which the input and output devices are separated from the computer unit according to one embodiment of the present invention. Figure 3 is a detailed configuration diagram of an intention processing unit (220) according to one embodiment of the present invention.

[0041] Referring to FIGS. 1 and 2, the meta-agent system of the present invention includes an input data collection unit (100) that collects input from an input device (11) of a computer unit (10), a meta-agent unit (200) that operates the computer unit (10) based on the input and generates an output, and an output data transmission unit (300) that transmits the output to an output device (12) of the computer unit (10). FIG. 1 shows a case where a computer unit (10) having an input device (11) and an output device (12) and an input data collection unit (100), a meta-agent unit (200), and an output data transmission unit (300) are configured separately, and FIG. 2 shows a case where a computer unit (10), an input device (11), an output device (12), an input data collection unit (100), a meta-agent unit (200), and an output data transmission unit (300) are configured separately, and there can be countless combinations of these configurations, from a case where the computer unit (10) includes an input device (11), an output device (12), an input data collection unit (100), a meta-agent unit (200), and an output data transmission unit (300) to the case of FIG. 2.

[0042] The computer unit (10) mentioned in the present invention can be implemented in a personal computer, tablet PC, smartphone, robot, etc. that has an operating system (OS) installed and can perform various multimedia processing. The input device (11) may include a camera, a microphone, and a touch screen. The input device (11) may be built into the main body of the computer unit (10) or may be separately connected to the outside of the main body of the computer unit (10) via a wire or wireless connection.

[0043] Next, the input data collection unit (100) can transmit the input to the meta agent unit (200). The input referred to in the present invention may include images and videos input from a camera, speech input from a microphone, location information input from a touch screen, mouse, and keyboard, and text input from a touch screen and keyboard. As described above, when the computer unit (10) is configured to include an input device (11), an output device (12), an input data collection unit (100), a meta agent unit (200), and an output data transmission unit (300), operation is possible even without the input data collection unit (100).

[0044] In operating the computer unit (10) based on a user voice command, the meta agent unit (200) is characterized by including a voice processing unit (210) that converts the voice input into text data based on a speech-to-text (STT) technology, an intent processing unit (220) that interprets the user intent from the text data, and a VQA unit (230) that recognizes an object from visual data including images and videos among the input using a visual question answering (VQA) model. In the case of operating the computer unit (10) based on the text input by the user, operation is possible even without the voice processing unit (210).

[0045] Referring to one embodiment of FIG. 3, the intent processing unit (220) is characterized by including an intent interpretation unit (221) that identifies a specific type of entity in the text data, classifies the entity by assigning it to a predefined category, and thereby determines the user's intent; an intent operation unit (222) that performs a specific task intended by the user, including activating an application installed in the computer unit (10), searching for an application, moving a cursor on a screen provided by the application, touching, clicking, scrolling, tapping, swiping, typing, etc., according to the entity classified by the intent interpretation unit (221); and a result generation unit (223) that generates a result of the task performed by the intent operation unit (222) in a preset format. If the task intended by the user is a touch, the intent operation unit (222) programmatically generates a touch event at a specific node or coordinate and transmits it to the application. The application accepts the touch event as if it were an actual touch and performs a response action to the touch, and the response action is displayed on the touch screen. Typically, the touch screen is regarded as an input device (11). However, as shown in the above touch example, the touch screen can be considered as an input device (11) and an output device (12) at the same time because it reacts (outputs) as if touched by the meta agent of the present invention even though no actual touch (input) has occurred. In addition to the touch screen, a keyboard, mouse, etc. can also be considered as an input device (11) and an output device (12) at the same time.

[0046] The term "application" as used herein may be interpreted differently depending on the operating system. If the computer unit (10) is equipped with Android or iOS, an operating system optimized for smartphones, it refers to applications distributed through the Google Play Store or App Store. If the computer unit (10) is equipped with the Windows operating system optimized for PCs, it refers to files, utilities, and browser icons displayed on the desktop.

[0047] First, the voice processing unit (210) can recognize only the user's voice from the voice based on speech recognition technology using a voice recognition utility such as Google Cloud API. In addition, the voice processing unit (210) can convert the recognized voice into text data and output the text data.

[0048] Next, the intention processing unit (220) can interpret or identify the user's intention from the text data based on artificial intelligence technology, natural language processing (NLP) or a large language model (LLM).

[0049] Specifically, in interpreting the user intent, the intent interpretation unit (221) can classify the sentence type (Type) in the text data into one of a question, an imperative, a declarative, and an exclamation. In addition, the intent interpretation unit (221) can identify and classify a specific type of entity (Named Entity) from the text data based on Named Entity Recognition (NER) technology such as spaCy, Google Cloud Natural Language API, Microsoft Azure Text Analytics, etc. The intent interpretation unit (221) is characterized in that it includes at least one of a place (Place), a person (Person), an object (Object), an organization name (Organization), a time (Time), an amount (Amount), an environment (Environment), and an event (Event) in a predefined category.

[0050] For example, the intention interpretation unit (221) can identify and classify the sentence type (Type) as 'interrogative sentence' and the time (Time) as 'now' for the text data 'What time is it now? '. The intention work unit (222) can activate the clock application installed in the computer unit (10) according to the classification result of time (Time). Among the output devices (12) of the computer unit (10), the monitor can display the clock application.

[0051] Additionally, the intent task unit (222) can retrieve the current time from an activated clock application. The result generation unit (223) can generate the task result in a preset format, such as "It is now 1:30 PM." The examples described above can be expressed in order as follows.

[0052] <text data>

[0053] What time is it now?

[0054] <Intention Interpretation Department>

[0055] type: interrogative sentence

[0056] time : now

[0057] Intentional Work Department

[0058] Activate the clock application

[0059] Find the current time on your watch

[0060] Current time transmission

[0061] <Result generation section>

[0062] It is now 1:30 PM.

[0063] In addition, the intention interpretation unit (221) can identify and classify the sentence type (Type) as 'command', the agent task (Work) as 'tell', and the place (Place) as 'here' for the text data 'Tell me where this is'. The intention work unit (222) can activate the map application installed in the computer unit (10) according to the place (Place). The monitor among the output devices (12) of the computer unit (10) can display the map application. In addition, the intention work unit (222) can search for the current location and address in the activated map application. The result generation unit (223) can generate the work result as 'This is in front of Gyeongbokgung Palace' according to the preset format. The above-described examples are expressed in order as follows.

[0064] <text data>

[0065] Tell me where this is

[0066] <Intention Interpretation Department>

[0067] type: statement

[0068] work : to inform

[0069] place : here

[0070] Intentional Work Department

[0071] Activate the map application

[0072] Search current location and address

[0073] Send current location and address

[0074] <Result generation section>

[0075] This is in front of Gyeongbokgung Palace.

[0076] In addition, the intention interpretation unit (221) can identify and classify the sentence type (Type) as 'command', the agent task (Work) as 'describe', and the place (Place) as 'Gyeongbokgung Palace' for the text data 'Explain about Gyeongbokgung Palace'. The intention work unit (220) can activate a browser application such as Chrome installed in the computer unit (10) according to the agent task (Work). Among the output devices (12) of the computer unit (10), a monitor can display the browser application. In addition, the intention work unit (222) can search for Gyeongbokgung Palace in the activated browser application. The result generation unit (223) can generate the task result as Gyeongbokgung Palace description information according to a preset format. The above-described examples are expressed in order as follows.

[0077] <text data>

[0078] Tell me about Gyeongbokgung Palace

[0079] <Intention Interpretation Department>

[0080] type: statement

[0081] work : to explain

[0082] place: Gyeongbokgung Palace

[0083] Intentional Work Department

[0084] Activate the browser application

[0085] Gyeongbokgung Palace Search

[0086] Gyeongbokgung Palace Information

[0087] <Result generation section>

[0088] Gyeongbokgung Palace is one of the palaces of the Joseon Dynasty located in Seoul, South Korea. ...Gyeongbokgung Palace is an important historical site representing Korean history and culture.

[0089] In operating the computer unit (10) based on text, video, and images, the intention task unit (222) is characterized in that it performs a search task by further utilizing objects recognized from the VQA unit (230). First, the VQA unit (230) can receive the text being input from the input data collection unit (100). The text may be text in English, Korean, or another language input from at least one of a touch screen and a keyboard among the input devices (11).

[0090] Meanwhile, the VQA unit (230) can directly determine the user's intention based on NLP or LLM technology after receiving the text input from the input data collection unit (100) or the text data converted into voice from the voice processing unit (210). Alternatively, the VQA unit (230) can also receive the user's intention after interpretation from the intent interpretation unit (221). In other words, the VQA unit (230) can determine the user's intention from text data without being limited to a specific method when operating the computer unit (10) based on text, video, and images.

[0091] At the same time, the VQA unit (230) can receive at least one of the input images and videos, along with text or voice, as necessary, from the input device (11), or from the input data collection unit (100), or via the intent interpretation unit (221). The image is a photograph composed of one frame, and the video is a moving image or video composed of multiple frames. The results of these collections are collectively called visual data, and the results of the text, voice, image, and video collections are collectively called multimodal data. The visual question answering (VQA) model referred to in the present invention is not a VQA model in a narrow sense that combines image processing and natural language processing, but a multimodal large language model (Multimoldal LLM) that includes a VQA model. Accordingly, the above VQA unit (230) can understand the content of images and videos and associate it with the context of the text, i.e., the user's intention, and furthermore, can also identify the user's intention from voice and video.

[0092] For example, the intent interpretation unit (221) can identify and classify the sentence type (Type) as 'interrogative sentence' and the object (Object) as 'this statue' for the text data 'What is this statue?'. The VQA unit (230) can obtain the pre-classified sentence type (Type) and object (Object) from the intent processing unit (220). At the same time, the VQA unit (230) can obtain a photograph of a sculpture. Then, the VQA unit (230) can recognize the sculpture as an object. Then, the intent work unit (222) can activate a browser application, search for a sculpture similar to the sculpture through a search operation, and search for sculpture information including the name, installation location, and manufacturer of the sculpture. In addition, the result generation unit (223) can generate the work result as sculpture name information according to a preset format. The above-described examples are expressed in order as follows.

[0093] <text data>

[0094] What is this statue? (Click on the picture)

[0095] <Intention Interpretation Department>

[0096] type: interrogative sentence

[0097] object: this statue

[0098] Intentional Work Department

[0099] Activate browser application

[0100] Enter and search photos into the application

[0101] Information on sculptures

[0102] <Result generation section>

[0103] This statue is of King Sejong of Joseon.

[0104] In addition, the intent interpretation unit (221) can identify and classify the sentence type (Type) as 'interrogative sentence', the event (Event) as 'soccer game', and the organization name (Organization) as 'winning team' for the text data 'Who won the soccer game today?'. The VQA unit (230) can obtain the sentence type (Type), event (Event), and organization name (Organization) classified in advance from the intent processing unit (220). At the same time, the VQA unit (230) can obtain a soccer game video and recognize Team A, Team B, the coach, and players as multiple objects. Then, the intent work unit (222) can activate a browser application, input photos and keywords for Team A, Team B, the coach, and players, and search for the corresponding soccer game information. The result generation unit (223) can generate the work result as the winning team name according to a preset format. The above-described examples are expressed in order as follows.

[0105] <text data>

[0106] Who won the soccer match today? (Soccer match video)

[0107] <Intention Interpretation Department>

[0108] type: interrogative sentence

[0109] event: soccer match

[0110] organization: winning team

[0111] Intentional Work Department

[0112] Activate browser application

[0113] Enter and search for photos and keywords for Team A, Team B, coaches, and players.

[0114] Information about the winning team

[0115] <Result generation section>

[0116] The Korean team won the soccer match today.

[0117] In controlling the computer unit (10), the meta agent unit (200) can perform a computer task corresponding to the identified user intention.

[0118] For example, the meta agent unit (200) can input objects and keywords on behalf of the user by operating the keyboard among the input devices (11). The meta agent unit (200) can recognize the current position of the mouse and then operate the mouse among the input devices (11) to move or click the mouse cursor on behalf of the user. The meta agent unit (200) can recognize the current position of the touch screen and then operate the touch screen among the input devices (11) to input or click specific objects and keywords on behalf of the user. Therefore, the present invention enables the computer unit (10) to operate according to the user's intention without the user performing any operation after the initial input to the input device (11), thereby replacing the user's computer work.

[0119] In performing a search task, the meta agent unit (200) can search for specific objects and keywords in an activated Chrome application and generate task results using a Kotlin utility on a smartphone running on the Android operating system. The meta agent unit (200) can directly search for specific objects and keywords in an activated Chrome application and generate task results using the Selenium and BeautifulSoup libraries under the Python programming language on a PC running on the Windows operating system. Therefore, the present invention can provide users with more diverse information than the conventional technology that provides information through web crawling technology on a web page.

[0120] Meanwhile, the meta agent unit (200) is characterized in that, if additional input is collected from the input device (11) by the user while performing a computer operation on behalf of the user, the additional input can be transmitted from the input data collection unit (200) and the work result is generated by reflecting the input. Accordingly, the present invention enables collaborative work rather than individual work by the user, and enables the user to generate work results more suitable to the user's intention.

[0121] Next, the output device (12) may include a monitor, a speaker, and other actuators that the user wishes to utilize. The actuators referred to in the present invention may be motors, actuators, etc. that enable the movement of a multi-joint robot or a humanoid robot. The output device (12) may be built into the main body of the computer unit (10), or may be separately connected to the outside of the main body of the computer unit (10) via a wire or wireless connection. In addition, as described above, a touch screen, a keyboard, a mouse, etc. may be considered as an input device (11) and an output device (12) at the same time.

[0122] Next, the output data transmission unit (300) can transmit the work result to the output device (12) in the order requested by the meta agent unit (200), or transmit the work result in real time as soon as it is generated by the meta agent unit (200). The work result referred to in the present invention may include text, images and videos output from a monitor, voice and sound output from a speaker, events for operations such as screen touch, and control commands for controlling an actuator. As described above, when the computer unit (10) is configured to include all of the input device (11), the output device (12), the input data collection unit (100), the meta agent unit (200), and the output data transmission unit (300), operation is possible even without the output data transmission unit (300).

[0123] Meanwhile, at least one of the meta agent unit (200) and the output data transmission unit (300) can convert the text-formatted work result into voice based on text-to-speech (TTS) technology and then transmit it to a speaker among the output devices (12). Accordingly, the user can check the work result not only in text format but also in voice format.

[0124] Therefore, according to the present invention, the meta-agent of the present invention can act as a virtual agent or secretary between the user and the computer unit (10). First, the present invention has a remarkable effect in that the computer unit (10) can search for and provide information required by the user on the Internet network using conversational commands based on spoken or text language.

[0125] In addition, the present invention has a remarkable effect of being able to perform computer operations on behalf of the user according to the user's intention, such as activating / deactivating applications and files, by converting a conversational command based on a voice language or text language containing the user's intention into a computer input / output control command.

[0126] Embodiments may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored on a computer-readable storage medium and executed by one or more processors.

[0127] Aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as program modules or components, being executed by a computer. Typically, program modules or components include routines, programs, objects, and data structures that perform particular tasks or implement particular data formats. Aspects of the subject matter described herein may also be practiced in distributed computing environments, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0128] Meanwhile, the structure that implements functions such as Virtual Network Computing (VNC) or Remote Desktop Protocol (RDP), which remotely control one computer from another, can be classified as an indirect control method based on a client-server structure, and has the problem that it requires a complex configuration such as a network, client-server program or hardware, and requires delegation of authority such as user accounts. In contrast, the method that controls only UI interaction with accessibility services (accessibility services, accessibility features, etc.) can be classified as a direct control method, and control is possible by activating the accessibility service on the user's device.

[0129] Therefore, in one embodiment of the present invention, a meta-agent system is implemented through (2) the direct control method among (1) the indirect control method based on the client-server structure and (2) the direct control method using the accessibility service described above. In this embodiment, in order to solve the above-mentioned problem, a meta-agent system is configured such that an input device (11), an output device (12), an input data collection unit (100), a meta-agent unit (200), and an output data transmission unit (300) are all included in the computer unit (10), so that a user-requested task can be performed based on software through a program using an application programming interface (API) and library provided by a target application, or an API or software development kit (SDK) provided by an OS, or multimodal LLM-based object detection including VQA. Accessibility services, a user interface automator, and other APIs, functions, and utilities can be used in the implementation of the meta-agent system. Accordingly, in the description of embodiments of the present invention, APIs and libraries are collectively referred to as modules, and APIs, libraries, functions, utilities, etc. are collectively referred to as packages. That is, an application-provided module is a software module that includes APIs and libraries provided by a target application necessary for running the target application, and an operating system-provided package is a software package that includes APIs, libraries, functions, utilities, etc. provided by the operating system.

[0130] FIG. 4 is a configuration diagram of an artificial intelligence meta-agent system based on language processing using computer input / output including a detailed configuration according to one embodiment of the present invention.

[0131] FIG. 4 shows a configuration in which all elements are included in the computer unit (10), and the input device (11), the output device (12), and the voice processing unit (210), the intention processing unit (220), and the VQA unit (230) of the meta-agent unit (200) are included in the computer unit (10), so that the input data collection unit (100) and the output data transmission unit (300) are omitted. In addition, FIG. 4 shows that the meta-agent unit (200) and the intention processing unit (220), which encompass the detailed configuration, are omitted by using dotted lines, and instead, the detailed components, the intention interpretation unit (221), the intention operation unit (222), and the result generation unit (223), are shown together.

[0132] Referring to the embodiment of FIG. 4, when text is input into the input device (11), the input text data (421) is transmitted to the intent interpretation unit (221), when voice is input, it is converted into text data (421) through the voice processing unit (210) and transmitted to the intent interpretation unit (221), and when a video or image is input, the input visual data (425) may be directly transmitted to the VQA unit (230) or may be transmitted to the VQA unit (230) together with the text data (421) or the interpreted user intent through the intent interpretation unit (221). In addition, the video or image input into the input device (11) may be directly transmitted to the VQA unit (230) as multimodal data (426) together with the text or voice. In the intention interpretation unit (221), the text data (421) can be interpreted as a user intention and transmitted to the intention work unit (222), and in the intention work unit (222), the user intention can be performed as a user interface (UI) operation for a target application (451) that matches the intention among several applications or an object detection operation in the VQA unit, and in the result generation unit (223), the result of the operation performed from the intention work unit (222) can be provided to the output device (12) in a preset format.

[0133]

[0134] Meta-agent embodiment for driving target applications

[0135] Currently, various operating systems, including Android and iOS, offer various convenient features, including artificial intelligence (AI), and they are continuously updating existing features and adding new ones. Examples include screen reader features like Android TalkBack and iOS VoiceOver, while voice command-based device control features like Android Voice Access and iOS Voice Control are available. Furthermore, representative voice assistants (VA) include Apple's Siri, Amazon's Alexa, Microsoft's Cortana, and Google Assistant or Google Gemini.

[0136] However, these are limited to some simple functions or some services, and some functions require checking the grid numbers on the screen or special gestures such as tapping, swiping, or using multiple fingers simultaneously, making them more difficult for the average person to use than a typical smartphone. In addition, it is difficult for those who are not familiar with digital functions or devices to use them, making it difficult for the elderly, who are digitally excluded, to use them.

[0137] Since most smart devices used by the elderly are based on the Android operating system, the following explanation focuses on examples using that operating system. However, the application of the present invention is not limited to Android, and it can also be applied to other operating systems, such as iOS.

[0138] Typically, in a computer system such as a smart phone, in order to integrate and run a target application (451) that includes other functions or services, it is necessary to utilize a software module that includes an API or library that provides the functions or services.

[0139] When a meta agent according to embodiments of the present invention drives various functions or services of a target application (451), it can utilize the target application (451) providing module as described above, but even if there is no target application (451) providing module as described above, a meta agent according to one embodiment of the present invention includes a function capable of driving the target application (451).

[0140] That is, the agent according to one embodiment of the present invention is not a simple VA, but a meta agent that sequentially provides the information required by the user through a conversation between the user and the agent, understands the user's intention, and automatically performs a function of performing physical operations such as touching, clicking, and typing, regardless of whether a target application (451) provision module is utilized according to the user's intention, thereby performing the entire series of tasks requested by the user.

[0141] To this end, the configurations of the meta agent described above with respect to FIGS. 1 to 4 will be described below to enable a user to easily perform an intended task for a target application (451), regardless of whether a module of a specific brand application is provided.

[0142] FIG. 5 is a drawing for explaining a concept of performing a task through a target application (451) according to one embodiment of the present invention.

[0143] First, the input / output unit (510) illustrated in FIG. 5 assumes a user equipment (User Equipment) such as a smartphone or tablet, and may include a microphone (510a), a touch screen (510b), an application (451), and a speaker (510c) as illustrated in FIG. 5. This input / output unit (510) can be viewed as a concept that includes all of the computer unit (10), input device (11), output device (12), input data collection unit (100), and output data transmission unit (300) described above with respect to FIG. 1. FIG. 5 illustrates, as an example, a case where a user inputs a voice command, “Select AAAAA,” through a microphone (510a) to select “AAAAA” in a specific brand application.

[0144] The meta agent unit (200) according to the present embodiment may include a voice processing unit (210) that converts the voice command as described above into text data via STT, and the converted text data may be transmitted to the intent interpretation unit (221) of the intent processing unit (220).

[0145] The intention interpretation unit (221) can extract information such as target UI elements and target tasks of the target application (451) based on NLP or LLM including NER and transmit the information to the intention task unit (222).

[0146] As described above, the intention task unit (222) according to the present embodiment is configured to perform a task intended by the user in the target application (451) based on the entity information extracted and classified from the intention interpretation unit (221) without utilizing a module (library, API, etc.) provided by the specific application. Therefore, Case 1 in FIG. 5 is indicated with a dotted line. If a module is provided from the target application (451), it may be more desirable to include the module within the meta agent program.

[0147] That is, the intention task unit (222) according to the present embodiment can be configured to perform a task (AAAAA selection) intended by the user in the target application (451) based on a package (accessibility service, etc.) provided by the operating system of the user device (Case 2) or based on object detection of the VQA unit (Case 3) when the module (library, API, etc.) of the target application (451) is not provided. However, the present invention does not need to exclude the case where a module is provided in the target application (451) (if agreed upon with the operator of the application), and therefore, the intention task unit (222) according to another embodiment of the present invention can be configured to perform a task (AAAAA selection) intended by the user in the target application (451) based on (1) a module (library, API, etc.) provided by the target application (451) corresponding to an object classified by the intention interpretation unit (221) as illustrated in FIG. 5, (2) an operating system provided package (accessibility service, etc.) (Case 2), or (3) based on object detection by the VQA unit (Case 3).

[0148] In addition, the intention task unit (222) according to one embodiment of the present invention, as illustrated in FIG. 5, (1) based on a module (library, API, etc.) provided by the target application (451) (Case 1) or (2) based on an operating system provided package (accessibility service, etc.) (Case 2), obtains target UI node information of the target application (451), and generates a click event for performing a task intended by the user on the target node, such that a reaction equivalent to clicking the target UI occurs on the target application (451). In addition, the intention task unit (222) according to one embodiment of the present invention, as illustrated in FIG. 5, may also obtain target image coordinate information of the target application (451) and generate a click event at the corresponding coordinates by utilizing a Computer Vision-based Object Detection API or a multimodal LLM API, etc., based on (3) object detection of the VQA unit (Case 3). The above intention interpretation unit (221) can classify objects based on NLP or LLM including NER to identify user intention, and extract information such as target UI elements and target tasks of the target application (451) and transmit the extracted information to the intention task unit (222). In addition, the VQA unit (230) can identify user intention from multimodal data (426) including text, voice, visual data, etc. based on multimodal LLM including a VQA model.

[0149] As described above, although no click actually occurred on the touch screen (510b), the target application (451) performs a response action to the click, so the section of the intention task unit (222) - target application (451) - touch screen (510b) is indicated by a dotted line.

[0150] In addition, as illustrated in FIG. 5, the result generation unit (223) can generate text such as “You selected AAAAA” or “There is no AAAAA. Please select another one.” to provide feedback to the user on the result of the click operation, convert this into voice through TTS, and transmit voice guidance to the user through the speaker (510c).

[0151] Although the embodiments of the above figures 1 to 5 are described by specific APIs, libraries, functions, utilities, etc., the implementation of the meta agent according to the present invention is not limited to these.

[0152]

[0153] Example of order execution through target application operation

[0154] As described above, the agent proposed according to embodiments of the present invention is a meta-agent capable of performing a whole series of tasks requested by a user by sequentially providing necessary information to the user through a conversation between the user and the agent, identifying the user's intention, and performing actions that achieve the same effect as physical actions such as touching, clicking, and typing on an actual system according to the user's intention.

[0155] To this end, as one embodiment, a process in which an elderly person who has difficulty ordering at a kiosk goes to a specific brand store, sits down, orders food through a conversation with a meta agent according to one embodiment of the present invention, and receives the food brought to a table designated by a store employee is described below. For convenience of explanation, the name of the meta agent proposed in this embodiment is referred to as MIA (Meta Intelligent Agent).

[0156] FIGS. 6 to 9 are drawings for explaining a scenario of ordering food of a specific brand using a meta agent according to one embodiment of the present invention.

[0157] FIG. 6 illustrates an example of a concept in which a disabled person, an elderly person, a general user, or a driver in a vehicle (630) sits at a table in a store of the brand and places an order without using a kiosk (640).

[0158] Meanwhile, as illustrated in FIG. 7a, a user (630) can activate the Mia app via the AI ​​voice recognition service on their smartphone. In the case of a specific brand app (620), which is assumed as an example of a target application (451) in this embodiment, the most recently used store is assumed to be set as the default store. Therefore, it is necessary to confirm the customer's current location or the store from which the customer wishes to place an order. At this time, Agent Mia (610) can guide the customer to the currently set store and three nearby stores, as illustrated in FIG. 7a.

[0159] After a store to order from is selected, a conversation can be conducted to specify the food (e.g., burger) desired by the user (630) as shown in FIG. 7b, and additional order details can be specified through the conversation as shown in FIG. 7c.

[0160] Figure 7d illustrates a conversation that conveys a specific order to the table where the user (630) is seated, and ultimately proceeds with payment. In the payment process, payment in the target application (451) is given priority. However, if payment in the target application (451) is difficult due to privacy issues, etc., the meta agent system may act as an agent for payment.

[0161] In this way, the meta agent system according to the present embodiment allows the user to easily place an order without using a kiosk, through a conversation with his or her smartphone, and without directly touching the screen or entering necessary information.

[0162] Meanwhile, FIG. 8 is a screen showing a voice conversation between Agent Mia (610) and a customer in text using the input device (11), input data collection unit (100), voice processing unit (210), output data transmission unit (300), output device (12), etc. of FIG. 1. The regular letters are the conversation between Agent Mia and the customer, and the large and bold letters are the result of recognizing what the customer says to Agent Mia (610) and converting it into text.

[0163] FIG. 8(a) is a screen showing the process in which the customer's input of the desired store into the input device (11) is recorded by the input data collection unit (100), FIG. 8(b) is a screen showing the process in which the voice is recognized by the voice processing unit (210) and converted into text, FIG. 8(c) is a screen showing the process in which the text data converted into "I will eat at Kyunghee University International Campus" by the intent interpretation unit (221) is identified as Kyunghee University International Campus and assigned to the store category, and FIG. 8(d) is a screen showing the intent interpretation unit (221) setting Kyunghee University International Campus as the target store and showing the agent Mia (610) guiding the customer through voice to change the store to Kyunghee University International Campus according to the customer's request.

[0164] Meanwhile, Fig. 9 is a screen showing the process of changing a store by activating a target application (620) in the intent work unit (222) according to the interpretation result of the intent interpretation unit (221). The screen is exemplified by changing a store by clicking Change Store on the order page of the target application (620), clicking the magnifying glass-shaped search icon in the upper right corner of the map page of Fig. 9(b), and clicking Select Store in the Kyunghee University International Campus row in the store search of Fig. 9(c). In addition, Fig. 9(d) is a screen showing the process of adding a menu to be ordered to the shopping cart, and shows that the store has been changed to Kyunghee University International Campus.

[0165] When the work in the intention task unit (222) is completed, the result generation unit (223) may generate text data such as “The store has been changed to Kyunghee University International Campus” if necessary and send it to the output data transmission unit (300), convert it into voice through TTS, and send it to the output device (12) to provide guidance to the customer, or the result generation unit (223) may convert the text data into voice through TTS, send it to the output device (12), and provide guidance to the customer. In the present embodiment, as shown in the screen of Fig. 8(d), the store change was provided to the customer in advance through voice guidance. In addition, the task of changing the store may be changed before starting a food and beverage-related conversation immediately after a store-related conversation between Agent Mia (610) and the customer, or may be changed immediately before putting food and beverage in the shopping cart after the store and food and beverage-related conversation is finished.

[0166] There are several ways to change a store other than the methods or order described above. One of them is to use the hierarchy of the AccessibilityNodeInfo object obtained by utilizing the accessibility service and the node information belonging to the object. Since the node information includes the class name, text, view ID, content description, and the screen border of the corresponding node, you can use this to find a node that contains the text "Kyunghee University International Campus" and has a button with the text "Change Store" in its subnode, and click the "Change Store" location directly.

[0167] FIG. 10 is a diagram for explaining a concept of utilizing accessibility services according to one embodiment of the present invention.

[0168] Specifically, FIG. 10 shows (a) the overall configuration of a screen consisting of five nodes, (b) the first node, (c) the second node, (d) the third node, (e) the fourth node, (f) the fifth node, and (g) the front part of a log file that represents the tree structure of the screen UI with indentation according to depth as an example.

[0169] The first node is a LinearLayout that takes up the entire screen and contains two TextViews as children, the second node is a TextView that displays the text "AAAAA" and contains a Button as a child, the third node is a clickable Button as a child of the TextView "AAAAA", the fourth node is a TextView that displays the text "BBBBB" and contains a Button as a child, and the fifth node is a clickable Button as a child of the TextView "BBBBB".

[0170] If the log for the first node is Node Info: Class=android.widget.LinearLayout, Text=null, ID=com.example:id / parentLayout, ContentDesc=null, Bounds=[0, 0, 1080, 1920], this log indicates that the first node is a LinearLayout with class name android.widget.LinearLayout, no text, view ID com.example:id / parentLayout, no content description, and bounds [0, 0, 1080, 1920]. LinearLayout is a parent layout that can contain other views, and in this case, it has two TextViews as child nodes.

[0171] If the log of the second node is Node Info: Class=android.widget.TextView, Text=AAAAA, ID=com.example:id / textView1, ContentDesc=AAAAA store, Bounds=[0, 400, 1080, 800], this log indicates that the second node is a TextView with a class name of android.widget.TextView, a text of AAAAA, a view ID of com.example:id / textView1, a content description of AAAAA store, and a bounds of [0, 400, 1080, 800]. The indentation in Figure 10(g) indicates that the second node has a child node, and that child node is the third node.

[0172] If the log of the third node is Node Info: Class=android.widget.Button, Text=Select STORE, ID=com.example:id / button1, ContentDesc=Click to select the store, Bounds=[540, 500, 1000, 700], then this log indicates that the third node is a Button, whose class name is android.widget.Button, whose text is Select STORE, whose view ID is com.example:id / button1, whose content description is Click to select the store, and whose bounds are [540, 500, 1000, 700].

[0173] If the log of the fourth node is Node Info: Class=android.widget.TextView, Text=BBBBB, ID=com.example:id / textView2, ContentDesc=BBBBB store, Bounds=[0, 800, 1080, 1200], this log indicates that the fourth node is TextView, whose class name is android.widget.TextView, whose text is BBBBB, whose view ID is com.example:id / textView2, whose content description is BBBBB store, and whose bounds are [0, 800, 1080, 1200]. The indentation shown in Fig. 10(g) indicates that the fourth node has a child node, and that child node is the fifth node.

[0174] If the log for the fifth node is Node Info: Class=android.widget.Button, Text=Select STORE, ID=com.example:id / button2, ContentDesc=Click to select the store, Bounds=[540, 900, 1000, 1100], then this log indicates that the fifth node is a Button, its class name is android.widget.Button, its text is Select STORE, its view ID is com.example:id / button2, its content description is Click to select the store, and its bounds are [540, 900, 1000, 1100].

[0175] As described in the example of the above-described Figure 10, the intent task unit (222) can utilize the hierarchical structure of the accessibility node information object of the accessibility service and the node information belonging to the object to find a target node and generate an event such as a click. Through this, even without a target application (451) providing module (library, API), the user can perform a desired task in the target application (451).

[0176] FIG. 11 is a diagram illustrating a meta-agent system that additionally includes a simplification processing function according to one embodiment of the present invention.

[0177] The ordering process exemplarily described with reference to FIGS. 7A to 7D has the advantage of allowing users to comfortably order by voice without worrying about the eyes of other users compared to using a kiosk in the store, but from the perspective of digitally challenged users such as the elderly, this ordering process may be too complicated and difficult.

[0178] Therefore, in one embodiment of the present invention, as illustrated in FIG. 11, it is proposed that at least one of the intention interpretation unit (221) or the intention task unit (222) additionally includes a simplification processing unit (1110a, 1110b) for simplifying information required for performing a task intended by a user in a target application (451).

[0179] For example, a user may input a command based on his or her personal work history, such as “I want to eat the burger I ate yesterday,” and this information may be stored and secured in a personalized DB (1120).

[0180] The simplified processing unit (1110a, 1110b) can use the DB (1120) record information of the user to order the burger, set menu, side menu, etc. ordered yesterday in the same manner in response to the user's simple voice, "I want to eat the burger I ate yesterday."

[0181] As another example, using the example of Fig. 9, if the customer answered “I’d like to eat at a nearby store” instead of “I’d like to eat at Kyunghee University International Campus,” there is a way to change the store to a nearby store by reading the distance that appears on the right side of the store name in the list of Fig. 9(c), and if it is recognized in advance that the list of Fig. 9(c) is sorted by distance, there is also a way to click on the change store in the first row. This simplified change of store can be processed by the simplified processing unit (1110a, 1110b) of the meta agent, but the simplified processing unit (1110a, 1110b) is conceptual and can be viewed as being performed by the intention processing unit (220) including the intention interpretation unit (221) / intention operation unit (222) or the meta agent unit (200).

[0182] While most applications use standard widgets, some use custom views or third-party libraries, which can result in unexpected class names that make user interface (UI) elements undetectable. In cases where UI elements are unidentifiable for the above reasons, the list page in Figure 9(c) is unavailable, or the display is in an unidentifiable foreign language, changing stores is only possible using the map screen in Figure 9(b).

[0183] In this case, the store change can be performed by sending a text from the intent processing unit (220) saying “Find the XX mark closest to the current location (blue circle) and find the corresponding coordinates” to the VQA unit (230), obtaining coordinate information through multimodal LLM-based object detection including VQA using an object detection API or a multimodal LLM API, and sending a touch event to the coordinates received from the VQA unit (230) to send a touch event to change the store on the opened page. In addition, the intent processing unit (220) can also be performed by sending a text from the VQA unit (230) saying “Find the XX mark closest to the current location (blue circle) and send a touch event, and find “Change store” on the opened page and send a touch event.”

[0184] One purpose of this embodiment is to provide a method for easily ordering through conversation with a language processing-based artificial intelligence meta-agent of the present invention, thereby enabling more elderly people to enjoy the benefits of digital technology without discrimination or exclusion, thereby solving social problems caused by the digital divide. Therefore, in order for the meta-agent according to one embodiment of the present invention to become more familiar to the elderly, it is proposed to provide a UI (1130) that the elderly can feel friendly with on the input / output unit (510) of the user device.

[0185] Recently, with the advent of an aging society, research has been actively conducted both domestically and internationally to improve digital accessibility for the digitally excluded elderly, using the Technology Acceptance Model (TAM) and Technology Acceptance. In 2023, Mingzhou Liu et al. expanded the Technology Acceptance Model (TAM) to create a model that includes usability needs such as perceived convenience, security / privacy, and internet self-efficacy, as well as emotional needs such as humanized interaction, perceived enjoyment, and perceived companionship. They empirically verified the model with Chinese elderly people who use voice assistants (VAs), and published the research results in “Older adults' intention to use voice assistants: Usability and emotional needs”, finding that companionship is the most important factor influencing the elderly's intention to use VAs. (Heliyon Volume 9, Issue 11, November 2023, e21932https: / doi.org / 10.1016 / j.heliyon.2023.e21932)

[0186] As one of the methods for increasing the sense of companionship, which is the most important factor influencing the intention of the elderly to use digital devices as shown above, the present invention proposes that instead of the conversation log screen of FIG. 8, the user selects the person (buddy) most familiar to the user, and an AI buddy (1130) synthesized with the buddy's voice and face converses with the user as a meta agent, thereby increasing the sense of companionship through a friendly face and voice.

[0187] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0188] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

[0189] Since the agent of the present invention is a meta agent that can sequentially provide the user with the information necessary for the user through a conversation between the user and the agent, understand the user's intention, and perform an action that obtains the same effect as a physical action such as touching, clicking, or typing on an actual system according to the user's intention, and thereby perform the entire series of tasks requested by the user, in addition to the store ordering described in the above embodiment, it can perform various types of tasks such as app installation, app operation, home automation such as home appliance control, and robot or smartwatch control.

[0190] Accordingly, the present invention can be utilized in most voice user interface (VUI) markets, including retail and e-commerce markets, home appliance markets, automotive and transportation markets, and healthcare markets.

Claims

1. In a language processing-based meta-agent system using computer input / output, An intention interpretation unit that identifies a specific type of object from input data collected from an input device of the computer and classifies the object by assigning the object to a predefined category; An intention task unit that activates an application installed on the computer and performs a task intended by the user in the activated application according to the object classified by the intention interpretation unit; and A meta-agent system including a result generation unit that provides the results of work performed from the above-mentioned intention work unit to an output device of the computer in a preset format.

2. In paragraph 1, It additionally includes a voice processing unit that converts input data collected from an input device of the computer into text data, A meta agent system, wherein the intention interpretation unit is configured to obtain the text data from the voice processing unit.

3. In paragraph 1, A meta-agent system further comprising a visual question answering (VQA) unit configured to detect objects from visual data among the above input data.

4. In paragraph 1, An input data collection unit that collects input data from an input device of the computer; and A meta-agent system further comprising an output data transmission unit that transmits the output data to an output device of the computer.

5. In paragraph 1, The above meta-agent system is a meta-agent system installed and operating on the computer.

6. In paragraph 1, The above intention interpretation section, A meta-agent system characterized in that it classifies entities based on natural language processing (NLP) including named entity recognition (NER) or a large language model (LLM) from the above input data.

7. In paragraph 3, The above VQA department, A meta-agent system characterized in that it detects an object based on a multimodal large language model (multimodal LLM) including a visual question answering (VQA) model from multimodal data among the above input data.

8. In paragraph 1, The above intention work unit is, A meta-agent system configured to perform the task intended by the user in the application, regardless of whether a module including an application library or an API (Application Programming Interface) corresponding to the object classified from the intent interpretation unit is provided.

9. In paragraph 8, The above intention work unit is, A meta-agent system configured to perform the task intended by the user in the application, (2) based on a package including an accessibility service provided by the operating system of the computer, or (3) based on object detection using a multimodal large language model (multimodal LLM) including a visual question answering (VQA) model, if the module of the application is not provided.

10. In paragraph 1, The above intention work unit is, A meta-agent system configured to perform the task intended by the user in the application based on (1) a module including an application library or API (Application Programming Interface) corresponding to the object classified from the intent interpretation unit, (2) a package including an accessibility service provided by the operating system of the computer, or (3) object detection using a multimodal large language model (Multimodal LLM) including a visual question answering (VQA) model.

11. In paragraph 10, The above intention work unit is, (1) When the application performs the task intended by the user based on the module of the application, or (2) based on the package provided by the operating system of the computer, A meta agent system that operates by obtaining target UI (User Interface) element node information of the above application.

12. In paragraph 10, The above intention work unit is, (3) When the application performs the task intended by the user based on object detection using a multimodal LLM including the above VQA model, A meta-agent system that operates by obtaining target object detection information of the above application.

13. In paragraph 1, A meta-agent system, wherein at least one of the intention interpretation unit and the intention task unit additionally includes a simplification processing unit for simplifying information required for performing the task intended by the user in the application.

14. In paragraph 1, A meta-agent system characterized in that, if additional input is collected through the input device while the meta-agent system is performing a computer operation on behalf of a user, the meta-agent system generates the work result by reflecting the additional input.

Images