A multi-agent interaction method and system based on smart glasses

Abstract

The present invention relates to a multi-agent interaction method and system based on smart glasses. The method comprises: S1, the root agent obtains the first multimodal data of the user's first-person perspective through the smart glasses; S2, the first multimodal data is parsed to obtain the demand data; S3, the corresponding first auxiliary agent is selected and awakened according to the demand data, and the demand data is sent to the corresponding first auxiliary agent; S4, the result data sent by the first auxiliary agent is received. The present invention utilizes the hardware advantages of smart glasses, and through the characteristics of smart glasses that can obtain the multimodal data of the user's first-person perspective in real time, multiple agents can interact with each other through the multimodal data of the user's first-person perspective obtained by the smart glasses, thereby improving the interaction efficiency between multiple agents and having a higher degree of intelligence.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-agent interaction method and system based on smart glasses. Background Art

[0002] Agent multimodal interaction is the ability of agents to interact with users by integrating multiple perception methods, such as integrating vision, hearing, touch and other modalities to achieve interaction between agents and users. Compared with traditional single interaction methods such as keyboard and mouse, multimodal interaction is closer to the natural interaction method of humans. Users can interact with agents through voice, gestures, expressions and other methods, so that agents can more accurately understand users' intentions and needs, and provide more efficient feedback and services.

[0003] Intelligent agents can achieve multimodal human-computer interaction in many fields. In the field of smart homes, intelligent agents can better understand and respond to user needs by integrating technologies such as voice recognition, facial recognition, and gesture recognition. For example, users can control the lights, temperature, and sound system in their homes through voice commands. In the field of healthcare, intelligent agents can assist doctors in diagnosing diseases and formulating treatment methods by combining technologies such as impact analysis, physiological signal monitoring, and voice recognition. In addition, intelligent agents can also be applied to entertainment, education, and other fields.

[0004] In the above scenarios, when intelligent agents are used for human-computer interaction, each intelligent agent can only directly interact with the other through the multimodal information perceived by each agent. It is impossible to achieve multimodal interaction between users and multiple intelligent agents at the same time, resulting in low interaction efficiency and low intelligence. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a multi-agent interaction method and system based on smart glasses, which utilizes smart glasses to obtain the user's first-person perspective information. During the multi-agent interaction, multimodal interaction between the user and multiple agents can be achieved through the first-person perspective information obtained in real time, and the degree of intelligence is better.

[0006] In one embodiment, the present invention provides a multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, and the control system integrates a root agent and multiple auxiliary agents; the multi-agent interaction method includes:

[0007] S1, the root agent obtains first multimodal data of a user's first-person perspective through smart glasses;

[0008] S2, parsing the first multimodal data to obtain demand data;

[0009] S3, selecting and waking up a corresponding first auxiliary agent according to the demand data, and sending the demand data to the corresponding first auxiliary agent, so that the first auxiliary agent obtains second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, and analyzes and processes the second multimodal data to obtain first result data; and the first auxiliary agent obtains third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain second result data; and the first auxiliary agent obtains fourth multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the fourth multimodal data to obtain third result data;

[0010] S4, receiving the second multimodal data and the first result data, the third multimodal data and the second result data and / or the fourth multimodal data and the third result data sent by the first auxiliary agent.

[0011] Furthermore, before step S1, the following steps are also included:

[0012] S0: The root agent obtains the start interaction instruction through the smart glasses.

[0013] Furthermore, the method further comprises:

[0014] S5, sending the second multimodal data and the first result data, the third multimodal data and the second result data, and / or the fourth multimodal data and the third result data to a corresponding third auxiliary agent, so that the third auxiliary agent saves the recorded data obtained after analyzing each result data and the corresponding multimodal data;

[0015] S6, the root agent receives the feedback result returned by the third auxiliary agent, and outputs the interaction result according to the feedback result.

[0016] In one embodiment, the present invention provides a multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, and the control system integrates a root agent and multiple auxiliary agents; the multi-agent interaction method includes:

[0017] S1, receiving demand data sent by a root agent; the demand data is obtained by the root agent parsing first multimodal data of a user's first-person perspective obtained through smart glasses;

[0018] S2, obtaining second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, and analyzing and processing the second multimodal data to obtain first result data;

[0019] S3A, output the first result data, and send the first result data to the root agent.

[0020] Further, the auxiliary agent is provided with a memory bank;

[0021] Step S2 specifically includes: analyzing and processing the second multimodal data, and acquiring first result data corresponding to the second multimodal data from the memory bank.

[0022] Furthermore, step S2 specifically includes:

[0023] S21, analyzing and processing the second multimodal data to obtain first intermediate data;

[0024] S22, selecting and waking up a corresponding second auxiliary agent according to the first intermediate data, and sending the second multimodal data and the first intermediate data to the corresponding second auxiliary agent, so that the second auxiliary agent analyzes and processes the first intermediate data to obtain the first result data;

[0025] S23, receiving the first result data sent by the second auxiliary agent.

[0026] Furthermore, after step S2, the method further includes:

[0027] S3B1, obtaining third multimodal data of the user's first-person perspective through the smart glasses, and analyzing and processing the third multimodal data in combination with the first result data to obtain second result data;

[0028] S3B2, output the second result data, and send the second result data to the root agent.

[0029] Furthermore, it also includes:

[0030] S4, obtaining fourth multimodal data of the user's first-person perspective through the smart glasses, and analyzing and processing the fourth multimodal data to obtain third result data;

[0031] S5, output the third result data, and send the third result data to the root agent.

[0032] In one embodiment, the present invention provides a multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, and the control system integrates a root agent and multiple auxiliary agents; the multi-agent interaction method includes:

[0033] S1, the root agent obtains first multimodal data of a user's first-person perspective through smart glasses;

[0034] S2, parsing the first multimodal data to obtain demand data;

[0035] S3, selecting and waking up a corresponding first auxiliary agent according to the demand data, and sending the demand data to the corresponding first auxiliary agent;

[0036] S4, the first auxiliary agent receives the corresponding demand data, and obtains second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, analyzes and processes the second multimodal data to obtain first result data, and outputs the first result data;

[0037] S5A, sending the first result data to the root agent;

[0038] S5B1, the first auxiliary intelligent agent obtains third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain second result data;

[0039] S5B2, sending the second result data to the root agent;

[0040] S6, the first auxiliary agent obtains fourth multimodal data of the user's first-person perspective through the smart glasses, analyzes and processes the fourth multimodal data to obtain third result data, and sends the third result data to the root agent;

[0041] S7, the root agent sends the second multimodal data and the first result data, the third multimodal data and the second result data, and / or the fourth multimodal data and the third result data to the corresponding third auxiliary agent, so that the third auxiliary agent saves the record data obtained after analyzing each result data and the corresponding multimodal data;

[0042] S8, the root agent receives the feedback result returned by the third auxiliary agent, and outputs the interaction result according to the feedback result.

[0043] In one embodiment, the present invention provides a multi-agent interactive system based on smart glasses, comprising:

[0044] Smart glasses and a control system; the control system integrates a root agent and multiple auxiliary agents;

[0045] The smart glasses collect multimodal data from the user's first-person perspective;

[0046] The root agent executes the aforementioned multi-agent interaction method based on smart glasses according to the multimodal data obtained by the smart glasses; and the auxiliary agent executes the aforementioned multi-agent interaction method based on smart glasses according to the multimodal data obtained by the smart glasses; the control system executes the aforementioned multi-agent interaction method based on smart glasses according to the multimodal data obtained by the root agent and the auxiliary agent.

[0047] The multi-agent interaction method based on smart glasses provided by the present invention utilizes the hardware advantages of smart glasses. Through the characteristics of smart glasses that can obtain multimodal data of the user's first-person perspective in real time, multiple agents can interact with each other through the multimodal data of the user's first-person perspective obtained by the smart glasses, thereby improving the interaction efficiency between multiple agents and achieving a higher level of intelligence.

[0048] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a module of a multi-agent interactive system based on smart glasses of the present invention;

[0050] Figure 2 It is a schematic diagram of the overall process of a multi-agent interaction method based on smart glasses of the present invention;

[0051] Figure 3 A schematic diagram of a multi-agent interaction method based on smart glasses according to the present invention;

[0052] Figure 4 The figure is a flow chart of a multi-agent interaction method based on smart glasses of the present invention. DETAILED DESCRIPTION

[0053] It should be clear that the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present invention.

[0054] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0055] In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0056] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

[0057] Smart glasses are a kind of wearable device that combines the visual assistance function of traditional glasses with the interaction and computing capabilities of modern smart devices. The present invention sets data acquisition modules such as cameras, microphones, and inertial sensors on smart glasses. While the user wears the smart glasses, the smart glasses can collect multimodal data such as vision, hearing, and movement from the user's first-person perspective through the data collection module, and realize the interaction between multiple intelligent agents through the multimodal data from the first-person perspective collected in real time by the data acquisition module during the multi-agent interaction process.

[0058] See also Figure 1 , Figure 1 It is a module schematic diagram of the multi-agent interactive system based on smart glasses of the present invention. The intelligent multimodal interactive system of the present invention includes: smart glasses 1 and a control system 2. The smart glasses 1 are provided with data acquisition modules such as a camera, a microphone, an inertial sensor, and a position sensor. While the user is wearing the smart glasses 1, the smart glasses 1 can collect multimodal data such as vision, hearing, and movement from the user's first-person perspective through the data acquisition modules carried thereon. An intelligent agent library is integrated in the control system 2, and the intelligent agent library includes a root intelligent agent and a plurality of auxiliary intelligent agents. The control system 2 can execute the multi-agent interactive method based on smart glasses described in an embodiment of the present invention according to the multimodal data from the first-person perspective collected by the smart glasses 1 when the root intelligent agent and the plurality of auxiliary intelligent agents interact, thereby improving the interactive effect.

[0059] The multimodal data collected by the smart glasses 1 of the present invention has the characteristics of "all-weather" and "first-person perspective". These two characteristics are relative to traditional smart terminals such as mobile phones and computers. Although traditional smart terminals such as mobile phones and computers can also be equipped with data acquisition modules such as cameras and microphones to collect multimodal data of users, due to the user's usage habits (users will not always install their mobile phones on their bodies, nor will they use their mobile phones during work, housework and exercise), traditional smart terminals such as mobile phones, computers and other terminals integrated with software intelligent bodies or other hardware intelligent bodies cannot obtain the user's first-person perspective information in real time, so they cannot obtain the user's needs in time, resulting in insufficient intelligence. A considerable number of users have the habit of wearing glasses all-weather, so smart glasses can easily realize the collection of users' first-person perspective data all-weather, and multiple intelligent bodies can interact using first-person perspective data, making their interaction more efficient.

[0060] See also Figure 2 In one embodiment, it is a flow chart of a multi-agent interaction method based on smart glasses of the present invention. The multi-agent interaction method based on smart glasses is implemented by a multi-agent interaction system, the multi-agent interaction system includes smart glasses and a control system corresponding to the smart glasses, the control system integrates an agent library, the agent library includes a root agent and multiple auxiliary agents; the multi-agent interaction method includes the following steps:

[0061] S1, the root agent obtains first multimodal data of a user's first-person perspective through smart glasses;

[0062] Specifically, the control system is integrated with an agent library, which includes a root agent and a plurality of corresponding auxiliary agents. The root agent is used to globally control the overall interaction process, and is used to control the start or end of this round of interaction; the auxiliary agent is used to receive the task data of the root agent and interact with the root agent and other auxiliary agents. Therefore, in this embodiment, step S0 is also included before step S1: the root agent obtains the start interaction instruction through the smart glasses. At this time, step S1 becomes, when the start interaction instruction is obtained, the first multimodal data of the user's first-person perspective is obtained through the smart glasses. It can be understood that when the root agent obtains the start interaction instruction, it further obtains the first multimodal data of the user's first-person perspective required in the interaction process. It can be understood that the start interaction instruction can be an audio instruction, a gesture instruction, etc., which is not limited here. For example, the audio instruction "Hi, start interaction"; the interaction is started when a preset gesture action is detected, etc.

[0063] Further, the multimodal data of the user's first-person perspective includes at least one of video data, image data, audio data, motion data and / or environmental data. When the start interaction instruction is obtained, the video data, image data, text data or environmental data of the first-person perspective are collected through the camera of the smart glasses, and the audio data of the first-person perspective can also be collected through the microphone of the smart glasses, and various types of motion data of the first-person perspective can also be collected through the motion sensor of the smart glasses. For example, in this embodiment, in the learning assistance scenario, when the user completes the homework while wearing the smart glasses, the user needs the smart glasses to assist in completing the homework together. At this time, when the microphone of the smart glasses collects the user's audio data "help me complete the homework together", the multimodal data of the first-person perspective is collected through the camera of the smart glasses; or when it is monitored that the user wearing the glasses is completing the homework, the user actively asks "Do you need assistance to complete the homework", and collects the audio data of the user's reply "need to help complete the homework together" through the microphone, or collects the video, image or text data of the user in completing the homework through the camera, so as to obtain the first multimodal data of the user's first-person perspective. In other embodiments, the first multimodal data of the user's first-person perspective may also be obtained through smart glasses in other scenarios, such as navigation scenarios, sports scenarios, blind guide scenarios, medical care scenarios, etc. The present invention does not limit the application scenarios, and this is only an example.

[0064] S2, parsing the first multimodal data to obtain demand data;

[0065] When the root agent obtains the first multimodal data of the user's first-person perspective through the smart glasses, the root agent parses and identifies the first multimodal data to obtain the required data. For example, when the root agent obtains the user's audio data "need assistance to complete the homework together", the audio data is parsed and identified to identify the auxiliary information support for problem-solving that the user needs to complete the homework. At this time, the required data is "need auxiliary information for problem-solving". Among them, for different multimodal data, different recognition algorithms are used to parse the multimodal data. For example, the existing audio data recognition algorithm or image data recognition algorithm can be used to parse the audio data, video data and image data respectively to obtain the required data. The use of specific recognition algorithms is not limited.

[0066] S3, selecting and waking up a corresponding first auxiliary agent according to the demand data, and sending the demand data to the corresponding first auxiliary agent;

[0067] Specifically, after obtaining the demand data, the root agent determines the service to be provided according to the demand data analysis and judgment, and selects and wakes up the first auxiliary agent that provides the corresponding service. For example, when the root agent obtains the demand data "needs auxiliary information for problem solving", it is determined that the auxiliary agent that provides the corresponding service is the picture-solving agent. At this time, the picture-solving agent is selected as the first auxiliary agent, and the picture-solving agent is woken up, and the demand data "needs auxiliary information for problem solving" is sent to the picture-solving agent.

[0068] S4, the first auxiliary agent receives the corresponding demand data, and obtains second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, analyzes and processes the second multimodal data to obtain first result data, and outputs the first result data;

[0069] Specifically, after receiving the demand data, the first auxiliary agent further obtains the second multimodal data from the user's first-person perspective through the smart glasses, and analyzes and processes the second multimodal data to obtain the first result data. It can be understood that the second multimodal data can also be video data, image data, audio data, action data and / or environmental data, etc. For example, in a learning assistance scenario, when the picture-solving agent receives the demand data "need for auxiliary information for problem solving", the camera of the smart glasses can be activated to capture image data of the current homework situation; the microphone of the smart glasses can also be activated to receive audio data of the user reading the current homework situation. In an embodiment of the present invention, the first multimodal data and the second multimodal data are mainly image data and audio data. In other scenarios, other types of second multimodal data from the user's first-person perspective can also be obtained.

[0070] Further, after obtaining the second multimodal data of the user's first-person perspective, the second multimodal data is analyzed and processed to obtain the first result data. In one embodiment, the first auxiliary agent is provided with a memory bank; the step S4 is specifically: analyzing and processing the second multimodal data, and obtaining the first result data corresponding to the second multimodal data from the memory bank. For example, when the camera of the smart glasses is started to capture the image data of the current homework situation, the real-time visual information of the user's current homework is obtained, and the picture-based problem-solving agent analyzes the image data of the current homework situation captured to confirm whether there is an error. It can be understood that the picture-based problem-solving agent is provided with a corresponding memory bank, and the memory bank stores the problem-solving idea data corresponding to the demand data. For example, when the image data of the current homework situation captured is analyzed and confirmed to have an error, the picture-based problem-solving agent analyzes the error part and obtains the corresponding first result data in the memory bank of the agent system. For the learning assistance scenario, the first result data is the problem-solving idea data, and the problem-solving idea data is output through the microphone of the smart glasses to assist the user in completing the homework.

[0071] Furthermore, in one embodiment, step S4 specifically includes:

[0072] S41, analyzing and processing the second multimodal data to obtain first intermediate data;

[0073] In this embodiment, when the first result data corresponding to the second multimodal data is not stored in the memory of the first auxiliary agent, the first auxiliary agent needs other auxiliary agents to cooperate to obtain the first result data. For example, in a learning assistance scenario, when the picture-solving agent receives the demand data "needs auxiliary information for problem-solving", it starts the camera of the smart glasses to capture the image data of the current homework situation, and then analyzes and processes the image data of the current homework situation to obtain the first intermediate data of "no relevant problem-solving ideas". At this time, it is necessary to query and search to obtain the corresponding problem-solving ideas data. It can be understood that when the second multimodal data includes text data, the corresponding first intermediate data can be obtained by performing semantic analysis on the second multimodal data. Specifically, it can be implemented using existing semantic recognition algorithms. The present invention is not limited.

[0074] S42, select and wake up the corresponding second auxiliary agent according to the first intermediate data, and send the second multimodal data and the first intermediate data to the corresponding second auxiliary agent, so that the second auxiliary agent analyzes and processes the first intermediate data to obtain the first result data, and sends the first result data to the first auxiliary agent.

[0075] Specifically, after obtaining the first intermediate data, the corresponding second auxiliary agent is selected and awakened according to the first intermediate data, and the second auxiliary agent is sent to the search agent. For example, when the first auxiliary agent obtains the first intermediate data of "no relevant solution ideas are obtained", the search agent is selected and awakened, and the second multimodal data "image data of the current operation situation" and the first intermediate data "no relevant solution ideas are obtained" are sent to the search agent, so that the search agent queries and searches to obtain the first result data corresponding to the second multimodal data.

[0076] S5A, sending the first result data to the root agent.

[0077] Furthermore, in one embodiment, the multi-agent interaction method based on smart glasses, after step S4, further comprises:

[0078] S5B1, the first auxiliary intelligent agent obtains third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain second result data;

[0079] Specifically, after the first auxiliary agent outputs the first result data, the user may need to give further requirements in combination with the first result data. At this time, the first auxiliary agent further obtains the third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain the second result data. It can be understood that when the third multimodal data includes text data, the corresponding second result data can also be obtained by semantically analyzing the second multimodal data. Specifically, it can be implemented using existing semantic recognition algorithms. For example, in a learning assistance scenario, when the picture-solving agent outputs problem-solving idea data through the microphone of the smart glasses to assist the user in completing the homework, the user may need to confirm whether the homework completed based on the above-mentioned problem-solving idea data is correct or whether there are other problem-solving ideas. At this time, the first auxiliary intelligent agent uses the camera of the smart glasses to capture the image data of the user's completed problem-solving, or simultaneously receives the audio data of "need to confirm the problem-solving result and provide other problem-solving ideas" collected by the microphone, and analyzes the image data of the problem-solving completion in combination with the problem-solving idea data to obtain the second result data; at this time, it can be understood that the second result data includes information on whether the problem-solving is correct and other problem-solving idea data different from the first result data. In this embodiment, the first auxiliary intelligent agent can obtain the second result data through the memory bank, or can obtain the second result data through collaborative processing by other auxiliary intelligent agents.

[0080] S5B2, sending the second result data to the root agent.

[0081] Furthermore, in one embodiment, the multi-agent interaction method based on smart glasses, after steps S5A and S5B2, further includes:

[0082] S6, the first auxiliary intelligent agent obtains the fourth multimodal data of the user's first-person perspective through the smart glasses, analyzes and processes the fourth multimodal data to obtain third result data, and sends the third result data to the root intelligent agent.

[0083] Specifically, after the first auxiliary agent outputs the first result data, the user may no longer have any further needs. At this time, the first auxiliary agent further obtains the fourth multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the fourth multimodal data to obtain the third result data. It can be understood that when the fourth multimodal data includes text data, the corresponding third result data can be obtained by semantically analyzing the second multimodal data. Specifically, it can be implemented by using the existing semantic recognition algorithm. For example, in the learning assistance scenario, when the picture-solving agent outputs the problem-solving idea data through the microphone of the smart glasses to assist the user in completing the homework, the user may no longer have other needs and needs to end this interaction. At this time, the first auxiliary agent shoots the problem-solving completion image data of the user's homework through the camera of the smart glasses, and does not receive the user's audio completion instruction within the preset time; or when the microphone collects the audio data of "I have completed my homework", it analyzes that it has completed the assistance task for the user's current needs and is ready to end the current interaction, and the third result data is obtained at this time. It can be understood that the third result data includes interaction completion data.

[0084] S7, the root agent sends the second multimodal data and the first result data, the third multimodal data and the second result data, and / or the fourth multimodal data and the third result data to the corresponding third auxiliary agent, so that the third auxiliary agent saves the record data obtained after analyzing each result data and the corresponding multimodal data;

[0085] It is understandable that when the root agent receives the third result data to confirm that the interaction is complete, it selects and wakes up the corresponding third auxiliary agent. Among them, the third auxiliary agent can be a recorder agent. At this time, the root agent sends the multimodal data in the interaction process and the corresponding result data to the recorder agent, and the recorder agent organizes and analyzes the multimodal data in the interaction process and the corresponding result data, generates and saves the organized record data. It is understandable that the recorder agent can upload and save the record data to the cloud by calling the cloud interface; it can also save the record data in the local memory of the smart glasses. This embodiment does not limit the storage method of the record data.

[0086] S8, the root agent receives the feedback result returned by the third auxiliary agent, and outputs the interaction result according to the feedback result.

[0087] Specifically, after the third auxiliary agent saves the generated recorded data, it feeds back a storage completion notification to the root agent. When the root agent receives the storage completion notification sent by the third auxiliary agent, it outputs the interaction result to the user through the microphone of the smart glasses. Specifically, the interaction result can be audio data "The task has been completed and recorded, welcome to use it next time".

[0088] See also Figure 3 , Figure 4 , Figure 3 A schematic flow chart of a multi-agent interaction method based on smart glasses executed by a root agent in an embodiment of the present invention. Figure 4 The figure is a flow chart of a multi-agent interaction method based on smart glasses executed by the first auxiliary agent in the embodiment of the present invention. The method flow is discussed in detail in the interaction method executed by the control system above, and will not be repeated here.

[0089] The multi-agent interaction method based on smart glasses provided in the above embodiment utilizes the hardware advantages of smart glasses. Through the characteristics of smart glasses that can obtain multimodal data from the user's first-person perspective in real time, multiple agents can interact with each other through the multimodal data from the user's first-person perspective obtained by the smart glasses, thereby improving the interaction efficiency between multiple agents and achieving a higher level of intelligence.

[0090] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, characterized in that: The control system is integrated with a root agent and a plurality of auxiliary agents; The multi-agent interaction method comprises: S1, the root agent obtains first multimodal data of a user's first-person perspective through smart glasses; S2, parsing the first multimodal data to obtain demand data; S3, select and wake up the corresponding first auxiliary intelligent agent according to the demand data, and send the demand data to the corresponding first auxiliary intelligent agent, so that the first auxiliary intelligent agent obtains the second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, and analyzes and processes the second multimodal data to obtain first intermediate data; and select and wake up the corresponding second auxiliary intelligent agent according to the first intermediate data, and send the second multimodal data and the first intermediate data to the corresponding second auxiliary intelligent agent, so that the second auxiliary intelligent agent analyzes and processes the first intermediate data to obtain first result data; and the first auxiliary intelligent agent obtains the third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain second result data; and the first auxiliary intelligent agent obtains the fourth multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the fourth multimodal data to obtain third result data; S4, receiving the second multimodal data and the first result data, the third multimodal data and the second result data and / or the fourth multimodal data and the third result data sent by the first auxiliary agent.

2. The multi-agent interaction method according to claim 1, characterized in that: Before step S1, the method further includes: S0: The root agent obtains the start interaction instruction through the smart glasses.

3. The multi-agent interaction method according to claim 2, characterized in that: The method further comprises: S5, sending the second multimodal data and the first result data, the third multimodal data and the second result data, and / or the fourth multimodal data and the third result data to a corresponding third auxiliary agent, so that the third auxiliary agent saves the recorded data obtained after analyzing each result data and the corresponding multimodal data; S6, receiving the feedback result returned by the third auxiliary agent, and outputting the interaction result according to the feedback result.

4. A multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, characterized in that: The control system is integrated with a root agent and a plurality of auxiliary agents; The multi-agent interaction method comprises: S1, receiving demand data sent by a root agent; the demand data is obtained by the root agent parsing first multimodal data of a user's first-person perspective obtained through smart glasses; S2, obtaining second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, and analyzing and processing the second multimodal data to obtain first result data; S3A, outputting the first result data, and sending the first result data to the root agent; Step S2 specifically includes: S21, analyzing and processing the second multimodal data to obtain first intermediate data; S22, selecting and waking up a corresponding second auxiliary agent according to the first intermediate data, and sending the second multimodal data and the first intermediate data to the corresponding second auxiliary agent, so that the second auxiliary agent analyzes and processes the first intermediate data to obtain the first result data; S23, receiving the first result data sent by the second auxiliary agent.

5. The multi-agent interaction method according to claim 4, characterized in that: The auxiliary agent is provided with a memory bank; Step S2 specifically includes: analyzing and processing the second multimodal data, and acquiring first result data corresponding to the second multimodal data from the memory bank.

6. The multi-agent interaction method according to claim 4, characterized in that: After step S2, the method further includes: S3B1, obtaining third multimodal data of the user's first-person perspective through the smart glasses, and analyzing and processing the third multimodal data in combination with the first result data to obtain second result data; S3B2, output the second result data, and send the second result data to the root agent.

7. The multi-agent interaction method according to claim 4, characterized in that: Also includes: S4, obtaining fourth multimodal data of the user's first-person perspective through the smart glasses, and analyzing and processing the fourth multimodal data to obtain third result data; S5, output the third result data, and send the third result data to the root agent.

8. A multi-agent interaction method based on smart glasses, wherein the smart glasses have a corresponding control system, characterized in that: The control system is integrated with a root agent and a plurality of auxiliary agents; The multi-agent interaction method comprises: S1, the root agent obtains first multimodal data of a user's first-person perspective through smart glasses; S2, parsing the first multimodal data to obtain demand data; S3, selecting and waking up a corresponding first auxiliary agent according to the demand data, and sending the demand data to the corresponding first auxiliary agent; S4, the first auxiliary agent receives the corresponding demand data, and obtains second multimodal data of the user's first-person perspective through the smart glasses according to the demand data, and analyzes and processes the second multimodal data to obtain first intermediate data; and selects and wakes up the corresponding second auxiliary agent according to the first intermediate data, and sends the second multimodal data and the first intermediate data to the corresponding second auxiliary agent, so that the second auxiliary agent analyzes and processes the first intermediate data to obtain first result data, and outputs the first result data; S5A, sending the first result data to the root agent; S5B1, the first auxiliary intelligent agent obtains third multimodal data of the user's first-person perspective through the smart glasses, and analyzes and processes the third multimodal data in combination with the first result data to obtain second result data; S5B2, sending the second result data to the root agent; S6, the first auxiliary agent obtains fourth multimodal data of the user's first-person perspective through the smart glasses, analyzes and processes the fourth multimodal data to obtain third result data, and sends the third result data to the root agent; S7, the root agent sends the second multimodal data and the first result data, the third multimodal data and the second result data, and / or the fourth multimodal data and the third result data to the corresponding third auxiliary agent, so that the third auxiliary agent saves the record data obtained after analyzing each result data and the corresponding multimodal data; S8, the root agent receives the feedback result returned by the third auxiliary agent, and outputs the interaction result according to the feedback result.

9. A multi-agent interactive system based on smart glasses, characterized in that: include: Smart glasses and a control system; the control system integrates a root agent and multiple auxiliary agents; The smart glasses collect multimodal data from the user's first-person perspective; The root agent executes the multi-agent interaction method based on smart glasses as described in any one of claims 1-3 according to the multimodal data obtained by the smart glasses; and the auxiliary agent executes the multi-agent interaction method based on smart glasses as described in any one of claims 4-7 according to the multimodal data obtained by the smart glasses; the control system executes the multi-agent interaction method based on smart glasses as described in claim 8 according to the multimodal data obtained by the root agent and the auxiliary agent.

Images