A multi-agent interactive control method, device and system based on smart glasses

Abstract

The present invention relates to a multi-agent interactive control method, device and system based on smart glasses. The present invention is based on the smart glasses integrated with multiple agents; the multi-agent interactive control method includes: S1, obtaining first multimodal data of the user's first-person perspective collected by the first agent through the smart glasses; S2, analyzing and processing the first multimodal data to obtain first result data; S3, controlling the interaction of the first agent according to the first result data. The present invention utilizes the hardware advantages of smart glasses, combined with the characteristics of smart glasses that can obtain multimodal data from the user's first-person perspective, actively controls the interaction of multiple agents through the multimodal data from the user's first-person perspective obtained by the smart glasses, and realizes active assistance to the user. It has a higher degree of intelligence and is more flexible and intelligent than traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-agent interactive control method, device and system based on smart glasses. Background Art

[0002] Agent multimodal interaction is the ability of agents to interact with users by integrating multiple perception methods, such as integrating vision, hearing, touch and other modalities to achieve interaction between agents and users. Compared with traditional single interaction methods such as keyboard and mouse, multimodal interaction is closer to the natural interaction method of humans. Users can interact with agents through voice, gestures, expressions and other methods, so that agents can more accurately understand users' intentions and needs, and provide more efficient feedback and services.

[0003] Intelligent agents can achieve multimodal human-computer interaction in many fields. In the field of smart homes, intelligent agents can better understand and respond to user needs by integrating technologies such as voice recognition, facial recognition, and gesture recognition. For example, users can control the lights, temperature, and sound system in their homes through voice commands. In the field of healthcare, intelligent agents can assist doctors in diagnosing diseases and formulating treatment methods by combining technologies such as impact analysis, physiological signal monitoring, and voice recognition. In addition, intelligent agents can also be applied to entertainment, education, and other fields.

[0004] In traditional smart devices, when receiving user instructions, a single agent in the smart device is usually triggered to respond to the user's instructions to complete the operation. For example, an online question-and-answer assistant can be installed on the smart terminal to conduct online questions and answers with the user when receiving the user's inquiry, thereby improving the interactive effect. However, in the prior art, the agent of the traditional smart device is only passively triggered to respond when receiving the user's instructions, and cannot actively assist the user, resulting in its low level of intelligence. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a multi-agent interactive control method, device and system based on smart glasses, which utilizes the hardware advantages of smart glasses and combines the characteristics of smart glasses that can obtain multimodal data from the user's first-person perspective. The multimodal data from the user's first-person perspective obtained by the smart glasses is used to actively control the multi-agents, thereby achieving active assistance to the user.

[0006] In one embodiment, the present invention provides a multi-agent interactive control method based on smart glasses, wherein the smart glasses are integrated with multiple agents; the method comprises:

[0007] S1, obtaining first multimodal data of a user's first-person perspective collected by a first agent through smart glasses;

[0008] S2, analyzing and processing the first multimodal data to obtain first result data;

[0009] S3: Control the interaction of the first agent according to the first result data.

[0010] Furthermore, the method further comprises:

[0011] S0, polling the status of each intelligent agent, and determining the first intelligent agent according to the status of the intelligent agent.

[0012] Further, step S3 includes:

[0013] S31, determining whether the first result data is preset data, and when it is determined that the first result data is preset data, returning to step S0, polling the status of each agent, and determining the first agent according to the status of the agent; when it is determined that the first result data is not preset data, entering step S32A;

[0014] S32A, output voice data information.

[0015] Further, in step S31, when it is determined that the first result data is not the preset data, the process proceeds to step S32B;

[0016] S32B, controlling the first intelligent agent to re-collect the second multimodal data of the user's first-person perspective through the smart glasses, and controlling the interaction of the first intelligent agent according to the second multimodal data.

[0017] Further, step S32B includes:

[0018] S32B1, obtaining second multimodal data of a user's first-person perspective collected by the first agent through smart glasses;

[0019] S32B2, analyzing and processing the second multimodal data to obtain second result data;

[0020] S32B3, controlling the interaction of the first intelligent agent according to the second result data.

[0021] Further, step S32B3 includes:

[0022] S32B31, selecting and waking up a corresponding second agent according to the second result data, and sending the second multimodal data and the second result data to the second agent, so that the second agent analyzes and processes the second multimodal data and the second result data to obtain feedback data information;

[0023] S32B32, output feedback data information.

[0024] Furthermore, the smart glasses are provided with an auxiliary working mode and a general working mode;

[0025] Step S0 is: when it is detected that the smart glasses enter the auxiliary working mode, the status of each intelligent body is polled at a preset interval, and the first intelligent body is determined according to the status of each intelligent body.

[0026] In one embodiment, the present invention provides a multi-agent interactive control device based on smart glasses, comprising:

[0027] A data acquisition module, which is used to acquire first multimodal data of a user's first-person perspective collected by a first agent through smart glasses;

[0028] A content analysis module, configured to analyze and process the first multimodal data to obtain first result data;

[0029] A control module is used to control the interaction of the first agent according to the first result data.

[0030] Furthermore, it also includes:

[0031] The polling module is used to poll the status of each intelligent agent and determine the first intelligent agent according to the status of the intelligent agent.

[0032] In one embodiment, the present invention provides a multi-agent interactive control system based on smart glasses, comprising:

[0033] Smart glasses and controllers;

[0034] The smart glasses collect multimodal data from the user's first-person perspective;

[0035] The controller executes the multi-agent interactive control method based on the smart glasses according to the multimodal data acquired by the smart glasses.

[0036] The present invention utilizes the hardware advantages of smart glasses and combines the characteristics of smart glasses that can obtain multimodal data from the user's first-person perspective. The multimodal data from the user's first-person perspective obtained by the smart glasses is used to actively control the interaction of multiple intelligent agents, thereby achieving active assistance to the user. It has a higher degree of intelligence and is more flexible and intelligent than traditional methods.

[0037] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a module schematic diagram of a multi-agent interactive control system based on smart glasses of the present invention;

[0039] Figure 2 A schematic diagram of a module of a multi-agent interactive control device based on smart glasses of the present invention;

[0040] Figure 3 It is a flow chart of the multi-agent interactive control method based on smart glasses of the present invention. DETAILED DESCRIPTION

[0041] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present invention.

[0042] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0043] In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0044] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

[0045] See also Figure 1 , Figure 1The module diagram of a multi-agent interactive control system based on smart glasses of the present invention. The multi-agent interactive control system of the present invention comprises: smart glasses 1 and a controller 2. The smart glasses 1 are provided with data acquisition modules such as a camera, a microphone, a motion sensor, and a position sensor. When the user wears the smart glasses 1 or the smart glasses are placed near the user's position, the smart glasses 1 can collect multi-modal data such as vision, hearing, and movement from the user's first-person perspective through the data collection module carried thereon; the smart glasses are integrated with various agents, and the controller executes the multi-agent interactive control method based on smart glasses of the present invention to control the interaction of each agent based on the collected multi-modal data from the user's first-person perspective, thereby realizing active response to the user and improving the degree of intelligence. In this embodiment, the controller 2 can be independent of the smart glasses 1, or it can be arranged inside the smart glasses 1 as a part of the smart glasses 1. The present invention does not limit the relationship between the smart glasses 1 and the controller 2.

[0046] See also Figure 2 and Figure 3 , Figure 2 is a module schematic diagram of the multi-agent interactive control device based on smart glasses of the present invention, Figure 3 It is a flow chart of the multi-agent interactive control method based on smart glasses of the present invention.

[0047] The multi-agent interactive control device based on smart glasses of the present invention comprises: a data acquisition module M1, a content analysis module M2 and a control module M3.

[0048] The data acquisition module M1 is used to execute step S1 to acquire first multimodal data of a user's first-person perspective collected by the first agent through smart glasses;

[0049] Specifically, the multimodal data includes video data, image data, and text data from the user's first-person perspective collected by the camera of the smart glasses, audio data from the first-person perspective collected by the microphone, action data collected by the motion sensor, and position data collected by the position sensor. The video data, image data, and text data may include video, images, and text of the surrounding environment, respectively. In one embodiment, the multi-agent interactive control device also includes a polling module M0, which is used to execute step S0, poll the status of each agent, and determine the first agent based on the status of the agent. Optionally, the status of each agent is polled at a preset interval, for example, the polling module M0 polls the status of each agent every 30 seconds. The preset interval here can be set according to actual conditions, and the present invention is not limited thereto.

[0050] When the smart glasses are worn by the user or placed near the user, the polling module M0 polls and monitors the status of various intelligent agents in the smart glasses. It can be understood that the status of various intelligent agents includes whether the intelligent agent is used normally and the functional role of the intelligent agent, etc. In different scenarios, different intelligent agents may be required to perform interactions to complete tasks. For example, in safety warning and assistance scenarios, it may be necessary to control the safety officer intelligent agent and the communication intelligent agent to perform interactions. At this time, when the safety officer intelligent agent is polled, the safety officer intelligent agent is used as the first intelligent agent; in the tour guide scenario, it may be necessary to control the navigation intelligent agent and the search intelligent agent to perform interactions. At this time, the navigation intelligent agent is used as the first intelligent agent.

[0051] Furthermore, in one embodiment, two working modes are set in the smart glasses control system, including an auxiliary working mode and a general working mode. When no assistance is needed, the smart glasses control system works in the general working mode; when real-time assistance is needed, the user turns on the auxiliary mode so that the smart glasses enter the auxiliary working mode. At this time, step S0 becomes, when it is detected that the smart glasses enter the auxiliary working mode, the status of each intelligent body is polled at a preset interval, and the first intelligent body is determined according to the status of each intelligent body.

[0052] The content analysis module M2 is used to execute step S2 to analyze and process the first multimodal data to obtain first result data;

[0053] Furthermore, in the auxiliary working mode, after determining the first intelligent agent according to the status of each intelligent agent, the first intelligent agent is controlled to start collecting the first multimodal data of the user's first-person perspective through the smart glasses, and the first multimodal data is analyzed and processed to obtain the first result data. It can be understood that it can analyze the first multimodal data according to the collected multimodal data using existing image recognition algorithms, video parsing algorithms, semantic analysis algorithms, etc. to obtain the first result data. For different first multimodal data, different recognition algorithms can be used to analyze, process and judge them. For example, existing image recognition algorithms and video parsing algorithms can be used to analyze and process image data and video data. When the image data and video data contain text, they can also be analyzed and processed in combination with semantic analysis algorithms. The present invention does not limit the specific algorithm selection.

[0054] For example, for the safety warning and assistance scenario, when the safety officer agent is polled, it is determined that the safety officer agent is the first agent, and the data acquisition module of the smart glasses is activated to collect the first multimodal data from the user's first-person perspective, such as activating the camera and motion sensor to shoot the environment data or motion data from the user's first-person perspective, and analyzing, processing and judging the environment data and motion data to obtain the first result data. At this time, the first result data may be that the user is normal, the user may be abnormal, etc. For the tour guide scenario, when the navigation agent is polled, it is determined that the navigation agent is the first agent, and the data acquisition module of the smart glasses is activated to collect the first multimodal data from the user's first-person perspective, such as activating the camera and position sensor to shoot the video data, image data or position data from the user's first-person perspective, and analyzing, processing and judging the video data, image data and position data to obtain the first result data. At this time, the first result data may be that the user is at home, the user is not at home and may be traveling, etc. For different scenarios, the first agent can be controlled to activate different sensors to obtain the required first multimodal data.

[0055] The control module M3 is used to execute step S3 to control the interaction of the first agent according to the first result data.

[0056] Furthermore, the control module M3 includes a circulation unit, an output unit, a control unit and a wake-up unit.

[0057] The loop unit is used to execute step S31, determine whether the first result data is preset data, and when it is determined that the first result data is preset data, return to step S0, poll the status of each intelligent agent, and determine the first intelligent agent based on the status of the intelligent agent; when it is determined that the first result data is not preset data, enter step S32A and step S32B.

[0058] Specifically, the preset data for each scenario is stored in the smart glasses, and the preset data for each scenario can be set according to actual needs. For example, in the safety warning and assistance scenario, the preset data can be data that the user is in a safe state, such as everything is normal for the user; after analyzing and processing the first multimodal data such as video data or image data, if it is monitored that the user is walking or resting normally, the first result data is that the user is normal at this time; if the motion sensor detects that the angle of the smart glasses and the user's posture change are abnormal, the first result data is that the user may be abnormal. In the tour guide scenario, the preset data can be data in a non-tour guide state such as the user at home; after analyzing and processing the first multimodal data such as video data or image data, if it is monitored that the user is reading, resting, etc. at home, the first result data is that the user is at home; if the user is monitored from the video data, image data or location data that the user is outside, the first result data is that the user is outside. After obtaining the first result data, it is determined whether the first result data is the preset data. When it is determined that the first result data is the preset data, it means that the user does not need to actively assist at this time, and returns to step S0 to poll the status of each intelligent body. When it is determined that the first result data is not the preset data, it indicates that the user may need assistance at this time, and then the process proceeds to step S32A and step S32B.

[0059] The output unit is used to execute step S32A to output voice data information.

[0060] Specifically, when it is determined that the first result data is not the preset data, it means that the user may need active assistance at this time, or it can be understood that the smart glasses control system actively provides assistance to the user, and at this time, the microphone outputs voice data information to the user. Understandably, in the safety warning and assistance scenario, the voice data information can be "Do you need assistance?", "There is danger ahead, please pay attention to avoid it", etc. In the tour guide scenario, the voice data information can be "Do you need to provide navigation route information?", "Do you need to provide explanation services?", etc.

[0061] The control unit is also used to execute step S32B, control the first intelligent agent to re-collect the second multimodal data of the user's first-person perspective through the smart glasses, and control the interaction of the first intelligent agent according to the second multimodal data.

[0062] Specifically, when it is determined that the first result is not the preset data, it indicates that the user may need to actively assist at this time. The control unit further executes step S32B1 through the data acquisition module M1 to obtain the second multimodal data of the user's first-person perspective collected by the first agent through the smart glasses; executes step S32B2 through the content analysis module M2 to analyze and process the second multimodal data to obtain the second result data; and executes step S32B3 to control the interaction of the first agent according to the second result data. It can be understood that it can use the existing image recognition algorithm, video parsing algorithm, semantic analysis algorithm, etc. to analyze the collected second multimodal data to obtain the second result data. For different second multimodal data, different recognition algorithms can be used to analyze, process and judge them. For example, the existing image recognition algorithm and video parsing algorithm can be used to analyze and process image data and video data. When the image data and video data contain text, they can also be analyzed and processed in combination with the semantic analysis algorithm. The present invention does not limit the specific algorithm selection.

[0063] Furthermore, when it is detected that the user may need active assistance or actively provide assistance to the user, the control unit controls the earphone or voice interface of the smart glasses to output voice data information, and actively asks the user whether he needs assistance. In this embodiment, when the control unit controls the output unit to output voice data information, it also controls the first intelligent body to collect the second multimodal data of the user's first-person perspective through the data acquisition module of the smart glasses, and analyzes and processes the second multimodal data to obtain the second result data, thereby improving the efficiency of actively providing assistance to the user. It can be understood that in the safety warning and assistance scenario, the second result data can be the user's confirmation of the existence of an abnormality, specific abnormal conditions, etc. In the tour guide scenario, the second result data can be the user's location information, environmental information, etc.

[0064] In other embodiments, after outputting the voice data information, if no feedback information from the user through the microphone is received within a preset time, the first intelligent agent is controlled to collect the second multimodal data of the user's first-person perspective through the data acquisition module of the smart glasses, and the second multimodal data is analyzed and processed to obtain the second result data. For example, in the safety warning and assistance mode, when it is monitored according to the first result data that the user may have a safety risk, the voice data information "whether assistance is needed" is output. At this time, if the user has an accident and cannot feedback information through the microphone, when it is judged that the user's feedback information is not received within the preset time, the first intelligent agent activates the data acquisition module again to collect data to obtain the second multimodal data, and analyzes the second multimodal data to confirm whether assistance is needed. For example, by analyzing and processing the captured video data, when it is seen that the current video data is the video data captured by the smart glasses in an oblique shooting state, it means that the user wearing the smart glasses may have fallen and needs to actively provide assistance. In the tour guide scenario, when it is monitored according to the first result data that the user may be outside, the voice data information "whether a navigation route is needed" or "whether an explanation service is needed" is output. When the feedback information received from the user through the microphone within the preset time is "yes", it means that the user wearing the smart glasses needs to actively provide assistance. At this time, the first intelligent agent is controlled to collect the second multimodal data of the user's first-person perspective through the data acquisition module of the smart glasses, and the second multimodal data is analyzed and processed to obtain the second result data. When the feedback information received from the user through the microphone within the preset time is "no", it means that the user wearing the glasses does not need to actively provide assistance. At this time, return to execute step S0, poll the status of each intelligent agent, and determine the first intelligent agent according to the status of the intelligent agent.

[0065] The awakening unit is used to execute step S32B31, select and awaken the corresponding second intelligent agent according to the second result data, and send the second multimodal data and the second result data to the second intelligent agent, so that the second intelligent agent analyzes and processes the second multimodal data and the second result data to obtain feedback data information.

[0066] For example, in the safety warning and assistance scenario, when the user is confirmed to have an abnormality based on the second result data, the communication agent is selected and awakened as the second agent, and the second multimodal data of the user's first-person perspective of the safety abnormality and the second result data are sent to the communication agent, so that the communication agent analyzes the above data and selects and calls the preset emergency contact number in the contact information of the smart glasses. In the tour guide scenario, when the user's tour location is confirmed based on the second result data, the search agent is selected and awakened as the second agent, and the image or video information of the current tour site collected from the user's first-person perspective and the second result data are sent to the search agent, so that the search agent analyzes the above data and searches the Internet for relevant introductions to the current tour site through wireless communication methods such as WIFI and Bluetooth.

[0067] The output unit is also used to execute step S32B32 and output feedback data information.

[0068] It can be understood that the feedback data information is data information that actively provides assistance to the user. For example, in the safety warning and assistance scenario, when the user is confirmed to have an abnormality according to the second result data, the communication agent is selected and awakened as the second agent, and the second multimodal data of the user's first-person perspective of the existence of safety abnormalities and the second result data are sent to the communication agent, so that the communication agent analyzes the above data, and when the preset emergency contact number is selected and dialed in the contact information of the smart glasses, the microphone is controlled to output feedback data information such as "the user has an abnormality and needs assistance" to the emergency contact when the call is connected. In the tour guide scenario, when the user's tour location is confirmed according to the second result data, the search agent is selected and awakened as the second agent, and the image or video information of the current tour holy place collected from the user's first-person perspective and the second result data are sent to the search agent, so that the search agent analyzes the above data, and when the relevant introduction of the current tour holy place is searched online through wireless communication methods such as WIFI and Bluetooth, the microphone is controlled to output the feedback data information such as the searched relevant introduction.

[0069] The present invention has the following technical effects: the present invention utilizes the hardware advantages of smart glasses, combined with the characteristics of smart glasses that can obtain multimodal data from the user's first-person perspective, and actively controls the interaction of multiple intelligent bodies through the multimodal data from the user's first-person perspective obtained by the smart glasses, thereby achieving active assistance to the user. It has a higher degree of intelligence and is more flexible and intelligent than traditional methods.

[0070] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A multi-agent interactive control method based on smart glasses, characterized in that: The smart glasses are integrated with a plurality of intelligent agents; the method comprises: S0, polling the status of each agent at a preset interval, and determining the first agent according to the status of the agent; S1, obtaining first multimodal data of a user's first-person perspective collected by a first agent through smart glasses; S2, analyzing and processing the first multimodal data to obtain first result data; the first result data is used to represent the result data corresponding to each of the different scenarios; S3, controlling the interaction of the first agent according to the first result data; Wherein, step S3 comprises: S31, determining whether the first result data is preset data; If the first result data is preset data, it is determined that the user does not need active assistance, and then the process returns to step S0 to poll the status of each agent and determine the first agent according to the status of the agent; If the first result data is not the preset data, it is determined that the user may need active assistance, and then the process proceeds to step S32A; S32A, output voice data information.

2. The multi-agent interactive control method according to claim 1, characterized in that: In step S31, when it is determined that the first result data is not the preset data, the process proceeds to step S32B; S32B, controlling the first intelligent agent to re-collect the second multimodal data of the user's first-person perspective through the smart glasses, and controlling the interaction of the first intelligent agent according to the second multimodal data.

3. The multi-agent interactive control method according to claim 2, characterized in that: Step S32B includes: S32B1, obtaining second multimodal data of a user's first-person perspective collected by the first agent through smart glasses; S32B2, analyzing and processing the second multimodal data to obtain second result data; S32B3, controlling the interaction of the first intelligent agent according to the second result data.

4. The multi-agent interactive control method according to claim 3, characterized in that: Step S32B3 includes: S32B31, selecting and waking up a corresponding second agent according to the second result data, and sending the second multimodal data and the second result data to the second agent, so that the second agent analyzes and processes the second multimodal data and the second result data to obtain feedback data information; S32B32, output feedback data information.

5. The multi-agent interactive control method according to claim 1, characterized in that: The smart glasses are provided with an auxiliary working mode and a general working mode; Step S0 is: when it is detected that the smart glasses enter the auxiliary working mode, the status of each intelligent body is polled at a preset interval, and the first intelligent body is determined according to the status of each intelligent body.

6. A multi-agent interactive control device based on smart glasses, characterized in that: include: A polling module, which is used to poll the status of each intelligent agent at a preset interval time, and determine the first intelligent agent according to the status of the intelligent agent; A data acquisition module, which is used to acquire first multimodal data of a user's first-person perspective collected by a first agent through smart glasses; A content analysis module, which is used to analyze and process the first multimodal data to obtain first result data; the first result data is used to represent the result data corresponding to each in different scenarios; a control module, configured to control the interaction of the first agent according to the first result data; Wherein, the control module includes: a circulation unit and an output unit; The loop unit is used to determine whether the first result data is preset data; If the first result data is preset data, it is determined that the user does not need active assistance, and then the polling module is called back; If the first result data is not the preset data, it is determined that the user may need active assistance, and then the output unit is called; The output unit is used to output voice data information.

7. A multi-agent interactive control system based on smart glasses, characterized in that: include: Smart glasses and controllers; The smart glasses collect multimodal data from the user's first-person perspective; The controller executes the multi-agent interactive control method based on smart glasses as described in any one of claims 1-5 according to the multimodal data acquired by the smart glasses.

Images