Mixed reality interaction method and system, electronic equipment and storage medium

Abstract

The invention discloses a mixed reality interaction method and system, electronic equipment and a storage medium. The system comprises an intelligent glasses terminal, a business server, an intelligent agent control server and an intelligent agent. The method is applied to the business server, and comprises the following steps: acquiring multi-dimensional state information currently acquired by an intelligent glasses terminal; arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information; the current multi-dimensional assistance information is sent to an agent control server side, so that the current multi-dimensional assistance information is transmitted to a currently started agent, and the agent controls the server side according to a workflow associated with the agent. And analyzing the current interaction information of the user sent by the intelligent glasses terminal in real time based on the current multi-dimensional assistance information to obtain a current analysis result, and feeding back the current analysis result to the intelligent glasses terminal.

Description

Technical Field

[0001] The present application relates to the field of mixed reality technology, and in particular to a mixed reality interaction method and system, electronic device, and storage medium. Background Art

[0002] Mixed reality glasses are wearable glasses that combine augmented reality (AR) and virtual reality (VR) technologies. Using sensors and cameras to capture real-world images, they seamlessly blend the virtual and real worlds, allowing users to experience various scenes in an immersive way.

[0003] To meet more user needs, current mixed reality glasses also support interactive features. Currently, the user receives interaction information through the mixed reality glasses and then sends this information to the backend. The backend then analyzes this information, determines the user's current interaction intent, and generates corresponding feedback, which is then fed back to the mixed reality glasses.

[0004] However, the existing methods follow a fixed processing logic for the interactive information sent by users, only analyzing the user's intention based on the current interactive information and generating corresponding feedback information, but fail to consider the user's current situation. Therefore, the accuracy of the analyzed user needs is low, and the corresponding feedback information cannot effectively meet the user's actual needs. Summary of the Invention

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a mixed reality interaction method and system, electronic device, and storage medium to solve the problem of inaccurate analysis and generation of information during interaction with mixed reality glasses in the prior art.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a mixed reality interaction method, comprising:

[0008] Obtain the multi-dimensional status information currently collected by the smart glasses terminal;

[0009] Arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information;

[0010] The current multi-dimensional assistance information is sent to the intelligent agent control server to pass the current multi-dimensional assistance information to the currently started intelligent agent, and the intelligent agent analyzes the current interaction information of the user sent in real time by the smart glasses terminal based on the current multi-dimensional assistance information according to its associated workflow, obtains the current analysis result and feeds it back to the smart glasses terminal.

[0011] Optionally, the above mixed reality interaction method further includes:

[0012] receiving a request to join a communication room sent by the smart glasses terminal;

[0013] In response to the request to join the communication room, request the agent control server to start the agent and join the agent to the communication room;

[0014] receiving a notification sent by the agent control server after the agent joins the communication room and an identifier of the communication room;

[0015] Feedback information is sent to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal joins the communication room.

[0016] Optionally, in the above-mentioned mixed reality interaction method, responding to the request to join the communication room and requesting the agent control server to start the agent and join the agent to the communication room includes:

[0017] Using the identifier of the agent requesting to join the communication room, a request to start the agent is sent to the agent control server to trigger the agent control server to start or create the corresponding agent according to the identifier of the agent, obtain the workflow associated with the agent, and add the agent to the newly created communication room.

[0018] Optionally, in the above-mentioned mixed reality interaction method, the sending feedback information to the smart glasses terminal based on the identifier of the communication room so that the smart glasses terminal joins the communication room includes:

[0019] generating token data of the communication room based on the identifier of the communication room;

[0020] The token data of the communication room is fed back to the smart glasses terminal, so that the smart glasses terminal uses the token data of the communication room to join the communication room.

[0021] Optionally, in the above-mentioned mixed reality interaction method, the step of obtaining the multi-dimensional state information currently collected by the smart glasses terminal includes:

[0022] Obtain current scene information and current glasses status information currently collected by the smart glasses terminal; wherein the current scene information at least includes the current field of view environment image of the smart glasses terminal, the current geographic location coordinates and POI points within a set range; the current glasses status information at least includes the current posture of the smart glasses terminal.

[0023] Optionally, the above mixed reality interaction method further includes:

[0024] Acquire, from the smart glasses terminal, multi-dimensional monitoring data currently collected by an external device and synchronized to the smart glasses terminal;

[0025] The current multi-dimensional monitoring data is added to the currently acquired multi-dimensional status information.

[0026] Optionally, the above mixed reality interaction method further includes:

[0027] Obtaining historical communication information of the agent;

[0028] Analyzing and extracting the historical communication information to obtain current analysis data and current extraction data;

[0029] The current analysis data and the current extraction data are converted into structured information, and added to the current multi-dimensional assistance information.

[0030] A second aspect of the present application provides a mixed reality interactive system, comprising:

[0031] Smart glasses terminal, business server, intelligent agent control server and intelligent agent;

[0032] The smart glasses terminal is used to collect multi-dimensional state information, send the currently collected multi-dimensional state information to the business service end, and send the user's current interaction information to the intelligent agent in real time;

[0033] The business service end is used to compile the currently acquired multi-dimensional state information into structured information, obtain current multi-dimensional assistance information, and send the current multi-dimensional assistance information to the agent control server end;

[0034] The agent control server is used to start the agent and transmit the current multi-dimensional assistance information to the currently started agent;

[0035] The intelligent agent is used to analyze the user's current interaction information based on the current multi-dimensional assistance information, obtain the current analysis result and feed it back to the smart glasses terminal.

[0036] A third aspect of the present application provides an electronic device, including:

[0037] memory and processor;

[0038] Wherein, the memory is used to store programs;

[0039] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the mixed reality interaction method as described in any one of the above.

[0040] In a fourth aspect, the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the mixed reality interaction method as described in any one of the above.

[0041] The present application provides a mixed reality interaction method, which collects the current multi-dimensional state information through the smart glasses terminal and sends the current multi-dimensional state information to the business server. The business server then compiles the current multi-dimensional state information into structured information to obtain the current multi-dimensional assistance information, so that it can be used for subsequent analysis after processing using these multi-dimensional information. The business server sends the current multi-dimensional assistance information to the intelligent agent control server, so that the current multi-dimensional assistance information is passed to the currently started intelligent agent through the intelligent agent control server. The smart glasses terminal sends the user's current interaction information to the intelligent agent in real time, and the intelligent agent analyzes the user's current interaction information sent by the smart glasses terminal in real time based on the current multi-dimensional assistance information according to the associated workflow, obtains the current analysis results and feeds them back to the smart glasses terminal. Therefore, by analyzing the current interaction information based on the state information of multiple dimensions through the corresponding intelligent agent and workflow, the user's true intention can be more accurately analyzed, and accurate feedback information can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0043] Figure 1 A schematic diagram of the architecture of a mixed reality interaction system provided in an embodiment of the present application;

[0044] Figure 2 A flow chart of a mixed reality interaction method provided in an embodiment of the present application;

[0045] Figure 3 A flowchart of an initialization processing method provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of a method for processing historical information provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the architecture of a business service end provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] In this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0051] The embodiments of the present application provide a mixed reality interaction method to solve the problem in the prior art of inaccurate analysis and generation of information during interaction with mixed reality glasses.

[0052] In order to implement the mixed reality interaction method provided in the embodiment of the present application, a mixed reality interaction system is provided in the embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a smart glasses terminal, a business service terminal, an intelligent agent control service terminal and an intelligent agent (instance).

[0053] It should be noted that before the smart glasses terminal interacts with the intelligent agent in real time, the system needs to be initialized first.

[0054] Alternatively, as Figure 1As shown in the figure, the system initialization process is as follows: the smart glasses terminal requests the business server to obtain the token data Token for joining the communication room. The business server requests to start the corresponding agent instance and join it to the communication room. The agent control server starts or creates the corresponding agent and joins it to the communication room. The agent joins the communication room and synchronizes its status to the agent control server. After the agent joins the communication room, the agent control server returns the startup result to the business server. After receiving the startup result returned by the agent control server, the business server returns the Token for entering the communication room to the smart glasses terminal. The agent uses the communication room Token to join the communication room.

[0055] Specifically, during interaction, the agent collects multi-dimensional state information and sends it to the business server. The business server compiles the acquired multi-dimensional state information into structured information, obtains current multi-dimensional assistance information, and sends this information to the agent control server. The agent control server transmits this information to the agent. When the smart glasses terminal transmits the user's current interaction information to the agent through the communication room, the agent analyzes the user's current interaction information based on the current multi-dimensional assistance information according to its associated workflow, obtains the current analysis results, and feeds them back to the smart glasses terminal.

[0056] Specifically, you can select the ID of the intelligent agent in the development of the smart glasses terminal, select the workflow associated with the intelligent agent, and select a workflow from these workflows for matching execution to achieve analysis of the current interactive information.

[0057] The current interaction information can be any of text, audio, image, or video, and the current analysis results can also be any of text, audio, image, or video. Furthermore, the agent can set the current multi-dimensional assistance information as prompt words for the large model or multimodal model to help the large model or multimodal model output better results.

[0058] Alternatively, as Figure 1 As shown, the mixed reality interactive system can also include external devices connected to the smart glasses terminal, such as smartphones and smart watches, to synchronize their currently monitored multi-dimensional data with the smart glasses terminal in real time. For example, health check data can be synchronized. The smart glasses terminal combines this multi-dimensional monitoring data with its own collected information as multi-dimensional status information and feeds it back to the business server, thereby providing more dimensional information to assist the intelligent agent in interactive analysis and further improve the accuracy of the analysis results.

[0059] The embodiment of the present application provides a mixed reality interaction method, which is applied to a business service end. Figure 2As shown, an embodiment of the present application provides a mixed reality interaction method, comprising the following steps:

[0060] S201: Acquire multi-dimensional status information currently collected by the smart glasses terminal.

[0061] It should be noted that the smart glasses terminal may specifically be a terminal of mixed reality glasses, that is, a pre-developed application is installed on the mixed reality glasses.

[0062] Multi-dimensional state information refers to information across multiple dimensions, which can help the model more accurately analyze user needs and generate accurate feedback. Specifically, the smart glasses terminal can be configured to collect information in various dimensions, such as environmental information and location information.

[0063] Therefore, in the embodiment of the present application, the smart glasses terminal collects the current multi-dimensional state information in real time. In addition, in order to use the collected data for model analysis, the smart glasses terminal first sends the current multi-dimensional state information to the service server for processing. At this time, the service server receives the multi-dimensional state information currently collected by the smart glasses terminal.

[0064] Optionally, in another embodiment of the present application, a specific implementation of step S201 includes:

[0065] Obtain the current scene information and current glasses status information currently collected by the smart glasses terminal.

[0066] The current scene information is information reflecting the current scene. The glasses status information is information about the status of the smart glasses terminal. Optionally, the current scene information includes at least an image of the current field of view of the smart glasses terminal, the current geographic location coordinates, and POI points within a set range. The current glasses status information includes at least the current posture of the smart glasses terminal.

[0067] Optionally, in another embodiment of the present application, the method may further include:

[0068] The multi-dimensional monitoring data currently collected by the external device and synchronized to the smart glasses terminal is obtained from the smart glasses terminal, and the current multi-dimensional monitoring data is added to the currently obtained multi-dimensional status information.

[0069] It should be noted that the information that can be collected by the smart glasses terminal is limited. Therefore, in the embodiments of the present application, the smart glasses terminal can also be connected to other devices to obtain data collected by these devices, so as to collect more dimensional information, accurately analyze the user's intentions, and further improve the accuracy of the analysis results. For example, external smartphones, smart watches, etc.

[0070] Specifically, the smart glasses terminal can obtain the multi-dimensional monitoring data currently collected by each external device and synchronized to the smart glasses terminal, and combine it with the multi-dimensional information collected by itself to form multi-dimensional status information and send it to the business server.

[0071] Optionally, since the user's heart rate and other information can reflect the user's status, and the user's status can reflect their current needs, in the embodiment of the present application, the user's current health monitoring data, such as heart rate, exercise, sleep, etc., collected by the smart monitoring device worn by the user is obtained.

[0072] In order to meet the user's customized needs and more accurately analyze the user's true intentions, the system will also obtain information entered by the user, such as the menstrual cycle and medication reminders actively entered by the user.

[0073] In order to enable the intelligent agent and allow the intelligent agent to interact with the smart glasses terminal in real time, an initialization process is required before executing step S201. Figure 3 As shown, an initialization processing method provided in an embodiment of the present application includes:

[0074] S301: Receive a request to join a communication room sent by a smart glasses terminal.

[0075] The request includes the identity of the agent.

[0076] It should be noted that in order for the smart glasses terminal and the agent to interact in real time, they must be added to the same communication room. Therefore, when the agent needs to be activated and both the smart glasses terminal and the agent join the communication room, the smart glasses terminal sends a request to join the communication room to the service server.

[0077] Since you want to join the same communication room and avoid other applications from joining, you can specifically request the business service to obtain the verification token for joining the communication room, and use the verification token of the communication room to join the corresponding specific communication room.

[0078] Optionally, to enable multi-dimensional instant communication between the smart glasses terminal and the intelligent agent, including text, voice, image, and video, the communication room can be a real-time communication (RTC) room. RTC refers to a technology that enables the instant transmission of audio, video, and data information over the Internet.

[0079] S302: respond to the request to join the communication room, request the agent control server to start the agent and add it to the communication room.

[0080] Specifically, the business server responds to the request of the smart glasses terminal to join the communication room, sends a start-up message to the agent control server and adds the agent to the communication room.

[0081] Optionally, in another embodiment of the present application, a specific implementation of step S302 includes:

[0082] Using the identity of the intelligent agent in the request to join the communication room, a request to start the intelligent agent is sent to the intelligent agent control server to trigger the intelligent agent control server to start or create the corresponding intelligent agent according to the identity of the intelligent agent, obtain the workflow associated with the intelligent agent, and add the intelligent agent to the newly created communication room.

[0083] Specifically, the agent control server searches for a workflow associated with the agent's identifier according to the agent's identifier, and initializes an agent instance, thereby starting the agent.

[0084] It should be noted that an agent is a computational intelligence unit used to achieve a specific goal, so the corresponding agent is pre-developed. Therefore, when a corresponding interaction is required, the smart glasses terminal requests the activation of the corresponding agent so that the corresponding goal can be achieved through the agent.

[0085] In order to target different interaction situations, the workflow of the agent is pre-configured so that the corresponding interaction analysis and feedback can be completed according to the workflow of the corresponding agent to ensure the accuracy of the results. For example, the following common workflow can be configured:

[0086] 1. Text input - large model processing - text output.

[0087] 2. Voice input - large model processing - text output.

[0088] 3. Voice input - large model processing - voice output.

[0089] 4. Audio and video input - multimodal large model processing - text output.

[0090] 5. Audio and video input - multimodal large model processing - audio output.

[0091] 6. Audio and video input - multimodal large model processing - audio output.

[0092] Therefore, in order to interact with the background through the smart glasses terminal, it is necessary to first start the agent requested by the smart glasses terminal and obtain the workflow associated with the agent.

[0093] S303: Receive the notification sent by the agent control server after the agent joins the communication room and the identifier of the communication room.

[0094] To inform the agent control server that the agent has joined the communication room and to further inform the business server, the agent synchronizes its joining information with the agent control server after joining the communication room. Upon receiving the agent's entry, the agent control server sends a notification of the agent joining the communication room and the room's identifier to the business server.

[0095] S304: Send feedback information to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal joins the communication room.

[0096] Specifically, the business service end feeds back relevant information of the communication room to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal can use this information to join the communication room, thereby adding the smart glasses terminal and the intelligent agent to the same room, so that the smart glasses terminal and the intelligent agent can interact in real time through the communication room.

[0097] Optionally, in another embodiment of the present application, a specific implementation of step S304 includes:

[0098] The token data of the communication room is generated based on the identifier of the communication room, and the token data of the communication room is fed back to the smart glasses terminal, so that the smart glasses terminal uses the token data of the communication room to join the communication room.

[0099] Specifically, after receiving notification that an agent has joined a communication room, the service server generates a corresponding token based on the room's identifier and then returns the token to the smart glasses terminal. The smart glasses terminal uses the token for verification and joins the communication room.

[0100] S202: Arrange the multi-dimensional status information into structured information to obtain current multi-dimensional assistance information.

[0101] It's important to note that because the collected multi-dimensional state information may not be directly input into the model for analysis, or because the data truly needed requires further processing, the business server organizes the current multi-dimensional state information into structured information. For example, trajectory prediction can be performed using environmental maps and poses to generate predicted trajectories for subsequent analysis. Alternatively, currently collected information can be aggregated with historically collected information.

[0102] Optionally, considering that previous data may be used when analyzing intent, that is, analysis is performed in conjunction with the context, the current multi-dimensional state information needs to be stored.

[0103] Optionally, in another embodiment of the present application, a method for processing historical information is further included, such as Figure 4As shown, the following steps are included:

[0104] S401. Obtain historical communication information of the intelligent agent.

[0105] Specifically, the agent control server can store each communication data of the agent, including the interaction information between the agent and the smart glasses terminal and multi-dimensional status information, so that the required historical communication information can be obtained from the database.

[0106] S402: Analyze and extract historical communication information to obtain current analysis data and current extraction data.

[0107] S403: Convert the current analysis data and the current extraction data into structured information, and add the structured information to the current multi-dimensional assistance information.

[0108] S203. Send the current multi-dimensional assistance information to the intelligent agent control server to pass the current multi-dimensional assistance information to the currently started intelligent agent. The intelligent agent analyzes the current interaction information of the user sent in real time by the smart glasses terminal based on the current multi-dimensional assistance information according to its associated workflow, obtains the current analysis results and feeds them back to the smart glasses terminal.

[0109] Specifically, users can send interactive information at any time through the smart glasses terminal. This interactive information can be in formats such as text, audio, video, and images. The smart glasses terminal then instantly sends the user's current interactive information to the agent via the communication room. The agent then selects the workflow corresponding to the format of the current interactive information from its associated workflows. Following the selected workflow, the agent invokes the model to analyze the user's current interactive information sent in real time by the smart glasses terminal based on the current multi-dimensional assistance information, obtains the current analysis results, and feeds them back to the smart glasses terminal.

[0110] Optionally, the model can include a large language model or a multi-model large model. Specifically, the corresponding model can be called according to the workflow, using prompt information generated based on the current multi-dimensional assistance information to analyze the current interaction information and determine the user's current intent. Based on the current multi-dimensional assistance information and user intent, the required feedback information, namely the current analysis result, is adaptively generated. After the agent completes the corresponding task according to the workflow, the model generates the current analysis result and immediately feeds it back to the smart glasses terminal via the communication room. The smart glasses terminal then displays the current analysis result to the user.

[0111] The present application provides a mixed reality interaction method, which collects the current multi-dimensional state information through the smart glasses terminal and sends the current multi-dimensional state information to the business server. The business server then compiles the current multi-dimensional state information into structured information to obtain the current multi-dimensional assistance information, so that it can be used for subsequent analysis after processing using these multi-dimensional information. The business server sends the current multi-dimensional assistance information to the intelligent agent control server, so that the current multi-dimensional assistance information is passed to the currently started intelligent agent through the intelligent agent control server. The smart glasses terminal sends the user's current interaction information to the intelligent agent in real time, and the intelligent agent analyzes the user's current interaction information sent by the smart glasses terminal in real time based on the current multi-dimensional assistance information according to the associated workflow, obtains the current analysis results and feeds them back to the smart glasses terminal. Therefore, by analyzing the current interaction information based on the state information of multiple dimensions through the corresponding intelligent agent and workflow, the user's true intention can be more accurately analyzed, and accurate feedback information can be generated.

[0112] Another embodiment of the present application provides a business service end, such as Figure 5 Shown, including:

[0113] The multi-dimensional information acquisition unit 501 is used to acquire the multi-dimensional status information currently collected by the smart glasses terminal.

[0114] The multi-dimensional information processing unit 502 is used to organize the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information.

[0115] The multi-dimensional information sending unit 503 sends the current multi-dimensional assistance information to the intelligent agent control server to pass the current multi-dimensional assistance information to the currently started intelligent agent. The intelligent agent analyzes the current interaction information of the user sent in real time by the smart glasses terminal based on the current multi-dimensional assistance information according to its associated workflow, obtains the current analysis results and feeds them back to the smart glasses terminal.

[0116] Optionally, the service server provided in another embodiment of the present application further includes:

[0117] The request receiving unit is used to receive a request to join a communication room sent by the smart glasses terminal.

[0118] The startup request unit is used to respond to the request to join the communication room, request the intelligent agent control server to start the intelligent agent and join the communication room.

[0119] The notification receiving unit is used to receive the notification sent by the agent control server after the agent joins the communication room and the identifier of the communication room.

[0120] The feedback unit is used to send feedback information to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal joins the communication room.

[0121] Optionally, in a business service end provided in another embodiment of the present application, the starting request unit includes:

[0122] The start request sub-unit is used to use the identifier of the intelligent agent requesting to join the communication room to send a request to start the intelligent agent to the intelligent agent control server, so as to trigger the intelligent agent control server to start or create the corresponding intelligent agent according to the identifier of the intelligent agent, obtain the workflow associated with the intelligent agent, and add the intelligent agent to the newly created communication room.

[0123] Optionally, in the service server provided in another embodiment of the present application, the feedback unit includes:

[0124] The token generating unit is used to generate token data of the communication room based on the identifier of the communication room.

[0125] The token feedback unit is used to feed back the token data of the communication room to the smart glasses terminal, so that the smart glasses terminal uses the token data of the communication room to join the communication room.

[0126] Optionally, in the business service end provided in another embodiment of the present application, the multi-dimensional information acquisition unit includes:

[0127] The multi-dimensional information acquisition subunit is used to obtain the current scene information and current glasses status information currently collected by the smart glasses terminal. The current scene information includes at least the current field of view image of the smart glasses terminal, the current geographic location coordinates, and the points of interest within a set range. The current glasses status information includes at least the current position of the smart glasses terminal.

[0128] Optionally, the service server provided in another embodiment of the present application further includes:

[0129] The monitoring data acquisition unit is used to obtain the multi-dimensional monitoring data currently collected by the external device and synchronized to the smart glasses terminal from the smart glasses terminal.

[0130] The adding unit is used to add the current multi-dimensional monitoring data to the currently acquired multi-dimensional status information.

[0131] Optionally, the service server provided in another embodiment of the present application further includes:

[0132] The historical information acquisition unit is used to obtain the historical communication information of the intelligent agent.

[0133] The historical information analysis unit is used to analyze and extract historical communication information to obtain current analysis data and current extraction data.

[0134] The supplement unit is used to convert the current analysis data and the current extracted data into structured information and supplement it into the current multi-dimensional assistance information.

[0135] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.

[0136] Another embodiment of the present application provides an electronic device, such as Figure 6 Shown, including:

[0137] Memory 601 and processor 602 .

[0138] The memory 601 is used to store programs.

[0139] The processor 602 is used to execute the program stored in the memory 601. When the program is executed, it is specifically used to implement the mixed reality interaction method provided by any one of the above embodiments.

[0140] Another embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the mixed reality interaction method provided in any of the above embodiments.

[0141] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0142] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A mixed reality interaction method, characterized in that: include: Obtain the multi-dimensional status information currently collected by the smart glasses terminal; Arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information; The current multi-dimensional assistance information is sent to the intelligent agent control server to pass the current multi-dimensional assistance information to the currently started intelligent agent, and the intelligent agent analyzes the current interaction information of the user sent in real time by the smart glasses terminal based on the current multi-dimensional assistance information according to its associated workflow, obtains the current analysis result and feeds it back to the smart glasses terminal.

2. The method according to claim 1, characterized in that Also includes: receiving a request to join a communication room sent by the smart glasses terminal; In response to the request to join the communication room, request the agent control server to start the agent and join the agent to the communication room; receiving a notification sent by the agent control server after the agent joins the communication room and an identifier of the communication room; Feedback information is sent to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal joins the communication room.

3. The method according to claim 2, characterized in that The step of responding to the request to join the communication room and requesting the agent control server to start the agent and join the agent to the communication room includes: Using the identifier of the agent requesting to join the communication room, a request to start the agent is sent to the agent control server to trigger the agent control server to start or create the corresponding agent according to the identifier of the agent, obtain the workflow associated with the agent, and add the agent to the newly created communication room.

4. The method according to claim 2, characterized in that The sending feedback information to the smart glasses terminal based on the identifier of the communication room so that the smart glasses terminal joins the communication room includes: generating token data of the communication room based on the identifier of the communication room; The token data of the communication room is fed back to the smart glasses terminal, so that the smart glasses terminal uses the token data of the communication room to join the communication room.

5. The method according to claim 1, wherein The obtaining of the multi-dimensional status information currently collected by the smart glasses terminal includes: Obtain current scene information and current glasses status information currently collected by the smart glasses terminal; wherein the current scene information at least includes the current field of view environment image of the smart glasses terminal, the current geographic location coordinates and POI points within a set range; the current glasses status information at least includes the current posture of the smart glasses terminal.

6. The method according to claim 1, wherein Also includes: Acquire, from the smart glasses terminal, multi-dimensional monitoring data currently collected by an external device and synchronized to the smart glasses terminal; The current multi-dimensional monitoring data is added to the currently acquired multi-dimensional status information.

7. The method according to claim 1, characterized in that Also includes: Obtaining historical communication information of the agent; Analyzing and extracting the historical communication information to obtain current analysis data and current extraction data; The current analysis data and the current extraction data are converted into structured information, and added to the current multi-dimensional assistance information.

8. A mixed reality interactive system, characterized in that: include: Smart glasses terminal, business server, intelligent agent control server and intelligent agent; The smart glasses terminal is used to collect multi-dimensional state information, send the currently collected multi-dimensional state information to the business service end, and send the user's current interaction information to the intelligent agent in real time; The business service end is used to compile the currently acquired multi-dimensional state information into structured information, obtain current multi-dimensional assistance information, and send the current multi-dimensional assistance information to the agent control server end; The agent control server is used to start the agent and transmit the current multi-dimensional assistance information to the currently started agent; The intelligent agent is used to analyze the user's current interaction information based on the current multi-dimensional assistance information, obtain the current analysis result and feed it back to the smart glasses terminal.

9. An electronic device, characterized in that: include: memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the mixed reality interaction method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that Used to store a computer program, which, when executed by a processor, is used to implement the mixed reality interaction method as described in any one of claims 1 to 7.

Images