A mixed reality interaction method and system, electronic device, storage medium

Abstract

The application discloses a mixed reality interaction method and system, an electronic device and a storage medium. The system comprises a smart glasses terminal, a business server, an agent control server and an agent. The method is applied to the business server and comprises the following steps: acquiring multi-dimensional state information collected by the smart glasses terminal currently; arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information; sending the current multi-dimensional assistance information to the agent control server, so as to transmit the current multi-dimensional assistance information to the agent started currently; and analyzing, by the agent, current interaction information of a user sent by the smart glasses terminal in real time based on a work flow associated with the agent and the current multi-dimensional assistance information, obtaining a current analysis result and feeding back the current analysis result to the smart glasses terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mixed reality, in particular to a mixed reality interaction method and system, an electronic device and a storage medium. BACKGROUND

[0002] Mixed reality glasses are wearable glasses that combine augmented reality (AR) and virtual reality (VR) technologies. Through sensors, cameras, etc., real images are collected, and the virtual world is seamlessly integrated with the real world, allowing users to experience various scenes in a lifelike way.

[0003] In order to meet the needs of users, the current mixed reality glasses also support interaction functions. Currently, the user receives the user's interaction information through the mixed reality glasses, and then sends the interaction information to the background. Then the background analyzes the interaction information to determine the user's current interaction intent, and generates corresponding feedback information and feeds it back to the mixed reality glasses.

[0004] However, the existing method analyzes the user's intent based on the current interaction information according to fixed processing logic, and generates corresponding feedback information, without considering the user's current situation, so the accuracy of the user's demand analysis is low, and the corresponding feedback information cannot effectively meet the user's actual needs. SUMMARY

[0005] Based on the above shortcomings of the prior art, the present application provides a mixed reality interaction method and system, an electronic device and a storage medium to solve the problem that the information analyzed and generated by the prior art is not accurate when the mixed reality glasses interact.

[0006] In order to achieve the above purpose, the present application provides the following technical solutions:

[0007] The first aspect of the present application provides a mixed reality interaction method, comprising:

[0008] Obtaining multi-dimensional state information currently collected by the intelligent glasses terminal;

[0009] Arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information;

[0010] Sending the current multi-dimensional assistance information to an intelligent agent control server to pass the current multi-dimensional assistance information to a currently started intelligent agent, and analyzing, by the intelligent agent, current interaction information of the user sent by the intelligent glasses terminal in real time based on the current multi-dimensional assistance information according to its associated workflow to obtain a current analysis result and feed it back to the intelligent glasses terminal.

[0011] Optionally, in the mixed reality interaction method, the method further comprises:

[0012] receiving a request for joining a communication room sent by the smart glasses terminal;

[0013] in response to the request for joining the communication room, requesting the agent control server to start an agent and add the agent to the communication room;

[0014] receiving a notification sent by the agent control server after the agent joins the communication room and an identifier of the communication room;

[0015] sending feedback information to the smart glasses terminal based on the identifier of the communication room, so that the smart glasses terminal joins the communication room.

[0016] Optionally, in the mixed reality interaction method, the method further comprises:

[0017] using the identifier of the agent that requests to join the communication room, sending a request for starting the agent to the agent control server, so as to trigger the agent control server to start or create the corresponding agent according to the identifier of the agent, obtain the workflow associated with the agent, and add the agent to the newly created communication room.

[0018] Optionally, in the mixed reality interaction method, the method further comprises:

[0019] generating token data of the communication room based on the identifier of the communication room;

[0020] feeding back the token data of the communication room to the smart glasses terminal, so that the smart glasses terminal joins the communication room using the token data of the communication room.

[0021] Optionally, in the mixed reality interaction method, the method further comprises:

[0022] obtaining current scene information and current glasses state information currently collected by the smart glasses terminal; wherein the current scene information at least includes a current field of view environment image, a current geographic position coordinate, and a POI point within a set range of the smart glasses terminal; and the current glasses state information at least includes a current pose of the smart glasses terminal.

[0023] Optionally, in the mixed reality interaction method, the method further comprises:

[0024] acquiring, from the smart glasses terminal, multi-dimensional monitoring data currently collected by an external device and synchronized to the smart glasses terminal;

[0025] adding the multi-dimensional monitoring data to the multi-dimensional state information currently acquired.

[0026] Optionally, in the mixed reality interaction method described above, the method further includes:

[0027] acquiring historical communication information of the intelligent agent;

[0028] analyzing and extracting the historical communication information to obtain current analysis data and current extraction data;

[0029] converting the current analysis data and the current extraction data into structured information and supplementing the structured information to the current multi-dimensional assistance information.

[0030] The second aspect of the application provides a mixed reality interaction system, including:

[0031] a smart glasses terminal, a business server, an intelligent agent control server and an intelligent agent;

[0032] The smart glasses terminal is configured to collect multi-dimensional state information, send the multi-dimensional state information currently collected to the business server, and send current interaction information of a user to the intelligent agent in real time.

[0033] The business server is configured to arrange the multi-dimensional state information currently acquired into structured information to obtain current multi-dimensional assistance information, and send the current multi-dimensional assistance information to the intelligent agent control server.

[0034] The intelligent agent control server is configured to start the intelligent agent, and deliver the current multi-dimensional assistance information to the intelligent agent currently started.

[0035] The intelligent agent is configured to analyze the current interaction information of the user based on the current multi-dimensional assistance information, obtain a current analysis result and feed back the current analysis result to the smart glasses terminal.

[0036] The third aspect of the application provides an electronic device, including:

[0037] a memory and a processor;

[0038] The memory is configured to store a program.

[0039] The processor is configured to execute the program, and the program, when executed, is specifically configured to implement the mixed reality interaction method described in any one of the above aspects.

[0040] The fourth aspect of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the mixed reality interaction method according to any one of the preceding aspects.

[0041] The present application provides a mixed reality interaction method, which collects current multi-dimensional state information through a smart glasses terminal, and sends the current multi-dimensional state information to a business service end, and then arranges the current multi-dimensional state information into structured information through the business service end to obtain current multi-dimensional assistance information, so that the multi-dimensional information can be used for subsequent analysis after processing. The business service end sends the current multi-dimensional assistance information to an agent control service end, so as to transmit the current multi-dimensional assistance information to the currently started agent through the agent control service end. The smart glasses terminal sends the current interaction information of the user to the agent in real time, and the agent analyzes the current interaction information of the user sent by the smart glasses terminal in real time based on the current multi-dimensional assistance information according to the associated workflow, obtains the current analysis result and feeds back to the smart glasses terminal. Therefore, through the corresponding agent and workflow, the current interaction information is analyzed based on the multi-dimensional state information, the real intention of the user can be more accurately analyzed, and accurate feedback information is generated. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0043] Figure 1 An architecture schematic diagram of a mixed reality interaction system provided by an embodiment of the present application;

[0044] Figure 2 A flowchart of a mixed reality interaction method provided by an embodiment of the present application;

[0045] Figure 3 A flowchart of an initialization processing method provided by an embodiment of the present application;

[0046] Figure 4 A flowchart of a history information processing method provided by an embodiment of the present application;

[0047] Figure 5 An architecture schematic diagram of a business service end provided by an embodiment of the present application;

[0048] Figure 6 An architecture schematic diagram of an electronic device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0051] The embodiments of the present application provide a mixed reality interaction method to solve the problem that the information analyzed and generated is not accurate enough when mixed reality glasses interact in the prior art.

[0052] In order to implement the mixed reality interaction method provided by the embodiments of the present application, a mixed reality interaction system is provided in the embodiments of the present application. As shown in Figure 1 The embodiments of the present application provide a smart glasses terminal, a business service end, an intelligent agent control service end and an intelligent agent (instance).

[0053] It should be noted that before the smart glasses terminal and the intelligent agent perform real-time interaction, the system needs to be initialized first.

[0054] Optionally, as shown in Figure 1As shown, the process of system initialization is as follows: the intelligent glasses terminal requests the business service end to obtain token data Token for joining the communication room. The business service end requests to start the corresponding agent instance and adds it to the communication room. The agent control service end starts or creates the corresponding agent and adds it to the communication room. The agent joins the communication room and synchronizes its state to the agent control service end. After the agent joins the communication room, the agent control service end returns the start result to the business service end. After receiving the start result returned by the agent control service end, the business service end returns the Token for entering the communication room to the intelligent glasses terminal. The agent joins the communication room by using the Token of the communication room.

[0055] Specifically, when interacting, the agent collects multi-dimensional state information and sends the currently collected multi-dimensional state information to the business service end. The business service end arranges the currently obtained multi-dimensional state information into structured information to obtain current multi-dimensional assistance information, and sends the current multi-dimensional assistance information to the agent control service end. The agent control service end transmits the current multi-dimensional assistance information to the agent. When the intelligent glasses terminal sends the current interaction information of the user to the agent through the communication room, the agent analyzes the current interaction information of the user based on the current multi-dimensional assistance information according to the workflow associated therewith, obtains a current analysis result and feeds back to the intelligent glasses terminal.

[0056] Specifically, the id of the selected agent in the development of the intelligent glasses terminal can be selected, the workflow associated with the agent can be selected, and one of the work flows can be selected for matching and execution to realize analysis of the current interaction information.

[0057] The current interaction information can be any one of text, audio, image and video, and the current analysis result can also be any one of text, audio, image and video. In addition, the agent can set the current multi-dimensional assistance information as a prompt word of a large model or a multi-modal model to assist the large model or the multi-modal model to better output a result.

[0058] Optionally, as shown in Figure 1 The mixed reality interaction system can also include an external device such as a smart phone, a smart watch, etc. connected to the intelligent glasses terminal, for synchronizing the current multi-dimensional data monitored thereby to the intelligent glasses terminal in real time. For example, health detection data is synchronized. The intelligent glasses terminal feeds back the multi-dimensional monitoring data and the information collected thereby to the business service end as multi-dimensional state information, thereby providing more dimensional information to assist the agent in interaction analysis and further improving the accuracy of the analysis result.

[0059] The embodiment of the present application provides a mixed reality interaction method applied to a business service end. As shown in Figure 2As shown, the mixed reality interaction method provided in the embodiment of the present application comprises the following steps:

[0060] S201, acquiring multi-dimension state information currently collected by the smart glasses terminal.

[0061] It should be noted that the smart glasses terminal can be a terminal of mixed reality glasses, that is, a pre-developed application is installed on the mixed reality glasses.

[0062] The multi-dimension state information is multi-dimension state information, and the state information refers to information that can assist the model in more accurately analyzing the user's demand and generating accurate feedback information. Each dimension of information that needs to be collected by the smart glasses terminal can be set according to the need. For example, environmental information, location information, etc.

[0063] Therefore, in the embodiment of the present application, the smart glasses terminal collects the current multi-dimension state information in real time. In order to use the collected data for model analysis, the smart glasses terminal first sends the current multi-dimension state information to the business service end for processing. At this time, the business service end receives the multi-dimension state information currently collected by the smart glasses terminal.

[0064] Optionally, in another embodiment of the present application, a specific implementation of step S201 comprises:

[0065] Acquiring current scene information and current glasses state information currently collected by the smart glasses terminal.

[0066] The current scene information is information reflecting the current scene. The glasses state information is information of the state of the smart glasses terminal. Optionally, the current scene information at least includes a current field of view environment image of the smart glasses terminal, a current geographic position coordinate, and a POI point within a set range. The current glasses state information at least includes a current pose of the smart glasses terminal.

[0067] Optionally, in another embodiment of the present application, it can further comprise:

[0068] Acquiring multi-dimension monitoring data currently collected by the external device and synchronized to the smart glasses terminal from the smart glasses terminal, and adding the current multi-dimension monitoring data to the currently acquired multi-dimension state information.

[0069] It should be noted that the information that can be collected by the smart glasses terminal is limited, so in the embodiment of the present application, the smart glasses terminal can also be connected to other devices, so as to acquire the data collected by these devices, so as to collect more dimension information, accurately analyze the user's intention, and further improve the accuracy of the analysis result. For example, external smart phones, smart watches, etc.

[0070] Specifically, the intelligent glasses terminal can acquire the multi-dimensional monitoring data currently collected by each external device and synchronized to the intelligent glasses terminal, and send the multi-dimensional state information composed of the multi-dimensional information collected by the intelligent glasses terminal to the business server.

[0071] Optionally, the user's state can be reflected by the user's heart rate and other information, and the user's state can reflect the user's current needs. Therefore, in the embodiments of the present application, the current health monitoring data of the user collected by the intelligent monitoring device worn by the user, such as heart rate, exercise, sleep, etc., is acquired.

[0072] In order to meet the user's self-defined needs and more accurately analyze the user's real intention, information input by the user, such as the menstrual cycle and medication reminders actively input by the user, is also acquired.

[0073] In order to enable the intelligent agent and allow the intelligent agent to interact with the intelligent glasses terminal in real time, initialization processing needs to be performed before step S201 is executed. As shown in Figure 3 The method for initialization processing provided by the embodiments of the present application includes:

[0074] S301, receiving a request for joining a communication room sent by the intelligent glasses terminal.

[0075] The request includes the identifier of the intelligent agent.

[0076] It should be noted that in order to enable the intelligent glasses terminal to interact with the intelligent agent in real time, the intelligent glasses terminal and the intelligent agent need to be added to the same communication room. Therefore, when the intelligent agent needs to be started, the intelligent glasses terminal sends a request for joining the communication room to the business server.

[0077] Since the same communication room needs to be joined and other applications need to be avoided, the business server can be requested to obtain a verification Token for joining the communication room, so that the intelligent agent can join the corresponding specific communication room by using the verification Token of the communication room.

[0078] Optionally, in order to enable the intelligent glasses terminal and the intelligent agent to support multi-dimensional instant communication of text, voice, image and video, the communication room can be a real-time audio / video (RTC) communication room. RTC refers to an instant transmission technology for audio / video and data information in the Internet.

[0079] S302, in response to the request for joining the communication room, requesting the intelligent agent control server to start the intelligent agent and add it to the communication room.

[0080] Specifically, the business server responds to the request of the intelligent glasses terminal to join the communication room, and sends a request to start the intelligent agent to the intelligent agent control server and adds the intelligent agent to the communication room.

[0081] Optionally, in another embodiment of the present application, a specific implementation of step S302 includes:

[0082] The identification of the intelligent agent in the request to join the communication room is used to send a request to start the intelligent agent to the intelligent agent control server, so as to trigger the intelligent agent control server to start or create a corresponding intelligent agent according to the identification of the intelligent agent, obtain the workflow associated with the intelligent agent, and add the intelligent agent to the newly created communication room.

[0083] Specifically, the intelligent agent control server finds the workflow associated with the identification according to the identification of the intelligent agent, and initializes an intelligent agent instance, thereby starting the intelligent agent.

[0084] It should be noted that the intelligent agent is a computing intelligent unit for achieving a specific goal, so the corresponding intelligent agent is developed in advance. Therefore, when the corresponding interaction needs to be achieved, the intelligent glasses terminal requests to start the corresponding intelligent agent, so as to achieve the corresponding goal through the intelligent agent.

[0085] In order to target different interaction situations, the workflow of the intelligent agent is configured in advance, so that the corresponding interaction analysis and feedback can be completed according to the workflow of the corresponding intelligent agent, and the accuracy of the result is ensured, for example, the following common workflows can be configured:

[0086] 1. Text input - large model processing - text output.

[0087] 2. Speech input - large model processing - text output.

[0088] 3. Speech input - large model processing - speech output.

[0089] 4. Audio and video input - multi-modal large model processing - text output.

[0090] 5. Audio and video input - multi-modal large model processing - audio output.

[0091] 6. Audio and video input - multi-modal large model processing - audio output.

[0092] Therefore, in order to interact with the background through the intelligent glasses terminal, the intelligent agent requested by the intelligent glasses terminal needs to be started, and the workflow associated with the intelligent agent needs to be obtained.

[0093] S303, receiving the notification sent by the intelligent agent control server after the intelligent agent joins the communication room and the identification of the communication room.

[0094] In order to enable the agent control server to know that the agent has joined the communication room, and can further inform the business server. Therefore, the agent synchronizes the information of joining the communication room to the agent control server after joining the communication room. The agent control server sends the notification of the agent joining the communication room and the identification of the communication room to the business server after receiving the information of the agent joining the communication room.

[0095] S304, sending feedback information to the intelligent glasses terminal based on the identification of the communication room, so that the intelligent glasses terminal joins the communication room.

[0096] Specifically, the business server feeds back the related information of the room to the intelligent glasses terminal based on the identification of the communication room, so that the intelligent glasses terminal can use the information to join the communication room, so as to join the intelligent glasses terminal and the agent in the same room, so that the intelligent glasses terminal and the agent can interact in real time through the communication room.

[0097] Optionally, in another embodiment of the present application, a specific implementation of step S304 comprises:

[0098] Generating token data of the communication room based on the identification of the communication room, and feeding back the token data of the communication room to the intelligent glasses terminal, so that the intelligent glasses terminal uses the token data of the communication room to join the communication room.

[0099] Specifically, after the business server receives the notification of the agent joining the communication room, it generates a corresponding token based on the identification of the communication room, and then feeds back the token to the intelligent glasses terminal. The intelligent glasses terminal uses the token to verify and join the communication room.

[0100] S202, arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information.

[0101] It should be noted that the collected multi-dimensional state information may not be directly input into the model for analysis, or the data that is really needed is data that needs to be further processed. Therefore, the current multi-dimensional state information is arranged into structured information by the business server. For example, the trajectory prediction is obtained by using the environment map and the pose to do data trajectory prediction, which is used for subsequent analysis. Or the current collected information is summarized with the historical collected information.

[0102] Optionally, considering that previous data may also be used when analyzing the intention, that is, the context is combined for analysis, the current multi-dimensional state information needs to be stored.

[0103] Optionally, in another embodiment of the present application, a further processing method for historical information is further included, such as Figure 4As shown, comprising the following steps:

[0104] S401, obtaining historical communication information of the agent.

[0105] Specifically, the agent control server can store the communication data of the agent each time, which specifically includes the interaction information of the agent and the intelligent glasses terminal and the multi-dimensional state information. Therefore, the required historical communication information can be obtained from the database.

[0106] S402, analyzing and extracting the historical communication information to obtain current analysis data and current extraction data.

[0107] S403, converting the current analysis data and the current extraction data into structured information, and supplementing it to the current multi-dimensional assistance information.

[0108] S203, sending the current multi-dimensional assistance information to the agent control server to transmit the current multi-dimensional assistance information to the currently started agent, and the agent analyzes the current interaction information of the user sent by the intelligent glasses terminal in real time based on the current multi-dimensional assistance information according to the associated workflow, obtains the current analysis result and feeds back to the intelligent glasses terminal.

[0109] Specifically, the user can send interaction information at any time through the intelligent glasses terminal. The interaction information can be in the form of text, audio, video, and image. At this time, the intelligent glasses terminal sends the current interaction information of the user to the agent through the communication room. At this time, the agent selects the workflow corresponding to the format of the current interaction information from its associated workflow, and calls the model to analyze the current interaction information of the user sent by the intelligent glasses terminal in real time based on the current multi-dimensional assistance information according to the selected workflow, obtains the current analysis result and feeds back to the intelligent glasses terminal.

[0110] Optionally, the model can include a large language model or a multi-model large model. Specifically, the corresponding model can be called according to the workflow, and the current interaction information is analyzed by using the prompt information generated based on the current multi-dimensional assistance information to obtain the current intention of the user. Then, based on the current multi-dimensional assistance information and the user's intention, the information that needs to be fed back, i.e. the current analysis result, is adaptively generated. The agent executes the corresponding task according to the workflow and completes the task, and the model generates the current analysis result and feeds back to the intelligent glasses terminal through the communication room in real time. The intelligent glasses terminal displays the current analysis result to the user.

[0111] The application provides a mixed reality interaction method, current multi-dimension state information is collected by a smart glasses terminal, and the current multi-dimension state information is sent to a business service end, then the current multi-dimension state information is arranged into structured information by the business service end, current multi-dimension assistance information is obtained, so that the multi-dimension information can be used for subsequent analysis after processing. The business service end sends the current multi-dimension assistance information to an intelligent agent control service end, so that the current multi-dimension assistance information is transmitted to a current started intelligent agent by the intelligent agent control service end. The smart glasses terminal sends current interaction information of a user to the intelligent agent in real time, and the intelligent agent analyzes the current interaction information of the user sent by the smart glasses terminal in real time based on the current multi-dimension assistance information according to an associated work flow, obtains a current analysis result and feeds back to the smart glasses terminal. Therefore, the current interaction information is analyzed based on multi-dimension state information by the corresponding intelligent agent and work flow, the real intention of the user can be more accurately analyzed, and accurate feedback information is generated.

[0112] Another embodiment of the application provides a business service end, as shown in the figure, comprising: Figure 5

[0113] A multi-dimension information acquisition unit 501 is configured to acquire multi-dimension state information collected by a smart glasses terminal.

[0114] A multi-dimension information processing unit 502 is configured to arrange the multi-dimension state information into structured information, and obtain current multi-dimension assistance information.

[0115] A multi-dimension information sending unit 503 is configured to send the current multi-dimension assistance information to an intelligent agent control service end, so that the current multi-dimension assistance information is transmitted to a current started intelligent agent, the intelligent agent analyzes current interaction information of a user sent by the smart glasses terminal in real time based on the current multi-dimension assistance information according to an associated work flow, obtains a current analysis result and feeds back to the smart glasses terminal.

[0116] Optionally, in the business service end provided by another embodiment of the application, further comprising:

[0117] A request receiving unit is configured to receive a join communication room request sent by the smart glasses terminal.

[0118] A start request unit is configured to request the intelligent agent control service end to start an intelligent agent and add it to a communication room in response to the join communication room request.

[0119] A notification receiving unit is configured to receive a notification and an identifier of the communication room sent by the intelligent agent control service end after the intelligent agent is added to the communication room.

[0120] ​The feedback unit is configured to send feedback information to the smart glasses terminal based on the identification of the communication room, so that the smart glasses terminal joins the communication room.

[0121] Optionally, in the service server provided in another embodiment of the present application, the starting request unit comprises:

[0122] The starting request sub-unit is configured to send a request for starting the agent to the agent control service server by using the identification of the agent that requests to join the communication room, so as to trigger the agent control service server to start or create a corresponding agent according to the identification of the agent, obtain a workflow associated with the agent, and add the agent to the newly created communication room.

[0123] Optionally, in the service server provided in another embodiment of the present application, the feedback unit comprises:

[0124] The token generation unit is configured to generate token data of the communication room based on the identification of the communication room.

[0125] The token feedback unit is configured to feed back the token data of the communication room to the smart glasses terminal, so that the smart glasses terminal joins the communication room by using the token data of the communication room.

[0126] Optionally, in the service server provided in another embodiment of the present application, the multi-dimensional information acquisition unit comprises:

[0127] The multi-dimensional information acquisition sub-unit is configured to acquire current scene information and current glasses state information currently collected by the smart glasses terminal. The current scene information at least includes a current field of view environment image, a current geographic position coordinate, and a POI point within a set range of the smart glasses terminal. The current glasses state information at least includes a current pose of the smart glasses terminal.

[0128] Optionally, in the service server provided in another embodiment of the present application, further comprising:

[0129] The monitoring data acquisition unit is configured to acquire multi-dimensional monitoring data currently collected by the external device and synchronized to the smart glasses terminal from the smart glasses terminal.

[0130] The adding unit is configured to add the current multi-dimensional monitoring data to the currently acquired multi-dimensional state information.

[0131] Optionally, in the service server provided in another embodiment of the present application, further comprising:

[0132] The historical information acquisition unit is configured to acquire historical communication information of the agent.

[0133] The historical information analysis unit is configured to analyze and extract the historical communication information to obtain current analysis data and current extraction data.

[0134] a supplement unit, configured to transform the current analysis data and the current extraction data into structured information, and supplement the structured information into the current multi-dimensional assistance information.

[0135] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can be correspondingly referred to the implementation process of the corresponding steps in the above method embodiments, which will not be described here.

[0136] Another embodiment of the present application provides an electronic device, as shown in the accompanying drawings, comprising: Figure 6

[0137] a memory 601 and a processor 602.

[0138] The memory 601 is configured to store a program.

[0139] The processor 602 is configured to execute the program stored in the memory 601, and the program, when executed, is specifically configured to implement the mixed reality interaction method provided in any one of the above embodiments.

[0140] Another embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is configured to implement the mixed reality interaction method provided in any one of the above embodiments.

[0141] The computer storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0142] ​Those skilled in the art will further realize that the mechanisms of the various examples described herein are capable of being implemented using any number of combinations of the described features. Accordingly, these examples are not limited to the mechanisms described herein, but rather, the intent is to cover all modifications and alternatives equivalent thereto. The preceding description of the examples is illustrative, and not restrictive. Many other examples will be apparent to those of skill in the art upon reviewing the above description. The scope of the examples should, therefore, be determined not with reference to the above description, but instead should be given to the appended claims, along with their full scope of equivalents.

[0143] The above description of disclosed examples is intended to be illustrative, and not restrictive. Many other examples will be apparent to those of skill in the art upon reviewing the above description. The scope of examples should, therefore, be determined not with reference to the above description, but instead should be given to the appended claims, along with their full scope of equivalents.

Claims

1. A mixed reality interaction method, characterized by, The method comprises the following steps: obtaining current scene information and current glasses state information collected by the intelligent glasses terminal, and taking the current scene information and the current glasses state information as multi-dimensional state information; wherein the current scene information at least includes a current field of view environment image, a current geographic position coordinate and a POI point within a set range of the intelligent glasses terminal; the current glasses state information at least includes a current pose of the intelligent glasses terminal; arranging the multi-dimensional state information into structured information to obtain current multi-dimensional assistance information; sending the current multi-dimensional assistance information to an intelligent agent control server to pass the current multi-dimensional assistance information to a currently started intelligent agent, and selecting, by the intelligent agent, a workflow corresponding to a format of current interaction information from a workflow associated with the intelligent agent, and calling a model to analyze current interaction information of a user sent by the intelligent glasses terminal in real time based on the current multi-dimensional assistance information according to the workflow to obtain a current analysis result and feed back to the intelligent glasses terminal.

2. The method of claim 1, wherein, The method further comprises the following steps: receiving a request for joining a communication room sent by the intelligent glasses terminal; in response to the request for joining the communication room, requesting the intelligent agent control server to start an intelligent agent and add it to the communication room; receiving a notification sent by the intelligent agent control server after the intelligent agent joins the communication room and an identifier of the communication room; based on the identifier of the communication room, sending feedback information to the intelligent glasses terminal to make the intelligent glasses terminal join the communication room.

3. The method of claim 2, wherein, The step of, in response to the request for joining the communication room, requesting the intelligent agent control server to start an intelligent agent and add it to the communication room, comprises the following steps: using an identifier of an intelligent agent requesting to join the communication room to send a request for starting the intelligent agent to the intelligent agent control server to trigger the intelligent agent control server to start or create a corresponding intelligent agent according to the identifier of the intelligent agent, obtain a workflow associated with the intelligent agent, and add the intelligent agent to a newly created communication room.

4. The method of claim 2, wherein, The step of, based on the identifier of the communication room, sending feedback information to the intelligent glasses terminal to make the intelligent glasses terminal join the communication room, comprises the following steps: generating token data of the communication room based on the identifier of the communication room; feeding back the token data of the communication room to the intelligent glasses terminal to make the intelligent glasses terminal join the communication room using the token data of the communication room.

5. The method of claim 1, wherein, The method further comprises the following steps: obtaining, from the intelligent glasses terminal, multi-dimensional monitoring data currently collected by an external device and synchronized to the intelligent glasses terminal; adding the current multi-dimensional monitoring data to the multi-dimensional state information currently obtained.

6. The method of claim 1, wherein, The method further comprises the following steps: obtaining historical communication information of the intelligent agent; analyzing and extracting the historical communication information to obtain current analysis data and current extraction data; converting the current analysis data and the current extraction data into structured information and supplementing them into the current multi-dimensional assistance information.

7. A mixed reality interactive system, characterized by The method comprises the following steps: an intelligent glasses terminal, a business server, an intelligent agent control server and an intelligent agent; The intelligent glasses terminal is configured to collect current scene information and current glasses state information collected by the intelligent glasses terminal, and send the collected current scene information and current glasses state information as multi-dimensional state information to the service server, and send current interaction information of a user to the intelligent agent in real time, wherein the current scene information at least includes a current field of view environment image, a current geographic position coordinate, and a POI point within a set range of the intelligent glasses terminal, and the current glasses state information at least includes a current pose of the intelligent glasses terminal. The service server is configured to arrange the acquired multi-dimensional state information into structured information to obtain current multi-dimensional assistance information, and send the current multi-dimensional assistance information to the intelligent agent control server. The intelligent agent control server is configured to start the intelligent agent, and deliver the current multi-dimensional assistance information to the started intelligent agent. The intelligent agent is configured to select a workflow corresponding to a format of the current interaction information from an associated workflow, and analyze the current interaction information of the user based on the current multi-dimensional assistance information according to the workflow to obtain a current analysis result and feed back to the intelligent glasses terminal.

8. An electronic device, comprising: The method comprises: a memory and a processor; The memory is configured to store a program; The processor is configured to execute the program, and when the program is executed, specifically used to implement the mixed reality interaction method of any one of claims 1 to 6.

9. A computer storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the mixed reality interaction method of any one of claims 1 to 6 is implemented.

Images