Multi-user collaborative interaction method and device

By using behavior prediction models to predict and interact with the behavior of downline users in a mixed reality environment, the lack of time asynchronous interaction design is solved, collaborative interaction and information transmission in different time periods are realized, and efficiency and User experience.

CN119024955APending Publication Date: 2024-11-26BEIJING INSTITUTE FOR GENERAL ARTIFICIAL INTELLIGENCE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310596008.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, multi-person interaction in mixed reality environments mainly consider spatial synchronization, lack of effective time asynchronous interaction design, and cannot effectively solve the problem of multi-person interaction efficiency during different working hours.

Method used

By monitoring the target mixed reality environment, calling the user's target behavior prediction model, predicting the behavior of downline users, controlling the virtual avatar to perform corresponding behaviors, interacting with other users, and updating the response information when the user re-onlines, realizing Time asynchronous collaborative interaction.

Benefits of technology

It realizes coordinated interaction and information transmission in a mixed reality environment for users at different time periods, improves the efficiency and work efficiency of coordinated interaction, and ensures uninterrupted information transmission and reception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024955A_ABST
    Figure CN119024955A_ABST
Patent Text Reader

Abstract

The invention provides a multi-user cooperative interaction method and device.The multi-user cooperative interaction method comprises the steps that a target mixed reality environment is monitored, multiple users in the target mixed reality environment interact through respective intelligent agents and / or the users themselves, and the intelligent agents are virtualized agents or agents with bodies; if it is monitored that a first user in the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user, and obtaining first predicted behavior data; and according to the first predicted behavior data, controlling a virtual avatar of the first user to execute a corresponding behavior, and interacting with an intelligent agent of at least one second user in the plurality of users and / or the user. A plurality of users can perform collaborative interaction and information transmission in the same mixed reality environment asynchronously, collaborative interaction in different time dimensions is realized, and the efficiency of collaborative interaction is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and in particular to a multi-user collaborative interaction method and device. Background Art

[0002] Mixed reality technology has developed rapidly since the 1950s. Currently, more and more users are working and interacting in a shared mixed reality collaborative space or environment. Multiple users can perform real-time interactive information transmission in a mixed reality environment, such as assigning tasks and collaborative discussions.

[0003] However, the existing technology for multi-person interaction in a mixed reality environment mainly considers spatial synchronization, that is, multiple users are in the same mixed reality environment at the same time to complete tasks such as teaching, collaboration, and social interaction; but the design of temporal asynchronous interaction is lacking. There is currently no good solution for multi-person interaction in different working hours and time zones. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0004] In view of the problems existing in the prior art, the embodiments of the present invention provide a multi-user collaborative interaction method and device.

[0005] The present invention provides a multi-user collaborative interaction method, comprising:

[0006] Monitoring a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0007] If it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user to obtain first predicted behavior data;

[0008] According to the first predicted behavior data, the virtual avatar of the first user is controlled to perform corresponding behavior and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0009] According to a multi-user collaborative interaction method provided by the present invention, after controlling the first user's virtual avatar to perform a corresponding behavior according to the first predicted behavior data, the method further includes:

[0010] When it is detected that the first user is back online, obtaining first current behavior data corresponding to the first user;

[0011] According to the first current behavior data, the intelligent agent of the first user is controlled to perform corresponding behavior, and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0012] According to a multi-user collaborative interaction method provided by the present invention, after controlling the first user's virtual avatar to perform a corresponding behavior according to the first predicted behavior data, the method further includes:

[0013] When it is detected that the first user is back online, displaying response information in the first predicted behavior data to the first user, the response information being used to respond to interactive information sent by the intelligent agent of the second user and / or the user himself;

[0014] receiving an update instruction from the first user, and updating the response information according to the update instruction;

[0015] The updated response information is transmitted to the second user.

[0016] According to a multi-user collaborative interaction method provided by the present invention, the transmitting the updated response information to the second user includes:

[0017] If the second user is in an offline state, cache the updated response information;

[0018] When the second user comes back online, the updated response information is transmitted to the second user.

[0019] According to a multi-user collaborative interaction method provided by the present invention, after the response information is updated according to the update instruction, the method further includes:

[0020] The target behavior prediction model is updated according to the updated response information.

[0021] According to a multi-user collaborative interaction method provided by the present invention, if it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user, before obtaining the first predicted behavior data, the method further includes:

[0022] When the first user is online, obtaining second current behavior data of the first user;

[0023] The target behavior prediction model is updated according to the second current behavior data.

[0024] According to a multi-user collaborative interaction method provided by the present invention, updating the target behavior prediction model according to the second current behavior data includes:

[0025] determining whether the second current behavior data only includes the behavior data of the first user;

[0026] If not, the behavior data of the first user in the second current behavior data is filtered out, and the target behavior prediction model is updated according to the behavior data of the first user.

[0027] According to a multi-user collaborative interaction method provided by the present invention, updating the target behavior prediction model according to the second current behavior data includes:

[0028] determining whether the second current behavior data is the behavior data of the first user;

[0029] If so, the target behavior prediction model is updated according to the second current behavior data.

[0030] According to a multi-user collaborative interaction method provided by the present invention, updating the target behavior prediction model according to the second current behavior data includes:

[0031] Acquire first current modality data in the target mixed reality environment;

[0032] Inputting the first current modality data into the target behavior prediction model to obtain second predicted behavior data;

[0033] According to the difference between the second current behavior data and the second predicted behavior data, the model parameters of the target behavior prediction model are adjusted to obtain the updated target behavior prediction model.

[0034] According to a multi-user collaborative interaction method provided by the present invention, before controlling the first user's virtual avatar to perform a corresponding behavior according to the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes:

[0035] The offline information of the first user is fed back to online users among the multiple users.

[0036] According to a multi-user collaborative interaction method provided by the present invention, after controlling the first user's virtual avatar to perform corresponding behavior according to the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes:

[0037] If it is detected that the multiple users are all offline, for each user, calling the target behavior prediction model corresponding to the user to predict the behavior of the user to obtain third predicted behavior data;

[0038] According to the third predicted behavior data, the user's virtual avatar is controlled to perform a corresponding behavior.

[0039] According to a multi-user collaborative interaction method provided by the present invention, if it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user, before obtaining the first predicted behavior data, the method further includes:

[0040] If it is detected that the first user is online for the first time, an initial behavior prediction model is allocated to the first user;

[0041] Acquire third current behavior data of the first user and second current modality data in the target mixed reality environment;

[0042] Inputting the second current modality data into the initial behavior prediction model to obtain fourth predicted behavior data;

[0043] According to the difference between the third current behavior data and the fourth predicted behavior data, the model parameters of the initial behavior prediction model are adjusted to obtain a target behavior prediction model.

[0044] According to a multi-user collaborative interaction method provided by the present invention, the step of assigning an initial behavior prediction model to the first user includes:

[0045] Acquire a general behavior prediction model and behavior pattern data of the first user, wherein the behavior pattern data is user behavior data and / or preference data uploaded by the first user;

[0046] The model parameters of the general behavior prediction model are adjusted according to the behavior pattern data to obtain an initial behavior prediction model corresponding to the first user.

[0047] The present invention also provides a multi-user collaborative interaction method, comprising:

[0048] Monitoring a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0049] If it is monitored that the intelligent agent and / or the user of the second user sends interaction information to the first user, the virtual avatar of the first user is called to feedback response information to the intelligent agent and / or the user of the second user for the interaction information, the first user is any user in the multiple users who is offline, and the second user is any user in the multiple users except the first user;

[0050] When the first user comes back online, an update instruction from the first user for the response information is received, the response information is updated according to the update instruction, and the updated response information is transmitted to the second user.

[0051] According to a multi-user collaborative interaction method provided by the present invention, calling the first user's virtual avatar to feed back response information to the second user's intelligent agent and / or the user himself regarding the interaction information includes:

[0052] calling a target behavior prediction model corresponding to the first user to determine response information for the interaction information;

[0053] The virtual avatar of the first user is controlled to feed back the response information to the intelligent agent of the second user and / or the user himself.

[0054] According to a multi-user collaborative interaction method provided by the present invention, before transmitting the updated response information to the second user, the method further includes:

[0055] If the second user is offline when the virtual avatar of the second user sends the interaction information, the interaction information is displayed to the second user.

[0056] The present invention also provides a multi-user collaborative interaction device, comprising:

[0057] A first monitoring module is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0058] The prediction module is configured to, if it is detected that a first user among the multiple users is offline, call a target behavior prediction model corresponding to the first user to predict the behavior of the first user and obtain first predicted behavior data;

[0059] The first control module is configured to control the virtual avatar of the first user to perform corresponding behavior according to the first predicted behavior data, and interact with the intelligent agent of at least the second user among the multiple users and / or the user himself.

[0060] The present invention also provides a multi-user collaborative interaction device, comprising:

[0061] A second monitoring module is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0062] A first feedback module is configured to, if monitoring that the intelligent agent and / or the user of the second user sends interaction information to the first user, call the virtual avatar of the first user to feedback response information to the intelligent agent and / or the user of the second user, wherein the first user is any user in the multiple users who is in an offline state, and the second user is any user among the multiple users except the first user;

[0063] The first transmission module is configured to receive an update instruction from the first user for the response information when the first user comes back online, update the response information according to the update instruction, and transmit the updated response information to the second user.

[0064] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the multi-user collaborative interaction methods described above is implemented.

[0065] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the multi-user collaborative interaction method described in any one of the above methods is implemented.

[0066] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-user collaborative interaction method as described above is implemented.

[0067] The multi-user collaborative interaction method and device provided by the present invention monitors the target mixed reality environment, and when the first user among multiple users is detected to be offline, calls the target behavior prediction model corresponding to the first user to predict the behavior of the first user, obtains the first predicted behavior data, and controls the virtual avatar of the first user to perform the corresponding behavior according to the first predicted behavior data, and interacts with the intelligent agent and / or the user himself of at least one second user among multiple users, so that users in different time periods (partially online and partially offline) can interact with multiple users in different time periods through behavior prediction based on the virtual avatar and the target behavior prediction model, such as a user can interact with past or future users and transmit information, so that multiple users can "not simultaneously" perform collaborative interaction and information transmission in the same mixed reality environment, realize collaborative interaction in different time dimensions, and improve the efficiency of collaborative interaction. In addition, through the virtual avatar and the target behavior prediction model, users can arrange their time in the mixed reality environment more flexibly, and realize uninterrupted information sending and receiving, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 This is one of the flow charts of the multi-user collaborative interaction method provided by the present invention;

[0070] Figure 2 This is the second flow chart of the multi-user collaborative interaction method provided by the present invention;

[0071] Figure 3 This is the third flow chart of the multi-user collaborative interaction method provided by the present invention;

[0072] Figure 4 This is the fourth flow chart of the multi-user collaborative interaction method provided by the present invention;

[0073] Figure 5 This is the fifth flow chart of the multi-user collaborative interaction method provided by the present invention;

[0074] Figure 6 This is the sixth flow chart of the multi-user collaborative interaction method provided by the present invention;

[0075] Figure 7 It is one of the structural schematic diagrams of the multi-user collaborative interaction device provided by the present invention;

[0076] Figure 8 This is the second structural diagram of the multi-user collaborative interaction device provided by the present invention;

[0077] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] In order to facilitate a clearer understanding of the embodiments of the present invention, some relevant background knowledge is first introduced as follows.

[0080] In recent years, with the increasing amount of data and stronger computing power, artificial intelligence has made great progress in behavior prediction. User behavior prediction technology predicts user behavior based on behavior prediction input information, user attribute data, current environment data, and attribute data of the object of behavior execution. In this way, the user's past behavior habits and behavior rules can be fitted to the user's behavior model, and the user's future behavior can be predicted.

[0081] Mixed reality technology has developed rapidly since the 1950s. Currently, more and more users are working and interacting in a shared mixed reality collaborative space or environment. Mixed reality environments mainly include virtual reality environments and augmented reality environments. In a virtual reality environment, multiple users' avatars can communicate, cooperate and interact in a virtual environment at the same time; in an augmented reality environment, the user's body or its virtual avatar can communicate, cooperate and interact in a physical environment. Multiple users can perform real-time interactive information transmission in a mixed reality environment, such as assigning tasks, collaborative discussions, etc.

[0082] Time-space synchronized mixed reality means that multiple different users enter the same mixed reality environment at the same time period for real-time interaction and information transfer. The time-space synchronized mixed reality environment allows multiple users in different locations to appear in the same mixed reality environment, providing support for multi-user information transfer and interaction, including interaction with virtual avatars, multi-user remote interaction, etc.

[0083] However, the existing technology for multi-person interaction in a mixed reality environment mainly considers spatial synchronization, that is, multiple users are in the same mixed reality environment at the same time to complete teaching, collaboration, social and other tasks; but the design of temporal asynchronous interaction is lacking. There is currently no good solution for multi-person interaction in different working hours and time zones. In addition, the behavior prediction model is not actually converted to the virtual body, which makes it difficult to use for collaborative interaction and inconvenient to improve the efficiency of interaction between users in different working time periods. In order to solve the above problems, the present invention provides a multi-user collaborative interaction method and device to solve the problem of inconsistent online time of multiple users in a mixed reality environment and improve the efficiency of interaction.

[0084] Combine the following Figure 1-Figure 8 The multi-user collaborative interaction method and device of the present invention are described.

[0085] Figure 1 This is one of the flow charts of the multi-user collaborative interaction method provided by the present invention, see Figure 1 As shown, it includes steps 101 to 103, wherein:

[0086] Step 101: Monitor a target mixed reality environment, in which multiple users interact through their own intelligent agents and / or the users themselves, and the intelligent agents are virtual avatars or embodied intelligent entities.

[0087] Users can be natural persons or artificial intelligence virtual persons. Virtual persons have intelligent activities and corresponding reasoning and perception abilities, and can have a variety of specific characteristics such as personality. Virtual persons can be derived from humans, animals, or preset values.

[0088] First of all, it should be noted that the execution subject of the present invention can be any electronic device or user himself that performs multi-user collaborative interaction. The electronic device can be any one of a smart phone, a smart watch, a desktop computer, a laptop computer, a VR reality device, a mixed display device, etc. The present invention can be implemented by equipping one or more total computing terminals (execution subjects) and each user with a corresponding mixed reality device.

[0089] Specifically, the target mixed reality environment is any mixed reality environment. A virtual avatar can refer to a simulated image of a user, which can be a human image, an animal image, an image of daily necessities, etc. It can also be a pure artificial intelligence robot, which has intelligent activities and corresponding reasoning and perception capabilities, and can have a variety of specific characteristics such as personality. An embodied intelligent body, that is, embodied intelligence or an intelligent body (which can be biological or mechanical), interacts with the environment and, through its own learning, develops the ability to understand and transform the objective world.

[0090] In actual applications, there may be multiple mixed reality environments, and an execution subject may monitor and perform other operations on at least one mixed reality environment. For each mixed reality environment, the steps of the multi-user collaborative interaction method provided by the present invention may be performed separately.

[0091] Multiple users can interact in the target mixed reality environment (any mixed reality environment). For each user, the user can interact with other users through a virtual avatar, can interact with other users through an embodied intelligent agent, and the user himself can also interact with other users.

[0092] In addition, in the target mixed reality environment, different users can interact with other users in different ways, for example, some users interact through intelligent agents, while other users interact based on the users themselves; different users can also interact with other users in the same way, for example, all users interact through intelligent agents.

[0093] Step 102: If it is monitored that a first user among the multiple users is offline, a target behavior prediction model corresponding to the first user is called to predict the behavior of the first user to obtain first predicted behavior data.

[0094] Specifically, the behavior prediction model is a model that predicts user behavior based on user behavior prediction technology, and the algorithm of the behavior prediction model can be but is not limited to the Long Short Term Memory (LSTM) algorithm, the GPT algorithm, the GPT-4 algorithm, the first-order or multi-order Markov decision process algorithm, etc. The target behavior prediction model is a behavior prediction model that is trained based on at least one type of data such as the behavior, preference, and habit of the first user.

[0095] In actual applications, any user interacting in the target mixed reality environment may go offline due to user needs (such as handling an emergency) or unexpected failures (such as power outages). At this time, the user is the first user.

[0096] The execution entity monitors the online status of each user in the target mixed reality environment, for example, monitoring the online status of the user through the user's online flag. If the user's online flag is 1, it means that the user is online, and if the user's online flag is 0, it means that the user is offline. For example, parsing the status information sent by the user, when the status information sent by the user is received as login information, it means that the user is online after the current time, until the status information sent by the user is received again, and the status information received again is offline information. Correspondingly, when the status information sent by the user is received as offline information, it means that the user is offline after the current time, until the status information sent by the user is received again, and the status information received again is login information.

[0097] Furthermore, during the time when the first user is monitored to be offline, the target behavior prediction model corresponding to the first user can be called according to the user identifier of the first user, and the behavior of the first user can be predicted according to the environmental data in the target mixed reality environment, such as the interactive behavior data sent by other users and / or the scene data of the target mixed reality environment, to obtain first predicted behavior data.

[0098] It should be noted that each user in the target mixed reality environment corresponds to a target behavior prediction model, that is, there is a one-to-one correspondence between the target behavior prediction model and the user, that is, the target behavior prediction model is specific.

[0099] Step 103: Based on the first predicted behavior data, control the virtual avatar of the first user to perform corresponding behavior, and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0100] Specifically, the second user refers to any user other than the first user in the target mixed reality environment.

[0101] In actual applications, the first user who is offline cannot interact with other users in the target mixed reality environment through the user himself and the embodied intelligent body. In order to ensure that the first user can also interact with other users, on the basis of obtaining the first predicted behavior data, the first user's virtual avatar is further controlled based on the first predicted behavior data, so that the first user's virtual avatar performs corresponding behaviors or operations and interacts with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0102] It should be noted that each user can set an avatar, that is, set a favorite avatar as his or her own avatar: for example, uploading the avatar that the user wants; or selecting the avatar that the user wants from the avatar library; or after determining the desired avatar, adjusting some areas of the avatar, such as adding ornaments, changing hair color, and adjusting size, etc. When the user does not set an avatar, the default avatar is used as the avatar of the user.

[0103] The target behavior prediction model is generated online in real time based on the real user's past behavior norms, behavior habits, preferences and other data. When the real user leaves the mixed reality environment, the virtual avatar with the target behavior prediction model as the behavior data input can become the real user's virtual avatar in the mixed reality environment, replacing the currently absent real user and continuing to perform corresponding behaviors in the mixed reality environment based on the real user's behavior norms, behavior habits and preferences.

[0104] The multi-user collaborative interaction method provided by the present invention monitors the target mixed reality environment, and when the first user among multiple users is detected to be offline, calls the target behavior prediction model corresponding to the first user to predict the behavior of the first user, obtains the first predicted behavior data, and controls the virtual avatar of the first user to perform the corresponding behavior according to the first predicted behavior data, and interacts with the intelligent agent and / or the user himself of at least one second user among multiple users, so that users in different time periods (partially online and partially offline) can interact with multiple users in different time periods through behavior prediction based on the virtual avatar and the target behavior prediction model, such as a user can interact with past or future users and transmit information, so that multiple users can "not simultaneously" perform collaborative interaction and information transmission in the same mixed reality environment, realize collaborative interaction in different time dimensions, and improve the efficiency of collaborative interaction. In addition, through the virtual avatar and the target behavior prediction model, users can arrange their time in the mixed reality environment more flexibly, and realize uninterrupted information sending and receiving, thereby improving work efficiency.

[0105] In one or more optional embodiments of the present invention, after the first user comes online again, the first user can interact with the intelligent agent of at least one second user and / or the user himself based on the first predicted behavior data of the first user. That is, after controlling the virtual avatar of the first user to perform a corresponding behavior according to the first predicted behavior data, the method further includes:

[0106] When it is detected that the first user is back online, obtaining first current behavior data corresponding to the first user;

[0107] According to the first current behavior data, the intelligent agent of the first user is controlled to perform corresponding behavior, and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0108] Specifically, the current behavior data refers to data corresponding to the user's current behavior, such as squatting, waving, speaking, and blinking, etc. The first current behavior data refers to the current behavior data of the user after he or she comes back online.

[0109] In actual applications, since the first user is back online, the first user has the corresponding first current behavior data. At this time, the method of using the target behavior prediction model to achieve collaborative interaction is converted to achieving collaborative interaction based on the first current behavior data.

[0110] After the first user comes back online, first current behavior data corresponding to the first user is received, such as obtaining the first current behavior data through at least one of various sensors, inputs, and instructions, etc. Then, the first current behavior data is used to control the intelligent agent of the first user, so that the intelligent agent of the first user performs corresponding behaviors or operations, and interacts with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0111] In this way, after the user comes back online, he or she can interact through the user's first current behavior data to achieve a change in the interaction mode, making the collaborative interaction more humane and closer to the user behavior, thereby improving the reliability of the collaborative interaction and user stickiness.

[0112] In one or more optional embodiments of the present invention, after the first user comes online again, the first user can interact with at least one second user's intelligent agent and / or the user himself based on the interaction settings of the first user. In this way, the interaction mode after going online again can be set based on the user's preferences, which can improve user stickiness.

[0113] Exemplarily, if the interaction of the first user is set to be performed through an intelligent agent, the intelligent agent can be controlled to interact based on the behavior data of the first user. That is, after controlling the first user's virtual avatar to perform corresponding behaviors according to the first predicted behavior data, it also includes: when detecting that the first user is back online, obtaining the first current behavior data corresponding to the first user; according to the first current behavior data, controlling the first user's intelligent agent to perform corresponding behaviors, and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself. In this way, after the user is back online, the user interacts through the user's first current behavior data to achieve a conversion of the interaction mode, making the collaborative interaction more humane and closer to the user's behavior, thereby improving the reliability of the collaborative interaction and user stickiness.

[0114] Exemplarily, if the interaction of the first user is set to be performed through the user himself, the first user can interact through the user himself. That is, after controlling the first user's virtual avatar to perform the corresponding behavior according to the first predicted behavior data, it also includes: when it is detected that the first user is back online, the first user interacts with the intelligent agent of at least one second user among the multiple users and / or the user himself through the user's own behavior. In this way, after the user is back online, the user interacts through the user himself to realize the conversion of the interaction mode, making the collaborative interaction more humane and closer to the user behavior, thereby improving the reliability and user stickiness of the collaborative interaction.

[0115] Optionally, after the first user comes back online, the target behavior prediction model can be used to replace the response made by the first user before the first user comes online and displayed to the first user, who can make modifications and adjustments and then feedback to the second user. That is, after controlling the first user's avatar to perform the corresponding behavior according to the first predicted behavior data, the method further includes:

[0116] When it is detected that the first user is back online, displaying response information in the first predicted behavior data to the first user, the response information being used to respond to interactive information sent by the intelligent agent of the second user and / or the user himself;

[0117] receiving an update instruction from the first user, and updating the response information according to the update instruction;

[0118] The updated response information is transmitted to the second user.

[0119] Specifically, the update instruction refers to the instruction generated by the first user performing an update operation such as adjusting and modifying the response information. Interaction information refers to information generated by a user asking another user questions or communicating with another user, and response information is feedback on the interaction information. For example, if the interaction information is the sentence "What do you like to do in your daily life", the response information may be the sentence "I like to listen to music and read books in my daily life".

[0120] In actual applications, when the first user is offline, the target behavior prediction model of the first user predicts response information, thereby responding to the interactive information sent by the second user's intelligent agent and / or the user himself. However, since the target behavior prediction model may be inconsistent with the actual response of the first user, after the first user is back online, the response information is displayed to the first user. When the response information is inconsistent with the first user's actual response to the interactive information, the first user adjusts and modifies the response information and other update operations. Accordingly, the execution body structure receives the update instruction, then updates the response information, obtains the updated response information, and then transmits the updated response information to the second user.

[0121] In this way, by displaying the response information in the first predicted behavior data to the first user after the first user comes back online, and then updating the response information to ensure that the second user obtains the correct response information, the efficiency and reliability of the interaction can be improved, thereby improving user satisfaction.

[0122] Optionally, after obtaining the updated response information, the updated response information may be directly transmitted to the second user, so that the second user can view the correct response information in a timely manner.

[0123] Optionally, the updated response information is transmitted to the second user, and a specific implementation process may be as follows:

[0124] If the second user is in an offline state, cache the updated response information;

[0125] When the second user comes back online, the updated response information is transmitted to the second user.

[0126] Specifically, the offline state is also the offline state.

[0127] In actual applications, when the updated response information is obtained, it can be determined whether the second user is offline: if not, the updated response information is directly delivered to the second user for the convenience of the second user to view; if so, the updated response information is not delivered first, but the updated response information is cached first, and the updated response information is delivered to the second user after the second user is back online. In this way, when the second user is in an offline state, the updated response information is cached first, which can prevent the prediction behavior model corresponding to the second user from using the updated response information as interaction information and performing erroneous interaction, thereby ensuring the smooth progress of the interaction, improving the reliability of the interaction, and thus improving user satisfaction.

[0128] In addition, since the response information is updated, it indicates to a certain extent that the matching degree between the target behavior prediction model and the first user does not reach the ideal value. At this time, the target behavior prediction model can be updated based on the updated response information, that is, after the response information is updated according to the update instruction, it also includes:

[0129] The target behavior prediction model is updated according to the updated response information.

[0130] In practical applications, the updated response information can be used as a label, and the model parameters of the target behavior prediction model can be adjusted according to the difference between the updated response information and the response information before the update to achieve the update of the target behavior prediction model. This is conducive to improving the robustness and prediction accuracy of the target behavior prediction model.

[0131] In one or more optional embodiments of the present invention, when the first user is online, the target behavior prediction model can be updated based on the behavior data of the first user, that is, if it is monitored that the first user among the multiple users is offline, the target behavior prediction model corresponding to the first user is called to predict the behavior of the first user, and before obtaining the first predicted behavior data, the method further includes:

[0132] When the first user is online, obtaining second current behavior data of the first user;

[0133] The target behavior prediction model is updated according to the second current behavior data.

[0134] Specifically, the second current behavior data refers to the current behavior data before the first user goes offline, that is, the current behavior data of the first user each time when the first user is online.

[0135] In actual applications, when the first user is online, the first user interacts with at least one second user's intelligent agent and / or the user himself based on the first user's second current behavior data. At this time, the first user's target behavior prediction model can be in an idle state or a closed state. However, in order to improve the matching degree between the target behavior prediction model and the first user, that is, to improve the accuracy of the target behavior prediction model in predicting the first user's behavior, when the first user is online, the target behavior prediction model can be updated based on the second current behavior data, that is, the target behavior prediction model is called to predict the first user's behavior, and the model parameters of the target behavior prediction model are adjusted according to the difference between the predicted behavior data and the second current behavior data to achieve the update of the target behavior prediction model.

[0136] For example, see Figure 2 , Figure 2 This is the second flow chart of the multi-user collaborative interaction method provided by the present invention: Figure 2 The horizontal line in the middle represents the timeline of the user in the mixed reality environment. The solid line represents the time period when the user is "online" or "online" in the mixed reality environment, and the dotted line represents the time period when the user is "offline" or "offline" in the mixed reality environment. In the "online" state, that is, the real user performs a series of operations such as moving and interacting in the mixed reality environment based on the user's own will, among which, it is also necessary to follow the time period T n The target behavior prediction model is updated in real time with the behavior data of the real user in the mixed reality environment; in the "offline" state, such as when the real user is offline at time t1, the target behavior prediction model generates the first predicted behavior data, that is, generates the behavior of the virtual avatar at time t, and the virtual avatar performs the behavior, that is, the virtual avatar performs a series of operations such as movement and interaction in the mixed reality environment on behalf of the real user. Although the real user is offline from t1 to t2, the virtual avatar can still replace the real user in the mixed reality environment. When the real user goes online again at time t2, the real user continues to perform activities and interactions in the mixed reality environment.

[0137] It should be noted that there may be multiple users using the same user (first user) account to interact. For example, Xiao A's father and mother communicate with Xiao A through Xiao A's father's account. At this time, the second current behavior data contains the behavior data of multiple users, and the behavior data of the first user needs to be filtered out for updating the target behavior prediction model. That is, the target behavior prediction model is updated according to the second current behavior data. The specific implementation process can be as follows:

[0138] determining whether the second current behavior data only includes the behavior data of the first user;

[0139] If not, the behavior data of the first user in the second current behavior data is filtered out, and the target behavior prediction model is updated according to the behavior data of the first user.

[0140] In actual applications, when updating the target behavior prediction model based on the second current behavior data, first determine whether the second current behavior data only contains the behavior data of the first user: if so, it means that only the first user uses the first user's account to interact, and the target behavior prediction model can be directly updated based on the second current behavior data; if not, it means that multiple users use the first user's account to interact, and at this time, the behavior data of the first user in the second current behavior data is filtered out, and then the target behavior prediction model is updated based on the filtered behavior data of the first user.

[0141] In this way, by screening the behavior data of the first user in the second current behavior data, filtering out the behavior data of non-first users in the second current behavior data, and then updating the target behavior prediction model, it is possible to avoid the target behavior prediction model learning the behavioral habits of non-first users, resulting in a decrease in the matching degree between the target behavior prediction model and the first user, which is conducive to improving the prediction accuracy of the target behavior prediction model.

[0142] In addition, there may be other users who use the first user's account to interact, for example, Xiao A's mother communicates with Xiao A through Xiao A's father's account, so it is necessary to determine whether the second current behavior data is the behavior data of the first user. That is, the target behavior prediction model is updated according to the second current behavior data, and the specific implementation process can be as follows:

[0143] determining whether the second current behavior data is the behavior data of the first user;

[0144] If so, the target behavior prediction model is updated according to the second current behavior data.

[0145] In actual applications, when updating the target behavior prediction model based on the second current behavior data, first determine whether the second current behavior data is the behavior data of the first user: if so, it means that the first user uses the first user's account to interact, and the target behavior prediction model can be directly updated based on the second current behavior data; if not, it means that a non-first user uses the first user's account to interact, and there is no need to update the target behavior prediction model.

[0146] Specifically, determining whether the second current behavior data is the behavior data of the first user includes: obtaining the historical behavior data of the first user; calculating the similarity between the historical behavior data and the second current behavior data; if the similarity is greater than a similarity threshold, confirming that the second current behavior data is the behavior data of the first user, and if the similarity is less than or equal to a similarity threshold, confirming that the second current behavior data is not the behavior data of the first user.

[0147] In this way, by judging whether the second current behavior data is the behavior data of the first user, it is possible to avoid the target behavior prediction model learning the behavior habits of non-first users, resulting in a decrease in the matching degree between the target behavior prediction model and the first user, which is conducive to improving the prediction accuracy of the target behavior prediction model.

[0148] Optionally, the target behavior prediction model is updated according to the second current behavior data, and the specific implementation process may be as follows:

[0149] Acquire first current modality data in the target mixed reality environment;

[0150] Inputting the first current modality data into the target behavior prediction model to obtain second predicted behavior data;

[0151] According to the difference between the second current behavior data and the second predicted behavior data, the model parameters of the target behavior prediction model are adjusted to obtain the updated target behavior prediction model.

[0152] Specifically, the current modal data is the environmental data in the target mixed reality environment at the current moment, including the interactive behavior data and scene data generated by the user. The modal data can be unimodal data or multimodal data. The first current modal data refers to the current modal data of the target mixed reality environment before the first user goes offline, that is, the current modal data of the target mixed reality environment every time the first user is online.

[0153] In practical applications, when updating the target behavior prediction model according to the second current behavior data, the first current modal data in the target mixed reality environment can be obtained, and the first current modal data can be used as input to enable the target behavior prediction model to perform behavior prediction and obtain the second predicted behavior data. The second current behavior data is used as a label, and the model parameters of the target behavior prediction model are adjusted according to the difference between the second current behavior data and the second predicted behavior data to achieve the update of the target behavior prediction model. In this way, when the first user is online, it is beneficial to improve the target behavior prediction model to continuously learn the behavior preferences of the first user, and improve robustness and prediction accuracy.

[0154] Normally, when a user in the target mixed reality environment goes offline, other users will not be notified. However, in order to ensure the quality of interaction, before controlling the first user's virtual avatar to perform corresponding behavior based on the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, it also includes:

[0155] The offline information of the first user is fed back to online users among the multiple users.

[0156] In actual applications, when the first user goes offline, the offline information of the first user is fed back to other online users, so that online users can know that the first user has gone offline and the behavior data of interacting with him is generated by the target behavior prediction model of the first user, so as to avoid the target behavior prediction model from generating erroneous behaviors that affect other online users, thereby improving the quality of interaction and user satisfaction. In addition, the feedback of offline information can be set so that online users can clearly know whether they are interacting with the first user or the target behavior prediction model corresponding to the first user.

[0157] In one or more optional embodiments of the present invention, after controlling the first user's virtual avatar to perform corresponding behavior according to the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes:

[0158] If it is detected that multiple users in the target mixed reality environment are offline, the target mixed reality environment is closed.

[0159] In this way, when all users are offline, it greatly indicates that the interaction based on the user's will has ended. At this time, closing the target mixed reality environment and ending the collaborative interaction can reduce the monitoring of the target mixed reality environment and greatly reduce the data processing volume of the execution subject. In addition, the closing instruction for the target mixed reality environment can be triggered by the last offline user, such as the last offline user clicking the close button of the target mixed reality environment to close it, or detecting that the last offline user has gone offline and automatically triggering the closing instruction.

[0160] In one or more optional embodiments of the present invention, after controlling the first user's virtual avatar to perform corresponding behavior according to the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes:

[0161] If it is detected that the multiple users are all offline, for each user, calling the target behavior prediction model corresponding to the user to predict the behavior of the user to obtain third predicted behavior data;

[0162] According to the third predicted behavior data, the user's virtual avatar is controlled to perform a corresponding behavior.

[0163] In actual applications, when all users are offline, the target behavior prediction model of each user can be called to predict the behavior of each user, obtain the third predicted behavior data, and then control the virtual avatar of each user to perform the corresponding behavior based on the third predicted behavior data, so as to realize the interaction between the virtual avatars of each user. In this way, when all users are offline, the interaction can continue without restriction, and the interactive content between the virtual avatars can provide inspiration for users who come back online later, thereby improving user satisfaction.

[0164] It should be noted that when the first user goes online for the first time, an initial behavior prediction model is assigned to the first user, and then the initial behavior prediction model learns the behavior data of the first user and becomes the target behavior prediction model. That is, if it is monitored that the first user among the multiple users is offline, the target behavior prediction model corresponding to the first user is called to predict the behavior of the first user, and before obtaining the first predicted behavior data, it also includes:

[0165] If it is detected that the first user is online for the first time, an initial behavior prediction model is allocated to the first user;

[0166] Acquire third current behavior data of the first user and second current modality data in the target mixed reality environment;

[0167] Inputting the second current modality data into the initial behavior prediction model to obtain fourth predicted behavior data;

[0168] According to the difference between the third current behavior data and the fourth predicted behavior data, the model parameters of the initial behavior prediction model are adjusted to obtain a target behavior prediction model.

[0169] Specifically, the third current behavior data refers to the current behavior data of the user when the user first goes online. The second current modality data refers to the current modality data of the target mixed reality environment when the first user goes online for the first time.

[0170] In actual applications, if the first user goes online for the first time, the first user does not have a corresponding target behavior prediction model at this time, and an initial behavior prediction model can be assigned to the first user. Then, the second current modal data in the target mixed reality environment and the third current behavior data of the first user are obtained, and the second current modal data is used as input to enable the initial behavior prediction model to perform behavior prediction and obtain the fourth predicted behavior data. Using the third current behavior data as a label, the model parameters of the initial behavior prediction model are adjusted according to the difference between the third current behavior data and the fourth predicted behavior data to obtain the target behavior prediction model. In this way, the corresponding target behavior prediction model can be quickly set for the first user, and the prediction efficiency and accuracy of the target behavior prediction model can be improved.

[0171] In addition, the initial behavior prediction model assigned to the first user may be the same as the initial behavior prediction model assigned to other users, that is, a general behavior prediction model based on the initial behavior prediction model; the initial behavior prediction model assigned to the first user may also be different from the initial behavior prediction model assigned to other users, that is, different initial behavior prediction models are assigned to different users, that is, the initial behavior prediction model is assigned to the first user, and the specific implementation process may be as follows:

[0172] Acquire a general behavior prediction model and behavior pattern data of the first user, wherein the behavior pattern data is user behavior data and / or preference data uploaded by the first user;

[0173] The model parameters of the general behavior prediction model are adjusted according to the behavior pattern data to obtain an initial behavior prediction model corresponding to the first user.

[0174] In actual applications, when the first user goes online for the first time, the first user will upload the behavior pattern data on the upload page, or select the corresponding option on the user behavior selection page and / or the preference selection page, and then the execution subject obtains the behavior pattern data of the first user. At the same time, the execution subject obtains the general behavior prediction model. The model parameters of the general behavior prediction model are adjusted using the behavior pattern data of the first user, and then the initial behavior prediction model corresponding to the first user is obtained.

[0175] In this way, users are divided into initial behavior prediction models based on their behavior pattern data, which ensures the matching degree between the initial behavior prediction model and the user to a certain extent, thereby improving the prediction accuracy of the target behavior prediction model.

[0176] Combine the following Figure 3 The multi-user collaborative interaction method provided by the present invention is further described. Figure 3 This is the third flow chart of the multi-user collaborative interaction method provided by the present invention: opening a target mixed reality environment and monitoring the target mixed reality environment, wherein the target mixed reality environment includes two users: A and B.

[0177] For A, A enters the target mixed reality environment, A interacts with B, and updates A's target behavior prediction model in real time; determines whether A is offline, that is, monitors A's online status; if not, A interacts with B, and updates A's target behavior prediction model in real time; if so, generates the behavior of virtual avatar a based on A's target behavior prediction model, and interacts with B, that is, calls A's corresponding target behavior prediction model to predict A's behavior, obtains first predicted behavior data, and controls A's virtual avatar a to perform corresponding behavior and interaction according to the first predicted behavior data; determines whether A is online; if so, A interacts with B, and updates A's target behavior prediction model in real time; if not, generates the behavior of virtual avatar a based on A's target behavior prediction model.

[0178] For B, B enters the target mixed reality environment, and B interacts with A and updates B's target behavior prediction model in real time; it is determined whether B is offline, that is, B's online status is monitored. If not, B interacts with A and updates B's target behavior prediction model in real time. If so, the behavior of the virtual avatar b is generated based on B's target behavior prediction model and interacts with A; it is determined whether B is online. If so, B interacts with A and updates B's target behavior prediction model in real time. If not, the behavior of the virtual avatar b is generated based on B's target behavior prediction model.

[0179] In addition, no matter A or B, no matter online or offline, their interaction process may include sending interaction information, caching interaction information, receiving response information, and updating response information.

[0180] Figure 4 This is the fourth flow chart of the multi-user collaborative interaction method provided by the present invention, see Figure 4 As shown, it includes steps 401 to 403, wherein:

[0181] Step 401: Monitor a target mixed reality environment, in which multiple users interact through their own intelligent agents and / or the users themselves, and the intelligent agents are virtual avatars or embodied intelligent entities.

[0182] It should be noted that the implementation process of step 401 is the same as that of the above step 101, and will not be repeated here to avoid repetition.

[0183] Step 402: If it is monitored that the intelligent agent of the second user and / or the user himself sends interaction information to the first user, the virtual avatar of the first user is called to feedback response information regarding the interaction information to the intelligent agent of the second user and / or the user himself, the first user is any user among the multiple users who is offline, and the second user is any user among the multiple users except the first user.

[0184] In actual applications, the execution subject monitors each user in the target mixed reality environment. When the first user is detected to be offline, the second user's intelligent agent and / or the user himself sends interaction information to the first user. Since the first user is not online, the first user cannot feedback the interaction information sent by the second user through the user himself and the embodied intelligent body in the target mixed reality environment. In order to avoid interaction interruption and ensure that the first user can still interact collaboratively with the second user, further, the first user's virtual avatar responds to the interaction information, that is, the first user's virtual avatar is called to feedback the response information to the second user's intelligent agent and / or the user himself.

[0185] Step 403: When the first user comes back online, receiving an update instruction from the first user for the response information, updating the response information according to the update instruction, and transmitting the updated response information to the second user.

[0186] Specifically, the update instruction refers to an instruction generated by the first user performing an update operation such as adjustment or modification on the response information.

[0187] In actual applications, when the first user is offline, the first user's virtual avatar responds to the interactive information sent by the second user's intelligent agent and / or the user himself. However, since the response fed back by the virtual avatar may be different from the actual response of the first user, after the first user comes back online, the response information is displayed to the first user. When the response information is different from the first user's actual response to the interactive information, the first user adjusts and modifies the response information and other update operations. Accordingly, the execution main structure receives the update instruction, and then updates the response information to obtain the updated response information, and then transmits the updated response information to the second user.

[0188] Information transfer in a mixed reality environment refers to the transfer of interactive information between multiple users in a mixed reality environment. Figure 5 , Figure 5This is the fifth flow chart of the multi-user collaborative interaction method provided by the present invention: Generally, there are two users in the mixed reality environment: A and B. A's timeline shows A's interaction process, and B's timeline shows B's interaction process. Generally, B conveys information θ1 to A at time t. After receiving information θ1 (interaction information), A can delete, modify, add or reply to B's information θ1 to obtain information θ2 (response information). Thereafter, A transmits information θ2 to B, realizing a two-way information transmission.

[0189] In the multi-user collaborative interaction method of this embodiment, when a user goes offline, his virtual avatar continues to act in the timeline with the user's behavior habits and can receive interactive information from other users. Based on the received interactive information, the virtual avatar can interact with other users, that is, feedback corresponding information. When the user goes online again, the user can modify and update the response information and then pass it to other users.

[0190] For example, see Figure 6 , Figure 6 This is the sixth flow chart of the multi-user collaborative interaction method provided by the present invention: There are two users in a mixed reality environment: A and B. The time A when A is "offline" and "online" in the mixed reality environment is shown in A's timeline, and the time B when B is "offline" and "online" in the mixed reality environment is shown in B's timeline. Among them, the solid line part represents the user "online" in the mixed reality environment, and the dotted line part represents the user "offline" in the mixed reality environment. The virtual avatar of the user replaces the user in the mixed reality environment. Activities and interactions. At time t0, A is offline, and A's virtual avatar replaces A's activities and interactions. Then at time t1, B initiates interaction with A's virtual avatar and transmits interaction information θ1 (for example, a meeting was held). At this time, the real user A is offline, and the virtual avatar participates in the interaction, that is, feedback response information θ2, and writes the interaction information θ1 and response information θ2 into the cache area. Then, at time t2, B is offline, and B's virtual avatar replaces its activities. When A comes online at time t3, the execution subject transmits the information sent by B at time t1 and the interaction process at time t1 to A, that is, the interaction information θ1 and the response information θ2 are displayed to A. User A can modify and update the response information θ2 and write the updated response information θ'2 into the cache area. When B comes online at time t4, the response information θ'2 is displayed to B, realizing time-asynchronous interaction and two-way information transmission.

[0191] The multi-user collaborative interaction method provided by the present invention monitors the target mixed reality environment, and when the intelligent agent of the second user and / or the user himself sends interactive information to the first user, calls the virtual avatar of the first user to feedback the response information to the intelligent agent of the second user and / or the user himself for the interactive information, and when the first user is back online, receives the update instruction of the first user for the response information, updates the response information according to the update instruction, and transmits the updated response information to the second user, so that users in different time periods (partially online and partially offline) can interact with multiple users in different time periods based on the virtual avatar, such as a user can interact with past or future users and transmit information, so that multiple users can "not simultaneously" perform collaborative interaction and information transmission in the same mixed reality environment, realizing collaborative interaction in different time dimensions and improving the efficiency of collaborative interaction. In addition, through the virtual avatar, users can arrange their time in the mixed reality environment more flexibly, and realize uninterrupted information sending and receiving, thereby improving work efficiency. And by displaying the response information in the first predicted behavior data to the first user after the first user comes back online, and then updating the response information to ensure that the second user obtains the correct response information, the efficiency and reliability of the interaction can be improved, thereby improving user satisfaction.

[0192] In one or more optional embodiments of the present invention, in order to improve the accuracy of the response information, the response information may be predicted by the target behavior prediction model corresponding to the first user. That is, the virtual avatar of the first user is called to feedback the response information to the intelligent agent of the second user and / or the user himself in response to the interaction information. The specific implementation process may be as follows:

[0193] calling a target behavior prediction model corresponding to the first user to determine response information for the interaction information;

[0194] The virtual avatar of the first user is controlled to feed back the response information to the intelligent agent of the second user and / or the user himself.

[0195] In actual applications, when the first user who is offline is monitored and the interaction information sent by the intelligent agent of the second user and / or the user himself is received, the target behavior prediction model corresponding to the first user can be called according to the user ID of the first user, and the behavior of the first user can be predicted according to the interaction information, thereby obtaining response information. Furthermore, the virtual avatar of the first user is controlled based on the response information, so that the virtual avatar of the first user feeds back the response information to the intelligent agent of the second user and / or the user himself. Since the target behavior prediction model is a behavior prediction model trained based on at least one type of data such as the behavior, preferences and habits of the first user, it has a high degree of match with the first user. The response information predicted by the target behavior prediction model can reduce the difference between the response information and the actual response of the first user to a certain extent, improve the accuracy of the response information, and thus effectively improve the interaction efficiency between users.

[0196] It should be noted that each user in the target mixed reality environment corresponds to a target behavior prediction model, that is, there is a one-to-one correspondence between the target behavior prediction model and the user, that is, the target behavior prediction model is specific.

[0197] In an optional embodiment of the present invention, if the second user is online when the intelligent agent and / or the user himself of the second user sends interaction information, it means that the interaction information is sent based on the second user's own wishes, that is, the second user is clear and understands the interaction information, and at this time there is no need to display the interaction information to the second user.

[0198] In another optional embodiment of the present invention, before transmitting the updated response information to the second user, the method further includes:

[0199] If the second user is offline when the virtual avatar of the second user sends the interaction information, the interaction information is displayed to the second user.

[0200] In actual applications, if the second user is not online, he cannot send interactive information through the user himself and / or the embodied intelligent body, but can send interactive information through the virtual avatar. In this way, there are two situations for the second user to send interactive information through the virtual avatar: one is that the second user is online, at this time, the interactive information is sent based on the second user's own will, and there is no need to show the interactive information to the second user; the other is that the second user is offline, at this time, the interactive information is generated by the target behavior prediction model corresponding to the second user, and sent through the second user's virtual avatar, which means that the second user does not understand the content of the interactive information. At this time, the interactive information needs to be shown to the second user so that the second user can understand the interactive information and avoid the second user receiving the response information. After the information is received, it is impossible to obtain the interactive process. This is conducive to improving the quality of interaction and thus improving user stickiness.

[0201] Optionally, after updating the response information according to the update instruction, the method further includes: updating a target behavior prediction model corresponding to the first user according to the updated response information.

[0202] It should be noted that the multi-user collaborative interaction method provided in this embodiment and the multi-user collaborative interaction method described above can refer to each other.

[0203] The multi-user collaborative interaction device provided by the present invention is described below. The multi-user collaborative interaction device described below and the multi-user collaborative interaction method described above can be referred to each other.

[0204] Figure 7 is one of the structural diagrams of the multi-user collaborative interaction device provided by the present invention, such as Figure 7 As shown, the multi-user collaborative interaction device 700 includes: a first monitoring module 701, a prediction module 702 and a first control module 703, wherein:

[0205] A first monitoring module 701 is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0206] The prediction module 702 is configured to, if it is detected that a first user among the multiple users is offline, call a target behavior prediction model corresponding to the first user to predict the behavior of the first user and obtain first predicted behavior data;

[0207] The first control module 703 is configured to control the virtual avatar of the first user to perform corresponding behavior according to the first predicted behavior data, and interact with the intelligent agent of at least the second user among the multiple users and / or the user himself.

[0208] The multi-user collaborative interaction method and device provided by the present invention enables users in different time periods (partially online, partially offline) to interact in different time periods through behavior prediction based on virtual avatars and target behavior prediction models. For example, a user can interact with past or future users and transmit information, so that multiple users can "not simultaneously" perform collaborative interaction and information transmission in the same mixed reality environment, achieving collaborative interaction in different time dimensions and improving the efficiency of collaborative interaction. In addition, through virtual avatars and target behavior prediction models, users can arrange their time in the mixed reality environment more flexibly, and achieve uninterrupted information sending and receiving, thereby improving work efficiency.

[0209] Optionally, the device further includes a first acquisition module configured to:

[0210] When it is detected that the first user is back online, obtaining first current behavior data corresponding to the first user;

[0211] According to the first current behavior data, the intelligent agent of the first user is controlled to perform corresponding behavior, and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

[0212] Optionally, the device further includes a second transmission module configured to:

[0213] When it is detected that the first user is back online, displaying response information in the first predicted behavior data to the first user, the response information being used to respond to interactive information sent by the intelligent agent of the second user and / or the user himself;

[0214] receiving an update instruction from the first user, and updating the response information according to the update instruction;

[0215] The updated response information is transmitted to the second user.

[0216] Optionally, the second transmission module is further configured to:

[0217] If the second user is in an offline state, cache the updated response information;

[0218] When the second user comes back online, the updated response information is transmitted to the second user.

[0219] Optionally, the device further includes a first updating module configured to:

[0220] The target behavior prediction model is updated according to the updated response information.

[0221] Optionally, the device further includes a second updating module configured to:

[0222] When the first user is online, obtaining second current behavior data of the first user;

[0223] The target behavior prediction model is updated according to the second current behavior data.

[0224] Optionally, the second updating module is further configured to:

[0225] determining whether the second current behavior data only includes the behavior data of the first user;

[0226] If not, the behavior data of the first user in the second current behavior data is filtered out, and the target behavior prediction model is updated according to the behavior data of the first user.

[0227] Optionally, the second updating module is further configured to:

[0228] determining whether the second current behavior data is the behavior data of the first user;

[0229] If so, the target behavior prediction model is updated according to the second current behavior data.

[0230] Optionally, the second updating module is further configured to:

[0231] Acquire first current modality data in the target mixed reality environment;

[0232] Inputting the first current modality data into the target behavior prediction model to obtain second predicted behavior data;

[0233] According to the difference between the second current behavior data and the second predicted behavior data, the model parameters of the target behavior prediction model are adjusted to obtain the updated target behavior prediction model.

[0234] Optionally, the device further includes a second feedback module configured to:

[0235] The offline information of the first user is fed back to online users among the multiple users.

[0236] Optionally, the device further includes a second control module configured to:

[0237] If it is detected that the multiple users are all offline, for each user, calling the target behavior prediction model corresponding to the user to predict the behavior of the user to obtain third predicted behavior data;

[0238] According to the third predicted behavior data, the user's virtual avatar is controlled to perform a corresponding behavior.

[0239] Optionally, the device further includes an adjustment module configured to:

[0240] If it is detected that the first user is online for the first time, an initial behavior prediction model is allocated to the first user;

[0241] Acquire third current behavior data of the first user and second current modality data in the target mixed reality environment;

[0242] Inputting the second current modality data into the initial behavior prediction model to obtain fourth predicted behavior data;

[0243] According to the difference between the third current behavior data and the fourth predicted behavior data, the model parameters of the initial behavior prediction model are adjusted to obtain a target behavior prediction model.

[0244] Optionally, the adjustment module is further configured to:

[0245] Acquire a general behavior prediction model and behavior pattern data of the first user, wherein the behavior pattern data is user behavior data and / or preference data uploaded by the first user;

[0246] The model parameters of the general behavior prediction model are adjusted according to the behavior pattern data to obtain an initial behavior prediction model corresponding to the first user.

[0247] The multi-user collaborative interaction device provided by the present invention is described below. The multi-user collaborative interaction device described below and the multi-user collaborative interaction method described above can be referred to each other.

[0248] Figure 8 This is the second structural diagram of the multi-user collaborative interaction device provided by the present invention. Figure 8 As shown, the multi-user collaborative interaction device 800 includes: a second monitoring module 801, a first feedback module 802 and a first transmission module 803, wherein:

[0249] A second monitoring module 801 is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents;

[0250] The first feedback module 802 is configured to, if it is detected that the intelligent agent and / or the user of the second user sends interaction information to the first user, call the virtual avatar of the first user to feedback response information to the intelligent agent and / or the user of the second user for the interaction information, wherein the first user is any user in the multiple users who is in an offline state, and the second user is any user in the multiple users except the first user;

[0251] The first transmission module 803 is configured to receive an update instruction from the first user for the response information when the first user comes back online, update the response information according to the update instruction, and transmit the updated response information to the second user.

[0252] The multi-user collaborative interaction device provided by the present invention enables users in different time periods (partially online, partially offline) to interact with each other in different time periods based on virtual avatars. For example, a user can interact with past or future users and transmit information, so that multiple users can "not simultaneously" perform collaborative interaction and information transmission in the same mixed reality environment, realizing collaborative interaction in different time dimensions and improving the efficiency of collaborative interaction. In addition, through virtual avatars, users can arrange their time in the mixed reality environment more flexibly, and realize uninterrupted information sending and receiving, thereby improving work efficiency.

[0253] Optionally, the first feedback module 802 is further configured to:

[0254] calling a target behavior prediction model corresponding to the first user to determine response information for the interaction information;

[0255] The virtual avatar of the first user is controlled to feed back the response information to the intelligent agent of the second user and / or the user himself.

[0256] Optionally, the device further comprises a display module configured to:

[0257] If the second user is offline when the virtual avatar of the second user sends the interaction information, the interaction information is displayed to the second user.

[0258] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the multi-user collaborative interaction method.

[0259] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0260] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-user collaborative interaction method provided by the above methods.

[0261] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the multi-user collaborative interaction method provided by the above methods.

[0262] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0263] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-user collaborative interaction method, characterized in that: include: Monitoring a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents; If it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user to obtain first predicted behavior data; According to the first predicted behavior data, the virtual avatar of the first user is controlled to perform corresponding behavior and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

2. The multi-user collaborative interaction method according to claim 1, characterized in that: After controlling the first user's avatar to perform a corresponding behavior according to the first predicted behavior data, the method further includes: When it is detected that the first user is back online, obtaining first current behavior data corresponding to the first user; According to the first current behavior data, the intelligent agent of the first user is controlled to perform corresponding behavior, and interact with the intelligent agent of at least one second user among the multiple users and / or the user himself.

3. The multi-user collaborative interaction method according to claim 1, characterized in that: After controlling the first user's avatar to perform a corresponding behavior according to the first predicted behavior data, the method further includes: When it is detected that the first user is back online, displaying response information in the first predicted behavior data to the first user, the response information being used to respond to interactive information sent by the intelligent agent of the second user and / or the user himself; receiving an update instruction from the first user, and updating the response information according to the update instruction; The updated response information is transmitted to the second user.

4. The multi-user collaborative interaction method according to claim 3, characterized in that: The transmitting the updated response information to the second user comprises: If the second user is in an offline state, cache the updated response information; When the second user comes back online, the updated response information is transmitted to the second user.

5. The multi-user collaborative interaction method according to claim 3 or 4, characterized in that: After the response information is updated according to the update instruction, the method further includes: The target behavior prediction model is updated according to the updated response information.

6. The multi-user collaborative interaction method according to claim 1, characterized in that: If it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user, before obtaining the first predicted behavior data, the method further includes: When the first user is online, obtaining second current behavior data of the first user; The target behavior prediction model is updated according to the second current behavior data.

7. The multi-user collaborative interaction method according to claim 6, characterized in that: The updating of the target behavior prediction model according to the second current behavior data includes: determining whether the second current behavior data only includes the behavior data of the first user; If not, the behavior data of the first user in the second current behavior data is filtered out, and the target behavior prediction model is updated according to the behavior data of the first user.

8. The multi-user collaborative interaction method according to claim 6, characterized in that: The updating of the target behavior prediction model according to the second current behavior data includes: determining whether the second current behavior data is the behavior data of the first user; If so, the target behavior prediction model is updated according to the second current behavior data.

9. The multi-user collaborative interaction method according to any one of claims 6 to 8, characterized in that: The updating of the target behavior prediction model according to the second current behavior data includes: Acquire first current modality data in the target mixed reality environment; Inputting the first current modality data into the target behavior prediction model to obtain second predicted behavior data; According to the difference between the second current behavior data and the second predicted behavior data, the model parameters of the target behavior prediction model are adjusted to obtain the updated target behavior prediction model.

10. The multi-user collaborative interaction method according to claim 1, characterized in that: Before controlling the first user's avatar to perform a corresponding behavior according to the first predicted behavior data and interacting with the intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes: The offline information of the first user is fed back to online users among the multiple users.

11. The multi-user collaborative interaction method according to claim 1, characterized in that: After controlling the first user's avatar to perform a corresponding behavior according to the first predicted behavior data and interacting with an intelligent agent of at least one second user among the multiple users and / or the user himself, the method further includes: If it is detected that the multiple users are all offline, for each user, calling the target behavior prediction model corresponding to the user to predict the behavior of the user to obtain third predicted behavior data; According to the third predicted behavior data, the user's virtual avatar is controlled to perform a corresponding behavior.

12. The multi-user collaborative interaction method according to claim 1, characterized in that: If it is monitored that a first user among the multiple users is offline, calling a target behavior prediction model corresponding to the first user to predict the behavior of the first user, before obtaining the first predicted behavior data, the method further includes: If it is detected that the first user is online for the first time, an initial behavior prediction model is allocated to the first user; Acquire third current behavior data of the first user and second current modality data in the target mixed reality environment; Inputting the second current modality data into the initial behavior prediction model to obtain fourth predicted behavior data; According to the difference between the third current behavior data and the fourth predicted behavior data, the model parameters of the initial behavior prediction model are adjusted to obtain a target behavior prediction model.

13. The multi-user collaborative interaction method according to claim 12, characterized in that: The allocating an initial behavior prediction model to the first user includes: Acquire a general behavior prediction model and behavior pattern data of the first user, wherein the behavior pattern data is user behavior data and / or preference data uploaded by the first user; The model parameters of the general behavior prediction model are adjusted according to the behavior pattern data to obtain an initial behavior prediction model corresponding to the first user.

14. A multi-user collaborative interaction method, characterized in that: include: Monitoring a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents; If it is monitored that the intelligent agent and / or the user of the second user sends interaction information to the first user, the virtual avatar of the first user is called to feedback response information to the intelligent agent and / or the user of the second user for the interaction information, the first user is any user in the multiple users who is offline, and the second user is any user in the multiple users except the first user; When the first user comes back online, an update instruction from the first user for the response information is received, the response information is updated according to the update instruction, and the updated response information is transmitted to the second user.

15. The multi-user collaborative interaction method according to claim 14, characterized in that: The calling of the first user's avatar to feed back response information to the second user's intelligent agent and / or the user himself regarding the interaction information includes: calling a target behavior prediction model corresponding to the first user to determine response information for the interaction information; The virtual avatar of the first user is controlled to feed back the response information to the intelligent agent of the second user and / or the user himself.

16. The multi-user collaborative interaction method according to claim 14 or 15, characterized in that: Before transmitting the updated response information to the second user, the method further includes: If the second user is offline when the virtual avatar of the second user sends the interaction information, the interaction information is displayed to the second user.

17. A multi-user collaborative interaction device, characterized in that: include: A first monitoring module is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents; The prediction module is configured to, if it is detected that a first user among the multiple users is offline, call a target behavior prediction model corresponding to the first user to predict the behavior of the first user and obtain first predicted behavior data; The first control module is configured to control the virtual avatar of the first user to perform corresponding behavior according to the first predicted behavior data, and interact with the intelligent agent of at least the second user among the multiple users and / or the user himself.

18. A multi-user collaborative interaction device, characterized in that: include: A second monitoring module is configured to monitor a target mixed reality environment in which a plurality of users interact through respective intelligent agents and / or the users themselves, wherein the intelligent agents are avatars or embodied agents; A first feedback module is configured to, if monitoring that the intelligent agent and / or the user of the second user sends interaction information to the first user, call the virtual avatar of the first user to feedback response information to the intelligent agent and / or the user of the second user, wherein the first user is any user in the multiple users who is in an offline state, and the second user is any user among the multiple users except the first user; The first transmission module is configured to receive an update instruction from the first user for the response information when the first user comes back online, update the response information according to the update instruction, and transmit the updated response information to the second user.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-user collaborative interaction method as described in any one of claims 1 to 13 or 14 to 16 is implemented.

20. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-user collaborative interaction method as described in any one of claims 1 to 13 or 14 to 16 is implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-user collaborative interaction method as described in any one of claims 1 to 13 or 14 to 16 is implemented.