User interaction method and device capable of realizing reality and virtuality

Abstract

The invention provides a user interaction method and device capable of realizing reality and virtuality, and relates to the technical field of computers.The method comprises the steps that first user information of a first user is obtained, and a first role is generated; generating a second role by obtaining second user information of a second user; generating a first special effect animation taking the first role as a leading angle by tracking the action and positioning of the first user in real time; generating a second special effect animation taking a second role as a main angle by tracking the action and positioning of a second user in real time; and based on the first special effect animation and the second special effect animation, realizing real-time interaction of the first user and the second user in the real world through a virtual image. According to the user interaction method capable of realizing reality and virtuality provided by the invention, the interaction experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a user interaction method and device capable of realizing real - virtual interaction. Background Art

[0002] Virtual Reality (VR) technology is a technology that comprehensively utilizes computer graphics systems and various control interfaces to generate an interactive three - dimensional environment on a computer and provide an immersive experience for users. The basic implementation method of virtual reality is mainly based on computer technology, and comprehensively utilizes the latest development achievements of various high - tech technologies such as three - dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. With the help of devices such as computers, a realistic virtual world with a variety of sensory experiences such as three - dimensional vision, touch, and smell is generated, so that people in the virtual world have a feeling of being on the scene.

[0003] In the prior art, only human - machine interaction can be realized, that is, the interaction between the user and the virtual reality scene. By providing rich operable virtual elements (such as virtual buttons) for the user in the virtual reality scene, the user can trigger corresponding operations in the virtual scene by selecting these operable elements provided in the scene, and interact with the virtual reality scene. However, the interaction between people cannot be realized, resulting in a poor user interaction experience. Summary of the Invention

[0004] The present invention provides a user interaction method and device capable of realizing real - virtual interaction, so as to solve the technical problem that in the prior art, the interaction between people in the real world in the form of virtual images cannot be realized, resulting in a poor user interaction experience.

[0005] The present invention provides a user interaction method capable of realizing real - virtual interaction, including the following steps: Obtain the first user information of the first user and the second user information of the second user; Based on the first user information, generate a first character; and, based on the second user information, generate a second character; By real - time tracking the actions and positions of the first user, generate a first special - effect animation with the first character as the protagonist; and, by real - time tracking the actions and positions of the second user, generate a second special - effect animation with the second character as the protagonist; Based on the first special - effect animation and the second special - effect animation, perform user interaction between the first user and the second user.

[0006] According to the user interaction method capable of realizing real - virtual interaction provided by the present invention, the generating a first character based on the first user information includes: Based on the first user information, obtain a first model corresponding to the first user instruction through the user instruction of the first user; the first model is generated by performing bone point matching on the three-dimensional pose model of the first user and an animation character model; Based on the first model, generate a first character by performing real-time motion capture on the first user.

[0007] According to a user interaction method capable of realizing real-virtual provided by the present invention, the generating a second character based on the second user information includes: Based on the second user information, obtain a second model corresponding to the second user instruction through the user instruction of the second user; the second model is generated by performing bone point matching on the three-dimensional pose model of the second user and an animation character model; Based on the second model, generate a second character by performing real-time motion capture on the second user.

[0008] According to a user interaction method capable of realizing real-virtual provided by the present invention, the performing user interaction between the first user and the second user based on the first special effect animation and the second special effect animation includes: When the first user is the main user, generate an interaction video based on the first special effect animation and the second special effect animation, wherein the first special effect animation is in the first-person perspective and the second special effect animation is in the third-person perspective; When the second user is the main user, generate an interaction video based on the first special effect animation and the second special effect animation, wherein the second special effect animation is in the first-person perspective and the first special effect animation is in the third-person perspective.

[0009] According to a user interaction method capable of realizing real-virtual provided by the present invention, the first user information includes the three-dimensional pose model of the first user and the role information of multiple animation characters.

[0010] According to a user interaction method capable of realizing real-virtual provided by the present invention, the second user information includes the three-dimensional pose model of the second user and the role information of multiple animation characters.

[0011] According to a user interaction method capable of realizing real-virtual provided by the present invention, the role information includes an animation character model, a role special effect animation, and an interaction instruction.

[0012] The present invention also provides a user interaction device capable of realizing real-virtual, including the following modules: An acquisition module, configured to acquire the first user information of the first user and the second user information of the second user; The first generation module is used to generate a first character based on the first user information, and generate a second character based on the second user information. The second generation module is used to generate a first special effect animation with the first character as the protagonist by tracking the actions and positions of the first user in real time, and generate a second special effect animation with the second character as the protagonist by tracking the actions and positions of the second user in real time. The interaction module is used to perform user interaction between the first user and the second user based on the first special effect animation and the second special effect animation.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, the method for realizing real virtual user interaction as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for realizing real virtual user interaction as described in any one of the above is implemented.

[0015] A method for realizing real virtual user interaction provided by the present invention includes generating a first character by obtaining the first user information of the first user, generating a second character by obtaining the second user information of the second user, generating a first special effect animation with the first character as the protagonist by tracking the actions and positions of the first user in real time, generating a second special effect animation with the second character as the protagonist by tracking the actions and positions of the second user in real time, and then realizing the real-time interaction between the first user and the second user in the real world with virtual avatars based on the first special effect animation and the second special effect animation, improving the user interaction experience. Description of the Drawings

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of a method for realizing real virtual user interaction provided by the present invention.

[0018] Figure 2 It is a structural diagram of a device for realizing real virtual user interaction provided by the present invention.

[0019] Figure 3 It is a structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0020] In the prior art, the technologies that enable users to interact in a virtual world mainly include virtual reality technology, augmented reality technology, mixed reality technology, and non-wearable real-time motion capture technology.

[0021] Virtual reality technology simulates the external environment through a computer simulation system. The main simulation objects include the environment, human characters, physical properties, processes, and systems, etc., providing users with a multi-information, three-dimensional dynamic, and interactive simulation experience.

[0022] Augmented Reality (AR) technology integrates the virtual environment generated by a computer with the real environment around the user by means of optoelectronic display technology, interaction technology, various sensor technologies, and computer graphics and multimedia technologies, making users believe from the sensory effect that the virtual environment is an integral part of the real environment around them. Augmented reality has the characteristics of combining virtual and real, real-time interaction, and three-dimensional registration.

[0023] Mixed Reality (MR) is a technology that deeply integrates the virtual world with the real environment. It combines virtual reality and augmented reality, enabling real-time interaction between virtual objects generated by a computer and the real world, capable of superimposing virtual elements onto the real environment and endowing these virtual objects with physical properties of the real world (such as real-time physical interaction and dynamic reaction).

[0024] The non-wearable real-time motion capture technology is based on advanced deep learning vision algorithms, human key point detection, pose estimation algorithms, motion tracking algorithms, and data smoothing and post-processing algorithms. It performs visual detection and recognition of the human body and key parts on the images of multi-view cameras, and can achieve a capture range of at least 27 m³ without wearing motion capture devices, realizing 360° dead-angle-free human motion capture including full-body movements, finger movements, and facial expression movements. There is no need to perform complex motion calibration, nor wear motion capture devices. As long as the user enters the monitoring area, motion capture can be carried out quickly and without obstacles.

[0025] However, there are some defects in the above-mentioned technologies in the process of enabling users to interact in a virtual world.

[0026] In the process of implementation, virtual reality technology will bring users into the virtual world, resulting in users being unaware of the real-world situation. Acting rashly is likely to cause danger. Therefore, to ensure safety, the user's behavior can only be fixed within a specified space range, so it has high requirements for the surrounding environment and is not convenient. In addition, in the virtual world, due to the imperfection of the existing technology, there are many problems such as discomfort and unreality in the interaction between users and virtual things. People can only interact with virtual objects within a certain space, resulting in a poor user experience.

[0027] Augmented reality technology mainly superimposes or stores virtual things near real-world objects and interacts with them. There may be a risk of diverting the user's attention and causing safety problems. Moreover, when interacting, it requires operating in mid-air without tactile feedback, making it not easy to operate successfully. In addition, when projecting the content of devices such as mobile phones and tablets onto glasses, augmented reality technology also has problems such as software incompatibility and poor display effects, and it can only achieve interaction between people and virtual objects.

[0028] Based on virtual reality technology and augmented reality technology, mixed reality technology can only achieve human-computer interaction during the process of enabling users to interact in a virtual world, and the implementation method is complex.

[0029] For non-wearable real-time motion capture technology, the target needs to enter the monitoring area, and a omnidirectional camera or a single camera is used to monitor the target data in real time. There are greater limitations on the activity range, higher requirements for related devices, and poor accuracy and high latency. In addition, non-wearable real-time motion capture technology only maps real human movements to character modeling and drives the modeling to perform corresponding actions, which are displayed on a computer. Only other special effects can be achieved through subsequent editing operations, etc. It is only applicable to related fields such as animation production and lacks actual user interactivity.

[0030] In order to enable users to perform real-time interactions in the real world in a virtual image, and at the same time be able to use it freely in various places in the real world, improve the interaction experience, and meet the needs of users to transform into anime or animation characters and have fun in the real world, the present invention provides a user interaction method that can achieve real-world virtualization.

[0031] To make the purpose, technical solution, and advantages of the present invention clearer, the technical solution in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0032] The following combines Figures 1 to 3 Describe a user interaction method and device that can achieve real-world virtualization according to the present invention.

[0033] Figure 1 is a schematic flowchart of a user interaction method that can achieve real-world virtualization provided by the present invention. As Figure 1 shown, the method includes the following steps 101 to step 104: Step 101, obtain the first user information of the first user and the second user information of the second user.

[0034] Specifically, a user interaction method for realizing virtual reality provided by the present invention can be implemented through some electronic devices. For example, through VR glasses, and then in combination with devices such as mobile phones, tablets, and laptops to jointly achieve it.

[0035] In the embodiment of the present invention, VR glasses are taken as an example for illustration. Multiple users can participate in user interaction by wearing VR glasses, and the number is not limited. Here, two users (i.e., the first user and the second user) are taken as an example for illustration. The first user and the second user each wear a VR glasses. The VR glasses scan the wearer in advance to obtain the three-dimensional information of the wearer, detect the bone points of the human body, and generate a three-dimensional pose model of the wearer in the 360-degree direction through three-dimensional reconstruction technology.

[0036] In addition, in order to enrich the user experience, the role information of multiple animation characters can also be stored in advance, including animation character models, animation character action special effects, animation character voice information, animation character transformation instructions, animation character transformation videos, animation character skill videos, and trigger actions, etc. Among them, these animation characters can be official animation characters that have been produced or released, or animation characters independently designed by users.

[0037] The three-dimensional pose model of the first user and the role information of multiple animation characters are used as the first user information and stored in the application of the first user's mobile phone (it can also be devices such as tablets or laptops, and mobile phones are taken as an example for illustration in the embodiment of the present invention), providing a basis for subsequent animation display in the VR glasses worn by the first user.

[0038] The three-dimensional pose model of the second user and the role information of multiple animation characters are used as the second user information and stored in the application of the second user's mobile phone (it can also be devices such as tablets or laptops, and mobile phones are taken as an example for illustration in the embodiment of the present invention), providing a basis for subsequent animation display in the VR glasses worn by the second user.

[0039] In the embodiment of the present invention, in order to make the VR glasses lightweight and simplify the observation operation of the mobile phone application, the mobile phone and the VR glasses are wirelessly connected. It can be understood that in special cases where the wireless network is poor or there is no condition for wireless connection, the mobile phone and the VR glasses can also be wired connected.

[0040] Step 102: Generate a first character based on the first user information; and generate a second character based on the second user information.

[0041] Specifically, the user can pre-store their own three-dimensional information in the mobile application to generate a corresponding three-dimensional pose model, so as to perform bone point matching (i.e., model matching) between the three-dimensional pose model and various animated character models in advance. By scaling the animated character model proportionally, it can be fully adapted to the user's three-dimensional pose model, and then a virtual character model corresponding to the user is generated and stored in the mobile application for later use. During the actual interaction process, the user can directly download or call the already matched virtual character model from the mobile application.

[0042] According to the animated character pre-selected by the first user or the animated character determined by the user instruction sent by the first user in real time, obtain the virtual character model corresponding to the user from the first user's mobile application, and then perform real-time motion capture on the first user, so as to generate the first character based on the virtual character model. Through the first character, it can represent the personal image of the first user in the virtual world.

[0043] Similarly to the first character, according to the animated character pre-selected by the second user or the animated character determined by the user instruction sent by the second user in real time, obtain the virtual character model corresponding to the user from the second user's mobile application, and then perform real-time motion capture on the second user, so as to generate the second character based on the virtual character model. Through the second character, it can represent the personal image of the second user in the virtual world.

[0044] In the embodiment of the present invention, by scaling the animated character model proportionally in advance to generate a virtual character model adapted to the user's three-dimensional pose model, which represents the personal image of the user in the virtual world, it can not only facilitate subsequent motion capture of the user and accurately present it one-to-one in the real world, but also reduce real-time calculation, shorten the time required for user interaction, improve efficiency, and enhance the user experience.

[0045] Step 103: Generate a first special effect animation with the first character as the protagonist by real-time tracking the actions and positions of the first user; and generate a second special effect animation with the second character as the protagonist by real-time tracking the actions and positions of the second user.

[0046] Specifically, through the sensors built into the VR glasses worn by the first user, the behavior actions and position changes of the first user are tracked in real time, the actions and positions of the first user are captured, and they are synchronously mapped onto the first character. Taking the first character as the protagonist, a first special effect animation is generated and projected and displayed on the VR glasses worn by the first user. In this case, the perspective of the first user is the protagonist perspective synchronized with the first character. Similarly, through the sensors built into the VR glasses worn by the second user, the behavior actions and position changes of the second user are tracked in real time, the actions and positions of the second user are captured, and they are synchronously mapped onto the second character. Taking the second character as the protagonist, a second special effect animation is generated and projected and displayed on the VR glasses worn by the second user. In this case, the perspective of the second user is the protagonist perspective synchronized with the second character.

[0047] In the embodiments of the present invention, for any user, the user can issue a voice command or directly make a specific gesture through the microphone on the VR glasses. After the VR glasses receive the voice command or monitor the specific gesture, the corresponding first character or second character can perform animation transformation or skill release according to the voice command or the specific gesture. The VR glasses convert the change information (including transformation and skill release, etc.) of the character corresponding to any user in the virtual world into a special effect animation (such as the first special effect animation or the second special effect animation), and then display the special effect animation from the protagonist perspective and the special effect animation of other users seen from the third-person perspective on the VR glasses worn by any user himself.

[0048] Step 104: Based on the first special effect animation and the second special effect animation, perform user interaction between the first user and the second user.

[0049] Specifically, in some embodiments, the interaction between users can have a sequence. For example, the first user accesses first and makes the first character in the virtual world transform through a voice command, and then makes the first character in the virtual world release a skill through a gesture. The VR glasses convert the change information (including transformation and skill release, etc.) of the first character into the first special effect animation.

[0050] The second user then synchronously displays the first special effect animation through the VR glasses worn by himself. The second user can make corresponding responses according to the first special effect animation. For example, issue a voice command to make the second character in the virtual world transform. At the same time, in order to counter the skill of the first character, the second user can also make the second character in the virtual world release a skill through a gesture. The VR glasses worn by the second user will convert the change information (including transformation and skill release, etc.) of the second character into the second special effect animation.

[0051] On a predetermined position or interface of the VR glasses of the first user, display the first characteristic animation from the perspective of the protagonist of the first character and the special effect animation of the second user from the third-person perspective; on a predetermined position or interface of the VR glasses of the second user, display the second characteristic animation from the perspective of the protagonist of the second character and the special effect animation of the first user from the third-person perspective. That is to say, in the embodiments of the present invention, from any user's perspective, not only can the 360-degree animated character partial body modeling and skill special effect animation of oneself be seen from the first-person perspective (i.e., the protagonist's perspective), but also, in combination with real-time positioning, by capturing the full-body movements of other users, the 360-degree animated character full-body modeling and skill special effect animation of other users can be seen from the third-person perspective. All the special effect animations are played in one video.

[0052] In addition, for user information, it is mainly stored in the user's mobile application. Before use, only the selected animated character modeling information needs to be transmitted to the VR glasses for use. After completing user interaction, all the information obtained by the VR glasses this time can be deleted by closing, restarting the VR glasses or directly deleting, thereby reducing the storage requirements of the VR glasses.

[0053] In the embodiments of the present invention, through the first special effect animation and the second special effect animation, the first character representing the first user's character image interacts with the second character representing the second user's character image in the real world, enhancing the user's interaction experience. In addition, through the lightweight VR glasses in the embodiments of the present invention, they can be used anywhere in the real world, are safer and more convenient to use, can ensure compatibility, have low latency, and enable the user to have a strong sense of immersion and a more real interaction experience, thus meeting the needs of users to transform into anime characters and have fun in the real world.

[0054] A user interaction method capable of realizing reality virtualization provided by the present invention includes: generating a first character by obtaining the first user information of the first user; generating a second character by obtaining the second user information of the second user; generating a first special effect animation with the first character as the protagonist by real-time tracking the actions and positioning of the first user; generating a second special effect animation with the second character as the protagonist by real-time tracking the actions and positioning of the second user; and then realizing the real-time interaction between the first user and the second user in the real world based on the first special effect animation and the second special effect animation, enhancing the user's interaction experience, being not only safe and convenient, but also meeting the needs of users to transform into anime characters and have fun in the real world.

[0055] Optionally, the generating a first character based on the first user information includes: Based on the first user information, obtain a first model corresponding to the first user's instruction through the user instruction of the first user; the first model is generated by performing bone point matching between the three-dimensional pose model of the first user and the animation character model; Based on the first model, generate a first character by performing real-time motion capture on the first user.

[0056] Specifically, the first user stores their own three-dimensional information in the mobile application in advance to generate a corresponding three-dimensional pose model, so that the bone point matching between the three-dimensional pose model and various animation character models can be carried out in advance. By equally scaling up or down the animation character model, it is fully adapted to the user's three-dimensional pose model, and then a virtual character model corresponding to the user (i.e., the first model) is generated and stored in the mobile application for later use. During the actual interaction process, the user can directly download or call the already matched first model from the mobile application.

[0057] After the first user wears the VR glasses, the first model can be obtained from the first user's mobile application according to the animation character pre-selected by the first user or the animation character determined by the real-time user instruction of the first user, and then real-time motion capture is performed on the first user, so that a first character is generated based on the first model. Through the first character, the personal image of the first user in the virtual world can be represented.

[0058] In the embodiment of the present invention, by pre-performing model matching, equally scaling up or down the animation character model, a first model corresponding to the first user's instruction is generated, representing the personal image of the first user in the virtual world, so that subsequent motion capture of the user can be facilitated, a first character is generated, and it is accurately presented one-to-one in the real world, shortening the time required for user interaction and improving the interaction experience of users.

[0059] Optionally, the generating a second character based on the second user information includes: Based on the second user information, obtain a second model corresponding to the second user's instruction through the user instruction of the second user; the second model is generated by performing bone point matching between the three-dimensional pose model of the second user and the animation character model; Based on the second model, generate a second character by performing real-time motion capture on the second user.

[0060] Specifically, the second user pre-stores their own three-dimensional information into the mobile application to generate a corresponding three-dimensional pose model, thereby enabling the pre-match of the skeletal points between the three-dimensional pose model and various animation character models in advance. By equally scaling up or down the animation character model, it can be fully adapted to the user's three-dimensional pose model, and then a virtual character model corresponding to the user (i.e., the second model) is generated and stored in the mobile application for later use. During the actual interaction process, the user can directly download or call the second model that has been successfully matched from the mobile application.

[0061] After the second user wears the VR glasses, according to the animation character pre-selected by the second user or the animation character determined by the user instructions issued by the second user in real time, the second model is obtained from the second user's mobile application, and then real-time motion capture is performed on the second user, thereby generating a second character based on the second model. Through the second character, the personal image of the second user in the virtual world can be represented.

[0062] In the embodiment of the present invention, by pre-performing model matching, equally scaling up or down the animation character model, a second model corresponding to the second user's instructions is generated to represent the personal image of the second user in the virtual world, thereby facilitating subsequent motion capture of the user, generating a second character, and accurately presenting it one-to-one in the real world, shortening the time required for user interaction and improving the interaction experience of users.

[0063] Optionally, the user interaction between the first user and the second user based on the first special effect animation and the second special effect animation includes: When the first user is the main user, an interaction video is generated based on the first special effect animation and the second special effect animation, where the first special effect animation is in the first-person perspective and the second special effect animation is in the third-person perspective; When the second user is the main user, an interaction video is generated based on the first special effect animation and the second special effect animation, where the second special effect animation is in the first-person perspective and the first special effect animation is in the third-person perspective.

[0064] Specifically, the interaction between users can have a sequence. For example, the first user accesses first and makes the first character in the virtual world transform through a voice command, and then makes the first character in the virtual world emit a skill through a gesture. The VR glasses convert these change information (including transformation and skill emission, etc.) of the first character into the first special effect animation.

[0065] The second user synchronously displays the first special effect animation through the VR glasses worn by himself / herself. The second user can make corresponding responses according to the first special effect animation. For example, the second user can issue a voice command to make the second character in the virtual world transform. At the same time, in order to respond to the skills of the first character, the second user can also make the second character in the virtual world issue skills through gesture actions. The VR glasses worn by the second user will convert the change information of the second character (including transformation and issuing skills, etc.) into the second special effect animation.

[0066] On a predetermined position or interface on the VR glasses of the first user, the first special effect animation from the perspective of the first character as the protagonist and the special effect animation of the second user from the third-person perspective are displayed; on a predetermined position or interface on the VR glasses of the second user, the second special effect animation from the perspective of the second character as the protagonist and the special effect animation of the first user from the third-person perspective are displayed.

[0067] In some embodiments, when there are only the first user and the second user using VR glasses for interaction, for the first user, an interactive video is generated with the first special effect animation as the first-person perspective and the second special effect animation as the third-person perspective, and is played on the VR glasses worn by the first user; for the second user, an interactive video is generated with the second special effect animation as the first-person perspective and the first special effect animation as the third-person perspective, and is played on the VR glasses worn by the second user.

[0068] In other embodiments, when there are multiple users (more than two) using VR glasses for interaction, for any one of the users, an interactive video is generated with the special effect animation generated with their own character as the protagonist as the first-person perspective and the special effect animations sent by other received users as the third-person perspective, and is played on the VR glasses worn by themselves. Specifically, from the perspective of any user, not only can they see the partial body modeling of their own 360-degree animated character and the skill special effect animation from the first-person perspective (i.e., the protagonist perspective), but also, combined with real-time positioning, they can see the full-body modeling of the 360-degree animated characters of other users and the skill special effect animations from the third-person perspective by capturing the full-body actions of other users.

[0069] In the embodiments of the present invention, through the first special effect animation and the second special effect animation, the first character representing the first user's character image interacts with the second character representing the second user's character image in the real world, improving the user's interaction experience. In addition, through the lightweight VR glasses in the embodiments of the present invention, not only can they be used anywhere in the real world, which is safer and more convenient, but also they can be compatible with multiple people at the same time, with low latency, making the user immersion strong and the interaction experience more real, so as to meet the needs of users to transform into anime characters and have entertainment in the real world.

[0070] Optionally, the first user information includes the three-dimensional pose model of the first user and the character information of multiple animation characters.

[0071] Specifically, the VR glasses worn by the first user are used to scan the wearer in advance to obtain the three-dimensional information of the wearer, perform bone point detection on the human body, and generate a three-dimensional pose model of the wearer in all 360-degree directions through three-dimensional reconstruction technology.

[0072] To enrich the user experience, the character information of multiple animation characters can be stored in advance, including animation character models, animation character action special effects, animation character voice information, animation character transformation instructions, animation character transformation videos, animation character skill videos, and trigger actions, etc. Among them, these animation characters can be official animation characters that have been produced or released, or animation characters independently designed by users.

[0073] In the embodiment of the present invention, the three-dimensional pose model of the first user and the character information of multiple animation characters are used as the first user information and stored in the application of the first user's mobile phone (which can also be devices such as a tablet or a laptop computer), providing a basis for subsequent animation display in the VR glasses worn by the first user.

[0074] Optionally, the second user information includes the three-dimensional pose model of the second user and the character information of multiple animation characters.

[0075] Specifically, the VR glasses worn by the second user are used to scan the wearer in advance to obtain the three-dimensional information of the wearer, perform bone point detection on the human body, and generate a three-dimensional pose model of the wearer in all 360-degree directions through three-dimensional reconstruction technology.

[0076] To enrich the user experience, the character information of multiple animation characters can be stored in advance, including animation character models, animation character action special effects, animation character voice information, animation character transformation instructions, animation character transformation videos, animation character skill videos, and trigger actions, etc. Among them, these animation characters can be official animation characters that have been produced or released, or animation characters independently designed by users.

[0077] In the embodiment of the present invention, the three-dimensional pose model of the second user and the character information of multiple animation characters are used as the second user information and stored in the application of the second user's mobile phone (which can also be devices such as a tablet or a laptop computer), providing a basis for subsequent animation display in the VR glasses worn by the second user.

[0078] Optionally, the character information includes an animation character model, a character special effect animation, and an interaction instruction.

[0079] Specifically, to enrich the user experience, role information of multiple animation characters can be stored in advance. The role information may include animation character models, character special effect animations (such as transformation special effect videos and action special effect videos, etc.), and interaction instructions (such as transformation voice instructions and skill action instructions, etc.). Among them, these animation characters can be official animation characters that have been produced or released, or animation characters independently designed by users.

[0080] The following describes a user interaction device capable of realizing real virtuality provided by the present invention. The user interaction device capable of realizing real virtuality described below can be correspondingly referred to with the method for realizing real virtuality described above.

[0081] Based on any of the above embodiments, Figure 2 is a schematic structural diagram of a user interaction device capable of realizing real virtuality provided by the present invention, as Figure 2 shown. An embodiment of the present invention provides a user interaction device capable of realizing real virtuality, including an acquisition module 201, a first generation module 202, a second generation module 203, and an interaction module 204, where: The acquisition module 201 is configured to acquire first user information of a first user and second user information of a second user; the first generation module 202 is configured to generate a first role based on the first user information, and generate a second role based on the second user information; the second generation module 203 is configured to generate a first special effect animation with the first role as the protagonist by real-time tracking the actions and positions of the first user, and generate a second special effect animation with the second role as the protagonist by real-time tracking the actions and positions of the second user; the interaction module 204 is configured to perform user interaction between the first user and the second user based on the first special effect animation and the second special effect animation.

[0082] A user interaction device capable of realizing real virtuality provided by the present invention generates a first role by acquiring first user information of a first user, generates a second role by acquiring second user information of a second user, generates a first special effect animation with the first role as the protagonist by real-time tracking the actions and positions of the first user, generates a second special effect animation with the second role as the protagonist by real-time tracking the actions and positions of the second user, and then realizes real-time interaction between the first user and the second user in the real world with virtual images based on the first special effect animation and the second special effect animation, improving the user's interaction experience, which is not only safe and convenient, but also can meet the needs of users to transform into anime characters and have entertainment in the real world.

[0083] Figure 3 Illustrates a schematic structural diagram of an electronic device, as Figure 3As shown in the figure, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logical instructions in the memory 330 to execute a user interaction method that can realize virtual reality. The method includes: Obtain the first user information of the first user and the second user information of the second user; Generate a first character based on the first user information; and generate a second character based on the second user information; Generate a first special effect animation with the first character as the protagonist by tracking the actions and positions of the first user in real time; and generate a second special effect animation with the second character as the protagonist by tracking the actions and positions of the second user in real time; Perform user interaction between the first user and the second user based on the first special effect animation and the second special effect animation.

[0084] In addition, when the logical instructions in the above-mentioned memory 330 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the user interaction method that can realize virtual reality provided by the above-mentioned methods. The method includes: Obtain the first user information of the first user and the second user information of the second user; Generate a first character based on the first user information; and generate a second character based on the second user information; By tracking the actions and positions of the first user in real time, a first special effect animation with the first character as the protagonist is generated; and by tracking the actions and positions of the second user in real time, a second special effect animation with the second character as the protagonist is generated. Based on the first special effect animation and the second special effect animation, user interaction between the first user and the second user is performed.

[0086] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the method for realizing user interaction between reality and virtuality provided by the above various methods. The method includes: Obtain the first user information of the first user and the second user information of the second user. Based on the first user information, a first character is generated; and based on the second user information, a second character is generated. By tracking the actions and positions of the first user in real time, a first special effect animation with the first character as the protagonist is generated; and by tracking the actions and positions of the second user in real time, a second special effect animation with the second character as the protagonist is generated. Based on the first special effect animation and the second special effect animation, user interaction between the first user and the second user is performed.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] It should be noted that in this document, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not preclude the existence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0090] It should be further noted that in the present invention, terms such as "first", "second", etc. are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms may be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object may be one or more.

[0091] "Determining B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "determining B only based on A", and should also include: "determining B based on A and C", "determining B based on A, C and E", "determining C based on A and further determining B based on C", etc. Additionally, it may also include using A as a condition for determining B. For example, "when A meets the first condition, use the first method to determine B"; for another example, "when A meets the second condition, determine B"; for yet another example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it may also be using A as a condition for the factor of determining B. For example, "when A meets the first condition, use the first method to determine C and further determine B based on C", etc.

[0092] The term "plurality" in the present invention means two or more, and other quantifiers are similar thereto.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user interaction method for realizing augmented reality, characterized in that including: Obtain the first user information of the first user and the second user information of the second user; Generate a first character based on the first user information; And generate a second character based on the second user information; Generate a first special effect animation with the first character as the protagonist by real-time tracking of the actions and positions of the first user; and generate a second special effect animation with the second character as the protagonist by real-time tracking of the actions and positions of the second user; Perform user interaction between the first user and the second user based on the first special effect animation and the second special effect animation.

2. The user interaction method for realizing virtual reality according to claim 1, wherein The generating the first character based on the first user information includes: Based on the first user information, obtain a first model corresponding to a first user instruction through the user instruction of the first user; the first model is generated by matching bone points between the three-dimensional pose model of the first user and an animation character model; Generate a first character based on the first model by performing real-time action capture on the first user.

3. The user interaction method for realizing virtual reality according to claim 1, wherein The generating the second character based on the second user information includes: Based on the second user information, obtain a second model corresponding to a second user instruction through the user instruction of the second user; the second model is generated by matching bone points between the three-dimensional pose model of the second user and an animation character model; Generate a second character based on the second model by performing real-time action capture on the second user.

4. The user interaction method for realizing virtual reality according to claim 1, characterized in that, The performing user interaction between the first user and the second user based on the first special effect animation and the second special effect animation includes: In the case where the first user is the main user, generate an interactive video based on the first special effect animation and the second special effect animation, wherein the first special effect animation is in the first-person perspective and the second special effect animation is in the third-person perspective; In the case where the second user is the main user, generate an interactive video based on the first special effect animation and the second special effect animation, wherein the second special effect animation is in the first-person perspective and the first special effect animation is in the third-person perspective.

5. The user interaction method for realizing virtual reality according to claim 1, wherein The first user information includes the three-dimensional pose model of the first user and the role information of multiple animation characters.

6. The user interaction method for realizing virtual reality according to claim 1, characterized in that The second user information includes the three-dimensional pose model of the second user and the role information of multiple animation characters.

7. The user interaction method capable of realizing virtual reality according to claim 5 or 6, characterized in that The role information includes an animation character model, a role special effect animation, and an interaction instruction.

8. A user interaction device capable of realizing virtual reality, characterized in that, including: An obtaining module, configured to obtain the first user information of the first user and the second user information of the second user; A first generating module, configured to generate a first character based on the first user information; And generate a second character based on the second user information; A second generating module, configured to generate a first special effect animation with the first character as the protagonist by real-time tracking of the actions and positions of the first user; and generate a second special effect animation with the second character as the protagonist by real-time tracking of the actions and positions of the second user; An interaction module, configured to perform user interaction between the first user and the second user based on the first special effect animation and the second special effect animation.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the user interaction method for realizing virtual reality as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the user interaction method for realizing virtual reality as described in any one of claims 1 to 7.

Images