Multi-User Virtual Interaction Method, Device and AR Device in AR Application

Abstract

Embodiments of the present application disclose a multi-user virtual interaction method, device, and AR device in an augmented reality (AR) application. In the embodiments of the present application, by using the structured information of the designated parts of different users at the same moment, 3D virtual parts corresponding to each user are determined, and through the 3D virtual parts corresponding to each user, the operation behaviors of each user in the AR application scenario are recognized to determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario and return the change amount to the AR devices corresponding to different users. The AR devices draw and display the 3D virtual object according to the received change amount, which realizes that even if there is a sequence relationship in the interaction of multiple users with the same 3D virtual object in the AR application scenario, such as editing, multiple users can operate on the same 3D virtual object at the same moment.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and particularly to a multi-user virtual interaction method, apparatus, and AR device in augmented reality (AR: Augmented Reality) applications. Background Art

[0002] Augmented reality, also known as augmented reality, is a technology that combines real-world information and virtual-world information content. Augmented reality can simulate and process entity information that is difficult to experience in the spatial scope of the real world on the basis of technologies such as computers, superimpose virtual information content on the real world for effective application, and can be perceived by the human senses during this process, thereby realizing a sensory experience beyond reality. After the real environment and virtual objects overlap, they can exist simultaneously in the same picture and space.

[0003] However, in current AR application scenarios, there is a sequence relationship among the interactions of multiple users with the same 3D virtual object in the AR application scenario, such as editing, etc., and they can only interact sequentially, and it is impossible to enable multiple users to operate on the same 3D virtual object at the same time. Summary of the Invention

[0004] Embodiments of this application disclose a multi-user virtual interaction method, apparatus, and AR device in augmented reality (AR) applications to enable multiple users to interact with the same 3D virtual object simultaneously in an AR application scenario.

[0005] Embodiments of this application provide a multi-user virtual interaction method in augmented reality (AR) applications. The method includes:

[0006] Obtain structured information of specified parts of different users at the same moment in the same AR application scenario;

[0007] According to the structured information of the specified part of each user obtained, drive the 3D virtual part template corresponding to the specified part to deform to obtain the 3D virtual part corresponding to the user; the 3D virtual part is used to match the current posture of the specified part of the user;

[0008] According to the 3D virtual parts corresponding to each user, identify the operation behaviors of each user in the AR application scenario, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario;

[0009] Return the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

[0010] An embodiment of the present application provides a multi-person virtual interaction device in an augmented reality (AR) application. The device is applied to a cloud server, and the method includes:

[0011] An acquisition unit, configured to acquire structured information of specified parts of different users at the same moment in the same AR application scenario;

[0012] A processing unit, configured to drive a 3D virtual part template corresponding to the specified part to deform according to the structured information of the specified part of each acquired user, so as to obtain a 3D virtual part corresponding to the user; the 3D virtual part is used to match the current posture of the specified part of the user;

[0013] A determination unit, configured to identify the operation behaviors of each user in the AR application scenario according to the 3D virtual parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario;

[0014] A sending unit, configured to return the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

[0015] An embodiment of the present application provides an AR device, which includes: an AR scene generator, an interaction device, a processor, and a display;

[0016] The AR scene generator is configured to generate an AR application scenario;

[0017] The interaction device is configured to interact with an external device to acquire structured information of specified parts of different users at the same moment in the same AR application scenario;

[0018] The processor is configured to drive a 3D virtual part template corresponding to the specified part to deform according to the structured information of the specified part of each acquired user, so as to obtain a 3D virtual part corresponding to the user; the 3D virtual part is used to match the current posture of the specified part of the user; and,

[0019] Identify the operation behaviors of each user in the AR application scenario according to the 3D virtual parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario; and return the change amount to the AR devices corresponding to different users;

[0020] The display is configured to draw and display the 3D virtual object according to the received change amount.

[0021] As can be seen from the above technical solution, the cloud server uses the structured information of the specified parts of different users at the same moment to determine the 3D virtual parts corresponding to each user. Through the 3D virtual parts corresponding to each user, the operation behaviors of each user in the AR application scenario are identified, so as to determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario and return the change amount to the AR devices corresponding to different users. The AR devices draw and display the 3D virtual object according to the received change amount, which realizes that even if there is a sequence relationship in the interactions of multiple users with the same 3D virtual object in the AR application scenario, such as editing, etc., multiple users can operate on the same 3D virtual object at the same moment.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this specification, and are used together with the specification to explain the principles of this specification.

[0024] Figure 1 It is a flowchart of the method provided by the embodiment of this application;

[0025] Figure 2 It is a schematic diagram of the interaction between the AR device and the cloud server provided by the embodiment of this application;

[0026] Figures 3a to 3c It is a schematic diagram of the 3D virtual hand provided by the embodiment of this application;

[0027] Figure 4 It is a schematic diagram of the operation gesture provided by the embodiment of this application;

[0028] Figure 5 It is a structural diagram of the device provided by the embodiment of this application;

[0029] Figure 6 It is a structural diagram of the electronic device provided by the embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] In order to realize the virtual interaction of multiple users with the same 3D virtual object simultaneously in the AR application scenario, this embodiment breaks the existing method of identifying the operations of the specified parts of users in the AR application scenario by the terminal (taking the AR device as an example), and places the operations of identifying the specified parts of users, etc. on the cloud server, and the cloud server and the AR device cooperate with each other to complete.

[0031] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application and to make the above-mentioned objectives, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] See Figure 1 , Figure 1 which is the flowchart of the method provided in the embodiment of the present application. As an embodiment, this method is applied to a cloud server or an AR device with similar server functions, and the present embodiment does not specifically limit this. As Figure 1 shown, this process may include the following steps:

[0033] Step 101, obtain the structured information of the specified parts of different users at the same moment in the same AR application scenario.

[0034] As an embodiment, the specified part structured information may be: the coordinate information of the upper joint points of the left hand and / or the right hand. Optionally, the coordinate information here is 3D coordinate information or 2D coordinate information.

[0035] As another embodiment, the specified part structured information here may also be information such as the lengths, thicknesses, and included angles of each finger joint on the left hand and / or the right hand, and the present embodiment does not specifically limit this.

[0036] In this embodiment, the structured information of the specified parts of different users at the same moment in the same AR application scenario is reported by the AR devices corresponding to different users, which will be described by examples below and will not be elaborated here for the time being. Figure 2 Taking the AR device as an AR glasses as an example, examples are shown of different AR glasses reporting the structured information of the specified parts of different users at the same moment in the same AR application scenario to the cloud server ( Figure 2 marked as cloud server 20).

[0037] Step 102, according to the structured information of the specified part of each user obtained, drive the 3D virtual part template corresponding to the specified part to deform, and obtain the 3D virtual part corresponding to this user; the 3D virtual part is used to match the current posture of the specified part of this user.

[0038] In this embodiment, the 3D virtual part template is pre-constructed, and the present embodiment does not specifically describe how to construct the 3D virtual part template.

[0039] In this embodiment, when the above-mentioned specified part structured information is the 2D coordinate information of the upper joint points of the left hand and / or the right hand, before performing this step 102, the obtained 2D coordinate information can be further converted into 3D coordinate information. As for how to convert 2D coordinate information into 3D coordinate information, it is similar to the existing coordinate conversion, and the present embodiment does not specifically limit this.

[0040] Step 103: Identify the operation behaviors of each user in the AR application scenario based on the 3D virtual body parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario.

[0041] For example, in an AR game scenario of different users playing basketball, based on this Step 103, the change amount of the 3D basketball, such as speed, acceleration, movement direction, position, etc., can be calculated in real time according to the collision between the virtual 3D basketball and the 3D virtual body parts of each user, so as to realize that in this AR game scenario, different speeds and different contact positions of the user hitting the basketball will trigger different motion states of the basketball, ensuring that this interaction experience approaches the real experience.

[0042] For another example, in multi-person activities such as multi-person surgical operations, multi-person construction of ceramic products, and multi-person shooting games, they are all similar to the above-mentioned AR basketball game scenario, and the specific application scenarios are not specifically limited in this embodiment.

[0043] Step 104: Return the above change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

[0044] When the AR device receives the above change amount, it will redraw and display the 3D virtual object based on the change amount. Still taking the AR game scenario of different users playing basketball as an example, if the change amount is the updated position of the basketball, the AR device will draw the 3D basketball at the updated position of the basketball.

[0045] So far, the Figure 1 shown process is completed.

[0046] Through Figure 1 As can be seen from the shown process, in this embodiment, the cloud server uses the specified part structured information of different users at the same moment to determine the 3D virtual body parts corresponding to each user, identifies the operation behaviors of each user in the AR application scenario through the 3D virtual body parts corresponding to each user, so as to determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario and return the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount, which realizes that even if there is a sequence relationship in the interaction such as editing of the same 3D virtual object in the AR application scenario by multiple users, multiple users can operate on the same 3D virtual object at the same moment.

[0047] Next, taking the specified part structured information as the hand joint point information as an example, the method provided by the embodiment of the present application will be described:

[0048] For the AR device side:

[0049] The AR device performs the following operations to enable different AR devices to send the hand joint point information of different users at the same moment in the same AR application scenario to the cloud server:

[0050] Image acquisition:

[0051] The acquired images such as color images, grayscale images, infrared images, depth images, etc. The present embodiment does not specifically limit the format and quantity of the acquired images.

[0052] Hand detection:

[0053] Detect the hand area (such as the left hand area and / or the right hand area) from the acquired images, and obtain the pixel coordinates of the hand center point in the hand area (if the input image is a depth image, 3D coordinates are obtained). For example, if the left hand area or the right hand area is detected in the acquired images, obtain the pixel coordinates of the hand center point in the left hand area or the right hand area (if the input image is a depth image, 3D coordinates are obtained). If both the left hand area and the right hand area are detected in the acquired images, obtain the pixel coordinates of the hand center point in the left hand area and the right hand area respectively (if the input image is a depth image, 3D coordinates are obtained).

[0054] 3D joint point detection:

[0055] Intercept the detected hand area from the acquired images, and detect the joint points from the intercepted hand area to obtain the hand joint point coordinate information (such as 3D joint point coordinate information). Taking the hand area as the left hand area as an example, detect the joint points from the intercepted hand area of the left hand to obtain the left hand joint point coordinate information (such as 3D joint point coordinate information); then taking the hand area as the right hand area as an example, detect the joint points from the intercepted hand area of the right hand to obtain the right hand joint point coordinate information (such as 3D joint point coordinate information).

[0056] After that, the AR device will send the hand joint point information to the cloud server. The hand joint point information at least includes the above-mentioned hand joint point coordinate information such as the left hand joint point coordinate information (such as 3D joint point coordinate information) and / or the right hand joint point coordinate information (such as 3D joint point coordinate information).

[0057] As an embodiment, the above-mentioned hand joint point information may further include time information. Here, the time information may be the time of the above-mentioned acquired images. Also, if the above-mentioned AR device is the AR device corresponding to the user who initiates or constructs the above-mentioned AR application scenario, the above-mentioned hand joint point information further includes the 3D coordinate information of the 3D virtual object in the above-mentioned AR application scenario.

[0058] Finally, through the above description, different AR devices are enabled to send the hand joint point information of different users at the same moment in the same AR application scenario to the cloud server.

[0059] For the cloud server side:

[0060] When the cloud server receives the hand joint point information sent by the AR device, the following operations are performed:

[0061] 3D hand reconstruction:

[0062] In this embodiment, according to the hand joint point coordinate information (such as 21 3D coordinate information on the left or right hand), for example Figure 3a as shown, a basic hand skeleton can be constructed, for example Figure 3b as shown, according to this skeleton, a 3D virtual hand template can be driven to deform to obtain a 3D virtual hand in the current hand posture, for example Figure 3c as shown.

[0063] Natural interaction:

[0064] Using the simulation attributes such as collision, elasticity, and friction of the 3D game engine, the interaction with the same 3D virtual object is realized based on the physical attributes such as the physical collision, movement speed, and friction of the contact surface between multiple 3D virtual hands and 3D virtual objects.

[0065] For example, still taking the AR game scenario of different users playing basketball as an example, the 3D game engine calculates the speed, acceleration, movement direction, and position of the 3D basketball in real time according to the collision between the virtual 3D basketball and the 3D virtual hand restored from the user's real hand. Different speeds and contact positions of the user hitting the basketball will trigger different motion states of the basketball, and this interaction experience approaches the real experience.

[0066] As another embodiment, when realizing the interaction with the same 3D virtual object based on the physical attributes such as the physical collision, movement speed, and friction of the contact surface between multiple 3D virtual hands and 3D virtual objects, the operation attributes of each user's operation behavior in the AR application scenario that have been obtained can also be referred to. Here, the operation attributes such as the strength and direction of the operation behavior can ensure that the change amount of the above-mentioned 3D virtual object calculated in real time is more accurate. For example, the user hitting the basketball causes changes in the basketball such as speed, acceleration, movement direction, and position, ensuring that the interaction experience approaches the real experience.

[0067] Optionally, in this embodiment, after natural interaction, the position and speed of the 3D virtual object will change, and the change amount (including but not limited to position, speed, etc.) is sent to the AR device in the above AR application scenario. Any AR device that receives the above change amount will redraw and display the 3D virtual object according to the change amount. For example, if the change amount is the updated position of the 3D virtual object, the AR device will draw the corresponding 3D virtual object at the updated position. Another example is that if the change amount is the speed of the 3D virtual object, in the application, the rendering frame rate may be higher than the frame rate of natural interaction in the cloud. Therefore, before the state of the 3D virtual object is updated, the new position of the 3D virtual object can be calculated locally using the speed and acceleration, and the 3D virtual object is drawn and displayed at the new position. Finally, it is realized that even if multiple users have a sequence relationship in the interaction with the same 3D virtual object in the AR application scenario, such as editing, multiple users can operate on the same 3D virtual object at the same time.

[0068] It should be noted that, in this embodiment, the cloud server can further identify the action corresponding to the specified part according to the obtained structured information of the specified part of each user; and respond to the action in the AR application scenario;

[0069] As an embodiment, when the action indicates a menu click in the AR application scenario, such as Figure 4 the click shown, respond to the menu click to provide the menu clicked in the AR application scenario.

[0070] As another embodiment, when the action indicates a grasping operation in the AR application scenario and the joint points of the specified part are inside the 3D virtual object, respond to the grasping operation to provide the 3D virtual object that has been grasped.

[0071] By identifying the actions corresponding to the specified parts of different users as described above, it is also realized that even if multiple users have a sequence relationship in the interaction with the same 3D virtual object in the AR application scenario, such as editing, multiple users can operate on the same 3D virtual object at the same time on the cloud server side.

[0072] The method provided in the embodiments of the present application has been described above. Next, the device provided in the embodiments of the present application will be described:

[0073] See Figure 5 Figure 5 which is the structure diagram of the device provided in the embodiments of the present application. This device is applied to a cloud server or an AR device with a similar server function. As Figure 5 shown, the device may include:

[0074] ​An acquisition unit, configured to acquire structured information of a specified part of different users at the same moment in the same AR application scenario;

[0075] A processing unit, configured to drive a 3D virtual part template corresponding to the specified part to deform according to the structured information of the specified part of each acquired user, so as to obtain a 3D virtual part corresponding to the user; the 3D virtual part is used to match the current posture of the specified part of the user;

[0076] A determination unit, configured to identify the operation behaviors of each user in the AR application scenario according to the 3D virtual parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario;

[0077] A sending unit, configured to return the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

[0078] As an embodiment, the acquisition of the structured information of the specified part of different users at the same moment in the same AR application scenario includes:

[0079] Receiving the structured information of the specified part of each user corresponding to the AR device in the AR application scenario at the first moment; wherein, the structured information of the specified part reported by the AR devices corresponding to different users is obtained from the images collected by different AR devices at the first moment.

[0080] As an embodiment, the driving of the 3D virtual part template corresponding to the specified part to deform according to the structured information of the specified part of each acquired user, so as to obtain a 3D virtual part corresponding to the user includes:

[0081] For the structured information of the specified part of each acquired user, constructing a part skeleton according to the structured information;

[0082] Searching for a 3D virtual part template that matches the specified part in the constructed 3D virtual part template;

[0083] Deforming the found 3D virtual part template according to the part skeleton to obtain a 3D virtual part corresponding to the user.

[0084] As an embodiment, the determination of the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario includes:

[0085] Utilize the configured physical properties through the deployed 3D game engine to simulate the operation behaviors of each user in the AR application scenario to determine the change amount that causes the 3D virtual object to change; or, utilize the configured physical properties through the deployed 3D game engine and, according to the operation attributes of each user's operation behaviors obtained in the AR application scenario, simulate the operation behaviors of each user in the AR application scenario to determine the change amount that causes the 3D virtual object to change;

[0086] Wherein, the change amount at least includes at least one of the following: speed, acceleration, movement direction, position; the physical properties at least include: collision, friction of the contact surface, elasticity.

[0087] As an embodiment, the processing unit further identifies the action corresponding to the specified part according to the structured information of the specified part of each user obtained; and responds to the action in the AR application scenario;

[0088] Wherein, when the action indicates a menu click in the AR application scenario, respond to the menu click to provide the menu clicked in the AR application scenario; when the action indicates a grabbing operation in the AR application scenario and the joint point of the specified part is inside the 3D virtual object, respond to the grabbing operation to provide the grabbed 3D virtual object.

[0089] As an embodiment, the specified part structured information at least includes: coordinate information of the upper joint points of the left hand and / or the right hand;

[0090] The coordinate information is 3D coordinate information or 2D coordinate information;

[0091] When the coordinate information is 2D coordinate information, before driving the deformation of the 3D virtual part template corresponding to the specified part according to the structured information of the specified part of each user obtained, it further includes: converting the obtained 2D coordinate information into 3D coordinate information.

[0092] Thus far, the structure description of the Figure 5 shown device is completed.

[0093] Correspondingly, the present application also provides the Figure 5 hardware structure of the shown device. Refer to Figure 6 , the hardware structure may include an AR scene generator, an interaction device, a processor, and a display.

[0094] In this embodiment, the AR scene generator is used to generate an AR application scenario, for example, finally generate the corresponding AR application scenario through the modeling, management, and rendering of the AR application scenario.

[0095] An interaction device, such as a handle, a grip sensor, a stylus, a remote control, a touch screen, a button, etc., is used to implement the input and output of sensory signals and environmental control operation signals. Applied to this embodiment, it can obtain the structured information of the specified parts of different users at the same moment in the same AR application scenario by interacting with external devices.

[0096] A processor, configured to drive the 3D virtual part template corresponding to the specified part to deform according to the obtained structured information of the specified part of each user, so as to obtain the 3D virtual part corresponding to the user; the 3D virtual part is used to match the current posture of the specified part of the user; and,

[0097] According to the 3D virtual parts corresponding to each user, identify the operation behaviors of each user in the AR application scenario, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario; and return the change amount to the AR devices corresponding to different users;

[0098] The display, such as a transmissive head-mounted display, etc., is configured to draw and display the 3D virtual object according to the received change amount. In this embodiment, the display is mainly responsible for displaying the signals after virtual and real fusion.

[0099] In this embodiment, the AR device further includes a tracker, such as an eye tracking device, for obtaining the eye movement changes of the user to ensure that the interaction device can obtain the above signals in real time.

[0100] As an embodiment, the above interaction device further obtains the operation attributes of each user when performing operation behaviors in the AR application scenario. Here, the operation attributes can be the force, direction, etc. of performing the operation behavior, and this embodiment does not specifically limit them.

[0101] When the above interaction device further obtains the operation attributes of each user when performing operation behaviors in the AR application scenario, then as an embodiment, the processor can use the configured physical attributes through the deployed 3D game engine, and simulate the operation behaviors of each user in the AR application scenario according to the operation attributes to determine the change amount that causes the 3D virtual object to change. The change amount includes at least one of the following at least one: speed, acceleration, movement direction, position; the physical attributes include at least: collision, friction of the contact surface, elasticity.

[0102] As an embodiment, the processor drives the 3D virtual part template corresponding to the specified part to deform according to the obtained structured information of the specified part of each user, so as to obtain the 3D virtual part corresponding to the user, including:

[0103] For the structured information of the specified part of each obtained user, construct a part skeleton according to the structured information;

[0104] In the constructed 3D virtual part template, find the 3D virtual part template that matches the specified part;

[0105] Deform the found 3D virtual part template according to the part skeleton to obtain the 3D virtual part corresponding to the user.

[0106] As an embodiment, the structured information of the specified parts of different users at the same moment in the same AR application scenario obtained by the above interactive device includes:

[0107] Receive the structured information of the specified part of each user corresponding to the AR device in the AR application scenario at the first moment; wherein, the structured information of the specified part reported by the AR devices corresponding to different users is obtained from the images collected by different AR devices at the first moment.

[0108] As an embodiment, the processor further identifies the action corresponding to the specified part according to the structured information of the specified part of each obtained user; and responds to the action in the AR application scenario;

[0109] Wherein, when the action indicates a menu click in the AR application scenario, respond to the menu click to provide the menu clicked in the AR application scenario; when the action indicates a grabbing operation in the AR application scenario and the joint points of the specified part are inside the 3D virtual object, respond to the grabbing operation to provide the grabbed 3D virtual object.

[0110] As an embodiment, the structured information of the specified part at least includes: the coordinate information of the upper joint points of the left hand and / or the right hand;

[0111] The coordinate information is 3D coordinate information or 2D coordinate information;

[0112] When the coordinate information is 2D coordinate information, before driving the 3D virtual part template corresponding to the specified part to deform according to the structured information of the specified part of each obtained user, it further includes: converting the obtained 2D coordinate information into 3D coordinate information.

[0113] So far, the structure description of the Figure 6 shown AR device is completed.

[0114] Based on the same application concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above examples of the present application can be implemented.

[0115] Exemplarily, the above machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0116] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0117] For the convenience of description, the above devices are described by dividing them into various units according to functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0119] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate for implementing the processFigure 1 a process or multiple processes and / or blocks Figure 1 a device for the functions specified in a block or multiple blocks.

[0120] Furthermore, these computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device that implements the process Figure 1 a process or multiple processes and / or blocks Figure 1 the functions specified in a block or multiple blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the process Figure 1 a process or multiple processes and / or blocks Figure 1 the functions specified in a block or multiple blocks.

[0122] The above are only embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A multi-user virtual interaction method in an augmented reality (AR) application, characterized in that, The method includes: Obtaining structured information of a specified part of different users at the same moment in the same AR application scenario; wherein, the structured information of the specified part of different users at the same moment is obtained by detecting images collected by AR devices corresponding to different users at the same moment; According to the obtained structured information of the specified part of each user, driving the 3D virtual part template corresponding to the specified part to deform to obtain the 3D virtual part corresponding to this user; the 3D virtual part is used to match the current posture of the specified part of this user; According to the 3D virtual parts corresponding to each user, identifying the operation behaviors of each user in the AR application scenario, and determining the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario; Returning the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

2. The method according to claim 1, characterized in that, The obtaining of the structured information of the specified part of different users at the same moment in the same AR application scenario includes: Receiving the structured information of the specified part of each user corresponding to the AR device in the AR application scenario at the first moment; wherein, the structured information of the specified part reported by AR devices corresponding to different users is obtained from the images collected by different AR devices at the first moment.

3. The method according to claim 1, wherein The driving the 3D virtual part template corresponding to the specified part to deform according to the obtained structured information of the specified part of each user to obtain the 3D virtual part corresponding to this user includes: For the obtained structured information of the specified part of each user, constructing a part skeleton according to the structured information; Searching for the 3D virtual part template that matches the specified part in the already constructed 3D virtual part template; Deforming the found 3D virtual part template according to the part skeleton to obtain the 3D virtual part corresponding to this user.

4. The method according to claim 1, characterized in that, The determining of the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario includes: Using the configured physical properties through the deployed 3D game engine to simulate the operation behaviors of each user in the AR application scenario to determine the change amount that causes the 3D virtual object to change; or, using the configured physical properties through the deployed 3D game engine and according to the operation properties of each user performing operation behaviors in the AR application scenario, simulating the operation behaviors of each user in the AR application scenario to determine the change amount that causes the 3D virtual object to change; Wherein, the change amount at least includes at least one of the following: speed, acceleration, movement direction, position; the physical properties at least include: collision, friction of the contact surface, elasticity.

5. The method according to claim 1, characterized in that The method further includes: Identifying the action corresponding to the specified part according to the obtained structured information of the specified part of each user; Responding to the action in the AR application scenario; Wherein, when the action indicates a menu click in the AR application scenario, the menu click is responded to provide the menu clicked in the AR application scenario; when the action indicates a grasping operation in the AR application scenario and the joint points of the specified part are inside the 3D virtual object, the grasping operation is responded to provide the grasped 3D virtual object.

6. The method according to any one of claims 1 to 5, characterized in that The structured information of the specified part at least includes: coordinate information of joint points on the left hand and / or the right hand; The coordinate information is 3D coordinate information or 2D coordinate information; When the coordinate information is 2D coordinate information, before driving the deformation of the 3D virtual part template corresponding to the specified part according to the structured information of the specified part of each obtained user, it further includes: converting the obtained 2D coordinate information into 3D coordinate information.

7. A multi-person virtual interaction device in an augmented reality (AR) application, characterized in that, The device includes: An obtaining unit, configured to obtain the structured information of the specified parts of different users at the same moment in the same AR application scenario; wherein, the structured information of the specified parts of different users at the same moment is detected from the images collected by the AR devices corresponding to different users at the same moment; A processing unit, configured to drive the deformation of the 3D virtual part template corresponding to the specified part according to the structured information of the specified part of each obtained user, to obtain the 3D virtual part corresponding to this user; the 3D virtual part is used to match the current posture of the specified part of this user; A determining unit, configured to identify the operation behaviors of each user in the AR application scenario according to the 3D virtual parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario; A sending unit, configured to return the change amount to the AR devices corresponding to different users, and the AR devices draw and display the 3D virtual object according to the received change amount.

8. An AR device, characterized in that, The AR device includes: an AR scene generator, an interaction device, a processor, and a display; The AR scene generator is configured to generate an AR application scenario; The interaction device is configured to interact with an external device to obtain the structured information of the specified parts of different users at the same moment in the same AR application scenario; The processor is configured to drive the deformation of the 3D virtual part template corresponding to the specified part according to the structured information of the specified part of each obtained user, to obtain the 3D virtual part corresponding to this user; the 3D virtual part is used to match the current posture of the specified part of this user; wherein, the structured information of the specified parts of different users at the same moment is detected from the images collected by the AR devices corresponding to different users at the same moment; and, Identify the operation behaviors of each user in the AR application scenario according to the 3D virtual parts corresponding to each user, and determine the change amount of the 3D virtual object that changes based on the operation behaviors of each user in the AR application scenario according to the operation behaviors of each user in the AR application scenario; and, return the change amount to the AR devices corresponding to different users; The display is configured to draw and display the 3D virtual object based on the received change amount.

9. The AR device according to claim 8, wherein, The processor drives the deformation of the 3D virtual part template corresponding to the specified part according to the structured information of the specified part of each user obtained, and the obtained 3D virtual part corresponding to the user includes: For the structured information of the specified part of each user obtained, a part skeleton is constructed according to the structured information; Search for the 3D virtual part template that matches the specified part in the already constructed 3D virtual part template; Deform the found 3D virtual part template according to the part skeleton to obtain the 3D virtual part corresponding to the user.

10. The AR device according to claim 8, wherein The processor determines the change amount of the 3D virtual object that changes based on the operation behavior of each user in the AR application scenario, including: Using the configured physical properties through the deployed 3D game engine to simulate the operation behavior of each user in the AR application scenario to determine the change amount that causes the 3D virtual object to change; or, further obtaining the operation attributes of each user performing the operation behavior in the AR application scenario through the interaction device, using the configured physical properties through the deployed 3D game engine, and simulating the operation behavior of each user in the AR application scenario according to the operation attributes to determine the change amount that causes the 3D virtual object to change; The change amount at least includes at least one of the following: speed, acceleration, movement direction, position; the physical properties at least include: collision, friction of the contact surface, elasticity.

Images