AR display method and device, wearable component and storage medium
Through the combination of wearable components and AR devices, AR animation display and interaction across devices is realized, solving the limitations of real-time rendering and interaction in the existing technology, and improving the application efficiency and interactivity of AR technology in the fields of education and medical care. .
Patent Information
- Application Number
- CN202310611171.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
The existing AR technology has technical limitations in real-time rendering and interaction, making it difficult to realize real-time virtual animation display and interaction across devices, especially in the transmission and feedback of actions between different users.
The user's skeletal animation information is collected through wearable components, and the AR device is used for information transmission and binding, so as to realize cross-device AR animation display, support recording, playback and copying functions, and use multi-line lidar, binocular fish-eye camera and The AR processor performs spatial positioning and action binding.
It realizes user skeletal animation information transmission and cross-device AR animation display between different AR devices, supports teaching, interaction and action feedback, and improves the application efficiency and interactivity of AR technology in education, medical care and other fields.
Smart Images

Figure CN119027556A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of communication technology, and in particular to an AR display method, device, wearable component and storage medium. Background Art
[0002] With the rapid development of information technology, augmented reality (AR) has become increasingly popular.
[0003] In related technologies, AR technology supports superimposing a pre-set virtual model on the real world in the space where the user's AR device is located, and displaying AR animation. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides an augmented reality (AR) display method, device, wearable component and storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, an augmented reality (AR) display method is provided, including: obtaining skeletal animation information of a second user sent by a second AR device, the skeletal animation information of the second user being collected by a wearable component worn by the second user and transmitted to the second AR device; binding the skeletal animation information of the second user to a character model, and driving the character model to display the AR animation of the second user in a space where the first user wears the first AR device.
[0006] In one embodiment, the method further includes: recording and storing the AR animation of the second user.
[0007] In one embodiment, the method further includes: in response to receiving a playback instruction to play back the AR animation of the second user, playing back the stored AR animation of the second user.
[0008] In one embodiment, the method further includes: in response to receiving a copying instruction to copy the AR animation of the second user, monitoring the copying action of the first user; analyzing the similarity between the copying action and the action displayed by the AR animation of the second user; if the similarity is less than a threshold, issuing a prompt message, wherein the prompt message is used to prompt that the similarity between the copying action and the action displayed by the AR animation is less than the threshold.
[0009] In one embodiment, the method also includes: obtaining skeletal animation information of the first user collected by a wearable component worn by the first user; sending the skeletal animation information of the first user to a second AR device, and the second AR device driving the display of the AR animation of the first user based on the skeletal animation information of the first user.
[0010] In one embodiment, the wearable component includes multiple AR detection components, and the multiple AR detection components are arranged at the skeleton key point positions for generating skeleton animation information.
[0011] According to a second aspect of an embodiment of the present disclosure, an augmented reality (AR) display device is provided, including: an acquisition unit, used to acquire skeletal animation information of a second user sent by a second AR device, the skeletal animation information of the second user being collected by a wearable component worn by the second user and transmitted to the second AR device; a processing unit, used to bind the skeletal animation information of the second user to a character model, and drive the character model to display the AR animation of the second user in a space where the first user wears the first AR device.
[0012] In one implementation, the processing unit is further configured to: record and store the AR animation of the second user.
[0013] In one implementation, the processing unit is further configured to: in response to receiving a playback instruction to play back the AR animation of the second user, play back the stored AR animation of the second user.
[0014] In one embodiment, the processing unit is also used to: in response to receiving a copying instruction to copy the AR animation of the second user, monitor the copying action of the first user; analyze the similarity between the copying action and the action displayed by the AR animation of the second user; if the similarity is less than a threshold, issue a prompt message, and the prompt message is used to prompt that the similarity between the copying action and the action displayed by the AR animation is less than the threshold.
[0015] In one embodiment, the acquisition unit is also used to: acquire skeletal animation information of the first user collected by a wearable component worn by the first user; the device also includes: a sending unit, which is used to send the skeletal animation information of the first user to a second AR device, and the second AR device drives the display of the AR animation of the first user based on the skeletal animation information of the first user.
[0016] According to a third aspect of an embodiment of the present disclosure, there is provided a wearable component, including:
[0017] A plurality of AR detection components are arranged at the positions of the skeleton key points for generating the skeleton animation information.
[0018] According to a fourth aspect of the embodiments of the present disclosure, an augmented reality (AR) device is provided, including:
[0019] a processor; a memory for storing instructions executable by the processor;
[0020] The processor is configured to call the instructions stored in the memory to execute the AR display method as described in the first aspect or any one of the embodiments of the first aspect.
[0021] According to a fifth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of an AR device, the AR device can execute the AR display method as described in the first aspect or any one of the embodiments of the first aspect.
[0022] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: the first AR device used by the first user obtains the skeletal animation information of the second user sent by the second AR device used by the second user, thereby realizing the transmission of the skeletal animation information of users between different AR devices. The first AR device binds the skeletal animation information of the second user to the character model, thereby realizing the display of the AR animation of the second user in the space where the first user wears the first AR device, thereby realizing the AR animation display across devices.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0025] Figure 1 The figure is a flowchart of an augmented reality (AR) display method according to an exemplary embodiment.
[0026] Figure 2 is a schematic diagram of a wearable component according to an exemplary embodiment.
[0027] Figure 3A The present invention is a schematic diagram showing a process of binding skeletal animation information with a character model according to an exemplary embodiment.
[0028] Figure 3B The present invention is a schematic diagram showing a flow chart of binding skeletal animation information with a character model according to an exemplary embodiment.
[0029] Figure 4 A schematic diagram of a process of performing AR animation display based on a 3D character model in an exemplary embodiment of the present disclosure is shown.
[0030] Figure 5 A trajectory effect diagram shown in an exemplary embodiment of the present disclosure is shown.
[0031] Figure 6The figure is a flowchart of an AR display method according to an exemplary embodiment.
[0032] Figure 7 The figure is a flowchart of playing back an AR animation according to an exemplary embodiment.
[0033] Fig. 8A The figure is a flowchart of copying an AR animation according to an exemplary embodiment.
[0034] Figure 8B The figure is a flowchart of copying an AR animation according to an exemplary embodiment.
[0035] Fig. 9 The figure is a flowchart of an AR display method according to an exemplary embodiment.
[0036] Fig.10 A schematic diagram of an AR display implementation process shown in an exemplary embodiment of the present disclosure is shown.
[0037] Fig.11 A schematic diagram of a process of online AR teaching between a student and a coach is shown.
[0038] Fig.12 The figure is a schematic diagram of an AR display scene according to an exemplary embodiment.
[0039] Fig.13 A flowchart of an implementation of watching an AR animation is shown according to an exemplary embodiment of the present disclosure.
[0040] Fig.14 The figure is a schematic diagram of an AR display scene according to an exemplary embodiment.
[0041] Fig.15 is a block diagram of an AR display device according to an exemplary embodiment.
[0042] Fig.16 It is a block diagram of a device for AR display according to an exemplary embodiment. DETAILED DESCRIPTION
[0043] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0044] The AR display method provided by the embodiments of the present disclosure is applied to the field of virtual reality technology, for example, to AR scenes.
[0045] In an AR scenario, the AR display function is mainly implemented based on an AR device, where the AR device may be, for example, AR glasses.
[0046] Among them, AR devices have functions such as scene construction, object tracking, and binding of skeletal animation information to character models. For example, AR devices include: multi-line laser radar, high-precision binocular fisheye camera and AR processor. Among them, the multi-line laser radar can build a 3D point cloud map of the scene, and the perceived environmental information features are rich, the positioning and matching are stable, and it can adapt to wide outdoor scenes and complex indoor scenes. The high-precision binocular fisheye camera can provide real-time image data and track the real-time environment. It can expand the scene faster, prevent drift, and provide more complete scene data, thereby completing the tracking between people and objects. The AR processor can calculate spatial positioning (Simultaneous localization and mapping, SLAM) based on the laser radar and camera data, and calculate the posture of the wearable device in real time to complete the binding and update of skeletal movements.
[0047] In related technologies, AR devices add virtual models to the real environment to display corresponding AR animations. Among them, the display of AR animations by AR devices can achieve the purpose of reducing props and demonstrations. For example, in the medical industry, virtual human skeleton models, heart models, etc. are placed on the desktop; for example, AR is used to place a dancing two-dimensional model animation on the ground, which can play the set animation.
[0048] However, due to the technical limitations of hardware and software, computers are unable to truly solve the problems of real-time rendering and interaction. The application of AR technology in today's society is mainly limited to games, advertising, medical care, education and other fields.
[0049] In view of this, an embodiment of the present disclosure provides an augmented reality (AR) display method, which transmits skeletal animation information acquired by the wearable body parts to the AR device by communicating with the AR device via wearable body parts, and interacts with the skeletal animation information through different AR devices to achieve cross-device AR animation display.
[0050] In the disclosed embodiment, the first AR device and the second AR device are two different AR devices that can communicate and interact. Different users wearing different AR devices are called the first user and the second user. The first user wears the first AR device, and the second user wears the second AR device.
[0051] Figure 1 is a flowchart of an AR display method according to an exemplary embodiment. Figure 1 As shown, the AR display method is used in an AR device and includes the following steps.
[0052] In step S11, skeletal animation information of the second user sent by the second AR device is obtained.
[0053] The skeletal animation information of the second user is determined by the second AR device based on the posture data of the second user collected by the wearable component worn by the second user.
[0054] In the disclosed embodiment, the second user performs a corresponding action, and the posture data corresponding to the action is obtained through the wearable component worn by the second user. The wearable component worn by the second user communicates and interacts with the second AR device to transmit the posture data of the second user to the second AR device. The second AR device determines the skeletal animation information of the second user based on the posture data of the second user.
[0055] The second AR device sends the skeletal animation information of the second user to the first AR device. The first AR device obtains the skeletal animation information of the second user sent by the second AR device.
[0056] In step S12, the skeleton animation information of the second user is bound to the character model, and the character model is driven to display the AR animation of the second user in the space where the first user wears the first AR device.
[0057] In the disclosed embodiments, the role model can be understood as a model used to generate a role displayed in an AR animation. For example, it can be a 3D model. For example, the role model can be a humanoid model. For example, it can be a cartoon model, a realistic model, a robot model, etc.
[0058] In the embodiment of the present disclosure, the role model may be determined by user selection on the AR application running on the second AR device.
[0059] Among them, the AR device drives (or can also be called running) the role model and can display AR animation.
[0060] In the disclosed embodiment, the first AR device receives the skeletal animation information sent by the second AR device, and binds it to the character model, which can be understood as a character model that integrates the skeletal animation information of the second user. The first AR device runs the character model that integrates the skeletal animation information of the second user, so as to realize the display of the AR animation in the space where the first user wears the AR device. That is, the display of the skeletal animation information of the second user at the first AR device is realized, and the AR animation display across devices is realized. For example, in a teaching scenario, the coach is the second user and the student is the first user, so the demonstration can be carried out without the second user arriving at the site, thereby realizing the effect of interactive feedback between the coach and the student.
[0061] The AR display method involved in the above embodiment will be described below in the embodiments of the present disclosure.
[0062] In one implementation of the disclosed embodiment, the wearable component includes a plurality of AR detection components, and the plurality of AR detection components are arranged at the positions of the skeleton key points for generating the skeleton animation information.
[0063] Among them, the positions of the key points of the skeleton can be, for example, the left hand, the right hand, the left upper arm, the right upper arm, the left lower arm, the head, the torso, the buttocks, the left thigh, the right thigh, the left calf and the right calf.
[0064] It can be understood that the wearable component provided in the embodiment of the present disclosure can be understood as a motion capture clothing, referred to as motion capture clothing for short.
[0065] Figure 2 FIG. 1 is a schematic diagram of a wearable component according to an exemplary embodiment. Figure 2 As shown, the wearable component can be worn on the user as motion capture clothing, and multiple AR detection components are set at the key points of the user's skeleton. Among them, the AR detection component can be understood as a physical inertial sensor. The AR detection component can be used to record, track, and measure the posture data of the part. The posture data can be used to subsequently generate skeleton key frame data.
[0066] In the disclosed embodiment, the skeleton key frame data may also be referred to as skeleton animation information.
[0067] In the disclosed embodiment, the skeleton animation information is bound to the character model, so as to achieve the fusion of the skeleton animation information and the character model, which can also be understood as skeleton redirection.
[0068] Figure 3A FIG. 1 is a schematic diagram showing a process of binding skeletal animation information to a character model according to an exemplary embodiment. Figure 3A As shown, the AR device collects the motion information of the wearable components in the left hand, right hand, left upper arm, right upper arm, left lower arm, head, torso, buttocks, left thigh, right thigh, left calf and right calf to complete the acquisition of posture data of relevant joints such as the human hand, upper arm, lower arm, head, torso, buttocks, thigh and calf. Based on the posture data, skeletal animation information is generated. The skeletal animation information is bound to the 3D character model selected by the user to complete the skeletal redirection of the selected 3D character model.
[0069] Figure 3B FIG. 1 is a flow chart showing a process of binding skeletal animation information to a character model according to an exemplary embodiment. Figure 3B As shown, the wearable component collects posture data, generates skeletal animation information based on the collected posture data, and binds the skeletal animation information to the 3D character model to redirect the skeletal animation information.
[0070] The 3D character model involved in the embodiments of the present disclosure can be understood as the character model involved in the above embodiments. The two names are often used interchangeably, and those skilled in the art should understand the consistency of their meanings.
[0071] The 3D character model can be selected by the user. The user selects a favorite 3D model on the virtual screen through the AR device, where the gender is optional, and the humanoid model has cartoon models, realistic models, robot models, etc. The 3D character model selected by the user is bound to the skeleton animation information to complete the skeleton redirection. Subsequently, AR animation display is performed based on the 3D character model that has completed the skeleton redirection.
[0072] In the disclosed embodiment, the process of driving the 3D character model to perform AR animation display can also be understood as the process of animation drawing.
[0073] Figure 4 FIG. 1 shows a schematic diagram of a process of performing AR animation display based on a 3D character model in an exemplary embodiment of the present disclosure. Figure 4 As shown, the user selects a 3D character model on the virtual screen. The 3D character model is bound to the skeleton animation information to complete the skeleton redirection. For the 3D character model that has completed the skeleton redirection, the drawing engine can be used to draw the model, that is, to complete the animation drawing (AR animation display).
[0074] It is understandable that, during the AR animation display process, the joints that require auxiliary lines can be configured, and the motion trajectory of the human joints can be calculated, and then the drawing of the motion trajectory can be completed based on the drawing engine.
[0075] It is understandable that the process of performing AR animation display in the disclosed embodiment also involves spatial positioning SLAM calculation. For example, 3D laser radar is used for spatial positioning. Through a high-precision binocular fisheye camera, the space is scanned and calculated to complete the map creation and the position of the glasses on the map. When performing real-time model animation drawing, a 3D character model is created based on the posture data input by the motion capture clothing, the skeleton data is generated and bound, and finally the model animation is driven according to the dynamic skeleton data. The AR device will also analyze each action and provide motion trajectory lines when drawing the model's arms, hands, legs, and feet, which the user can choose to display and hide.
[0076] in, Figure 5 FIG. 2 shows a trajectory effect diagram shown in an exemplary embodiment of the present disclosure. Figure 5As shown in the figure, if some actions are too complicated, the AR device can choose to configure auxiliary lines on certain joints to observe the action trajectory more intuitively. The motion trajectory of the joint is calculated by binding the 3D model selected by the user with the skeletal animation information and redirecting the updated 3D model. The model and motion trajectory are drawn using the drawing engine in combination with the updated 3D model, and the user can choose to show or hide it.
[0077] It can be understood that the AR device involved in the above embodiment can be the first AR device or the second AR device.
[0078] In the disclosed embodiment, the following description will still be based on the process of performing AR display by the first AR device.
[0079] In one implementation of the disclosed embodiment, the first AR device may record and store the AR animation of the second user generated, so as to facilitate subsequent operations based on the recorded and stored AR animation, such as playback and copying.
[0080] Figure 6 is a flowchart of an AR display method according to an exemplary embodiment. Figure 6 As shown, the AR display method is used in a first AR device and includes the following steps.
[0081] Steps S21 and S22 are consistent with steps S11 and S12 and will not be described again here.
[0082] In step S23, the AR animation of the second user is recorded and stored.
[0083] In the disclosed embodiment, the first AR device can automatically record 3D holographic video data of the actions performed by the second user, and the first user can choose whether to store it in the AR device and generate an animation archive for later viewing and learning.
[0084] In one implementation of the disclosed embodiment, an AR device supporting a playback function is provided.
[0085] Figure 7 is a flowchart of playing back an AR animation according to an exemplary embodiment. Figure 7 As shown, the playback AR animation is used in the first AR device, including the following steps.
[0086] In step S31, a playback instruction for playing back the AR animation of the second user is obtained.
[0087] In step S32, the recorded and stored AR animation of the second user is played back.
[0088] In the disclosed embodiment, the skeletal animation information analyzed and calculated by the second user is bound to the 3D model selected by the first user and the skeleton is redirected. The newly formed 3D model is rendered in the virtual space mapped after the first user wears the AR device. The first AR device can automatically record and save the process animation of the action performed by the second user. When the first user forgets the action or wants to watch it again, the previously stored animation archive can be played back for review.
[0089] The playback instruction involved in the embodiments of the present disclosure may be an instruction issued by a user to the first AR device. The instruction may be an instruction for playing a stored AR animation, or an instruction for controlling the playback of the AR animation during playback.
[0090] Among them, in the disclosed embodiment, when the 3D holographic video animation saved by the AR device is played back, there are three speed modes to choose from: fast, slow, and normal speed. For example: when a user feels very proficient in a certain part of the action, he can choose to play it quickly, and when encountering an unfamiliar action, he can resume playing at normal speed; when it feels that a certain part of the action is difficult to imitate, he can choose to play it slowly, where the AR device configures auxiliary lines for the action made by the second user, and the first user can observe the action made by the second user in 360 degrees, so as to better perform the corresponding action, and resume normal speed after practicing and becoming familiar with it.
[0091] In the disclosed embodiment, the first AR device supports a copying function, and the user can select an archived animation and copy it in AR.
[0092] Fig. 8A is a flowchart of copying AR animation according to an exemplary embodiment. Fig. 8A As shown, the copying AR animation is used in the first AR device, including the following steps.
[0093] In step S41 , a copying instruction for copying the AR animation of the second user is received.
[0094] In step S42 , in response to receiving a copy instruction to copy the AR animation of the second user, the copy animation is displayed.
[0095] In one implementation of the disclosed embodiment, the copying action of the first user is monitored and a similarity analysis can be performed so that when there is a large difference in the copying, prompt information is given and correction is made in time.
[0096] Figure 8B is a flowchart of copying AR animation according to an exemplary embodiment. Figure 8B As shown, the copying AR animation is used in the first AR device, including the following steps.
[0097] in, Figure 8B exist Fig. 8A The following steps are also included:
[0098] In step S43 , the similarity between the copied action and the action displayed by the AR animation of the second user is analyzed.
[0099] In step S44, if the similarity is less than the threshold, a prompt message is issued.
[0100] The prompt information is used to prompt that the similarity between the copied action and the action displayed by the AR animation is less than a threshold.
[0101] It is understandable that in the embodiment of the present disclosure, if the similarity is greater than the threshold, normal AR animation playback can be maintained for the user to copy.
[0102] In the disclosed embodiment, the skeletal animation information of the second user is bound to the 3D model selected by the first user and then mapped in the virtual space where the first user is located, and the first AR device can record and save the 3D holographic animation data of the second user. If the first user wants to make the action more standard, the first user can select the archived action video animation of the second user for copying. When copying, the AR device can provide help and reminders for similarity comparison. If the difference in action is greater than the set threshold, the AR device will remind. If the difference in action is less than the set threshold, the AR device will not remind.
[0103] Among them, in the disclosed embodiment, the sensor component of the wearable component contains a vibration device, and the sensor receives the terminal command of the AR device. During the user's copying process, the AR device will analyze and compare the current user's action with the action completion and similarity in the stored animation. When it is found that the difference in the action of a certain joint is greater than the set threshold, the AR device will issue a vibration command to the sensor at the corresponding position, thereby slightly vibrating to remind the user that the action is not standardized.
[0104] In one implementation of the disclosed embodiment, the first AR device may also adopt a method similar to the above embodiment to obtain posture data collected by the wearable component worn by the first user, and generate skeletal animation information of the first user, and send the skeletal animation information to the second AR device, and display the AR animation of the first user in the space where the second AR device is located.
[0105] Fig. 9 is a flowchart of an AR display method according to an exemplary embodiment. Fig. 9 As shown, the AR display method is used in an AR device and includes the following steps.
[0106] In step S51, skeletal animation information of the first user collected by a wearable component worn by the first user is obtained.
[0107] In the disclosed embodiment, the first user performs a corresponding action, the action is acquired through the wearable component, and the AR device calculates the skeletal animation information of the first user through data analysis transmitted by the wearable component, so that the first AR device can transmit the skeletal animation information of the first user to the AR device worn by the second user for interaction.
[0108] In step S52, the skeleton animation information of the first user is sent to the second AR device, and the second AR device drives and displays the AR animation of the first user based on the skeleton animation information of the first user.
[0109] In the disclosed embodiment, the second AR device receives the skeletal animation information sent by the first AR device, and binds it to a character model selected according to the hobbies through the AR device, to form a new 3D model that integrates the skeletal animation information of the first user. The model is mapped in the virtual space where the second user is located after wearing the AR device, and the demonstration can be carried out without the need for a real person to arrive at the scene, thereby achieving the effect of real-person interactive feedback.
[0110] If an online connection is required, an online application is initiated with the other party. After the other party agrees, the online connection is started. The skeletal animation information generated by the communication between the first AR device and the wearable component is transmitted to the second AR device through the network, bound to the selected 3D model, and animated after the skeleton is redirected, and presented in the virtual environment currently located by the second user for real-life interactive feedback. Similarly, the skeletal animation information generated by the communication between the AR device worn by the second user and the wearable component is also transmitted to the first AR device through the network, bound to the 3D model selected by the first user, and animated after the skeleton is redirected, and presented in the virtual environment currently located by the first user. If an online connection is not required, the AR device will create a 3D model based on the data transmitted from the user's wearable component, complete its own skeletal data generation and binding, and finally drive the model animation based on the dynamic skeletal data.
[0111] Based on the above embodiments of the present disclosure, one implementation of the embodiments of the present disclosure provides an AR display that supports cross-device AR display and supports functions such as recording, playback, and copying.
[0112] The AR display method provided by the embodiments of the present disclosure can be applied to a variety of scenarios where display information interaction is required, such as teaching scenarios based on AR devices to demonstrate body movements, or other action-related competitive fields.
[0113] Among them, wearable components and AR devices (first AR device and second AR device) bring users (first user and second user) in different geographical locations into the same space, so that coaches and demonstrators do not need to go to the scene in person, but can also achieve the real effect of real-time competitive interaction between two or more people on the same stage. For example, through AR glasses, you can find boxing coaches and fencing coaches online to achieve a real teaching experience of real-time interaction without leaving home. For another example, in the shooting of action movies, it is also particularly applicable to dangerous scenes where people are attacked by knives, guns, sticks and clubs. It can prevent actors from being in dangerous situations, let actors perform their actions in a safe place, and use AR to synthesize the performers into the filmed pictures.
[0114] In the embodiments of the present disclosure, the AR device is an AR glasses, the first user is a student, and the second user is a coach, and the AR display method involved in the above embodiments is exemplarily described in combination with a teaching scenario.
[0115] The AR display method provided in the embodiment of the present disclosure can be understood as an AR coaching system based on AR display, and the AR coaching system can be understood as an application running on an AR device.
[0116] Fig.10 FIG. 1 is a schematic diagram showing an AR display implementation process according to an exemplary embodiment of the present disclosure. Fig.10 As shown, the SLAM function is provided in the AR device (the first AR device or the second AR device). Among them, SLAM runs in the background as the core module of AR and is responsible for the spatial positioning of AR. The wearable component provides component data (posture data) to the AR device. The AR device generates skeletal animation data based on the component data. Among them, the first user and the second user can set up an online connection between the first AR device and the second AR device to perform AR animation drawing in the AR coaching system, and realize functions such as recording, playback, and copying.
[0117] In one example, Fig.11 A schematic diagram of the process of AR teaching between a student and a coach is shown. Fig.11As shown, after the user accesses the system, the AR teaching system provides the user with an option to choose whether to go online for cross-device interaction. If online is required, the accessing user initiates an online application with the other party, and the other party agrees to start online. Among them, in the embodiment of the present disclosure, after the user accesses the AR teaching system, the AR teaching system will determine whether the accessing user is a coach, that is, whether the accessing user is a student or a coach, and provide different AR display functions. If the accessing user is a coach, the coach demonstrates the action, and the wearable component worn by the coach detects the action to determine the posture data, and sends the posture data to the AR glasses worn by the coach. The AR glasses worn by the coach generate skeletal animation information based on the posture data, and send the skeletal animation information to the AR glasses worn by the student through the network. The AR glasses worn by the student receive the coach's skeletal animation information, bind the coach's skeletal animation information to the role model selected by the student, and complete the coach's skeletal redirection. The AR glasses worn by the student drive the role model after skeletal redirection, draw animation, and realize the display of the coach's demonstration action on the student side. If the accessing user is a student, the student is provided with a role model selection. The student selects a character model and binds it to the coach's skeletal animation information obtained by the AR glasses worn by the student, completes the coach's skeletal redirection, drives the character model after skeletal redirection, draws animation, and realizes the display of the coach's demonstrated movements on the student side.
[0118] Among them, in one implementation, when the access user is a student, the AR glasses worn by the student collect the student's movements. The wearable component worn by the student detects the movement to determine the posture data, and sends the posture data to the AR glasses worn by the student. The AR glasses worn by the student generate skeletal animation information based on the posture data, and send the skeletal animation information to the AR glasses worn by the coach through the network. The AR glasses worn by the coach receive the skeletal animation information of the student, bind the skeletal animation information of the student to the character model selected by the coach, and complete the skeletal redirection of the student. The AR glasses worn by the coach drive the character model after the skeletal redirection, draw the animation, and realize the display of the student's movements on the coach's side.
[0119] Fig.12 FIG. 1 is a schematic diagram of an AR display scene according to an exemplary embodiment. Fig.12As shown, the coach demonstrated the elbow flexion movement, and the wearable component worn by the coach detected the posture data corresponding to the elbow flexion movement, and sent the posture data to the AR glasses worn by the coach. The AR glasses worn by the coach generate skeletal animation information based on the posture data, and send the skeletal animation information to the AR glasses worn by the trainee through the network. The AR glasses worn by the trainee receive the skeletal animation information of the coach, bind the skeletal animation information of the coach to the character model selected by the trainee, and complete the skeletal redirection of the coach. The AR glasses worn by the trainee drive the character model after skeletal redirection, draw the animation of the coach's elbow flexion movement, and realize the display of the coach's demonstration of elbow flexion movement on the trainee side.
[0120] In the disclosed embodiment, in the AR coaching system, during the AR teaching process, the user can record the teaching process and generate an animation archive. For example, the coaching AR animation is recorded and stored. The user can then replay or copy the animation based on the generated animation archive. Among them, animation playback and copying can be understood as a viewing mode.
[0121] Fig.13 FIG. 1 shows an implementation flow chart of watching an AR animation according to an exemplary embodiment of the present disclosure. Fig.13 As shown, the user selects a teaching archive animation. The AR device reads the archive animation and determines whether the viewing mode is to play the archive animation. If so, the archive is played, that is, replayed. If the viewing mode is not to play the archive, but to copy, the archive animation corresponding to the copy is displayed.
[0122] The AR display method provided by the embodiment of the present disclosure is applied to AR-assisted teaching, which adds auxiliary lines and prompt information compared to the real world to help understand the teaching content. It also enables two people in different places to complete real-time teaching, which can give full play to its advantages and achieve the purpose of coaching teaching and student learning at home.
[0123] For example Fig.14 As shown in the example of a fencing teaching scenario, the user puts on AR glasses and motion capture equipment, and after connecting to the Internet, finds his coach or student. The coach and the student are just like being face to face in real life, and then conduct one-on-one teaching. After the coach demonstrates the movements, the student can choose to replay the movements, observe the movements, or copy the movements according to the trajectory and rhythm.
[0124] Based on the AR display method provided by the embodiment of the present disclosure, compared with the solution that cannot cope with the changing scenes and cannot achieve real-time real-person feedback interaction by superimposing the prefabricated virtual model in advance into the real world, it can build models in real time and complete the real-person dynamic effect of the model. For example, the camera and microphone on the AR glasses are used to collect data, and then communicate through the network to complete the end-to-end holographic audio-visual effect.
[0125] Moreover, when the AR display method of superimposing a prefabricated virtual model on the real world only uses AR glasses, the displayed human model cannot completely replicate the current user's body movements, that is, it cannot represent the same data as motion capture; although the motion capture device can completely replicate the current user's body movements, it is necessary to set up a positioning camera outside to capture the user's image and calculate the skeleton data in the computer, and finally draw it on the screen. Without the positioning camera, the human body posture cannot be calculated. However, the method for cross-device AR display provided by the embodiment of the present disclosure has both motion capture equipment and AR capabilities, using glasses and motion capture wearable clothing to complete motion capture and can process positioning and 3D display.
[0126] It should be noted that those skilled in the art can understand that the various implementation methods / embodiments involved in the embodiments of the present disclosure can be used in conjunction with the aforementioned embodiments or can be used independently. Whether used alone or in conjunction with the aforementioned embodiments, the implementation principle is similar. In the implementation of the present disclosure, some embodiments are described in terms of implementation methods used together. Of course, those skilled in the art can understand that such examples are not limitations of the embodiments of the present disclosure.
[0127] Based on the same concept, an embodiment of the present disclosure also provides an AR display device.
[0128] It is understandable that in order to realize the above functions, the AR display device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0129] Fig.15 is a block diagram of an AR display device 100 according to an exemplary embodiment. Fig.12 The device includes an acquisition unit 101 and a processing unit 102.
[0130] The acquisition unit 101 is used to acquire skeletal animation information of a second user sent by a second AR device, where the skeletal animation information of the second user is collected by a wearable component worn by the second user and transmitted to the second AR device.
[0131] The processing unit 102 is configured to bind the skeletal animation information of the second user to the character model, and drive the character model to display the AR animation of the second user in the space where the first user wears the first AR device.
[0132] In one implementation, the processing unit 102 is further configured to record and store the AR animation of the second user.
[0133] In one implementation, the processing unit 102 is further configured to, in response to receiving a playback instruction to play back the AR animation of the second user, play back the stored AR animation of the second user.
[0134] In one embodiment, the processing unit 102 is further configured to monitor the copying action of the first user in response to receiving a copying instruction to copy the AR animation of the second user. Analyze the similarity between the copying action and the action displayed in the AR animation of the second user. If the similarity is less than a threshold, a prompt message is issued, and the prompt message is used to prompt that the similarity between the copying action and the action displayed in the AR animation is less than the threshold.
[0135] In one implementation, the acquisition unit 101 is further configured to acquire skeletal animation information of the first user collected by a wearable component worn by the first user.
[0136] In one implementation, the AR display apparatus 100 further includes a sending unit 103, configured to send the skeleton animation information of the first user to the second AR device, and the second AR device drives the display of the AR animation of the first user based on the skeleton animation information of the first user.
[0137] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0138] An embodiment of the present disclosure also provides a wearable component, which includes multiple AR detection components, and the multiple AR detection components are arranged at the positions of skeletal key points for generating skeletal animation information.
[0139] In one embodiment, the wearable component is worn as motion capture clothing, and physical inertial sensors are set at key joints to record, track, and measure the movement trajectory of the parts.
[0140] In one embodiment, each AR component in the wearable component also includes a data communication device that communicates with the AR glasses and transmits data to the AR glasses for calculating model skeleton information.
[0141] In one embodiment, each AR component in the wearable component is also provided with a vibration device for receiving commands from the AR device and performing slight vibrations.
[0142] Among them, the wearable components can refer to the relevant description of the above embodiments, which will not be repeated here.
[0143] Fig.16 1 is a block diagram of an apparatus 200 for AR display according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0144] Reference Fig.16 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .
[0145] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.
[0146] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0147] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.
[0148] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0149] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.
[0150] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0151] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
[0152] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0153] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0154] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0155] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0156] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0157] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.
[0158] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0159] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0161] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. An augmented reality (AR) display method, characterized in that: Applied to a first AR device, where the first AR device is used by a first user, the method includes: Acquire skeletal animation information of a second user sent by a second AR device, where the skeletal animation information of the second user is determined by the second AR device based on posture data of the second user collected by a wearable component worn by the second user; The skeletal animation information of the second user is bound to the character model, and the character model is driven to display the AR animation of the second user in a space where the first user wears the first AR device.
2. The AR display method according to claim 1, characterized in that: The method further comprises: The AR animation of the second user is recorded and stored.
3. The AR display method according to claim 2, characterized in that: The method further comprises: In response to receiving a playback instruction to play back the AR animation of the second user, the stored AR animation of the second user is played back.
4. The AR display method according to claim 2, characterized in that: The method further comprises: In response to receiving a copying instruction to copy the AR animation of the second user, monitoring a copying action of the first user; Analyzing the similarity between the copied action and the action displayed by the AR animation of the second user; If the similarity is less than a threshold, a prompt message is issued, where the prompt message is used to prompt that the similarity between the copying action and the action displayed by the AR animation is less than a threshold.
5. The AR display method according to claim 1, characterized in that: The method further comprises: Acquire skeletal animation information of the first user collected by a wearable component worn by the first user; The skeleton animation information of the first user is sent to the second AR device, and the second AR device drives and displays the AR animation of the first user based on the skeleton animation information of the first user.
6. The AR display method according to claim 1 or 5, characterized in that: The wearable component includes multiple AR detection components, and the multiple AR detection components are arranged at the skeleton key point positions for generating skeleton animation information.
7. An augmented reality (AR) display device, characterized in that: include: an acquiring unit, configured to acquire skeletal animation information of a second user sent by a second AR device, where the skeletal animation information of the second user is determined by the second AR device based on posture data of the second user collected by a wearable component worn by the second user; A processing unit is used to bind the skeletal animation information of the second user to the character model, and drive the character model to display the AR animation of the second user in a space where the first user wears the first AR device.
8. The augmented reality (AR) display device according to claim 7, characterized in that: The processing unit is also used for: The AR animation of the second user is recorded and stored.
9. The augmented reality (AR) display device according to claim 7, characterized in that: The processing unit is also used for: In response to receiving a playback instruction to play back the AR animation of the second user, the stored AR animation of the second user is played back.
10. The augmented reality (AR) display device according to claim 7, characterized in that: The processing unit is also used for: In response to receiving a copying instruction to copy the AR animation of the second user, monitoring a copying action of the first user; Analyzing the similarity between the copied action and the action displayed by the AR animation of the second user; If the similarity is less than a threshold, a prompt message is issued, where the prompt message is used to prompt that the similarity between the copying action and the action displayed by the AR animation is less than a threshold.
11. The augmented reality (AR) display device according to claim 7, characterized in that: The acquisition unit is also used for: Acquire skeletal animation information of the first user collected by a wearable component worn by the first user; The device further comprises: a sending unit, The sending unit is used to send the skeletal animation information of the first user to the second AR device, so that the second AR device drives the display of the AR animation of the first user based on the skeletal animation information of the first user.
12. The augmented reality (AR) display device according to claim 7 or 11, characterized in that: The wearable component includes multiple AR detection components, and the multiple AR detection components are arranged at the skeleton key point positions for generating skeleton animation information.
13. A wearable component, characterized in that: include: A plurality of AR detection components are arranged at the positions of the skeleton key points for generating the skeleton animation information.
14. An augmented reality (AR) device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: The method is used to call the instruction stored in the memory to execute the AR display method according to any one of claims 1 to 6.
15. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by a processor of the AR device, the AR device is enabled to execute the AR display method according to any one of claims 1 to 6.
