Motion interaction method for virtual motion detection and projection
By predicting motion data in the user's virtual environment and rendering virtual objects using image projection, the problem of poor user interaction experience in the prior art is solved, and a better virtual reality interaction experience is achieved.
Patent Information
- Application Number
- CN202410809689.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2025-03-07
AI Technical Summary
The existing virtual reality motion interaction methods cannot fully consider the actual situation of the user, resulting in poor user interaction experience.
By obtaining the user's motion data in the virtual environment, predicting the user's motion direction, type and range, and rendering the virtual object using image projection to improve the interactive experience.
It realizes pre-rendering of virtual objects before user interaction, improves user experience, and quickly responds to user interaction through image projection.
Smart Images
Figure CN119576113A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual reality technology, and in particular to a motion interaction method for virtual motion detection and projection. Background Art
[0002] With the development of science and technology, technologies such as virtual reality and augmented reality are attracting more and more attention. Existing motion interaction methods render virtual objects through motion tracking so that users can interact based on virtual objects. For example, the device detects the position and direction of the user's head through built-in sensors (such as gyroscopes, accelerometers, and magnetometers) to ensure that the perspective in the virtual environment can follow the user's head movement in real time.
[0003] This motion interaction method cannot fully take the actual situation of the user into consideration, thus resulting in a poor user interaction experience. Summary of the invention
[0004] The purpose of this application is to provide a motion interaction method for virtual motion detection and projection, so as to enhance the user's virtual reality interaction experience.
[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a motion interaction method for virtual motion detection and projection, comprising: obtaining motion data of a user in a virtual environment; determining predicted motion data of the user according to the motion data of the user in the virtual environment, wherein the predicted motion data includes predicted motion direction, predicted motion type and predicted motion range, and the predicted motion type is used to characterize the predicted interactive action of the user on a virtual object in the virtual environment; determining to render the virtual object through image projection according to the predicted motion data; projecting a first virtual image corresponding to the virtual object to the virtual environment; and projecting a second virtual image corresponding to the virtual object to the virtual environment in response to detecting the interactive action of the user on the virtual object. .
[0006] In a possible implementation, obtaining the motion data of the user in the virtual environment includes: obtaining first motion data collected by a smart wearable device worn by the user; obtaining second motion data collected by an electronic device held by the user; obtaining third motion data collected by a motion data collection device in the real environment where the user is located; and determining the motion data of the user in the virtual environment based on the first motion data, the second motion data, and the third motion data.
[0007] In a possible implementation, determining the motion data of the user in the virtual environment based on the first motion data, the second motion data, and the third motion data includes: determining the motion data corresponding to the first body area of the user based on the first motion data; determining the motion data corresponding to the second body area of the user based on the second motion data, the second body area being a part of the first body area; determining the motion data corresponding to the third body area of the user based on the third motion data, the distance between the third body area and the first body area being greater than a preset distance; determining the motion data of the user in the virtual environment based on the motion data corresponding to the first body area of the user, the motion data corresponding to the second body area of the user, and the motion data corresponding to the third body area of the user.
[0008] In a possible implementation, the motion data of the user in the virtual environment includes: motion data corresponding to a first body area of the user, motion data corresponding to a second body area of the user, and motion data corresponding to a third body area of the user, and determining the predicted motion data of the user based on the motion data of the user in the virtual environment includes: converting the motion data corresponding to the first body area of the user into a first user motion image, the first user motion image being used to characterize the motion state of the first body area; converting the motion data corresponding to the second body area of the user into a second user motion image, the second user motion image being used to characterize the motion state of the second body area; converting the motion data corresponding to the third body area of the user into a third user motion image, the third user motion image being used to characterize the motion state of the third body area; determining a predicted user motion image based on the first user motion image, the second user motion image, the third user motion image and a pre-trained motion prediction model, the predicted user image being used to characterize the predicted motion state of the user as a whole; and determining the predicted motion data of the user based on the predicted user motion image.
[0009] In a possible implementation, the motion interaction method of virtual motion detection and projection also includes: obtaining a training data set, the training data set including multiple training samples, each training sample including: multiple sample images and sample labels, the multiple sample images respectively representing the motion states of different body regions, and the sample labels are used to represent predicted images for the multiple sample images; training the motion prediction model to be trained according to the training data set to obtain the pre-trained motion prediction model.
[0010] In a possible implementation, determining to render the virtual object through image projection based on the predicted motion data includes: if the predicted motion direction is directly in front of the user, and the predicted motion type represents that the user's predicted interaction action with respect to the virtual object in the virtual environment is a preset interaction action, and the predicted motion range is within the preset motion range, determining to render the virtual object through image projection; wherein the preset interaction action is: an action that does not change the position of the virtual object in the virtual environment, or an action that changes the position of the virtual object in the virtual environment without changing the position of the user in the virtual environment.
[0011] In a possible implementation, the first virtual image is a virtual image corresponding to an initial state of the virtual object, and the motion interaction method of virtual motion detection and projection further includes: determining a predicted state of the virtual object based on the first virtual image and an interactive action of the user with respect to the virtual object; and determining a second virtual image corresponding to the virtual object based on the initial state of the virtual object and the predicted state of the virtual object.
[0012] In a possible implementation, determining the second virtual image corresponding to the virtual object based on the initial state of the virtual object and the predicted state of the virtual object includes: acquiring virtual images corresponding to multiple pre-configured states of the virtual object, wherein the virtual images corresponding to the multiple states are arranged in order of state change; determining, from the virtual images corresponding to the multiple states, a virtual image corresponding to an intermediate state between the initial state and the predicted state, and determining a virtual image corresponding to the predicted state; and determining the second virtual image corresponding to the virtual object based on the virtual image corresponding to the intermediate state and the virtual image corresponding to the predicted state.
[0013] In a possible implementation, the motion interaction method for virtual motion detection and projection also includes: obtaining the interaction time of the user with the virtual object, where the interaction time is determined according to the interaction time corresponding to multiple interaction actions; if the interaction time of the user with the virtual object is less than a preset interaction time, adding a projection tag to the virtual object, and the projection tag is used to indicate that the virtual object is rendered through image projection when the virtual object is rendered next time.
[0014] In a possible implementation, the motion interaction method of virtual motion detection and projection further includes: in response to not detecting the user's interactive action on the virtual object within a preset time period, rendering the virtual object according to the user's motion data in the virtual environment.
[0015] Compared with the prior art, the motion interaction method of virtual motion detection and projection provided by the present application has the following technical effects:
[0016] First, the user's motion data in the virtual environment is obtained. Based on the motion data, the user's motion direction, motion type and motion range are predicted. Based on the prediction results, when it is determined that image projection can be used for virtual object rendering, the virtual object rendering is realized by using image projection. Then, when the user's interactive action is detected, the virtual object is rendered again by image projection to respond to the user's interaction. In this way, the user's interaction is predicted, and the virtual object rendering is realized by image projection based on the prediction results, which is equivalent to pre-rendering the virtual object before the user interaction, so that the user experience is better; and the rendering of the virtual object by image projection can quickly respond to the user's interaction. Therefore, this technical solution can enhance the user's virtual reality interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a flow chart of a motion interaction method of virtual motion detection and projection according to an embodiment of the present application;
[0018] Figure 2 is a schematic structural diagram of a motion tracking system according to an embodiment of the present application;
[0019] Figure 3 is a schematic diagram of body region division according to an embodiment of the present application;
[0020] Figure 4 is an example diagram of a motion image according to an embodiment of the present application;
[0021] Figure 5 is a structural schematic diagram of a motion interaction device for virtual motion detection and projection according to an embodiment of the present application;
[0022] Figure 6 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] The specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present application is not limited by the specific implementation methods.
[0024] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.
[0025] Smart wearable devices are advanced technology products that create an immersive virtual environment by simulating the user's visual, auditory and even tactile experience, making the user feel as if they are actually there.
[0026] The visual display principles of smart wearable devices include: Stereoscopic imaging: Smart wearable devices can be equipped with two screens, corresponding to the user's left eye and right eye respectively, to simulate the stereoscopic vision of the human eye in the real world by displaying slightly different images (stereoscopic images). Wearable display: Users view images through wearable displays. This design can reduce interference from the external environment and enhance immersion. High refresh rate: In order to reduce visual delay and blur, smart wearable devices usually have a high refresh rate (usually above 90Hz) to provide smooth image updates.
[0027] Existing motion interaction methods render virtual objects through motion tracking so that users can interact based on virtual objects. For example, the device detects the position and direction of the user's head through built-in sensors (such as gyroscopes, accelerometers, and magnetometers) to ensure that the perspective in the virtual environment can follow the user's head movement in real time.
[0028] This motion interaction method cannot fully take the actual situation of the user into consideration, thus resulting in a poor user interaction experience.
[0029] Through research, it is found that this motion interaction method is based on real-time detection results to respond to user interactions, and thus cannot fully meet user needs. For example, although the current user's state is in a small activity range, the user may need a larger activity range at the next moment. If a real-time interactive response method is used, it is not possible to immediately respond to user interactions in a larger activity range.
[0030] Based on this, an embodiment of the present application provides a technical solution, which realizes motion interaction based on virtual motion detection and projection to enhance the user's virtual reality interaction experience.
[0031] It can be understood that the technical solution can be applied to smart wearable devices, or control devices of smart wearable devices, etc., and is not limited here.
[0032] Please refer to Figure 1 , is a flow chart of a motion interaction method for virtual motion detection and projection according to an embodiment of the present application, the method comprising:
[0033] Step 101, obtaining the motion data of the user in the virtual environment.
[0034] Step 102: Determine predicted motion data of the user based on the motion data of the user in the virtual environment. The predicted motion data includes predicted motion direction, predicted motion type and predicted motion range. The predicted motion type is used to characterize the predicted interactive action of the user on the virtual object in the virtual environment.
[0035] Step 103: Determine, based on the predicted motion data, whether to render the virtual object through image projection.
[0036] Step 104: project a first virtual image corresponding to the virtual object onto the virtual environment.
[0037] Step 105 : In response to detecting an interactive action of the user on the virtual object, projecting a second virtual image corresponding to the virtual object into the virtual environment.
[0038] It can be understood that steps 101 to 105 are a motion interaction process for a virtual object, and motion interaction can also be achieved for other virtual objects through the motion interaction process.
[0039] In step 101, the motion data of the user in the virtual environment can be understood as the motion data at the current moment. The current moment can be one moment or multiple moments. If it is multiple moments, it can be understood as the current time period.
[0040] Furthermore, the motion data in the virtual environment can be understood as the motion data generated by the user based on the virtual environment, which can be mapped to the real environment, or has a corresponding correspondence with the real environment.
[0041] In some embodiments, the motion data may be collected by a motion tracking module provided by the smart wearable device.
[0042] In other embodiments, the motion data may be collected by a variety of devices.
[0043] Therefore, please refer to Figure 2 , is a schematic diagram of the structure of a motion tracking system according to an embodiment of the present application, such as Figure 2 As shown, the motion tracking system includes a smart wearable device, an electronic device and a motion data acquisition device.
[0044] Among them, the smart wearable device is a device worn by the user, which can be a head-mounted smart wearable device. The electronic device can be an electronic device held by the user, such as a mobile phone, a handle, etc. The motion data acquisition device can be a motion tracking device set in a real environment, which is set at a fixed position and used to track the user's position, such as a laser radar, etc.
[0045] In some embodiments, assuming that the real environment corresponds to a corresponding activity range, the above-mentioned motion data acquisition device can be set at the edge, surroundings, fixed corners, etc. of the activity range to ensure that human motion data can be collected.
[0046] Furthermore, as an optional implementation, step 101 includes: obtaining first motion data collected by a smart wearable device worn by a user; obtaining second motion data collected by an electronic device held by the user; obtaining third motion data collected by a motion data collection device in the real environment where the user is located; and determining the motion data of the user in the virtual environment based on the first motion data, the second motion data and the third motion data.
[0047] In this implementation, the motion data collected by the three devices are obtained respectively, and the motion data of the user in the virtual environment can be determined based on the integration of the three motion data.
[0048] As an optional implementation, the motion data of the user in the virtual environment includes: motion data corresponding to a first body area of the user, motion data corresponding to a second body area of the user, and motion data corresponding to a third body area of the user.
[0049] Please refer to Figure 3 , is a schematic diagram of a body region division according to an embodiment of the present application, such as Figure 3 As shown, the body area of the human body is divided into three parts, the first part is the upper body, that is, the first body area, the second part is the hand, that is, the second body area, and the third part is the lower body, that is, the third body area. Therefore, the second body area is part of the first body area.
[0050] It can be understood that since human interaction focuses on the hands and body parts, dividing the hand area separately can facilitate the prediction of human interaction actions. Also, separating the upper body and the lower body can facilitate the prediction of human interaction actions.
[0051] Furthermore, based on the first motion data, the second motion data and the third motion data, the motion data of the user in the virtual environment is determined, including: based on the first motion data, determining the motion data corresponding to the first body area of the user; based on the second motion data, determining the motion data corresponding to the second body area of the user, the second body area is part of the first body area; based on the third motion data, determining the motion data corresponding to the third body area of the user, the distance between the third body area and the first body area is greater than a preset distance; based on the motion data corresponding to the first body area of the user, the motion data corresponding to the second body area of the user and the motion data corresponding to the third body area of the user, the motion data of the user in the virtual environment is determined.
[0052] In the embodiment of the present application, the motion data may include: position and posture. Therefore, for each motion tracking device, the motion data collected by it can be understood as a motion matrix, in which a vector for representing the position and a vector for representing the posture are included. Moreover, different vectors of the motion matrix correspond to different human body positions.
[0053] Therefore, according to the first motion data, the motion data belonging to the first body region, i.e., the motion data corresponding to the first body region, can be screened out. According to the second motion data, the motion data belonging to the second body region, i.e., the motion data corresponding to the second body region, can be screened out. According to the third motion data, the motion data belonging to the third body region, i.e., the motion data corresponding to the third body region, can be screened out.
[0054] Furthermore, the motion data corresponding to the first body area of the user, the motion data corresponding to the second body area of the user, and the motion data corresponding to the third body area of the user are integrated again to obtain a new motion matrix, which is the motion data of the user in the virtual environment.
[0055] In addition, it can be understood that the above-mentioned motion data may involve coordinate conversion, etc., and this part can refer to mature technologies in the field.
[0056] Furthermore, the motion data of the user in the virtual environment can also be represented by a motion matrix, which can represent the position and posture, and can also represent the motion state of the user at a corresponding moment.
[0057] In step 102, the predicted motion data of the user is determined based on the motion data of the user in the virtual environment. The predicted motion data includes a predicted motion direction, a predicted motion type, and a predicted motion range, wherein the predicted motion direction and the predicted motion type can be associated with the aforementioned gesture. The predicted motion type is used to characterize the predicted interactive action of the user on the virtual object in the virtual environment.
[0058] Predicted movement direction, such as forward, left, right, etc. Predicted interactive actions, such as walking, running, waving, kicking, etc. In some embodiments, these interactive actions can be preconfigured into different motion types so that the motion types can characterize these interactive actions.
[0059] The predicted movement range may represent the activity range of the user, which may be a range compared to the entire virtual environment. For example, if the virtual environment corresponds to 10 square meters in total, the predicted movement range is within the 10 square meters.
[0060] As an optional implementation, step 102 includes: converting motion data corresponding to a first body region of a user into a first user motion image, the first user motion image being used to characterize the motion state of the first body region; converting motion data corresponding to a second body region of a user into a second user motion image, the second user motion image being used to characterize the motion state of the second body region; converting motion data corresponding to a third body region of a user into a third user motion image, the third user motion image being used to characterize the motion state of the third body region; determining a predicted user motion image based on the first user motion image, the second user motion image, the third user motion image and a pre-trained motion prediction model, the predicted user motion image being used to characterize the predicted motion state of the user as a whole; and determining the predicted motion data of the user based on the predicted user motion image.
[0061] In this embodiment, the motion data corresponding to each body area can be converted into a corresponding user motion image, so that the user motion image can be predicted based on multiple motion images.
[0062] The predicted user motion image may represent the overall motion state of the user at the next moment, and the next moment may be a moment or a time period.
[0063] For examples, please refer to Figure 4 , is an example diagram of a motion image according to an embodiment of the present application, such as Figure 4 As shown, the first user motion image is used to represent the motion state of the upper body, the second user motion image is used to represent the motion state of the hands, and the third user motion image is used to represent the motion state of the lower body.
[0064] It can be understood that by converting the three types of motion data into images, the pre-trained motion prediction model can predict the user's motion data.
[0065] Among them, the pre-trained motion prediction model can be some models commonly used in image processing, such as neural network models, large models, etc.
[0066] In some embodiments, the image output by the motion prediction model can represent the motion state of the user at the next moment, and the predicted motion direction and predicted motion range can be determined based on the image, the first user motion image, the second user motion image, and the third user motion image. For example, the predicted user motion image is compared with the integrated image obtained by the three user motion images to determine the motion direction and motion range.
[0067] In addition, the type of motion can be directly determined based on the predicted motion image. For example, the type of motion can be determined based on the posture of the user in the predicted motion image.
[0068] Regarding the pre-trained motion prediction model, the motion image of the whole person at the next moment can be predicted based on the motion images corresponding to different body regions at the current moment.
[0069] As an optional implementation, the training of the motion prediction model includes: obtaining a training data set, the training data set includes multiple training samples, each training sample includes: multiple sample images and sample labels, the multiple sample images respectively represent the motion states of different body regions, and the sample labels are used to represent the predicted images for the multiple sample images; training the motion prediction model to be trained according to the training data set to obtain a pre-trained motion prediction model.
[0070] The sample label may be the predicted position label and predicted posture label of the pixel at the next moment.
[0071] It can be understood that these training samples can be converted into corresponding vectors before being input into the model for training, and the vectors can represent the positions and postures of different pixels. Therefore, the sample labels can be directly configured as the predicted position labels and predicted posture labels of the pixels at the next moment, without having to configure them in the form of images.
[0072] Through the above training, the first user motion image, the second user motion image, and the third user motion image are input into the model for training, and the model can output the predicted user motion image to represent the overall predicted motion state of the user.
[0073] Furthermore, in step 103, it is determined to render the virtual object through image projection according to the predicted motion data.
[0074] In the embodiments of the present application, image projection can be implemented based on a smart wearable device or based on an external projection device.
[0075] Regarding smart wearable devices, image rendering can be achieved by image projection. Regarding external projection equipment, virtual images can be directly projected in a virtual environment to present a rendering effect of a virtual object.
[0076] However, the effect of image projection may not be as stereoscopic as that of a three-dimensional model. Therefore, image projection is not used for all types of interaction. Therefore, in the embodiment of the present application, image projection may have corresponding conditions.
[0077] Furthermore, it is necessary to determine whether the virtual object is rendered by image projection, and if so, step 103 is executed. If not, the virtual object is rendered according to a conventional virtual object rendering method, such as a 3D model rendering method, which can refer to mature technologies in the art.
[0078] As an optional implementation, step 103 includes: if the predicted movement direction is directly in front of the user, and the predicted movement type represents that the user's predicted interactive action with respect to the virtual object in the virtual environment is a preset interactive action, and the predicted movement range is within the preset movement range, determining to render the virtual object through image projection; wherein the preset interactive action is: an action that does not change the position of the virtual object in the virtual environment, or an action that changes the position of the virtual object in the virtual environment without changing the position of the user in the virtual environment.
[0079] In some embodiments, the virtual object may be a virtual object that the current user is interacting with, or may be a virtual object that is about to be interacted with at the next moment. If it is a virtual object that is about to be interacted with at the next moment, the virtual object that is about to be interacted with at the next moment may be predicted based on the virtual object that the current user is interacting with. For example, if the virtual object that the user is interacting with is a tree, the interactive object at the next moment may be a virtual object around the tree, such as grass, animals, etc. Regarding the prediction of virtual objects, predictions can be made through pre-configured virtual object interaction rules, which can characterize the general interaction order of users for multiple interactive virtual objects in a virtual environment.
[0080] In this implementation, if the predicted movement direction is directly in front of the user, it means that the user will interact directly in front of the user. At this time, if the image is projected directly in front of the user, it is relatively easy to achieve, so it can be used as one of the conditions.
[0081] Also, if the user's action does not change the position of the virtual object in the virtual environment, the position change of the virtual object will not be involved, which is also easier to achieve through image projection and can also be used as one of the conditions.
[0082] In addition, although the user's actions change the position of the virtual object in the virtual environment, the predicted user range is within the preset motion range and does not involve changes in the activity range. This can also be achieved through image projection and can also be used as one of the conditions.
[0083] Therefore, in the above case, it can be determined that the virtual object is rendered by image projection.
[0084] Furthermore, in step 104, a first virtual image corresponding to the virtual object is projected onto the virtual environment, that is, virtual object rendering is achieved through image projection technology.
[0085] For example, if the virtual object is a 2D virtual object, such as an interactive interface, the virtual interactive interface can be directly projected to the corresponding position in the virtual environment. If the virtual object is a 3D virtual object, such as an animal, a virtual image with a 3D effect corresponding to the virtual object can be projected to the corresponding position in the virtual environment, so that the virtual object seen by the user through the smart wearable device is also 3D.
[0086] It can be understood that when the first virtual image corresponding to the virtual object is projected into the virtual environment, the preliminary rendering of the virtual object can be completed, and this rendering is pre-rendering based on the predicted data, so the user may not necessarily interact with the virtual object.
[0087] Therefore, in step 105, in response to detecting the user's interactive action on the virtual object, a second virtual image corresponding to the virtual object is projected into the virtual environment.
[0088] In this case, it is equivalent to that the user has performed an interaction with the virtual object. At this time, the rendering of the virtual object can continue to respond to the user's rendering.
[0089] As an optional implementation, the first virtual image is a virtual image corresponding to the initial state of the virtual object. Then, through user interaction, the initial state of the virtual object may be changed, and a new rendering effect needs to be presented. Then, it is necessary to first determine the second virtual image corresponding to the virtual object.
[0090] As an optional implementation, determining a second virtual image corresponding to a virtual object includes: determining a predicted state of the virtual object based on the first virtual image and an interactive action of a user on the virtual object; and determining the second virtual image corresponding to the virtual object based on an initial state of the virtual object and the predicted state of the virtual object.
[0091] In this embodiment, the state of the virtual object is predicted, and the second virtual image is determined according to the predicted state and the initial state.
[0092] In some embodiments, based on the first virtual image, the user's interactive action on the virtual object can be simulated in the background, and the predicted state is determined according to the simulation result. This simulation is a simulation independent of the user's interaction.
[0093] In other embodiments, interaction results corresponding to different interaction actions may be preconfigured, and the state of the virtual object may be predicted based on the preconfigured interaction results and the first virtual image.
[0094] In some other embodiments, multiple interactive actions that can be implemented can be configured for the virtual object, and the state changes corresponding to the multiple interactive actions can be configured. According to the configuration, the state of the virtual object can be predicted.
[0095] As an optional implementation, determining a second virtual image corresponding to the virtual object according to an initial state of the virtual object and a predicted state of the virtual object includes: acquiring virtual images corresponding to a plurality of states of a pre-configured virtual object, and arranging the virtual images corresponding to the plurality of states in order of state change; determining, from the virtual images corresponding to the plurality of states, a virtual image corresponding to an intermediate state between the initial state and the predicted state, and determining a virtual image corresponding to the predicted state; determining the second virtual image corresponding to the virtual object according to the virtual image corresponding to the intermediate state and the virtual image corresponding to the predicted state.
[0096] In this embodiment, virtual images corresponding to multiple states of a virtual object are pre-configured. For example, for an interactive interface, different interactive operations can be configured, and virtual images corresponding to the interactive operations can be configured. Furthermore, the virtual images corresponding to the multiple states are arranged in the order of state changes.
[0097] Furthermore, the virtual image corresponding to the intermediate state between the initial state and the predicted state can be determined from the virtual images corresponding to the multiple states. Also, the virtual image corresponding to the predicted state can be determined, wherein if the predicted state belongs to the pre-configured multiple states, the virtual image corresponding to the predicted state can be directly determined. If the predicted state does not belong to the pre-configured multiple states, since the state change caused by one interactive action is not large, the state in the first order among the multiple states can be determined as the predicted state.
[0098] Among them, the number of virtual images corresponding to the intermediate state is not limited to one, and is usually multiple. The specific number can be determined according to the projection refresh rate. If the projection refresh rate is high, the number can be large, and if the refresh rate is low, the number can be small.
[0099] Then, the virtual image corresponding to the intermediate state and the virtual image corresponding to the predicted state are both determined as the second virtual image corresponding to the virtual object. When projecting the second virtual image, the virtual image corresponding to the intermediate state is projected first, and then the virtual image corresponding to the predicted state is projected.
[0100] In this way, the state change of the virtual object presented to the user can be gradual, thereby improving the user's interactive experience.
[0101] Furthermore, in the presence of interaction, the method may also include: obtaining the user's interaction time for the virtual object, the interaction time being determined according to the interaction time corresponding to multiple interaction actions; if the user's interaction time for the virtual object is less than a preset interaction time, adding a projection tag to the virtual object, the projection tag being used to indicate that the virtual object is rendered through image projection the next time the virtual object is rendered.
[0102] In this implementation, if the user's interaction time with the virtual object is short, it means that the user's interaction efficiency is high, and the virtual object is more suitable for rendering by image projection. Therefore, a projection tag can be added to the virtual object, and the projection tag is used to indicate that the virtual object is rendered by image projection the next time the virtual object is rendered, without further judgment.
[0103] The preset interaction time may be an interaction time that satisfies basic interaction efficiency, for example, 2 to 5 seconds.
[0104] In some embodiments, an average value or median value of interaction durations corresponding to a plurality of interaction actions may be determined as the user's interaction duration with the virtual object.
[0105] In some embodiments, the method further includes: in response to not detecting the user's interactive action on the virtual object within a preset time period, rendering the virtual object according to the user's motion data in the virtual environment.
[0106] In this embodiment, if the user's interactive action on the virtual object is not detected within the preset time, it means that the virtual object prediction may be wrong, or the predicted motion data is wrong, or the user cannot complete the corresponding interactive action.
[0107] Therefore, the virtual object can be rendered according to the motion data of the user in the virtual environment, that is, the virtual object is rendered using the motion data at the current moment in a non-image projection manner, such as a conventional rendering manner.
[0108] Furthermore, after rendering, if no user interaction action on the virtual object is detected, the rendered virtual object is replaced, or the user is prompted to provide feedback on why there is no response.
[0109] Therefore, through this implementation, it is possible to avoid the situation where a user cannot be responded to in a timely manner in the event of a prediction error or other error, thereby achieving effective handling of the error situation.
[0110] It can be seen from the introduction of the embodiments of the present application that the user's motion data in the virtual environment is first obtained, and based on the motion data, the user's motion direction, motion type and motion range are predicted. Based on the prediction results, when it is determined that image projection can be used for virtual object rendering, image projection is used to render the virtual object, and then when the user's interactive action is detected, the virtual object is rendered again through image projection to respond to the user's interaction. In this way, the user's interaction is predicted, and the virtual object is rendered by image projection based on the prediction result, which is equivalent to pre-rendering the virtual object before the user's interaction, so that the user experience is better; and the rendering of the virtual object by image projection can quickly respond to the user's interaction. Therefore, this technical solution can enhance the user's virtual reality interactive experience.
[0111] Please refer to Figure 5 , the embodiment of the present application further provides a motion interaction device for virtual motion detection and projection, comprising:
[0112] The acquisition module 501 is used to acquire the motion data of the user in the virtual environment.
[0113] The interaction module 502 is used to determine the predicted motion data of the user according to the motion data of the user in the virtual environment, the predicted motion data including the predicted motion direction, the predicted motion type and the predicted motion range, the predicted motion type being used to characterize the predicted interactive action of the user on the virtual object in the virtual environment; determine to render the virtual object through image projection according to the predicted motion data; project a first virtual image corresponding to the virtual object to the virtual environment; and in response to detecting the interactive action of the user on the virtual object, project a second virtual image corresponding to the virtual object to the virtual environment.
[0114] The device corresponds to the aforementioned method, and the implementation of each module can refer to the aforementioned embodiment, which will not be repeated here.
[0115] Please refer to Figure 6 , an embodiment of the present application also provides an electronic device, which can be used to implement the aforementioned technical solution.
[0116] The electronic device includes a processor 601 and a memory 602, and the processor 601 and the memory 602 are communicatively connected.
[0117] The processor 601 and the memory 602 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements can be electrically connected through one or more communication buses or signal buses. The method steps executed by the aforementioned modules or interactive terminals respectively include at least one software function module that can be stored in the memory 602 in the form of software or firmware.
[0118] Processor 601 may be an integrated circuit chip with signal processing capability. Processor 601 may be a general-purpose processor, including a CPU (Central Processing Unit), NP (Network Processor), etc.; it may also be a digital signal processor, a dedicated integrated circuit, a readily available programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0119] The memory 602 can store various software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, that is, implements various steps in the embodiment of the present application.
[0120] The memory 602 may include but is not limited to RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.
[0121] Understandably, Figure 6 The structure shown is for illustration only. The electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations shown.
[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0126] The foregoing description of the specific exemplary embodiments of the present application is for the purpose of illustration and illustration. These descriptions are not intended to limit the present application to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present application and its practical application, so that those skilled in the art can realize and utilize the various exemplary embodiments of the present application and various selections and changes. The scope of the present application is intended to be limited by the claims and their equivalents.
Claims
1. A motion interaction method for virtual motion detection and projection, characterized in that: include: Obtaining the user's motion data in the virtual environment; Determining predicted motion data of the user according to the motion data of the user in the virtual environment, the predicted motion data including a predicted motion direction, a predicted motion type and a predicted motion range, the predicted motion type being used to characterize a predicted interactive action of the user with respect to a virtual object in the virtual environment; Determining, based on the predicted motion data, to render the virtual object by image projection; Projecting a first virtual image corresponding to the virtual object to the virtual environment; In response to detecting an interactive action of the user on the virtual object, a second virtual image corresponding to the virtual object is projected onto the virtual environment.
2. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The obtaining of the user's motion data in the virtual environment includes: Acquire first motion data collected by the smart wearable device worn by the user; Acquire second motion data collected by the electronic device held by the user; Acquire third motion data collected by a motion data collection device in the real environment where the user is located; The motion data of the user in the virtual environment is determined according to the first motion data, the second motion data and the third motion data.
3. The motion interaction method of virtual motion detection and projection according to claim 2, characterized in that: The determining, according to the first motion data, the second motion data, and the third motion data, the motion data of the user in the virtual environment includes: Determining, according to the first motion data, motion data corresponding to a first body region of the user; determining, according to the second motion data, motion data corresponding to a second body region of the user, the second body region being a part of the first body region; Determining, according to the third motion data, motion data corresponding to a third body region of the user, wherein a distance between the third body region and the first body region is greater than a preset distance; The motion data of the user in the virtual environment is determined based on the motion data corresponding to the first body area of the user, the motion data corresponding to the second body area of the user, and the motion data corresponding to the third body area of the user.
4. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The motion data of the user in the virtual environment includes: motion data corresponding to a first body region of the user, motion data corresponding to a second body region of the user, and motion data corresponding to a third body region of the user, and determining the predicted motion data of the user according to the motion data of the user in the virtual environment includes: converting the motion data corresponding to the first body region of the user into a first user motion image, where the first user motion image is used to represent the motion state of the first body region; converting the motion data corresponding to the second body region of the user into a second user motion image, where the second user motion image is used to represent the motion state of the second body region; converting the motion data corresponding to the third body region of the user into a third user motion image, wherein the third user motion image is used to represent the motion state of the third body region; Determine a predicted user motion image according to the first user motion image, the second user motion image, the third user motion image and a pre-trained motion prediction model, wherein the predicted user image is used to represent the predicted motion state of the user as a whole; The predicted motion data of the user is determined according to the predicted user motion image.
5. The motion interaction method of virtual motion detection and projection according to claim 4, characterized in that: The motion interaction method of virtual motion detection and projection also includes: Acquire a training data set, the training data set comprising a plurality of training samples, each training sample comprising: a plurality of sample images and a sample label, the plurality of sample images respectively representing the motion states of different body regions, the sample label being used to represent a predicted image for the plurality of sample images; The motion prediction model to be trained is trained according to the training data set to obtain the pre-trained motion prediction model.
6. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The step of determining, according to the predicted motion data, to render the virtual object by image projection comprises: If the predicted movement direction is directly in front of the user, and the predicted operation type indicates that the predicted interaction action of the user with respect to the virtual object in the virtual environment is a preset interaction action, and the predicted movement range is within the preset movement range, determining to render the virtual object through image projection; The preset interactive action is: an action that does not change the position of the virtual object in the virtual environment, or an action that changes the position of the virtual object in the virtual environment but does not change the position of the user in the virtual environment.
7. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The first virtual image is a virtual image corresponding to the initial state of the virtual object, and the motion interaction method of virtual motion detection and projection further includes: determining a predicted state of the virtual object according to the first virtual image and an interactive action of the user on the virtual object; A second virtual image corresponding to the virtual object is determined according to the initial state of the virtual object and the predicted state of the virtual object.
8. The motion interaction method of virtual motion detection and projection according to claim 7, characterized in that: The determining, according to the initial state of the virtual object and the predicted state of the virtual object, a second virtual image corresponding to the virtual object comprises: Acquire virtual images corresponding to a plurality of pre-configured states of the virtual object, wherein the virtual images corresponding to the plurality of states are arranged in a state change order; Determine, from the virtual images respectively corresponding to the multiple states, a virtual image corresponding to an intermediate state between the initial state and the predicted state, and determine a virtual image corresponding to the predicted state; A second virtual image corresponding to the virtual object is determined according to the virtual image corresponding to the intermediate state and the virtual image corresponding to the predicted state.
9. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The motion interaction method of virtual motion detection and projection also includes: Acquiring an interaction time of the user with respect to the virtual object, where the interaction time is determined according to interaction times corresponding to a plurality of interaction actions; If the user's interaction time with the virtual object is less than a preset interaction time, a projection tag is added to the virtual object, and the projection tag is used to instruct to render the virtual object through image projection when the virtual object is rendered next time.
10. The motion interaction method of virtual motion detection and projection according to claim 1, characterized in that: The motion interaction method of virtual motion detection and projection also includes: In response to not detecting an interactive action of the user on the virtual object within a preset time period, rendering the virtual object according to the motion data of the user in the virtual environment.