Virtual experience interaction control method based on virtual environment and related device

Abstract

The invention discloses a virtual experience interaction control method based on a virtual environment and a related device, and the method comprises the steps: loading a corresponding virtual interaction experience scene to a VR head-mounted display device based on a virtual experience game after an experience user wears the VR head-mounted display device and selects the virtual experience game; associating and generating a three-dimensional virtual character at a specified position of the virtual interaction experience scene based on the three-dimensional character model; acquiring a current motion trend based on capturing the current motion trend of the experiencing user; updating the position and the motion trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current motion trend to obtain an updated position and an updated motion trend of the three-dimensional virtual character; and adjusting interactive experience elements in the virtual interactive experience scene based on the updated motion trend, the updated position and the gaze tracking result. In the embodiment of the invention, seamless connection between the virtual environment and the real action is realized, so that the interactive experience is more natural and smoother.

Description

Technical Field

[0001] The present invention relates to the field of VR technology, and in particular to a virtual experience interactive control method based on a virtual environment and a related device. Background Art

[0002] Virtual reality technology (English name: Virtual Reality, abbreviated as VR); Virtual reality technology encompasses computers, electronic information, and simulation technology. Its basic implementation method is based on computer technology, utilizing and integrating the latest developments in three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other high-tech technologies. With the help of computers and other equipment, a realistic three-dimensional visual, tactile, olfactory and other sensory experiences are generated in the virtual world, so that people in the virtual world have an immersive feeling; with the continuous development of social productivity and science and technology, the demand for VR technology in all walks of life is growing; VR technology has also made great progress and has gradually become a new field of science and technology.

[0003] Existing VR technology is unable to establish a corresponding three-dimensional virtual character in the virtual interactive experience scene that needs to be experienced according to the three-dimensional character model corresponding to the experiencing user, and at the same time is unable to smoothly update and adjust the interactive experience elements in the virtual interactive experience scene in a timely manner according to the movement trend and line of sight tracking results; resulting in insufficient interactive smoothness during the virtual experience, and unable to provide a more ideal virtual interactive experience effect. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a virtual experience interaction control method and related devices based on a virtual environment, which realizes the establishment of a corresponding three-dimensional virtual character in a virtual interactive experience scene to be experienced according to the three-dimensional character model corresponding to the experience user, and realizes the seamless connection between the virtual environment and the real action, making the interactive experience more natural and smooth.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a virtual experience interaction control method based on a virtual environment, which is applied to a VR interaction system. The VR interaction system includes a virtual interaction device, a VR head-mounted display device, and a user motion capture space. The method includes: After the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; When the experience user enters the user motion capture space, the virtual interaction device generates a three-dimensional virtual character at a designated position of the virtual interaction experience scene based on the three-dimensional character model corresponding to the experience user; When the experience user starts to perform virtual experience interaction, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; The virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtains the updated position and the updated movement trend of the three-dimensional virtual character; The virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, the updated position and the sight tracking result.

[0006] Optionally, the virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game, including: The virtual interaction device indexes the virtual experience game selected by the experience user to the virtual interaction experience scene corresponding to the virtual experience game; The virtual interaction device loads the virtual interaction experience scene onto the VR head mounted display device based on a communication connection.

[0007] Optionally, the virtual interaction device generates a three-dimensional virtual character based on a three-dimensional character model corresponding to the experience user at a specified position of the virtual interaction experience scene, including: When the experience user enters the user motion capture space, the virtual interaction device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition processing on the experience user, multiple experience user acquisition images at different angles, and one experience user acquisition image corresponding to each angle; Input multiple images collected from different angles of the experience user into the three-dimensional coordinate extraction model respectively, perform three-dimensional space coordinate extraction processing, and obtain the three-dimensional space coordinate information corresponding to multiple experience users at different angles; A three-dimensional character model corresponding to a plurality of experience users at different angles is established based on the three-dimensional spatial coordinate information corresponding to the experience users, and a three-dimensional virtual character is generated at a specified position of a virtual interactive experience scene in the VR head mounted display device based on the three-dimensional character model, and the three-dimensional virtual character is dynamically associated with the three-dimensional character model.

[0008] Optionally, the establishing of the three-dimensional role model corresponding to the experience user based on the three-dimensional space coordinate information corresponding to the multiple experience users at different angles includes: Combining and processing the three-dimensional spatial coordinate information corresponding to multiple experience users at different angles to form the three-dimensional body information corresponding to the experience users; A three-dimensional character modeling algorithm is used to perform modeling processing based on the three-dimensional body information corresponding to the experience user, and a three-dimensional character model corresponding to the experience user is generated according to the modeling result.

[0009] Optionally, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user, including: The virtual interaction device controls multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experience user at the current moment, so as to obtain multiple images of the experience user collected at different angles at the current moment; Input multiple images collected by the experience users at different angles at the current moment into the three-dimensional coordinate extraction model to perform three-dimensional space coordinate extraction processing, and obtain the current three-dimensional space coordinate information corresponding to the multiple experience users at different angles; Based on the current three-dimensional spatial coordinate information corresponding to multiple experience users at different angles, a current three-dimensional character model corresponding to the experience user is established, and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment to obtain the current motion trend of the current three-dimensional character model corresponding to the experience user, and the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

[0010] Optionally, the virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend to obtain the updated position and the updated movement trend of the three-dimensional virtual character, including: Obtaining a position change of the current three-dimensional character model in the current motion compared with the three-dimensional character model at a previous moment; Based on the dynamic association between the three-dimensional character model and the three-dimensional virtual character, the position and movement trend of the three-dimensional virtual character in the virtual interactive experience scene are updated in real time using the posture change to obtain the updated position and updated movement trend of the three-dimensional virtual character.

[0011] Optionally, the virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, the updated position, and the sight tracking result, including: The virtual interaction device performs real-time sight tracking processing on the eyeballs of the experience user based on the VR head mounted display device to obtain a sight tracking result of the experience user; The virtual interactive device adaptively adjusts the interactive experience elements in the virtual interactive experience scene according to the progress of the virtual interactive experience scene based on the updated motion trend, the updated position and the sight tracking result, and the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0012] In addition, an embodiment of the present invention further provides a virtual experience interaction control device based on a virtual environment, which is applied to a VR interaction system. The VR interaction system includes a virtual interaction device, a VR head-mounted display device, and a user motion capture space. The device includes: Loading module: used for, after the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads the corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; A generating module: used for, when the experience user enters the user motion capture space, the virtual interaction device generates a three-dimensional virtual character at a specified position of the virtual interaction experience scene based on the three-dimensional character model corresponding to the experience user; Capturing module: used for, when the experience user starts to interact with the virtual experience, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; Update module: used for the virtual interaction device to update the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtain the updated position and updated movement trend of the three-dimensional virtual character; Adjustment module: used for the virtual interaction device to adaptively adjust the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result.

[0013] In addition, an embodiment of the present invention further provides a VR interaction system, which includes a virtual interaction device, a VR head-mounted display device, and a user motion capture space, and is configured to execute a virtual experience interaction control method as described in any one of the above.

[0014] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the virtual experience interaction control method as described in any one of the above.

[0015] In an embodiment of the present invention, a corresponding virtual interactive experience scene is loaded into a VR head mounted display device through a virtual experience game; then a three-dimensional virtual character is generated by associating a three-dimensional character model at a specified position in the virtual interactive experience scene; then the current motion trend of the experience user is captured and processed to obtain the current motion trend; then the position and motion trend of the three-dimensional virtual character in the virtual interactive experience scene are updated through the current motion trend to obtain an updated position and an updated motion trend of the three-dimensional virtual character; finally, the interactive experience elements in the virtual interactive experience scene are adjusted according to the updated motion trend, updated position and line of sight tracking results; thereby, a corresponding three-dimensional virtual character is established in the virtual interactive experience scene to be experienced according to the three-dimensional character model corresponding to the experience user, and a seamless connection between the virtual environment and the real action is achieved, making the interactive experience more natural and smooth. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a virtual experience interactive control method based on a virtual environment in an embodiment of the present invention; Figure 2 is a flow chart of a virtual experience interactive control method based on a virtual environment in another embodiment of the present invention; Figure 3 Schematic diagram of the structure of a virtual experience interactive control device based on a virtual environment in an embodiment of the present invention; Figure 4 It is a schematic diagram of the structural composition of the VR interaction system in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] For example, see Figure 1 , Figure 1 It is a flowchart of a virtual experience interactive control method based on a virtual environment in an embodiment of the present invention.

[0020] like Figure 1 As shown, a virtual experience interaction control method based on a virtual environment is applied to a VR interaction system, wherein the VR interaction system includes a virtual interaction device, a VR head mounted display device, and a user motion capture space, and the method includes: S101: After the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; In the specific implementation process of the present invention, the virtual interaction device loads the corresponding virtual interaction experience scene to the VR head-mounted display device based on the virtual experience game, including: the virtual interaction device indexes the virtual interaction experience scene corresponding to the virtual experience game based on the virtual experience game selected by the experience user; the virtual interaction device loads the virtual interaction experience scene to the VR head-mounted display device based on a communication connection.

[0021] Specifically, when the experience user needs to perform VR virtual interaction, it is first necessary to guide the experience user to wear the VR head-mounted display device, and then perform selection operations through the operation buttons on the VR head-mounted display device to complete the virtual experience game that the experience user needs to experience, and the VR head-mounted display device uploads the instructions of the virtual experience game selected by the user to the virtual interaction device. After the virtual interaction device receives the instructions of the virtual experience game selected by the user, it will index the virtual experience game selected by the experience user in the instructions to the virtual interaction experience scene corresponding to the virtual experience game, wherein the virtual interaction experience scene is the full scene map of the virtual experience game; and the virtual interaction device loads the virtual interaction experience scene onto the VR head-mounted display device through a communication connection.

[0022] The VR interactive system includes a virtual interactive device, a VR head-mounted display device and a user motion capture space, and the VR head-mounted display device and the user motion capture space are both communicated with the virtual interactive device, and a plurality of motion capture devices at different angles are arranged in the user motion capture space, and these motion capture devices are generally image acquisition devices.

[0023] S102: When the experience user enters the user motion capture space, the virtual interaction device generates a three-dimensional virtual character at a designated position of the virtual interaction experience scene based on a three-dimensional character model corresponding to the experience user; In the specific implementation process of the present invention, the virtual interaction device generates a three-dimensional virtual character based on the three-dimensional character model corresponding to the experience user at the specified position of the virtual interaction experience scene, including: when the experience user enters the user motion capture space, the virtual interaction device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition processing on the experience user, multiple experience user acquisition images at different angles, and each angle corresponds to an experience user acquisition image; the multiple experience user acquisition images at different angles are respectively input into the three-dimensional coordinate extraction model, and three-dimensional space coordinate extraction processing is performed to obtain three-dimensional space coordinate information corresponding to multiple experience users at different angles; based on the three-dimensional space coordinate information corresponding to multiple experience users at different angles, a three-dimensional character model corresponding to the experience user is established, and based on the three-dimensional character model, a three-dimensional virtual character is generated at the specified position of the virtual interaction experience scene in the VR head-mounted display device, and the three-dimensional virtual character is dynamically associated with the three-dimensional character model.

[0024] Furthermore, the three-dimensional character model corresponding to the experience user is established based on the three-dimensional spatial coordinate information corresponding to the multiple experience users at different angles, including: combining the three-dimensional spatial coordinate information corresponding to the multiple experience users at different angles to form the three-dimensional body information corresponding to the experience user; performing modeling processing using a three-dimensional character modeling algorithm based on the three-dimensional body information corresponding to the experience user, and generating the three-dimensional character model corresponding to the experience user according to the modeling result.

[0025] Specifically, when generating a three-dimensional virtual character, first, the experience user needs to enter the user motion capture space, and when the experience user enters the user motion capture space, the virtual interactive device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition and processing on the experience user, thereby obtaining multiple experience user captured images at different angles, wherein each angle corresponds to an experience user captured image; at this time, the three-dimensional space coordinates of each frame of the experience user captured image need to be extracted, and this needs to be achieved using a three-dimensional coordinate extraction model, wherein the three-dimensional coordinate extraction model can be a three-dimensional coordinate extraction model Vision. Transformer model, and it is a model that has been trained and converged. During the training process, the motion capture device at each angle collects images of the target in the user motion capture space to construct a target image set; and the corresponding position of the target in each image in the target image set is marked, generally manually through a rectangular frame for marking; and it is necessary to obtain the three-dimensional spatial coordinate information of the target in each image in the image set, and this three-dimensional spatial coordinate information is also manually marked, and the angle and device ID of the motion capture device corresponding to each image are also marked; at the same time, the position of each image in the image set is coupled with the three-dimensional spatial coordinate information, so that a training data set can be formed, and the model can be trained through the training data set until convergence.

[0026] Then, the multiple images of the experience users captured at different angles need to be input into the three-dimensional coordinate extraction model respectively, and the three-dimensional space coordinate extraction processing is performed, so that the three-dimensional space coordinate information corresponding to the multiple experience users at different angles can be obtained; in this way, a three-dimensional character model corresponding to the experience user can be established according to the three-dimensional space coordinate information corresponding to the multiple experience users at different angles, and a three-dimensional virtual character can be generated at a specified position of the virtual interactive experience scene in the VR head-mounted display device according to the three-dimensional character model, and the three-dimensional virtual character can be dynamically associated with the three-dimensional character model at the same time. In this way, when the action or position of the experience user in the user motion capture space changes, the action trend of the three-dimensional virtual character can be updated in time.

[0027] When establishing a three-dimensional character model, the first step is to combine and process the three-dimensional spatial coordinate information corresponding to multiple experience users at different angles to form the three-dimensional body information corresponding to the experience user; finally, the three-dimensional character modeling algorithm will be used to perform modeling processing based on the three-dimensional body information corresponding to the experience user. After the modeling is completed, the three-dimensional character model corresponding to the experience user will be generated based on the modeling results; when performing modeling, the three-dimensional body information is used as the key point of three-dimensional modeling, and then modeling fitting is performed to generate a three-dimensional character model.

[0028] S103: When the experience user starts to perform virtual experience interaction, the virtual interaction device captures the current movement trend of the experience user based on the user motion capture space to obtain the current movement trend of the experience user; In the specific implementation process of the present invention, the virtual interaction device captures and processes the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user, including: the virtual interaction device controls multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experience user at the current moment, and obtain multiple experience user captured images at different angles at the current moment; the multiple experience user captured images at different angles at the current moment are respectively input into the three-dimensional coordinate extraction model for three-dimensional space coordinate extraction processing to obtain the current three-dimensional space coordinate information corresponding to multiple experience users at different angles; based on the current three-dimensional space coordinate information corresponding to multiple experience users at different angles, a current three-dimensional character model corresponding to the experience user is established, and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment to obtain the current motion trend of the current three-dimensional character model corresponding to the experience user, and the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

[0029] Specifically, when it is necessary to capture the motion trend of the experiencing user, the virtual interactive device is required to control multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experiencing user at the current moment, so as to obtain multiple images of the experiencing user collected at different angles at the current moment; then the multiple images of the experiencing user collected at different angles at the current moment are respectively input into the three-dimensional coordinate extraction model for three-dimensional space coordinate extraction processing, so as to obtain the current three-dimensional space coordinate information corresponding to the multiple experiencing users at different angles; finally, the current three-dimensional character model corresponding to the experiencing user is established according to the current three-dimensional space coordinate information corresponding to the multiple experiencing users at different angles, and the specific method is the same as the above steps; and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment, so as to obtain the current motion trend of the current three-dimensional character model corresponding to the experiencing user, wherein the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

[0030] S104: the virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtains an updated position and an updated movement trend of the three-dimensional virtual character; In the specific implementation process of the present invention, the virtual interaction device updates the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend to obtain the updated position and updated movement trend of the three-dimensional virtual character, including: obtaining the posture change of the current three-dimensional character model in the current movement compared to the three-dimensional character model at the previous moment; based on the dynamic association relationship between the three-dimensional character model and the three-dimensional virtual character, using the posture change to perform real-time update processing on the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene to obtain the updated position and updated movement trend of the three-dimensional virtual character.

[0031] Specifically, it is necessary to obtain the posture change of the current three-dimensional character model in the current motion compared to the three-dimensional character model at the previous moment; and after obtaining the posture change of the current three-dimensional character model in the current motion compared to the three-dimensional character model at the previous moment, the dynamic association relationship between the three-dimensional character model and the three-dimensional virtual character will be used to utilize the posture change to perform real-time update processing on the position and movement trend of the three-dimensional virtual character in the virtual interactive experience scene, thereby obtaining the updated position and updated movement trend of the three-dimensional virtual character; after obtaining the updated position and updated movement trend of the three-dimensional virtual character, the interactive experience elements in the virtual interactive experience scene are updated and adjusted in a timely manner.

[0032] S105: The virtual interaction device adaptively adjusts the interaction experience elements in the virtual interaction experience scene based on the updated motion trend, the updated position, and the sight tracking result.

[0033] In the specific implementation process of the present invention, the virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result, including: the virtual interaction device performs real-time line of sight tracking processing on the eyeballs of the experience user based on the VR head-mounted display device to obtain the line of sight tracking result of the experience user; the virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene according to the progress setting of the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result, and the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0034] Specifically, after the experience user wears the VR head-mounted display device, the VR head-mounted display device will start the line of sight tracking function to track the experience user's line of sight. The specific implementation is to track and monitor the experience user's eyeballs, and infer the fitting based on the experience user's eye movement tracking monitoring results, so as to track the experience user's line of sight and obtain the experience user's line of sight tracking results; in order to make the experience user seamlessly connect the virtual environment with the real action and make the interactive experience more natural and smooth, it will be necessary to adaptively adjust the interactive experience elements in the virtual interactive experience scene according to the updated motion trend, updated position and line of sight tracking results according to the progress of the virtual interactive experience scene; the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0035] In an embodiment of the present invention, a corresponding virtual interactive experience scene is loaded into a VR head mounted display device through a virtual experience game; then a three-dimensional virtual character is generated by associating a three-dimensional character model at a specified position in the virtual interactive experience scene; then the current motion trend of the experience user is captured and processed to obtain the current motion trend; then the position and motion trend of the three-dimensional virtual character in the virtual interactive experience scene are updated through the current motion trend to obtain an updated position and an updated motion trend of the three-dimensional virtual character; finally, the interactive experience elements in the virtual interactive experience scene are adjusted according to the updated motion trend, updated position and line of sight tracking results; thereby, a corresponding three-dimensional virtual character is established in the virtual interactive experience scene to be experienced according to the three-dimensional character model corresponding to the experience user, and a seamless connection between the virtual environment and the real action is achieved, making the interactive experience more natural and smooth.

[0036] For example 2, please refer to Figure 2 , Figure 2 It is a flowchart of a virtual experience interactive control method based on a virtual environment in another embodiment of the present invention.

[0037] like Figure 2 A virtual experience interaction control method based on a virtual environment is applied to a VR interaction system, wherein the VR interaction system includes a virtual interaction device, a VR head mounted display device, and a user motion capture space, and the method includes: S201: After the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; S202: When the experience user enters the user motion capture space, the virtual interaction device controls a plurality of motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition processing on the experience user, and a plurality of experience user acquisition images at different angles, each angle corresponding to one experience user acquisition image; S203: inputting the multiple images collected by the experience users at different angles into the three-dimensional coordinate extraction model respectively, performing three-dimensional space coordinate extraction processing, and obtaining the three-dimensional space coordinate information corresponding to the multiple experience users at different angles; S204: establishing a three-dimensional role model corresponding to a plurality of experience users based on three-dimensional spatial coordinate information corresponding to the experience users at different angles, and generating a three-dimensional virtual role at a designated position of a virtual interactive experience scene in the VR head mounted display device based on the three-dimensional role model, wherein the three-dimensional virtual role is dynamically associated with the three-dimensional role model; S205: When the experience user starts to perform virtual experience interaction, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; S206: the virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtains an updated position and an updated movement trend of the three-dimensional virtual character; S207: The virtual interaction device performs real-time sight tracking processing on the eyeballs of the experience user based on the VR head mounted display device to obtain a sight tracking result of the experience user; S208: The virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene according to the progress of the virtual interactive experience scene based on the updated motion trend, the updated position and the sight tracking result, wherein the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0038] The specific implementation of the second embodiment can be found in the first embodiment, which will not be described in detail here.

[0039] For example 3, please refer to Figure 3 , Figure 3 It is a schematic diagram of the structural composition of a virtual experience interactive control device based on a virtual environment in an embodiment of the present invention.

[0040] like Figure 3 As shown, a virtual experience interactive control device based on a virtual environment is applied to a VR interactive system, wherein the VR interactive system includes a virtual interactive device, a VR head mounted display device and a user motion capture space, and the device includes: Loading module 301: configured to load the corresponding virtual interactive experience scene to the VR head mounted display device based on the virtual experience game after the experience user wears the VR head mounted display device and selects a virtual experience game; In the specific implementation process of the present invention, the virtual interaction device loads the corresponding virtual interaction experience scene to the VR head-mounted display device based on the virtual experience game, including: the virtual interaction device indexes the virtual interaction experience scene corresponding to the virtual experience game based on the virtual experience game selected by the experience user; the virtual interaction device loads the virtual interaction experience scene to the VR head-mounted display device based on a communication connection.

[0041] Specifically, when the experience user needs to perform VR virtual interaction, it is first necessary to guide the experience user to wear the VR head-mounted display device, and then perform selection operations through the operation buttons on the VR head-mounted display device to complete the virtual experience game that the experience user needs to experience, and the VR head-mounted display device uploads the instructions of the virtual experience game selected by the user to the virtual interaction device. After the virtual interaction device receives the instructions of the virtual experience game selected by the user, it will index the virtual experience game selected by the experience user in the instructions to the virtual interaction experience scene corresponding to the virtual experience game, wherein the virtual interaction experience scene is the full scene map of the virtual experience game; and the virtual interaction device loads the virtual interaction experience scene onto the VR head-mounted display device through a communication connection.

[0042] The VR interactive system includes a virtual interactive device, a VR head-mounted display device and a user motion capture space, and the VR head-mounted display device and the user motion capture space are both communicated with the virtual interactive device, and a plurality of motion capture devices at different angles are arranged in the user motion capture space, and these motion capture devices are generally image acquisition devices.

[0043] Generating module 302: configured to generate, by the virtual interaction device, a three-dimensional virtual character at a designated position of the virtual interaction experience scene based on a three-dimensional character model corresponding to the experience user when the experience user enters the user motion capture space; In the specific implementation process of the present invention, the virtual interaction device generates a three-dimensional virtual character based on the three-dimensional character model corresponding to the experience user at the specified position of the virtual interaction experience scene, including: when the experience user enters the user motion capture space, the virtual interaction device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition processing on the experience user, multiple experience user acquisition images at different angles, and each angle corresponds to an experience user acquisition image; the multiple experience user acquisition images at different angles are respectively input into the three-dimensional coordinate extraction model, and three-dimensional space coordinate extraction processing is performed to obtain three-dimensional space coordinate information corresponding to multiple experience users at different angles; based on the three-dimensional space coordinate information corresponding to multiple experience users at different angles, a three-dimensional character model corresponding to the experience user is established, and based on the three-dimensional character model, a three-dimensional virtual character is generated at the specified position of the virtual interaction experience scene in the VR head-mounted display device, and the three-dimensional virtual character is dynamically associated with the three-dimensional character model.

[0044] Furthermore, the three-dimensional character model corresponding to the experience user is established based on the three-dimensional spatial coordinate information corresponding to the multiple experience users at different angles, including: combining the three-dimensional spatial coordinate information corresponding to the multiple experience users at different angles to form the three-dimensional body information corresponding to the experience user; performing modeling processing using a three-dimensional character modeling algorithm based on the three-dimensional body information corresponding to the experience user, and generating the three-dimensional character model corresponding to the experience user according to the modeling result.

[0045] Specifically, when generating a three-dimensional virtual character, first, the experience user needs to enter the user motion capture space, and when the experience user enters the user motion capture space, the virtual interactive device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition and processing on the experience user, thereby obtaining multiple experience user captured images at different angles, wherein each angle corresponds to an experience user captured image; at this time, the three-dimensional space coordinates of each frame of the experience user captured image need to be extracted, and this needs to be achieved using a three-dimensional coordinate extraction model, wherein the three-dimensional coordinate extraction model can be a three-dimensional coordinate extraction model Vision. Transformer model, and it is a model that has been trained and converged. During the training process, the motion capture device at each angle collects images of the target in the user motion capture space to construct a target image set; and the corresponding position of the target in each image in the target image set is marked, generally manually through a rectangular frame for marking; and it is necessary to obtain the three-dimensional spatial coordinate information of the target in each image in the image set, and this three-dimensional spatial coordinate information is also manually marked, and the angle and device ID of the motion capture device corresponding to each image are also marked; at the same time, the position of each image in the image set is coupled with the three-dimensional spatial coordinate information, so that a training data set can be formed, and the model can be trained through the training data set until convergence.

[0046] Then, the multiple images of the experience users captured at different angles need to be input into the three-dimensional coordinate extraction model respectively, and the three-dimensional space coordinate extraction processing is performed, so that the three-dimensional space coordinate information corresponding to the multiple experience users at different angles can be obtained; in this way, a three-dimensional character model corresponding to the experience user can be established according to the three-dimensional space coordinate information corresponding to the multiple experience users at different angles, and a three-dimensional virtual character can be generated at a specified position of the virtual interactive experience scene in the VR head-mounted display device according to the three-dimensional character model, and the three-dimensional virtual character can be dynamically associated with the three-dimensional character model at the same time. In this way, when the action or position of the experience user in the user motion capture space changes, the action trend of the three-dimensional virtual character can be updated in time.

[0047] When establishing a three-dimensional character model, the first step is to combine and process the three-dimensional spatial coordinate information corresponding to multiple experience users at different angles to form the three-dimensional body information corresponding to the experience user; finally, the three-dimensional character modeling algorithm will be used to perform modeling processing based on the three-dimensional body information corresponding to the experience user. After the modeling is completed, the three-dimensional character model corresponding to the experience user will be generated based on the modeling results; when performing modeling, the three-dimensional body information is used as the key point of three-dimensional modeling, and then modeling fitting is performed to generate a three-dimensional character model.

[0048] Capturing module 303: used for, when the experience user starts to perform virtual experience interaction, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; In the specific implementation process of the present invention, the virtual interaction device captures and processes the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user, including: the virtual interaction device controls multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experience user at the current moment, and obtain multiple experience user captured images at different angles at the current moment; the multiple experience user captured images at different angles at the current moment are respectively input into the three-dimensional coordinate extraction model for three-dimensional space coordinate extraction processing to obtain the current three-dimensional space coordinate information corresponding to multiple experience users at different angles; based on the current three-dimensional space coordinate information corresponding to multiple experience users at different angles, a current three-dimensional character model corresponding to the experience user is established, and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment to obtain the current motion trend of the current three-dimensional character model corresponding to the experience user, and the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

[0049] Specifically, when it is necessary to capture the motion trend of the experiencing user, the virtual interactive device is required to control multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experiencing user at the current moment, so as to obtain multiple images of the experiencing user collected at different angles at the current moment; then the multiple images of the experiencing user collected at different angles at the current moment are respectively input into the three-dimensional coordinate extraction model for three-dimensional space coordinate extraction processing, so as to obtain the current three-dimensional space coordinate information corresponding to the multiple experiencing users at different angles; finally, the current three-dimensional character model corresponding to the experiencing user is established according to the current three-dimensional space coordinate information corresponding to the multiple experiencing users at different angles, and the specific method is the same as the above steps; and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment, so as to obtain the current motion trend of the current three-dimensional character model corresponding to the experiencing user, wherein the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

[0050] Update module 304: used for the virtual interaction device to update the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtain the updated position and updated movement trend of the three-dimensional virtual character; In the specific implementation process of the present invention, the virtual interaction device updates the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend to obtain the updated position and updated movement trend of the three-dimensional virtual character, including: obtaining the posture change of the current three-dimensional character model in the current movement compared to the three-dimensional character model at the previous moment; based on the dynamic association relationship between the three-dimensional character model and the three-dimensional virtual character, using the posture change to perform real-time update processing on the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene to obtain the updated position and updated movement trend of the three-dimensional virtual character.

[0051] Specifically, it is necessary to obtain the posture change of the current three-dimensional character model in the current motion compared to the three-dimensional character model at the previous moment; and after obtaining the posture change of the current three-dimensional character model in the current motion compared to the three-dimensional character model at the previous moment, the dynamic association relationship between the three-dimensional character model and the three-dimensional virtual character will be used to utilize the posture change to perform real-time update processing on the position and movement trend of the three-dimensional virtual character in the virtual interactive experience scene, thereby obtaining the updated position and updated movement trend of the three-dimensional virtual character; after obtaining the updated position and updated movement trend of the three-dimensional virtual character, the interactive experience elements in the virtual interactive experience scene are updated and adjusted in a timely manner.

[0052] Adjustment module 305: used for the virtual interaction device to adaptively adjust the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, updated position and sight tracking result.

[0053] In the specific implementation process of the present invention, the virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result, including: the virtual interaction device performs real-time line of sight tracking processing on the eyeballs of the experience user based on the VR head-mounted display device to obtain the line of sight tracking result of the experience user; the virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene according to the progress setting of the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result, and the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0054] Specifically, after the experience user wears the VR head-mounted display device, the VR head-mounted display device will start the line of sight tracking function to track the experience user's line of sight. The specific implementation is to track and monitor the experience user's eyeballs, and infer the fitting based on the experience user's eye movement tracking monitoring results, so as to track the experience user's line of sight and obtain the experience user's line of sight tracking results; in order to make the experience user seamlessly connect the virtual environment with the real action and make the interactive experience more natural and smooth, it will be necessary to adaptively adjust the interactive experience elements in the virtual interactive experience scene according to the updated motion trend, updated position and line of sight tracking results according to the progress of the virtual interactive experience scene; the interactive experience elements include landscapes, obstacles and light and shadow effects.

[0055] In an embodiment of the present invention, a corresponding virtual interactive experience scene is loaded into a VR head mounted display device through a virtual experience game; then a three-dimensional virtual character is generated by associating a three-dimensional character model at a specified position in the virtual interactive experience scene; then the current motion trend of the experience user is captured and processed to obtain the current motion trend; then the position and motion trend of the three-dimensional virtual character in the virtual interactive experience scene are updated through the current motion trend to obtain an updated position and an updated motion trend of the three-dimensional virtual character; finally, the interactive experience elements in the virtual interactive experience scene are adjusted according to the updated motion trend, updated position and line of sight tracking results; thereby, a corresponding three-dimensional virtual character is established in the virtual interactive experience scene to be experienced according to the three-dimensional character model corresponding to the experience user, and a seamless connection between the virtual environment and the real action is achieved, making the interactive experience more natural and smooth.

[0056] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the virtual experience interactive control method of any one of the above embodiments is implemented. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (EraSable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that can store or transmit information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.

[0057] An embodiment of the present invention further provides a computer application program, which runs on a computer and is used to execute the virtual experience interactive control method of any one of the above embodiments.

[0058] also, Figure 4 It is a schematic diagram of the structural composition of the VR interaction system in an embodiment of the present invention.

[0059] The embodiment of the present invention also provides a VR interactive system, such as Figure 4 The VR interactive system includes a virtual interactive device 402, a VR head mounted display device 401, and a user motion capture space 403. Those skilled in the art will appreciate that Figure 4 The structural components of the VR interactive system shown do not constitute a limitation on all devices and may include more or fewer components than shown in the figure, or combine certain components.

[0060] As an embodiment, the VR interaction system includes a virtual interaction device, a VR head-mounted display device, and a user motion capture space, and the VR interaction system is configured to execute the virtual experience interaction control method as described in any one of the above.

[0061] In an embodiment of the present invention, a corresponding virtual interactive experience scene is loaded into a VR head mounted display device through a virtual experience game; then a three-dimensional virtual character is generated by associating a three-dimensional character model at a specified position in the virtual interactive experience scene; then the current motion trend of the experience user is captured and processed to obtain the current motion trend; then the position and motion trend of the three-dimensional virtual character in the virtual interactive experience scene are updated through the current motion trend to obtain an updated position and an updated motion trend of the three-dimensional virtual character; finally, the interactive experience elements in the virtual interactive experience scene are adjusted according to the updated motion trend, updated position and line of sight tracking results; thereby, a corresponding three-dimensional virtual character is established in the virtual interactive experience scene to be experienced according to the three-dimensional character model corresponding to the experience user, and a seamless connection between the virtual environment and the real action is achieved, making the interactive experience more natural and smooth.

[0062] In addition, the above is a detailed introduction to a virtual experience interactive control method based on a virtual environment and related devices provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A virtual experience interactive control method based on a virtual environment, characterized in that: Applied to a VR interaction system, the VR interaction system includes a virtual interaction device, a VR head mounted display device and a user motion capture space, the method includes: After the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; When the experience user enters the user motion capture space, the virtual interaction device generates a three-dimensional virtual character at a designated position of the virtual interaction experience scene based on the three-dimensional character model corresponding to the experience user; When the experience user starts to perform virtual experience interaction, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; The virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtains the updated position and the updated movement trend of the three-dimensional virtual character; The virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, the updated position and the sight tracking result.

2. The virtual experience interactive control method according to claim 1, characterized in that: The virtual interaction device loads a corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game, including: The virtual interaction device indexes the virtual experience game selected by the experience user to the virtual interaction experience scene corresponding to the virtual experience game; The virtual interaction device loads the virtual interaction experience scene onto the VR head mounted display device based on a communication connection.

3. The virtual experience interactive control method according to claim 1, characterized in that: The virtual interaction device generates a three-dimensional virtual character based on the three-dimensional character model corresponding to the experience user at a specified position of the virtual interaction experience scene, including: When the experience user enters the user motion capture space, the virtual interaction device controls multiple motion capture devices at different angles set in the user motion capture space to simultaneously perform image acquisition processing on the experience user, multiple experience user acquisition images at different angles, and one experience user acquisition image corresponding to each angle; Input multiple images collected from different angles of the experience user into the three-dimensional coordinate extraction model respectively, perform three-dimensional space coordinate extraction processing, and obtain the three-dimensional space coordinate information corresponding to multiple experience users at different angles; A three-dimensional character model corresponding to a plurality of experience users at different angles is established based on the three-dimensional spatial coordinate information corresponding to the experience users, and a three-dimensional virtual character is generated at a specified position of a virtual interactive experience scene in the VR head mounted display device based on the three-dimensional character model, and the three-dimensional virtual character is dynamically associated with the three-dimensional character model.

4. The virtual experience interactive control method according to claim 3, characterized in that: The step of establishing a three-dimensional role model corresponding to a plurality of experience users based on three-dimensional space coordinate information corresponding to the experience users at different angles includes: Combining and processing the three-dimensional spatial coordinate information corresponding to multiple experience users at different angles to form the three-dimensional body information corresponding to the experience users; A three-dimensional character modeling algorithm is used to perform modeling processing based on the three-dimensional body information corresponding to the experience user, and a three-dimensional character model corresponding to the experience user is generated according to the modeling result.

5. The virtual experience interactive control method according to claim 1, characterized in that: The virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user, including: The virtual interaction device controls multiple motion capture devices at different angles in the user motion capture space to simultaneously collect and process the image of the experience user at the current moment, so as to obtain multiple images of the experience user collected at different angles at the current moment; Input multiple images collected by the experience users at different angles at the current moment into the three-dimensional coordinate extraction model to perform three-dimensional space coordinate extraction processing, and obtain the current three-dimensional space coordinate information corresponding to the multiple experience users at different angles; Based on the current three-dimensional spatial coordinate information corresponding to multiple experience users at different angles, a current three-dimensional character model corresponding to the experience user is established, and the current three-dimensional character model is compared with the three-dimensional character model at the previous moment to obtain the current motion trend of the current three-dimensional character model corresponding to the experience user, and the current motion trend is the posture change of the current three-dimensional character model compared with the three-dimensional character model at the previous moment.

6. The virtual experience interactive control method according to claim 1, characterized in that: The virtual interaction device updates the position and the movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtains the updated position and the updated movement trend of the three-dimensional virtual character, including: Obtaining a position change of the current three-dimensional character model in the current motion compared with the three-dimensional character model at a previous moment; Based on the dynamic association between the three-dimensional character model and the three-dimensional virtual character, the position and movement trend of the three-dimensional virtual character in the virtual interactive experience scene are updated in real time using the posture change to obtain the updated position and updated movement trend of the three-dimensional virtual character.

7. The virtual experience interactive control method according to claim 1, characterized in that: The virtual interaction device adaptively adjusts the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, the updated position, and the sight tracking result, including: The virtual interaction device performs real-time sight tracking processing on the eyeballs of the experience user based on the VR head mounted display device to obtain a sight tracking result of the experience user; The virtual interactive device adaptively adjusts the interactive experience elements in the virtual interactive experience scene according to the progress of the virtual interactive experience scene based on the updated motion trend, the updated position and the sight tracking result, and the interactive experience elements include landscapes, obstacles and light and shadow effects.

8. A virtual experience interactive control device based on a virtual environment, characterized in that: Applied to a VR interactive system, the VR interactive system includes a virtual interactive device, a VR head mounted display device and a user motion capture space, the device includes: Loading module: used for, after the experience user wears the VR head mounted display device and selects a virtual experience game, the virtual interaction device loads the corresponding virtual interaction experience scene to the VR head mounted display device based on the virtual experience game; A generating module: used for, when the experience user enters the user motion capture space, the virtual interaction device generates a three-dimensional virtual character at a specified position of the virtual interaction experience scene based on the three-dimensional character model corresponding to the experience user; Capturing module: used for, when the experience user starts to interact with the virtual experience, the virtual interaction device captures the current motion trend of the experience user based on the user motion capture space to obtain the current motion trend of the experience user; Update module: used for the virtual interaction device to update the position and movement trend of the three-dimensional virtual character in the virtual interaction experience scene based on the current movement trend, and obtain the updated position and updated movement trend of the three-dimensional virtual character; Adjustment module: used for the virtual interaction device to adaptively adjust the interactive experience elements in the virtual interactive experience scene based on the updated motion trend, updated position and line of sight tracking result.

9. A VR interactive system, comprising a virtual interactive device, a VR head mounted display device and a user motion capture space, characterized in that: The VR interaction system is configured to execute the virtual experience interaction control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the virtual experience interaction control method according to any one of claims 1 to 7.

Images