An interactive method and system based on VR digital content

Abstract

This invention relates to an interactive method and system based on VR digital content. The method includes: acquiring current user identity information and user-specific attribute information; collecting multi-dimensional user interaction information in real time; determining user intent and identifying candidate intent scenarios based on the user intent; determining the target scene and pre-loading the target scene data into the local client or VR device. This invention achieves rapid loading of the target scene through hierarchical, modular, and step-by-step loading, greatly reducing interaction stutters and improving the interactive experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of VR interaction, and particularly relates to an interaction method and system based on VR digital content. BACKGROUND

[0002] With the progress of Internet technology, human communication methods gradually move towards the virtual reality era. Virtual reality (VR) is a computer simulation technology that can create and experience a virtual world. It uses computers to generate an interactive three-dimensional dynamic view, and the simulation system of entity behavior can make users immerse in the environment. Virtual reality technology is a computer simulation system that can create and experience a virtual world. It uses computers to generate a simulated environment, which is a multi-source information fusion interactive three-dimensional dynamic view and entity behavior system simulation, making users immerse in the environment. Augmented reality is a technology that calculates the position and angle of the camera image in real time and adds corresponding images,

[0003] The goal of this technology is to overlay the virtual world onto the real world on a screen and enable interaction. With the increasing computing power of portable electronic devices, the applications of augmented reality are expected to become increasingly widespread. Currently, we experience many VR applications in industries such as gaming, real estate, film and television, education, and healthcare. VR video is essentially a spherical video containing 360° x 180° omnidirectional visual information, allowing viewers to change their viewing angle and select areas of interest. Because VR video covers an omnidirectional view, it requires higher resolution (8K and above) to ensure clarity and immersion. Users can experience exciting sporting events, tour world-famous landmarks, and enjoy immersive real-world learning without leaving home using VR videos. VR devices are mainly divided into two types: professional VR devices used for game control and browsing, which require a cable connection to a high-configuration computer and operation using a controller; and VR glasses, which require an application to be installed on a smart terminal to play VR content. User operation involves moving the body to operate buttons and interact with the VR scene. With the promotion of high-value applications such as VR sports, exhibitions, and education, VR video content is gradually moving towards ultra-high definition. High-resolution cameras and stitching technology enable the production of ultra-high-resolution VR videos, which will undoubtedly bring a better user experience. However, this also leads to a surge in scene data that needs to be displayed instantly or quickly in order to provide a better user experience. Remotely acquiring the data required for display will inevitably greatly reduce the user experience, and directly loading all the data into the VR device is obviously not feasible. Therefore, how to effectively record data during real-time user interaction to avoid stagnation and stuttering and improve the interactive experience is a current hot issue and a problem to be solved. This invention determines the set of scenes to be loaded based on multi-dimensional user interaction information. For different usage scenarios and predicted usage scenarios, it achieves rapid loading of target scenes by loading them in a hierarchical and modular manner, greatly reducing the stuttering feeling of interaction and improving the interactive experience. [Summary of the Invention]

[0004] To address the aforementioned problems in the prior art, this invention proposes an interactive method and system based on VR digital content, the method comprising:

[0005] Step S1: Construct a scene transition graph based on user historical data; the scene transition graph includes one or more scene nodes and edges formed by their transition connection relationships; edge l between the first node i and the second node j i,j weight wl i,j It equals the transition probability of the scene corresponding to the first node to the second node;

[0006] Step S2: Obtain the current user's identity information and inherent user attribute information;

[0007] Step S3: collecting multi-dimension user interaction information in real time;

[0008] Step S4: determining user intention and determining candidate intention scene based on user intention; the determination of user intention is specifically: obtaining intention feature vector based on user interaction data; obtaining intention feature vector corresponding to user inherent attribute information; looking up the reference table based on the intention feature vector to obtain the intention corresponding to the intention feature vector; the reference table is a reference table of the corresponding relationship between intention feature vector and intention;

[0009] Step S5: determining target scene according to current scene and candidate intention scene based on scene transition graph; loading target scene data to local client in advance; the current scene is the scene where the current user is located;

[0010] Step S6: loading all or part of modules in target scene to VR device based on historical operation record of user to scene module and historical stay time of target scene.

[0011] Further, the user interaction data includes body action data, eye action data and / or touch interaction data.

[0012] Further, each intention has one or more intention features.

[0013] Further, there are multiple reference tables, and the reference table corresponding to each different user inherent attribute is different; the reference table is a pre-stored reference table.

[0014] Further, based on user interaction indication, scene data located in VR device, local client and remote server is selected in sequence.

[0015] A VR digital content-based interaction system for implementing the above-mentioned VR digital content-based interaction method, the system is arranged on a cloud server, and obtains the required user historical data and user operation record included in the user historical data from a big data server;

[0016] Further comprising one or more local clients and one or more VR devices; each local client serves one or more VR devices; when providing VR digital content interaction, the local client can provide scene data to the VR device and manage and control the VR device.

[0017] A processor for running a program, wherein the program performs the VR digital content-based interaction method when running.

[0018] An execution device, characterized by comprising a processor, the processor and a memory are coupled, the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the interactive method based on VR digital content is realized.

[0019] A computer readable storage medium, comprising a program, when it is run on a computer, so that the computer executes the interactive method based on VR digital content.

[0020] A cloud server, characterized in that the cloud server is configured to execute the interactive method based on VR digital content.

[0021] The beneficial effects of the present application include:

[0022] (1) Based on multi-dimensional user interaction information to determine the scene set to be loaded, under the guidance of user personalized data, for different use cases and predicted use cases, through hierarchical, modular, step-by-step loading, the target scene is quickly loaded, greatly reducing the occurrence of pause, and improving the interactive experience;

[0023] (2) The scene transfer graph is proposed, the target scene is circled from the global range through the intention guess of the current user, the scene most likely to be used subsequently is selected according to the local loading capacity for preloading, the subsequent data reading speed is improved, and the influence of experience caused by network lag is avoided;

[0024] (3) Based on the stay time and user historical operation record, the VR device can load more scenes and scene modules conforming to the user operation habit, and the user can get good interactive response speed in the interaction process no matter which path is entered.

DRAWINGS

[0025] The drawings described herein are used to provide further understanding of the present application, constitute a part of the present application, but do not constitute improper limitation to the present application, in the drawings:

[0026] Figure 1 The schematic diagram of the interactive method based on VR digital content provided by the present application.

CONCRETE IMPLEMENTATION

[0027] The present application will be described in detail below in combination with the drawings and specific embodiments, the schematic embodiments and the description are only used to explain the present application, but do not constitute the limitation to the present application

[0028] The present application proposes an interactive method and system based on VR digital content, the method comprises the following steps:

[0029] Step S1: constructing a scene transition graph based on user historical data; the scene transition graph includes one or more scene nodes and the edge formed by the transition connection relationship between the scene nodes; the edge l between the first node i and the second node j i,j The weight wl i,j of the edge l is equal to the transition probability of the corresponding scene from the first node to the second node; the transition probability is obtained according to the analysis of the user historical data;

[0030] Step S2: obtaining current user identity information and user inherent attribute information; wherein: the user identity information includes user identification, such as mobile phone number, ID number, etc.; the user inherent attribute information includes attribute information that is not easy to change and is different from other users; such as gender, age (birthday), occupation, birthplace, etc.

[0031] Step S3: real-time collection of multi-dimensional user interaction information; wherein: the user interaction data includes one or more dimensions in the dimensions of body action data, eye action data, touch interaction data, etc.

[0032] Step S4: determining the user intent and determining the candidate intent scene based on the user intent; the switching or staying between the scenes by the user is with the existence of the internal intent, and the user tries to find the intent in the VR journey, so the scene switching performed by each person is around the intent; for example: the user intent to enter the flower house, the intent to eat, etc.

[0033] The determination of the user intent is specifically: obtaining an intent feature vector based on the user interaction data; obtaining an intent feature vector corresponding to the user inherent attribute information; looking up a reference table based on the intent feature vector to obtain an intent corresponding to the intent feature vector; the reference table is a reference table of the corresponding relationship between the intent feature vector and the intent; each intent has one or more intent features;

[0034] Preferably, the reference table is a pre-stored reference table; further, the reference table has a plurality of reference tables, and the reference table corresponding to each user inherent attribute is different, so that the preliminary distinction of different users can be formed;

[0035] The step S4 specifically includes the following steps:

[0036] Step S4A1: obtaining user interaction data of an unprocessed dimension;

[0037] Step S4A2: for each intent feature, judging whether there is a data value satisfying the intent feature in the user interaction data, if yes, setting the intent feature value corresponding to the intent feature in the intent feature vector to 1, otherwise, setting it to 0; after processing all the intent features, the intent feature vector corresponding to the user interaction data is obtained;

[0038] Of course, the setting method is various, as long as the intention and the non-intention are distinguished;

[0039] Step S4A3: judging whether all dimensions of user interaction data are processed, if yes, entering next step, otherwise, returning to step S4A1; the number of final intention feature vectors is same as the dimension of user interaction data;

[0040] Step S4A4: finding the intention set corresponding to the intention feature vector and the intention of the user inherent attribute based on the contrast table to obtain the intention set corresponding to the intention feature vector; combining the intention sets corresponding to all intention feature vectors to obtain the final intention set; the final intention is one or more;

[0041] Alternatively: obtaining the superposition vector of all intention feature vectors, finding the contrast table based on the superposition vector to obtain the intention set corresponding to the intention feature vector as the final intention set;

[0042] Alternatively: the determination of the user intention is specifically: inputting the user interaction information and the user inherent attribute into the intention determination model to obtain the user intention; the intention determination model is a neural network model, the N-dimensional user interaction information and the 1-dimensional user attribute information are converted into an N+1-dimensional vector after normalization, and the N+1-dimensional vector is taken as the input of the neural network model; the intention determination model is trained in advance by using user historical data;

[0043] Preferably, the intention determination model is a deep neural network model;

[0044] The determination of the candidate intention scene based on the user intention is specifically: pre-setting the association relationship between the user intention and the candidate intention scene, and finding and determining the candidate intention scene based on the association relationship and the user intention; the candidate intention scene can be a scene that can meet the user intention; the user intention here is one or more, and when there are multiple, one-to-one determination and superposition are needed;

[0045] The application determines the scene set to be loaded based on multi-dimensional user interaction information, and realizes the rapid loading of the target scene through hierarchical and modular progressive loading for different use cases and predicted use cases, greatly reduces the occurrence of the pause feeling, and improves the interactive experience;

[0046] Step S5: determining the target scene based on the current scene and the candidate intention scene based on the scene transition graph; the target scene data is loaded in advance to the local client; the current scene is the scene where the current user is located;

[0047] The step S5 specifically includes the following steps:

[0048] Step S51: based on the scene transition graph, all paths pt from the current scene to the candidate intent scene are obtained; at this time, all the scenes corresponding to the nodes included in the path are taken as candidate target scenes;

[0049] Step S53: the transition probability Pr of each path pt is calculated pt ;

[0050] Pr pt =∑wl i,j , wherein: the edge l i,j ∈pt;

[0051] Step S53: the transition threshold TRPR of the transition probability is obtained;

[0052] Preferably, the transition threshold is a preset value; so that the size of the local client storage space and the transition probability are inversely related; the larger the local client storage space is, the smaller the transition probability threshold is, and vice versa, the smaller the local client storage space is, the larger the transition probability threshold is; of course, a reasonable transition threshold can also be directly set;

[0053] Step S54: all the scenes on the path whose transition probability Pr pt is greater than the transition threshold TRPR are determined as target scenes;

[0054] Step S55: the scene data of the target scene is loaded in advance to the local client;

[0055] The application circumscribes the target scene in a relatively global range through the current user's intent guess, and selects the scene most likely to be used subsequently for preloading according to the local loading capacity, thereby improving the subsequent data reading speed and avoiding the experience influence caused by network lag

[0056] Step S6: based on the historical operation records of the user on the scene module and the historical residence time of the target scene, all or part of the modules in the target scene are loaded into the VR device; wherein: each scene includes one or more scene modules, and the scene module includes a scene interaction module for providing interactive services for the user;

[0057] The step S6 specifically includes the following steps:

[0058] Step S61: the use of the scene module in the scene by the current user is obtained from the historical operation records of the current user on various types of scene modules;

[0059] Preferably, the use is a statistical value (ItN c ) of the interaction times of various types of scene modules; wherein: ItN cThis is a statistical value of the number of interactions for type C scene modules; scene modules can be classified from multiple perspectives, such as by interaction type, by interaction purpose, or by user experience.

[0060] Step S62: Compare the usage scenario with the scene module type of the target scenario to obtain the matching degree MD;

[0061] Preferred method: Distribute the number of interactions between the current user and various types of scene modules, and the distribution of the number of various types of scene modules in the target scene (TPN). c Compare the results to determine the degree of match, where: TPN c It is the number of type C scene modules in the target scene; the more consistent the distribution of the two, the higher the degree of matching, and vice versa.

[0062] Replaceable: Distribution of current user interactions with various types of scene modules and distribution of importance of various types of scene modules in the target scene (TPN). c Compare the results to determine the degree of match, where: TPN c It represents the importance of type C scene modules in the target scene; the more consistent the distribution of the two, the higher the matching degree, and vice versa.

[0063] Preferred: The matching degree MD is calculated based on the following formula;

[0064]

[0065] in: yes The mean of ε is the minimum value;

[0066] Step S63: Calculate the attention index (IMS) of the target scene based on the matching degree and the historical dwell time of the target scene; specifically: determine the attention index (IMS) of the target scene based on the historical dwell time (TS) and the matching degree (MD) of the target scene, so that the higher the historical dwell time and the matching degree, the higher the attention index, and vice versa.

[0067] IMS = α × TS × MD;

[0068] Where: α is the adjustment value, which is the preset value;

[0069] Step S64: selecting all or part of the scene modules in the target scene based on the attention degree IMS of the target scene and loading into the VR device; specifically: selecting all the scene modules in the target scene whose attention degree is greater than or equal to the upper limit attention degree into the VR device; selecting the necessary scene modules in the target scene whose attention degree is less than or equal to the lower limit attention degree into the VR device; for the target scene whose attention degree is between the upper limit attention degree and the lower limit attention degree, selecting part of the scene modules to load into the VR device;

[0070] Preferably: the upper limit attention degree is greater than the lower limit attention degree; the necessary scene is a scene module that must be loaded to meet the scene use;

[0071] Alternatively: for the target scene whose attention degree is between the upper limit attention degree and the lower limit attention degree, according to the amount of attention degree, a corresponding number of part of the scene modules are selected to load into the VR device, so that the scene module quantity loaded by the one with high attention degree is more, and the scene module quantity loaded by the one with low attention degree is less;

[0072] After the scene data is loaded into the local client, each path to the scene involves relatively smooth data loading of the scene, but it is still not enough for the interactive operation with higher corresponding reaction speed requirement; the present application enables the VR device to load more scene modules in line with the user's operation habits based on the residence time and the user's historical operation record, and the user can get good interactive reaction speed no matter which scene-related path he enters; that is, all or part of the modules in the target scene can be determined to be loaded into the VR device based on the predicted interactive features of the target scene; at this time, the scene with very short residence time does not involve or design few user operations, so the loading quality has little effect on the user experience; and the user experience in this case can also be improved by combining technologies such as loading summary scene data first;

[0073] Based on the same inventive concept, the present application also provides an interactive system based on VR digital content, which is used to implement the above-mentioned interactive method based on VR digital content;

[0074] The system is arranged on a cloud server and obtains the required user historical data and user operation record included in the user historical data from a big data server;

[0075] The system further comprises one or more local clients and one or more VR devices; each local client serves one or more VR devices; when providing interactive VR digital content, the local client can provide scene data to the VR device and manage and control the VR device;

[0076] Preferably: the control is near-distance and middle-distance communication control;

[0077] The terms "data server", "cloud server", "VR device" or "local client" encompass all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of the foregoing. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0078] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and networks.

[0079] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of computer-implemented processes. Embodiments of the present application can also be embodied as computer readable storage devices containing computer readable code, such as a floppy disk, a CD ROM, a RAM, a ROM, or a magnetic tape. Accordingly, embodiments of the present application can be embodied as a computer program product storing computer readable code, which, when executed by a computer, implements an embodiment of the present application.

[0080] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0081] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0082] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0083] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, but are not intended to limit the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.

Claims

1. A VR digital content-based interaction method, characterized in that, The method comprises: Step S1: constructing a scene transition graph based on user historical data; the scene transition graph includes one or more scene nodes and the edge formed by the transition connection relationship thereof; the edge l between the first node i and the second node j i,j The weight wl i,j of the edge l is equal to the transition probability of the corresponding scene from the first node to the second node. Step S2: obtaining current user identity information and user inherent attribute information; Step S3: collecting multi-dimensional user interaction information in real time; Step S4: determining a user intention and determining a candidate intention scene based on the user intention; the determination of the user intention comprises: obtaining an intention feature vector based on user interaction data; obtaining an intention feature vector corresponding to the user inherent attribute information; and searching a reference table based on the intention feature vector to obtain an intention corresponding to the intention feature vector; the reference table is a reference table of corresponding relationships between intention feature vectors and intentions; Step S5: determining a target scene based on a current scene and the candidate intention scene based on a scene transition graph; and loading target scene data to a local client in advance; The current scene is a scene in which the current user is located; The step S5 specifically comprises: Step S51: obtaining all paths pt from the current scene to the candidate intention scene based on the scene transition graph; at this time, all nodes included in the paths are taken as candidate target scenes; Step S53: Calculate the transition probability Pr of each path pt pt ; Pr pt =∑wl i,j , where: edge l i,j ∈pt; Step S53: obtaining a transition threshold TRPR of the transition probability; The size of the local client storage space and the transition probability are in inverse correlation; the larger the local client storage space is, the smaller the transition probability threshold is, and vice versa, the smaller the local client storage space is, the larger the transition probability threshold is; Step S54: determining the transition probability Pr pt all scenes on the path greater than the transition threshold TRPR are target scenes; Step S55: loading the scene data of the target scene to the local client in advance; Step S6: loading all or part of modules in the target scene to a VR device based on historical operation records of the user on the scene module and historical residence time of the target scene. 2.The VR digital content-based interaction method according to claim 1, wherein, The user interaction data comprises body action data, eye action data and / or touch interaction data. 3.The VR digital content-based interaction method of claim 2, wherein, Each intention has one or more intention features. 4.The VR digital content-based interaction method according to claim 3, wherein, There are multiple reference tables, and the reference table corresponding to each different user inherent attribute is different; the reference table is a pre-stored reference table. 5.The VR digital content-based interaction method according to claim 4, wherein, The scene data in the VR device, the local client and the remote server is sequentially selected based on the user interaction indication.

6. A VR digital content-based interactive system for implementing the method of any of the preceding claims 1-5, characterized by, The system is arranged on a cloud server and obtains required user historical data and user operation records included in the user historical data from a big data server; The system further comprises one or more local clients and one or more VR devices; each local client serves one or more VR devices; When the VR device provides interaction of VR digital content, the local client can provide scene data to the VR device and manage and control the VR device.

7. A processor, comprising: The processor is used to run a program, wherein the program, when running, executes the VR digital content-based interaction method in any one of claims 1-5.

8. An execution device, characterized by The processor is coupled with the memory, and the memory stores program instructions; when the program instructions stored in the memory are executed by the processor, the VR digital content-based interaction method in any one of claims 1-5 is implemented.

9. A computer-readable storage medium, characterized in that, The program, when running on a computer, causes the computer to execute the VR digital content-based interaction method in any one of claims 1-5.

10. A cloud server, characterized by, The cloud server is configured to perform the VR digital content-based interaction method as claimed in any one of claims 1-5. The cloud server is configured to perform the VR digital content-based interaction method as claimed in any one of claims 1-5.

Images