A VR training integrated analysis method and system

Abstract

The present invention relates to a VR training integrated analysis method, platform and system, the method comprising: dividing the VR field of view into one or more sub-ranges; obtaining the focus track of the current user constituted by the sub-range where the user focus is located during the progress of the training content; obtaining a predicted focus set and its corresponding focus score based on the focus track of the current user; and selectively performing server-side rendering on the VR training content within the VR field of view based on the focus score. The present invention combines VR big data information, fully utilizes server-side resources and local resources, constructs differentiated VR training content for different users, and greatly improves the efficiency of VR training.

Description

[Technical field]

[0001] The present invention belongs to the field of virtual reality technology, and in particular relates to a VR training integrated analysis method and system. [Background technology]

[0002] Virtual reality technology is a computer simulation system in which a virtual world can be created and experienced. Virtual reality technology uses computers to generate a simulated environment, using high-tech means with computer technology as the core to generate a realistic virtual environment that integrates vision, hearing, touch, etc. Users can interact with objects in virtual reality through display terminals. The simulated environment is an interactive three-dimensional dynamic vision and system simulation of entity behavior that integrates multi-source information, which can achieve an immersive experience. The multi-source information involved in virtual reality includes real-time three-dimensional computer graphics technology, wide-angle (wide field of view) stereo display technology, tracking technology for the observer's head, eyes and hands, as well as tactile / force feedback, stereo, network transmission, voice input and output technology, etc. In addition, binocular stereo vision is also involved, which plays a major role.

[0003] The development of the "VR+" industry has promoted the in-depth application of VR virtual reality technology in various industries such as cities, tourism, culture and education, enterprises, commerce, film and television, etc. At present, VR technology is more deeply applied in cultural education and professional skills training, involving a variety of professions and fields.

[0004] As VR technology develops more and more vigorously, it actually brings great challenges to the hardware resources of VR presentation devices. VR presentation devices need to carry more and more complex and large amounts of information, and also need to perform a lot of calculations to complete rendering. To make matters worse, since the knowledge itself is updated and iterated very quickly, the training content is also updated very quickly, which brings a lot of system overhead. Therefore, how to effectively and scientifically transfer the software and hardware resource pressure of VR presentation terminals to the server side while ensuring the quality of VR presentation is a technical problem to be solved.

[0005] Based on the above problems, the present invention combines big data information, fully utilizes server-side resources and local resources, and constructs differentiated VR training content for different users, which greatly improves the efficiency of VR training. [Summary of the invention]

[0006] In order to solve the above problems in the prior art, the present invention proposes a VR training integrated analysis method and system, the method comprising:

[0007] Step S1: dividing the VR field of view into one or more sub-ranges; correspondingly, the rendering data of the VR field of view is divided into rendering data corresponding to the one or more sub-ranges; and a unique identifier is set for each sub-range;

[0008] Step S2: During the progress of the training content, the focus track of the current user formed by the sub-range where the user focus is located is obtained; the sub-range where the user focus is located during the progress of the training content is arranged in the time sequence of the training progress to form the focus track of the current user; the focus track is divided into time slices, and the focus change in each time slice is represented by a focus matrix, and the element value of each element (i, j) in the focus matrix is ​​used to represent the proportion of the time length that the user focus stays in the sub-range (i, j) in the time slice; correspondingly, all focus matrices corresponding to the focus track constitute a focus matrix sequence;

[0009] Step S3: obtaining a predicted focus set and its corresponding focus score based on the focus trajectory of the current user; specifically: based on the focus trajectory of the current user and the historical focus change information about the training content, predicting the focus change of the current user and its occurrence probability / prediction accuracy, thereby obtaining a predicted focus set, and determining the corresponding focus score based on the occurrence probability / prediction accuracy;

[0010] The step S3 specifically includes the following steps:

[0011] Step S31: acquiring a set of sample focus trajectories corresponding to the training content based on the training content;

[0012] Step S32: Obtain the sample focus trajectory with the highest matching degree with the focus trajectory of the current user in the sample focus trajectory set as the sample focus trajectory to be selected; specifically: use the following formula (1) to calculate the matching degree M of the sth sample focus trajectory: s ; Where: s is the sample focus track number; t is the time slice number corresponding to the training progress, (i, j) is the sub-range number corresponding to the focus, and ct is the time slice corresponding to the training progress of the current user; fs.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the s-th sample focus trajectory; fu.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the focus trajectory of the current user;

[0013] M s =∑ t＝1～ct ∑ i，j |fs.t i，j -fu.t i，j | (1);

[0014] Step S33: determining a predicted focus set and its focus score based on the target focus trajectory;

[0015] Step S4: Based on the focus score, the VR training content within the VR field of view is selectively rendered on the server side, and the selected VR training content is loaded into the rendering device; based on the training progress, the corresponding complete VR training content is constructed, and the complete training content is presented to the user.

[0016] Furthermore, the sizes of the sub-ranges are the same or different.

[0017] Furthermore, the field of view is evenly divided into one or more sub-ranges using a two-dimensional grid, and in this case, the sizes of the sub-ranges are the same.

[0018] Furthermore, according to the presentation characteristics of the VR field of view, the setting of the training content, and / or the different presentation methods of the training content in the field of view, the field of view is divided to obtain one or more sub-ranges.

[0019] Furthermore, the time slice is a time interval of a preset length.

[0020] A VR training integrated analysis system includes: a server and a presentation device; the server is used to implement the VR training integrated analysis method.

[0021] Furthermore, the time slice is 60s.

[0022] A VR training integrated analysis platform includes a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the VR training integrated analysis method is implemented.

[0023] A computer-readable storage medium includes a program, which, when executed on a computer, enables the computer to execute the VR training integrated analysis method.

[0024] A cloud server, wherein the cloud server is configured to execute the VR training integrated analysis method.

[0025] The beneficial effects of the present invention include:

[0026] (1) Divide the VR field of view into one or more sub-ranges; support thin presentation devices to present complex VR scene data by predicting the sub-ranges; and build differentiated VR training content for different users by combining big data information and making full use of server-side resources and local resources, thus greatly improving the efficiency of VR training.

[0027] (2) The changes in the user's observation focus during the training process are separated by time slices and represented by a focus matrix. The predicted focus set and its corresponding focus score are obtained as accurately as possible through quantitative calculation, which supports the scientific selection of key sub-ranges and loads their server-side rendering data, thereby providing differentiated VR training content that best suits the current user characteristics with the least software and hardware resource support;

[0028] Furthermore, the inertial matching is corrected through a dynamic matching threshold, and the inertial target focus matrix is ​​adjusted to a certain degree by increasing the threshold to impose an inertial penalty, thereby improving the calculation accuracy.

Brief Description of the Drawings

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 This is a schematic diagram of the VR training integrated analysis method provided by the present invention. [Specific implementation method]

[0031] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0032] The present invention proposes a VR training integrated analysis method and system, as shown in the attached Figure 1 As shown, the method comprises the following steps:

[0033] Step S1: dividing the VR field of view into one or more sub-ranges; correspondingly, the rendering data of the VR field of view is divided into rendering data corresponding to the one or more sub-ranges; and a unique identifier is set for each sub-range;

[0034] Preferably, the sizes of the sub-ranges are the same or different. Case 1: the visual field is evenly divided into one or more sub-ranges using a two-dimensional grid, and the sizes of the sub-ranges are the same. Case 2: the visual field is divided into one or more sub-ranges according to the presentation characteristics of the VR visual field and the presentation method of the training content in the visual field, and the sizes of the sub-ranges are the same or different. That is, the division method is not even at this time. A unique number (i, j) is set for each sub-range according to the position of the sub-range in the visual field.

[0035] Preferably: the determination of i, j is related to its position within the field of view;

[0036] Preferred: Therefore, the field of view range and sub-range are three-dimensional and four-dimensional space ranges;

[0037] Step S2: During the progress of the training content, the focus track of the current user formed by the sub-range where the user focus is located is obtained; the sub-range where the user focus is located during the progress of the training content is arranged in the time sequence of the training progress to form the focus track of the current user; the focus track is separated by time slices, and the focus change in each time slice is represented by a focus matrix, and the element value of each element (i, j) in the focus matrix is ​​used to represent the proportion of the length of time that the user focus stays in the sub-range (i, j) in the time slice; correspondingly, all focus matrices corresponding to the focus track constitute a focus matrix sequence;

[0038] Preferably, the time slice is a time interval of a preset length, for example, 1 to 60 seconds;

[0039] Alternatively: sampling the user focus in a time slice to obtain a focus track consisting of the sub-range where the user focus is located; in this case, the focus sampled in a time slice may be one or more, so each element in the focus track is a tuple or a multi-tuple, and the element in the tuple is a sub-range identifier; further: continuously sampling the user focus multiple times in a time slice to obtain multiple user focuses corresponding to a time slice, and using the multi-tuple consisting of the corresponding multiple sub-ranges as an element in the focus track;

[0040] Step S3: obtaining a predicted focus set and its corresponding focus score based on the focus trajectory of the current user; specifically: based on the focus trajectory of the current user and the historical focus change information about the training content, predicting the focus change of the current user and its occurrence probability / prediction accuracy, thereby obtaining a predicted focus set, and determining the corresponding focus score based on the occurrence probability / prediction accuracy;

[0041] Alternatively, the method of obtaining a predicted focus set and its corresponding focus score based on the focus trajectory is as follows: inputting the focus trajectory of the current user into a focus prediction model to determine a predicted focus set and its corresponding focus score corresponding to the user's focus; wherein: the focus prediction model is an artificial intelligence model; obtaining sample focus trajectories corresponding to training content in advance, and using the sample focus trajectories to train the focus prediction model; the output of the focus prediction model is a predicted focus set and its corresponding focus score;

[0042] The step S3 specifically includes the following steps:

[0043] Step S31: based on the training content, a set of sample focus trajectories corresponding to the training content is obtained; specifically, a set of sample focus trajectories corresponding to each training content is constructed in advance; when the sample users participate in the training for the training content, information on changes in their focus as the training progresses is collected, and representative change information is determined as sample focus change information; the sample focus change information is converted into a focus trajectory representation to obtain a sample focus trajectory; the sample focus trajectories corresponding to a plurality of sample focus change information constitute a set of sample focus trajectories;

[0044] Preferably: clustering is performed by taking the gap or distance between the information on the change of the focus as the training progresses as a clustering optimization target to obtain representative change information;

[0045] Step S32: Obtain the sample focus trajectory with the highest matching degree with the current user's focus trajectory in the sample focus trajectory set as the sample focus trajectory to be selected; specifically: use the following formula (1) to calculate the matching degree M of the sth sample focus trajectory: s ; Where: s is the sample focus track number; t is the time slice number corresponding to the training progress, for example: t is the specific time slice number; (i, j) is the focus corresponding sub-range number, that is, the element position number in the corresponding focus matrix; ct is the training progress of the current user; fs.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the s-th sample focus trajectory; fu.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the focus trajectory of the current user;

[0046] M s =∑ t＝1～ct ∑ i，j |fs.t i，j -fu.t i，j | (1);

[0047] Step S33: If the highest matching degree is greater than the matching degree threshold, the selected sample focus trajectory is used as the target focus trajectory; otherwise, the average focus trajectory is used as the target focus trajectory TG, which is expressed as TG = <[tg.t i，j ]>; Wherein: the element value of each focus matrix in the average focus trajectory is the element mean of the elements at the corresponding position in all focus matrices of the same degree of progress in the collected sample focus trajectories;

[0048] Preferably: the matching degree threshold is a static threshold, which is a preset value;

[0049] Alternatively: the matching threshold is a dynamic threshold; and the matching threshold Tr is calculated based on the following formulas (2)-(4); wherein: cl is a unit adjustment value; Mct-1 It is the degree of match between the focus matrix corresponding to the ct-1th training progress in the target focus track and the focus matrix corresponding to the ct-1th training progress in the current user's focus track; that is, the determination of the target focus track is based on a certain degree of inertia, and the inertia can be corrected by setting a dynamic threshold, and the inertia target focus matrix is ​​adjusted to a certain degree of threshold increase to impose an inertia penalty;

[0050] Tr = Tr × (1 + cl × jg) (2);

[0051]

[0052] M ct-1 =∑ i，j |tg.t i，j -fu.t i，j |, t = ct-1 (4);

[0053] Preferably: cl = 0.1 × Tr0; Tr0 is the initial value of the dynamic threshold, which can be set to be equal to the preset static threshold;

[0054] Step S34: determining a predicted focus set and its focus score based on the target focus trajectory; specifically: determining the predicted focus set and its corresponding focus score by using the element value in the focus matrix corresponding to the ct+1th training progress in the target focus trajectory; placing the sub-range corresponding to the element value exceeding the element mean in the focus matrix corresponding to the ct+1th training progress into the predicted focus set, and using the corresponding element value as the focus score;

[0055] Alternatively: put the sub-range corresponding to the N elements with the largest element values ​​in the focus matrix corresponding to the ct+1th training progress into the predicted focus set, and use the corresponding element values ​​as the focus score;

[0056] Alternatively, the step S34 specifically includes the following steps:

[0057] Step S341: the part of the target focus trajectory starting from the ct+1th training progress is called the sub-target focus trajectory;

[0058] Step S342: Obtain N1 focus matrices starting from the head of the sub-target focus track [sf.t i，j ], t=1~N1; calculate the focus scores of all sub-ranges (i, j) corresponding to N1 focus matrices according to the following formula (5), and select N2 sub-ranges with the highest focus scores to form a predicted focus set;

[0059]

[0060] That is, as the training progresses further, its corresponding focus score will decrease;

[0061] Preferably: N1 and N2 are both preset values, for example: set N1 = N2 = the average number of sub-ranges preloaded each time;

[0062] Step S4: selectively performing server-side rendering on VR training content within the VR field of view based on the focus score, and loading the selected VR training content into a rendering device; constructing corresponding complete VR training content based on the training progress, and presenting the complete training content to the user;

[0063] The selective server-side rendering is performed, and the selected VR training content is loaded into the rendering device; specifically, the size of the available preloaded space in the rendering device is obtained, and N3 sub-ranges are selected from the predicted focus set in the order of focus scores from large to small, wherein the rendering data of the N3 sub-ranges have not been preloaded, and the sum of the rendering data of the N3 sub-ranges is closest to the available preloaded space size; the server-side rendering is performed to obtain the rendering data of the N3 sub-ranges; and the rendering data of the N3 sub-ranges is preloaded into the rendering device;

[0064] Preferably: the server-side rendering is high-precision rendering;

[0065] Preferably: the presentation device is a VR presentation device;

[0066] Alternatively, the presentation device is a VR training device, such as a wearable device or a surround presentation device;

[0067] Preferably: when performing server-side rendering, the virtual object model of each virtual object created in the VR scene is rendered with high precision using rendering materials, the corresponding sub-range rendering data in the rendering data is obtained for loading, and the sub-range rendering data is loaded into the rendering device;

[0068] Replaceable: creating a virtual framework model corresponding to each sub-range, using rendering materials to render the sub-range that needs to be preloaded with high precision, and loading the rendering data of the sub-range into the rendering device;

[0069] The corresponding complete VR training content is constructed based on the training progress, and the complete training content is presented to the user; specifically: during the training progress, if the VR field of view involves a sub-range of server-side rendering that has not been preloaded, local rendering is performed locally on the rendering device, the sub-range is rendered in real time, and the rendering data corresponding to the sub-range is loaded; and the VR training field of view corresponding to the training progress is constructed based on the rendering data of the sub-range rendered by the server and the rendering data corresponding to the sub-range rendered locally; wherein: the server rendering is high-precision rendering, while the rendering device is low-precision rendering;

[0070] Based on the same inventive concept, the present invention also provides a VR training integrated analysis system, the system comprising: a server and a presentation device; the server is used to implement the above-mentioned VR training integrated analysis method; the presentation device is used to present VR training content within the VR field of view according to the training content;

[0071] Preferably: the server is a cloud server;

[0072] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of a programming language, including an assembled or interpreted language, a declarative or procedural language, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A VR training integrated analysis method, characterized in that: The method comprises: Step S1: dividing the VR field of view into one or more sub-ranges; correspondingly, the rendering data of the VR field of view is divided into rendering data corresponding to the one or more sub-ranges; Set a unique identifier for each sub-range; According to the presentation characteristics of the VR field of view and the presentation method of the training content in the field of view, the field of view is divided to obtain one or more sub-ranges; Step S2: during the progress of the training content, the focus track of the current user formed by the sub-range where the user focus is located is obtained; the sub-range where the user focus is located during the progress of the training content is arranged in the time sequence of the training progress to form the focus track of the current user; the focus track is divided into time slices, and the focus change in each time slice is represented by a focus matrix, and the element value of each element (i, j) in the focus matrix is ​​used to represent the proportion of the length of time that the user focus stays in the sub-range (i, j) in the time slice; Correspondingly, all focus matrices corresponding to the focus trajectory constitute a focus matrix sequence; Step S3: obtaining a predicted focus set and its corresponding focus score based on the focus trajectory of the current user; specifically: based on the focus trajectory of the current user and the historical focus change information about the training content, predicting the focus change of the current user and its occurrence probability / prediction accuracy, thereby obtaining a predicted focus set, and determining the corresponding focus score based on the occurrence probability / prediction accuracy; The step S3 specifically includes the following steps: Step S31: acquiring a set of sample focus trajectories corresponding to the training content based on the training content; Step S32: Obtain the sample focus trajectory with the highest matching degree with the current user's focus trajectory in the sample focus trajectory set as the sample focus trajectory to be selected; specifically: use the following formula (1) to calculate the matching degree M of the sth sample focus trajectory: s ; Where: s is the sample focus track number; t is the time slice number corresponding to the training progress, (i, j) is the sub-range number corresponding to the focus, and ct is the time slice corresponding to the training progress of the current user; fs.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the s-th sample focus trajectory; fu.t i，j is the element (i, j) in the focus matrix corresponding to the t-th training progress in the focus matrix sequence corresponding to the focus trajectory of the current user; M s =∑ t＝1～ct ∑ i，j |fs.t i，j -fu.t i，j | (1); Step S33: If the highest matching degree is greater than the matching degree threshold, the selected sample focus trajectory is used as the target focus trajectory; otherwise, the average focus trajectory is used as the target focus trajectory TG, which is expressed as TG = <[tg.t i，j ]>; Wherein: the element value of each focus matrix in the average focus trajectory is the element mean of the elements at the corresponding position in all focus matrices of the same degree of progress in the collected sample focus trajectories; The matching threshold is a dynamic threshold; and the matching threshold Tr is calculated based on the following formulas (2)-(4); where: cl is the unit adjustment value; M ct-1 is the degree of match between the focus matrix corresponding to the ct-1th training progress in the target focus trajectory and the focus matrix corresponding to the ct-1th training progress in the current user's focus trajectory; Tr = Tr × (1 + cl × jg) (2); (3)； M ct-1 =∑ i，j |tg.t i，j -fu.t i，j |,t=ct-1 (4); Step S34: determining a predicted focus set and its focus score based on the target focus trajectory; Step S4: Based on the focus score, the VR training content within the VR field of view is selectively rendered on the server side, and the selected VR training content is loaded into the rendering device; based on the training progress, the corresponding complete VR training content is constructed, and the complete training content is presented to the user.

2. The VR training integrated analysis method according to claim 1, characterized in that: The sizes of the sub-ranges may be the same or different.

3. The VR training integrated analysis method according to claim 2, characterized in that: The field of view is evenly divided into one or more sub-ranges using a two-dimensional grid. In this case, the sizes of the sub-ranges are the same.

4. The VR training integrated analysis method according to claim 3 is characterized in that: According to the presentation characteristics of the VR field of view, the setting of the training content, and / or the different presentation methods of the training content in the field of view, the field of view is divided to obtain one or more sub-ranges.

5. The VR training integrated analysis method according to claim 4 is characterized in that: The time slice is a time interval of a preset length.

6. A VR training integrated analysis system, characterized in that: include: Servers and rendering equipment; The server is used to implement the VR training integrated analysis method described in any one of claims 1 to 5.

7. The VR training integrated analysis system according to claim 6, characterized in that: The time slice is 60s.

8. A VR training integrated analysis platform, characterized in that: The invention comprises a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the VR training integrated analysis method described in any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that: The invention comprises a program which, when executed on a computer, enables the computer to execute the VR training integrated analysis method as described in any one of claims 1 to 5.

10. A cloud server, characterized in that: The cloud server is configured to execute the VR training integrated analysis method as described in any one of claims 1-5.

Images