Method and system for course production and reproduction based on virtual reality

Abstract

A method and system for curriculum production and reproduction based on virtual reality, applied to an electronic device. The method includes: receiving a 3D model; generating a model data packet corresponding to the 3D model, where the model data packet at least includes a plurality of objects applied to the 3D model; recording, by a plurality of virtual cameras, the actions of a user operating the objects in a virtual reality and generating an action video corresponding to the objects; and generating a curriculum data packet, where the curriculum data packet includes the model data packet and an animation packet including the action video.

Description

Technical Field

[0001] The present disclosure relates to a method and system for course production and reproduction, and particularly to a method and system for course production and reproduction based on virtual reality. Background Art

[0002] In the emerging multimedia teaching mode, VR teaching material courses and VR content editors developed with virtual reality (VR) technology can present an immersive simulated real working environment as if on the scene, and have been applied to operations of large-scale machines, industrial safety event drills, and simulation of human physiological phenomena. However, the production threshold of this type of teaching material courses is high, and professional inputs from engineers, 2D / 3D artists, and instructional designers are required to perform tasks such as script design, 3D model establishment, and VR simulation interaction setting. Moreover, the functions of VR content editors on the market are complex, time-consuming, and even require basic programming concepts, making it impossible for instructors who simply want to conduct technical operation training to design and produce teaching materials efficiently. In addition, there are also some solutions for recording VR teaching materials by using the method of recording the teaching process as a video for students to watch. However, the video only provides one perspective for students to watch, and students cannot watch the teaching process from various perspectives, resulting in students possibly being unable to understand the teaching process.

[0003] Therefore, a method and system for course production and reproduction based on virtual reality are needed to improve the above problems. Summary of the Invention

[0004] The content disclosed below is only exemplary and is not intended to be limiting in any way. In addition to the aspects, embodiments, and features described, other aspects, embodiments, and features will also be apparent by referring to the accompanying drawings and the following specific embodiments. That is, the content disclosed below is provided to introduce concepts, highlights, benefits, and novel and non-obvious technical advantages described herein. The selected, not all, embodiments will be described in further detail below. Therefore, the content disclosed below is not intended to be a necessary feature of the claimed subject matter, nor is it intended to be used in determining the scope of the claimed subject matter.

[0005] Therefore, the main objective of the present disclosure is to provide a method and system for course production and reproduction based on virtual reality to improve the above disadvantages.

[0006] The present disclosure provides a method for curriculum production and reproduction based on virtual reality, which is applied to an electronic device. The method includes: receiving a 3D model; generating a model data packet corresponding to the 3D model, where the model data packet at least includes a plurality of objects applied to the 3D model; recording, by a plurality of virtual cameras, the actions of a user operating the objects in a virtual reality and generating an action video corresponding to the objects; and generating a curriculum data packet, where the curriculum data packet includes the model data packet and an animation packet including the action video.

[0007] In some embodiments, the plurality of objects are classified into pure collision objects, base objects, extraction objects, physical objects, penetrative objects, and pseudo-physical objects.

[0008] In some embodiments, the virtual cameras record the coordinate rotation information and displacement information of the objects through a time difference.

[0009] In some embodiments, after recording the actions of the user operating the objects in the virtual reality, the method further includes: determining whether the objects move; when the objects move, marking the action video with a timestamp; and storing the action video marked with the timestamp.

[0010] In some embodiments, the method further includes: loading the curriculum data packet; and playing the animation packet included in the curriculum data packet to reproduce the actions of the user operating the objects in the virtual reality.

[0011] In some embodiments, the step of playing the animation packet included in the curriculum data packet further includes: putting the animation packet including the action video into an object pool; determining whether an action frame in the action video has the same first object as a to-be-played action frame; and when the action frame has the same first object as the to-be-played action frame, retaining the first object of the action frame in the object pool.

[0012] In some embodiments, the method further includes: establishing the virtual reality through a Unity3D engine.

[0013] In some embodiments, the curriculum data packet is an Assetbundle file.

[0014] In some embodiments, the animation packet at least includes an audio file, a preloaded object and animation pairing file, animation information corresponding to the objects, and an animation packet screenshot.

[0015] In some embodiments, the above-mentioned electronic device is a head-mounted display (HMD) device.

[0016] The present disclosure also proposes a system for virtual reality-based course production and reproduction, including: one or more processors and one or more computer storage media storing computer-readable instructions, wherein the above-mentioned processors use the above-mentioned computer storage media to execute: receiving a 3D model; generating a model data packet corresponding to the 3D model according to the 3D model, wherein the model data packet at least includes a plurality of objects applied to the 3D model; recording the actions of a user operating the objects in a virtual reality through a plurality of virtual cameras and generating an action video corresponding to the objects; and generating a course data packet, wherein the course data packet includes the model data packet and an animation packet including the action video. Description of the Drawings

[0017] Figure 1A It is a schematic diagram showing an instructor wearing a head-mounted display (HMD) device to operate virtual reality teaching materials in a virtual reality according to an embodiment of the present disclosure.

[0018] Figure 1B It is a schematic diagram showing a learner wearing an HMD device to view virtual reality teaching materials in a virtual reality according to an embodiment of the present disclosure.

[0019] Figure 2 It is a schematic diagram showing the formulation of a base object according to an embodiment of the present disclosure.

[0020] Figure 3 It is a schematic diagram showing the formulation of a collision object according to an embodiment of the present disclosure.

[0021] Figure 4 It is a schematic diagram showing the formulation of a cluster object according to an embodiment of the present disclosure.

[0022] Figure 5 It is a flowchart showing a method for virtual reality-based course production and reproduction according to an embodiment of the present disclosure.

[0023] Figure 6 It is a schematic diagram showing a unified modeling language (UML) of a virtual camera recording object movement according to an embodiment of the present disclosure.

[0024] Figure 7 It is a schematic diagram showing an electronic device marking the above-mentioned action video with time stamps according to an embodiment of the present disclosure.

[0025] Figure 8 It is a schematic diagram showing an electronic device playing an action movie according to a timestamp of an embodiment of the present disclosure.

[0026] Figures 9A to 9D are schematic diagrams showing six different types of objects according to an embodiment of the present disclosure.

[0027] Figure 10 It is a schematic diagram showing the playing of a course data packet according to an embodiment of the present disclosure.

[0028] Figure 11 It is a detailed flowchart showing a user creating a course data packet according to an embodiment of the present disclosure.

[0029] Figure 12 It is a detailed flowchart showing a user reproducing a course data packet according to an embodiment of the present disclosure.

[0030] Figure 13 It is a schematic diagram showing an exemplary operating environment for implementing an embodiment of the present disclosure.

[0031] Description of reference numerals:

[0032] 110: Instructor

[0033] 120: HMD device

[0034] 130: Learner

[0035] 200: Machine

[0036] 210: Cabinet

[0037] 500: Method flowchart

[0038] S505, S510, S515, S520: Steps

[0039] 910: Environment

[0040] 920: Cabinet

[0041] 930: Assembled object

[0042] 940: VR handle

[0043] 950: Marked object

[0044] 960: Hand tool

[0045] 1100: Flowchart

[0046] S1105, S1110, S1115, S1120, S1125, S1130, S1135, S1140, S1145: Steps

[0047] 1200: Flowchart

[0048] S1205, S1210, S1215, S1220, S1225, S1230: Steps

[0049] 1300: Computing device

[0050] 1310: Bus

[0051] 1312: Memory

[0052] 1314: Processor

[0053] 1316: Display element

[0054] 1318: I / O port

[0055] 1320: I / O element

[0056] 1322: Power supply Detailed implementation manners

[0057] In the following, various aspects of the present disclosure will be more fully described with reference to the accompanying drawings. However, the present disclosure can be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout the present disclosure. On the contrary, providing these aspects will make the present disclosure thorough and complete, and the present disclosure will fully convey the scope of the present disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art should realize that the scope of the present disclosure, whether implemented alone or in combination with any other aspect of the present disclosure, is intended to cover any aspect disclosed herein. For example, any number of the devices or methods proposed herein can be used to implement it. In addition, in addition to the multiple aspects of the present disclosure proposed herein, the scope of the present disclosure is more intended to cover devices or methods implemented using other structures, functions, or a combination of structures and functions. It should be understood that any aspect disclosed herein can be embodied by one or more elements of the claims.

[0058] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any aspect of the present disclosure or a design described as "exemplary" herein is not necessarily to be construed as preferred or superior to other aspects of the present disclosure or design. In addition, the same numerals indicate the same elements in all several figures, and unless otherwise specified in the description, the articles "a" and "the" include plural references.

[0059] It will be appreciated that when an element is referred to as being "connected" or "coupled" to another element, the element can be directly connected to or coupled to the other element or there may be intervening elements. Conversely, when the element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" and "directly between", "adjacent" and "directly adjacent", etc.).

[0060] Figure 1A FIG. is a schematic diagram showing a teacher 110 wearing a head-mounted display (HMD) device 120 to operate virtual reality teaching materials in a virtual reality according to an embodiment of the present disclosure.

[0061] In Figure 1A , the teacher 110 can convert real hand operation actions into virtual 3D images through the method and system for virtual reality-based course production and reproduction proposed by the present disclosure, so that the teacher 110 can record actions while operating in the virtual reality. The teacher 110 can produce virtual reality courses through the HMD device 120. The HMD device 120 can transmit data to a computing device (not shown in the figure) through a wired network or wireless communication or receive data from the computing device. The HMD device 120 and the computing device can respectively include a processor and a memory that can store programs.

[0062] The type range of the computing device is, for example, from small handheld devices (e.g., mobile phones / portable computers) to large mainframe systems (e.g., mainframe computers). Examples of portable computers include devices such as personal digital assistants (PDAs), laptop computers, etc. In another embodiment, the computing device can be used with other components, systems, subsystems, and / or devices other than those described herein. For example, the computing device can be integrated with the HMD device 120 into one device.

[0063] Figure 1B FIG. is a schematic diagram showing a learner 130 wearing an HMD device 120 to view virtual reality teaching materials in a virtual reality according to an embodiment of the present disclosure. The learner 130 can first receive the virtual reality teaching materials recorded by the teacher 110 through the HMD device 120, and enter the virtual reality through the method and device for virtual reality-based course production and reproduction proposed by the present disclosure to synchronously or asynchronously view the virtual 3D image operation actions of the teacher 110. In addition, the learner 130 can also freely explore from multiple angles or follow the operation steps of the teacher 110 to perform operations together.

[0064] It should be understood that Figure 1A and 1BEach component shown in the HMD device 120 and the computing device 130 in [reference] can be implemented via any type of computing device, such as the computing device 1300 described in [reference], as shown in [reference]. Figure 13 as described in [reference] Figure 13 as shown in [reference].

[0065] In order for users to directly start course production operations when entering the virtual reality, it is necessary to simulate the physical characteristics of the real world as much as possible in the virtual reality to prevent users from being confused due to some physical characteristics that violate the real world, resulting in a decline in the user's willingness to operate. The method and device for course production and reproduction based on virtual reality provided by the present disclosure design six types of object characteristics to enable users to feel the experience that conforms to the real world during operation, even if the objects do not fully conform to the real physical properties. The following is an example of assembling a 3D model, where the above 3D model is a machine.

[0066] To simulate the physical effects when assembling a machine in the virtual reality, the present disclosure uses the built-in 3D physics in the Unity3D engine. This Unity3D engine mainly integrates the open-source Nvidia PhysX physics engine to assist various simulations during development, such as rigid bodies, collisions, joint simulations, and character controllers. In the present disclosure, the objects that make up the 3D model will be divided into six different types of objects, namely pure collision objects, base objects, extraction objects, physical objects, penetrative objects, and pseudo-physical objects. Each type of object has different physical effects, as shown in Table 1.

[0067]

[0068] Table 1

[0069] And the effects after each type of object comes into contact with each other are shown in Table 2.

[0070]

[0071]

[0072]

[0073] Table 2

[0074] In order to construct a correct and realistic assembly simulation in virtual reality, in addition to adjusting the built-in parameters, the conditions for defining the assembly base must also be met. When objects are assembled, it often happens that the object to be assembled has been placed at the target position, but coordinate offsets or rotations are caused by collisions with other objects, thereby affecting the assembly result. Therefore, the present disclosure designs the conditions for the base object so that when a user assembles objects in virtual reality, an object must be used as the assembly base, and any object in contact with the base inherits the characteristics of the base object and enters the fixed mode. Figure 2 FIG. 179 is a schematic diagram showing the formulation of the base object according to an embodiment of the present disclosure. As shown in the figure, the user can set the cabinet part 210 as the base for assembling the entire machine platform 200. When the robotic arm, the electric lock piece, and the electric lock module come into contact with the base, they are fixed.

[0075] Figure 3 FIG. 180 is a schematic diagram showing the formulation of the collision object according to an embodiment of the present disclosure. In order to simulate the real collision effect of objects in virtual reality and save the energy consumption of the operation in the Unity3D engine, when the user assembles each object, the user can define the collision range of each object through the physical collider in the Unity3D engine to achieve a real virtual reality.

[0076] Figure 4 FIG. 181 is a schematic diagram showing the formulation of the cluster object according to an embodiment of the present disclosure. To meet the assembly requirements of the user's actual assembly, during the assembly process, the user can perform actions such as assembling objects and disassembling objects according to their own needs. The user can determine in real time which object has the function of pulling another object according to the mass or volume of the two objects. During the assembly process of the two objects, the user can use the Fixed Joint function in the Unity3D engine to move one object and assemble it on top of the other object. More specifically, the assembly process of the two objects is similar to forming the two objects into a parent-child relationship. According to the characteristics of the parent-child objects, the user can easily disassemble and reassemble the objects.

[0077] Figure 5 FIG. 182 is a flowchart 500 of a method for course production and reproduction based on virtual reality according to an embodiment of the present disclosure. This method can be executed in an electronic device such as the HMD device 120 or the computing device 130 shown in FIG. 1.

[0078] In step S505, the electronic device receives a 3D model. In step S510, the electronic device generates a model data packet corresponding to the 3D model according to the 3D model, where the model data packet at least includes a plurality of objects applied to the 3D model. In an embodiment, the plurality of objects are classified as pure collision objects, base objects, extraction objects, physical objects, penetrative objects, and pseudo-physical objects.

[0079] Next, in step S515, the electronic device records an action of a user operating the above object in a virtual reality through a plurality of virtual cameras, and generates an action video corresponding to the above object, wherein the above virtual cameras record the coordinate rotation information and displacement information of the above object through a time difference. In one embodiment, the above time difference is 10 milliseconds (ms). In another embodiment, the user can mount corresponding virtual cameras on each object to achieve the purpose of recording the movement of each object. Figure 6 FIG. is a schematic diagram of a Unified Modeling Language (UML) for a virtual camera to record object movement according to an embodiment of the present disclosure.

[0080] In step S520, the electronic device generates a course data packet, wherein the above course data packet includes the above model data packet and an animation packet including the above action video.

[0081] More specifically, the course data packet generated by the electronic device contains all the multimedia files required in the course, such as pictures, 3D models, audio files, etc. Each file is classified according to type and attribute for easy maintenance. The course data packet is shown in Table 2

[0082]

[0083]

[0084] Table 3

[0085] In one embodiment, the action video in the above course data packet is an Assetbundle file. More specifically, the action video and 3D model generated in step S515 will be imported into the Asset of the Unity3D engine, so that the action video and 3D model are packaged into the Assetbundle format. The Assetbundle function can translate various 3D file formats (such as.FBX,.obj,.gltf,.stl, etc.) to achieve the effect of saving the model loading space.

[0086] The animation packet included in the course data packet at least includes an audio file (.wav), a preloaded object and animation pairing file (.json), animation information corresponding to the above object (.pbani), and an animation packet screenshot (.jpg). The detailed content of the animation packet is shown in Table 3.

[0087]

[0088]

[0089] Table 4

[0090] In one embodiment, after step S515, the electronic device may further determine whether the above object moves. When the above object moves, the electronic device marks the above action video with a timestamp, and stores only the above action video marked with the above timestamp. Figure 7 FIG. is a schematic diagram showing that an electronic device marks the above action video with a timestamp according to an embodiment of the present disclosure. Assume that this 3D model has five objects, namely objects 1 to 5. The virtual camera that records the movement of objects 1 to 5 can record the coordinate rotation information and displacement information of objects 1 to 5 through a time difference (10 ms). As shown in the figure, the electronic device can also determine whether objects 1 to 5 move. When objects 1 to 5 move, the electronic device marks the above action video with a timestamp (mark ⊕), and records the coordinate rotation information and displacement information of objects 1 to 5 at this timestamp. For example, the electronic device determines that object 1 moves at time points 00 ms, 20 ms, 30 ms, 50 ms, 80 ms, and 100 ms respectively. The electronic device adds timestamps to the action video of object 1 at the above time points respectively, and stores only the above partial action video with timestamps. On the contrary, since object 1 does not move at other time points, the electronic device will not store the action video without a timestamp, so as to avoid continuous increase in file size and achieve optimization of reducing file size and energy consumption.

[0091] After step S520, the electronic device may upload the course data packet to a server for other learners to download. Other learners can download the course data packet from the server, load the above course data packet through the electronic device, and play the above animation packet included in the above course data packet to reproduce the above actions of the instructor operating the above object in the above virtual reality.

[0092] In one embodiment, before playing the animation packet, the electronic device may first put the above animation packet including the above action video into an object pool. The electronic device may then determine whether an action frame in the above action video has the same first object as a to-be-played action frame. When the above action frame has the same above first object as the above to-be-played action frame, the above first object of the above action frame is retained in the above object pool to achieve a low-latency and high-speed playback mode.

[0093] Figure 8 FIG. is a schematic diagram showing that an electronic device plays an action video with a timestamp according to an embodiment of the present disclosure. Similar Figure 7, assume that this 3D model has five objects, namely Object 1 to Object 5. The electronic device can directly play the coordinate rotation information and displacement information of Object 1 to Object 5 at the corresponding timestamps (mark ◎). For example, the timestamps of the action video of Object 2 are located at time points 00ms, 10ms, and 90ms respectively. When the time of the timer (Global Timer) of the electronic device is at time points 00ms, 10ms, and 90ms, the electronic device will play the coordinate rotation information and displacement information of Object 2 at the corresponding timestamps (mark ◎).

[0094] Figures 9A - 9D is a schematic diagram showing six different types of objects according to an embodiment of the present disclosure. FIGS. 9A to 9D are described by taking the assembly of a machine as an example. As Figure 9A shown, the electronic device will generate an environment 910 (pure collision object), a cabinet 920 (base object), and an assembled object 930 (physical object). As Figure 9B shown, the electronic device will generate a VR handle 940 (extracted object). As Figure 9C shown, the electronic device will generate a marker object 950 (penetrating object). As Figure 9D shown, the electronic device will generate a hand tool 960 (quasi-physical object). The user records the action video of operating the above objects in the virtual reality.

[0095] Figure 10 is a schematic diagram showing the playback of a course data packet according to an embodiment of the present disclosure. As shown in the figure, the blue part shows the movement of the objects in the course data packet played by the user. The electronic device can also perform playback functions such as fast-forwarding or rewinding the action video.

[0096] Figure 11 is a detailed flowchart 1100 showing the user creating a course data packet according to an embodiment of the present disclosure. In step S1105, the user wears the HMD device and enters the virtual reality for editing. In step S1110, the user receives a 3D model through the HMD device. In step S1115, the HMD device generates a model data packet corresponding to the above 3D model according to the above 3D model, where the above model data packet at least includes a plurality of objects applied to the above 3D model (in this step, the virtual camera will be placed on each object).

[0097] Next, in step S1120, the HMD device detects whether the user presses the recording key. When the HMD device detects that the user presses the recording key (Yes in step S1120), in step S1125, the virtual camera records the coordinate rotation information and displacement information of the above object through a time difference. When the HMD device does not detect that the user presses the recording key (No in step S1120), it returns to step S1120, and the HMD device continues to detect whether the user presses the recording key.

[0098] Next, in step S1130, the HMD device detects whether the user presses the stop recording key. When the HMD device detects that the user presses the stop recording key (Yes in step S1130), in step S1135, the virtual camera stops recording and generates an animated video. When the HMD device does not detect that the user presses the stop recording key (No in step S1130), it returns to step S1130, and the HMD device continues to detect whether the user presses the stop recording key.

[0099] In step S1140, the HMD device combines the model data packet and the animation packet into a course data packet. In step S1145, the HMD device uploads the course data packet to a server for storage.

[0100] Figure 12 FIG. 1200 is a detailed flowchart showing a user reproducing a course data packet according to an embodiment of the present disclosure. In step S1205, the user wears the HMD device and watches in virtual reality. In step S1210, the user downloads a course data packet from a server through the HMD device. In step S1215, the HMD device generates a 3D model in the above course data packet, where the above course data packet further includes a plurality of objects applied to the above 3D model and an action video corresponding to the above objects (in this step, a virtual player will be placed on each object).

[0101] Next, in step S1220, the HMD device detects whether the user presses the play key. When the HMD device detects that the user presses the play key (Yes in step S1220), in step S1225, the virtual player plays the animation. When the HMD device does not detect that the user presses the play key (No in step S1220), it returns to step S1220, and the HMD device continues to detect whether the user presses the play key.

[0102] Next, in step S1230, the HMD device detects whether the user presses the stop playback key or whether the animation has finished playing. When the HMD device detects that the user presses the stop playback key or the animation has finished playing (the "yes" in step S1230), the process ends. When the HMD device detects that the user does not press the stop playback key or the animation has not finished playing (the "no" in step S1230), the HMD device continues to detect whether the user presses the stop playback key or whether the animation has finished playing.

[0103] Through the method and device for virtual reality-based course production and reproduction of the present disclosure, the body and hand movements of the instructor can be captured. And the learner can see the viewing angle of the instructor looking at the object and the key hand movements when the video is replayed. The present disclosure not only records the step sequence of operating the object, but also records the movement trajectory of the object being operated. Therefore, the learner can see the movements of the instructor and the object movement trajectory in virtual reality. The learner can freely explore and learn from various angles, or watch and do at the same time. The learner can also freely choose to watch certain steps, improving the learning efficiency of the learner.

[0104] In addition, since the course data packet of the present disclosure is an Assetbundle file, all files in the course data packet can be read on multiple platforms. Furthermore, because the present disclosure uses the Unity3D engine, the electronic device only needs to load various virtual reality development toolkits (SDKs) during the process of making the course data packet, such as: AR Foundation, Vuforia, AR Kit, ARCore, etc., to present various virtual reality effects, such as: World Tracking, Plane Detection, Object Tracking, Image Tracking, etc.

[0105] For the embodiments of the present invention that have been described, the following describes an exemplary operating environment in which the embodiments of the present invention can be implemented. Specifically referring to Figure 13 , Figure 13 is a diagram showing an exemplary operating environment for implementing the embodiments of the present invention, which can generally be regarded as a computing device 1300. The computing device 1300 is only an example of a suitable computing environment and is not intended to imply any limitation on the use or functional scope of the present invention. The computing device 1300 should not be construed as having any dependence or requirement related to any one or combination of the elements shown.

[0106] The present invention can be implemented by computer program code or machine-usable instructions. The instructions can be computer-executable instructions for program modules, and the program modules are executed by a computer or other machines, such as a personal digital assistant or other portable devices. Generally speaking, program modules include routines, programs, objects, components, data structures, etc. A program module refers to the program code that executes a specific task or implements a specific abstract data type. The present invention can be implemented in various system configurations, including portable devices, consumer electronics, general-purpose computers, more professional computing devices, etc. The present invention can also be implemented in a distributed computing environment, processing devices connected by a communication network.

[0107] Reference Figure 13 。The computing device 1300 includes a bus 1310, a memory 1312, one or more processors 1314, one or more display elements 1316, an input / output (I / O) port 1318, an input / output (I / O) element 1320, and an illustrative power supply 1322 that are directly or indirectly coupled to the following devices. The bus 1310 represents an element that can be one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although Figure 13 each block of is shown as a straight line for simplicity, in fact, the boundaries of each element are not specific. For example, the rendering element of the display device can be regarded as an I / O element; the processor can have a memory.

[0108] The computing device 1300 generally includes various computer-readable media. Computer-readable media can be any available media accessible by the computing device 1300, which includes both volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer-readable media also includes volatile and non-volatile media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically-erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage devices, magnetic disks, magnetic tapes, magnetic tape storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and is accessible by the computing device 1300. Computer storage media does not include signals per se.

[0109] Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal form such as a carrier wave or other transmission mechanism, and includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct wired connection, and wireless media such as audio, radio frequency, infrared, and other wireless media. Combinations of the above media are included within the scope of computer-readable media.

[0110] The memory 1312 includes computer storage media in the form of volatile and non-volatile memory. The memory can be removable, non-removable, or a combination of both. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The computing device 1300 includes one or more processors that read data from various entities such as the memory 1312 or the I / O element 1320. The display element 1316 displays data indications to the user or other devices. Exemplary display elements include display devices, speakers, printing elements, vibrating elements, etc.

[0111] The I / O port 1318 allows the computing device 1300 to logically connect to other devices including the I / O component 1320, and some of such devices are built-in devices. Exemplary components include microphones, joysticks, game tables, dish satellite signal receivers, scanners, printers, wireless devices, etc. The I / O component 1320 can provide a natural user interface for processing user-generated gestures, sounds, or other physiological inputs. In some examples, these inputs can be transmitted to a suitable network component for further processing. The computing device 1300 can be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and combinations of these systems, to detect and identify objects. In addition, the computing device 1300 can be equipped with sensors (e.g., radar, lidar) to periodically sense the surrounding environment within a sensing range, generating sensor information representing its association with the surrounding environment. Furthermore, the computing device 1300 can be equipped with an accelerometer or gyroscope for detecting motion. The output of the accelerometer or gyroscope can be provided to the display of the computing device 1300.

[0112] In addition, the processor 1314 in the computing device 1300 can also execute the programs and instructions in the memory 1312 to present the actions and steps described in the above embodiments, or other descriptions of the content in the specification.

[0113] Any specific order or hierarchical steps of the programs disclosed herein are purely for illustrative purposes. Based on design preferences, it must be understood that any specific order or hierarchical steps of the programs can be rearranged within the scope disclosed in this document. The method claims present the elements of various steps in an example order, and thus should not be limited by the specific order or hierarchy shown herein.

[0114] The use of ordinal numbers such as "first", "second", "third", etc. to modify elements in the claims does not itself imply any priority, precedence, order among the elements, or order of the steps performed by the method, but is only used as an identifier to distinguish different elements with the same name (with different ordinal numbers).

[0115] Although the present disclosure has been disclosed as above with embodiments, it is not intended to limit the present disclosure. Any person skilled in the art can make some changes and modifications without departing from the concept and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to what is defined by the claims.

Claims

1. A method for virtual reality-based course production and reproduction, applied to an electronic device, the method comprising: Receiving a 3D model; Generating a model data packet corresponding to the 3D model, wherein the model data packet at least includes a plurality of objects applied to the 3D model; Recording, by a plurality of virtual cameras, the actions of a user operating the objects in a virtual reality, and generating an action video corresponding to the objects; and Generating a course data packet, wherein the course data packet includes the model data packet and an animation packet including the action video, and after recording the actions of the user operating the objects in the virtual reality, the method further comprises: Determining whether the objects move; When the objects move, marking the action video with a time stamp; and Storing the action video marked with the time stamp.

2. The virtual reality-based course production and reproduction method according to claim 1, wherein the plurality of objects are classified into pure collision objects, base objects, extraction objects, physical objects, penetrative objects, and pseudo-physical objects.

3. The virtual reality-based course production and reproduction method according to claim 1, wherein the virtual cameras record the coordinate rotation information and displacement information of the objects through a time difference.

4. The virtual reality-based course production and reproduction method according to claim 1, further comprising: Loading the course data packet; And Playing the animation packet included in the course data packet to reproduce the actions of the user operating the objects in the virtual reality.

5. The virtual reality-based course production and reproduction method according to claim 4, wherein the step of playing the animation packet included in the course data packet further comprises: Putting the animation packet including the action video into an object pool; Determining whether an action frame in the action video has the same first object as a to-be-played action frame; and When the action frame has the same first object as the to-be-played action frame, retaining the first object of the action frame in the object pool.

6. The virtual reality-based course production and reproduction method according to claim 1, further comprising: Establishing the virtual reality through a Unity3D engine.

7. The virtual reality-based course production and reproduction method according to claim 6, wherein the course data packet is an Assetbundle file.

8. The virtual reality-based course production and reproduction method according to claim 1, wherein the animation packet at least includes an audio file, a preloaded object and animation pairing file, animation information corresponding to the objects, and an animation packet screenshot.

9. The virtual reality-based course production and reproduction method according to claim 1, wherein the electronic device is a head-mounted display device.

10. A virtual reality-based course production and reproduction system, comprising: One or more processors; And One or more computer storage media storing computer-readable instructions, wherein the processor uses the computer storage media to execute: Receive a 3D model; Generate a model data packet corresponding to the 3D model based on the 3D model, wherein the model data packet at least includes a plurality of objects applied to the 3D model; Record, by a plurality of virtual cameras, an action of a user operating the objects in a virtual reality and generate an action video corresponding to the objects; and Generate a course data packet, wherein the course data packet includes the model data packet and an animation packet including the action video, and after recording the action of the user operating the objects in the virtual reality, the processor further executes: Determine whether the objects move; When the objects move, mark the action video with a timestamp; and Store the action video marked with the timestamp.

11. The virtual reality-based course production and reproduction system according to claim 10, wherein the plurality of objects are classified into pure collision objects, base objects, extraction objects, physical objects, penetrative objects, and pseudo-physical objects.

12. The virtual reality-based course production and reproduction system according to claim 10, wherein the virtual cameras record coordinate rotation information and displacement information of the objects through a time difference.

13. The virtual reality-based course production and reproduction system according to claim 10, the processor further executes: Load the course data packet; and Play the animation packet included in the course data packet to reproduce the action of the user operating the objects in the virtual reality.

14. The virtual reality-based course production and reproduction system according to claim 13, wherein the step of playing the animation packet included in the course data packet further includes: Put the animation packet including the action video into an object pool; Determine whether an action frame in the action video has the same first object as a to-be-played action frame; and When the action frame has the same first object as the to-be-played action frame, retain the first object of the action frame in the object pool.

15. The virtual reality-based course production and reproduction system according to claim 10, the processor further executes: Establish the virtual reality through a Unity3D engine.

16. The virtual reality-based course production and reproduction system according to claim 15, wherein the course data packet is an Assetbundle file.

17. The virtual reality-based course production and reproduction system according to claim 10, wherein the animation packet at least includes an audio file, a preloaded object and animation pairing file, animation information corresponding to the objects, and an animation packet screenshot.

18. The virtual reality-based course production and reproduction system according to claim 10, the system is applied to a head-mounted display device.

Images