Practical training management system and method based on virtual reality technology

Abstract

The invention relates to the field of intelligent management, and particularly discloses a practical training management system and method based on a virtual reality technology, and the method comprises the steps: obtaining a behavior operation video of a student in virtual practical training, and carrying out the key frame extraction of the behavior operation video, carrying out behavior feature analysis and association on each behavior operation video key frame by adopting a video key frame analysis technology and a processing algorithm based on deep learning, and carrying out full-time-domain-based fusion on behavior operation change context semantic understanding features obtained through differential calculation of a plurality of key frames; therefore, the judgment result of whether the operation of the student accords with the correct steps of the experiment is intelligently obtained, and a real-time feedback instruction is generated. Through the mode, practical training can be carried out in the virtual space, and meanwhile, the system can provide instant guidance and personalized feedback, so that the learning efficiency and effect are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent management, and more specifically, to a training management system and method based on virtual reality technology. Background Art

[0002] Practical training is a teaching method that focuses on practical operation and skill training. It corresponds to traditional theoretical teaching and pays more attention to students' practical ability and skill mastery. In other words, it focuses on cultivating students' hands-on ability and professional skills, so that students can demonstrate their professional ability in a real or simulated working environment.

[0003] However, since practical training usually requires a lot of physical equipment and venues, this may lead to uneven resource allocation. In addition, traditional practical training usually needs to be conducted at a specific time and place, which limits students' flexibility and accessibility and is not conducive to students learning according to their own schedule. Secondly, in traditional practical training, feedback usually relies on on-site observation and evaluation by teachers, which may need to be carried out after the end of the practical training, resulting in delayed feedback, which affects learning efficiency and quality.

[0004] Therefore, an optimized training management system is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a training management system and method based on virtual reality technology, which obtains the behavior operation video of the student in the virtual training, and extracts the key frame of the behavior operation video, and adopts the video key frame analysis technology and processing algorithm based on deep learning to analyze the behavior characteristics of each behavior operation video key frame and the semantic association between the context, and the sequence of the behavior operation change context semantic understanding feature vector obtained by differential calculation of multiple key frames is fused based on the full time domain, so as to intelligently obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment, and generate real-time feedback instructions. In this way, the training can be carried out in the virtual space, no longer limited by the size and availability of the physical space, and no longer subject to a specific time and place, thereby solving the problem of limited space for training teaching, and improving the flexibility of time and place. At the same time, the system can interact with students more effectively according to the specific operation behavior of the students, provide immediate guidance and personalized feedback, so as to improve learning efficiency and effect.

[0006] According to one aspect of the present application, a training management system based on virtual reality technology is provided, which includes: The behavior operation video acquisition module is used to acquire the behavior operation video of students in virtual training; A behavior operation key frame extraction feature capture module is used to extract key frames from the behavior operation video and then input the key frames into the behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; A behavior operation semantic understanding module, used for inputting the sequence of behavior operation feature vectors into a behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; A behavior operation context semantic understanding differential change calculation module, used to calculate the behavior operation context semantic understanding differential change of every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors; An operation change full-time domain feature fusion module, used to fuse the sequence of behavior operation change context semantic understanding feature vectors to obtain a behavior operation change context semantic understanding full-time domain feature vector; The judgment result generation module is used to understand the full-time domain feature vector based on the context semantics of the behavior operation change, obtain the judgment result, and generate a real-time feedback instruction based on the judgment result.

[0007] According to another aspect of the present application, a training management method based on virtual reality technology is provided, which includes: Obtain videos of students’ behaviors and operations in virtual training; Extracting key frames from the behavior operation video and inputting the key frames into a behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; Inputting the sequence of behavior operation feature vectors into a behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; Calculating the difference change of the behavior operation context semantic understanding at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors; The sequence of the behavior operation change context semantic understanding feature vectors is fused to obtain the behavior operation change context semantic understanding full time domain feature vector; Based on the context semantics of the behavior operation change, the full-time domain feature vector is understood to obtain a judgment result, and based on the judgment result, a real-time feedback instruction is generated.

[0008] Compared with the prior art, the present application provides a training management system and method based on virtual reality technology, which obtains the student's behavior operation video in virtual training, and extracts key frames of the behavior operation video, and adopts video key frame analysis technology and processing algorithm based on deep learning to analyze the behavior characteristics of each behavior operation video key frame and the semantic association between the context, and the sequence of behavior operation change context semantic understanding feature vectors obtained by differential calculation of multiple key frames is fused based on the full time domain, so as to intelligently obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment, and generate real-time feedback instructions. In this way, the training can be carried out in the virtual space, no longer limited by the size and availability of the physical space, and no longer subject to a specific time and place, thereby solving the problem of limited space for training teaching, and improving the flexibility of time and place. At the same time, the system can interact with students more effectively according to their specific operation behaviors, provide immediate guidance and personalized feedback, so as to improve learning efficiency and effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 It is a block diagram of a training management system based on virtual reality technology according to an embodiment of the present application.

[0011] Figure 2 It is a block diagram of a behavior operation key frame extraction feature capture module in a training management system based on virtual reality technology according to an embodiment of the present application.

[0012] Figure 3 It is a block diagram of a judgment result generation module in a training management system based on virtual reality technology according to an embodiment of the present application.

[0013] Figure 4 The flowchart of the training management method based on virtual reality technology according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0016] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0017] As shown in this disclosure, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0018] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0019] Practical training is a teaching method that focuses on practical operation and skill training. It corresponds to traditional theoretical teaching and pays more attention to students' practical ability and skill mastery. In other words, it focuses on cultivating students' hands-on ability and professional skills, so that students can demonstrate their professional ability in a real or simulated working environment.

[0020] However, since practical training usually requires a lot of physical equipment and venues, this may lead to limited practical training space. In addition, traditional practical training usually needs to be carried out at a specific time and place, which limits students' flexibility and accessibility and is not conducive to students learning according to their own schedule. Secondly, in traditional practical training, feedback usually relies on on-site observation and evaluation by teachers. This process may need to be carried out after the end of the practical training, resulting in delayed feedback, which affects learning efficiency and quality.

[0021] It should be understood that virtual reality (VR) is a technology that generates a three-dimensional environment through computer simulation, immersing users in a virtual world that looks real. This technology usually requires special hardware equipment, such as head-mounted displays (HMDs), handles, sensors, etc., as well as software programs to implement. With the development of technology, the application of virtual reality in practical training management will become more and more extensive. For example, in high-risk industries such as construction, mining or aviation, virtual reality can provide a safe training environment for employees to learn and practice responding to emergencies without actual danger, and provide timely feedback and understanding. Therefore, the application of virtual reality technology in practical training management can not only improve the efficiency and effectiveness of training, but also reduce costs and risks, and provide a richer and more personalized learning experience.

[0022] Therefore, a training management system based on virtual reality technology is expected, which obtains the behavior operation video of the students in the virtual training, extracts the key frames of the behavior operation video, and uses the video key frame analysis technology and processing algorithm based on deep learning to analyze the behavior characteristics of each behavior operation video key frame and the semantic association between the context, and the sequence of the behavior operation change context semantic understanding feature vector obtained by differential calculation of multiple key frames is fused based on the full time domain, so as to intelligently obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment, and generate real-time feedback instructions. In this way, the training can be carried out in the virtual space, no longer limited by the size and availability of the physical space, and no longer subject to a specific time and place, thereby solving the problem of limited space for training teaching, and improving the flexibility of time and place. At the same time, the system can interact with students more effectively according to their specific operation behaviors, provide instant guidance and personalized feedback, thereby improving learning efficiency and effectiveness.

[0023] Figure 1 FIG. 1 is a block diagram of a training management system based on virtual reality technology according to an embodiment of the present application. Figure 1As shown, according to the embodiment of the present application, the training management system 100 based on virtual reality technology includes: a behavior operation video acquisition module 110, which is used to acquire the behavior operation video of the student in the virtual training; a behavior operation key frame extraction feature capture module 120, which is used to extract the key frame of the behavior operation video and then input it into the behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; a behavior operation semantic understanding module 130, which is used to input the sequence of behavior operation feature vectors into the behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; a behavior operation context semantic understanding differential transformation module 130; The calculation module 140 is used to calculate the differential change of the behavior operation context semantic understanding at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vector to obtain a sequence of behavior operation change context semantic understanding feature vectors; the operation change full-time domain feature fusion module 150 is used to fuse the sequence of the behavior operation change context semantic understanding feature vectors to obtain the behavior operation change context semantic understanding full-time domain feature vector; the judgment result generation module 160 is used to obtain a judgment result based on the behavior operation change context semantic understanding full-time domain feature vector, and generate a real-time feedback instruction based on the judgment result.

[0024] In an embodiment of the present application, the behavior operation video acquisition module 110 is used to obtain the behavior operation video of the students in the virtual training. It should be understood that the behavior operation video of the students in the virtual training refers to the video record recorded when the students perform the training operation in the virtual reality environment. These videos record all the interactive actions, decision-making processes and skill applications of the students in the virtual training process. These behavior operation videos can evaluate whether the students' operations are accurate and whether they follow the correct steps and methods. Based on this, in the technical solution of the present application, the behavior operation video of the students in the virtual training is obtained, and the behavior operation features in the video are analyzed and extracted to generate real-time feedback to guide students to improve their operations immediately.

[0025] In the embodiment of the present application, the behavior operation key frame extraction feature capture module 120 is used to extract key frames from the behavior operation video and then input the key frames into the behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors. Figure 2 : is a block diagram of a behavior operation key frame extraction feature capture module in a training management system based on virtual reality technology according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 2As shown, the behavior operation key frame extraction feature capture module 120 includes: a behavior operation video key frame sampling unit 121, used to perform key frame sampling on the behavior operation video to obtain a sequence of behavior operation image key frames; a behavior operation key frame convolution encoding unit 122, used to input the sequence of behavior operation image key frames into a behavior operation feature extraction module based on a deep neural network to obtain a sequence of behavior operation feature vectors.

[0026] Specifically, the behavior operation video key frame sampling unit 121 is used to perform key frame sampling on the behavior operation video to obtain a sequence of behavior operation image key frames. Accordingly, considering that the behavior operation video contains a large amount of image frame information, some of the image frames play a key role in the analysis of the behavior operation, and contain important action captures of students during operation, but there are also some background frames or. Based on this, in the technical solution of the present application, key frame sampling is performed on the behavior operation video to obtain a sequence of behavior operation image key frames. It should be understood that key frame sampling is to extract frames with representative or key information from the video sequence. These frames can capture the main actions or events in the video, that is, key frames usually contain important actions or changes in the training process. Through key frames, behavioral features can be more easily identified and extracted, so as to better understand the semantics and context of student behavior.

[0027] Specifically, the behavior operation key frame convolution encoding unit 122 is used to input the sequence of the behavior operation image key frames into the behavior operation feature extraction module based on the deep neural network to obtain the sequence of the behavior operation feature vectors. In particular, the behavior operation feature extraction module based on the deep neural network is a behavior operation feature extraction module based on the convolutional neural network. It should be understood that each behavior operation image key frame in the sequence of the behavior operation image key frames expresses the key behavior operation visual feature information, and the convolutional neural network is particularly good at automatically learning and extracting complex features from image data. That is, in the behavior operation image, CNN can identify key visual patterns and action features. Therefore, in the technical solution of the present application, the sequence of the behavior operation image key frames is input into the behavior operation feature extraction module based on the convolutional neural network to obtain the sequence of the behavior operation feature vectors. In particular, the behavior operation feature extraction module based on the convolutional neural network here is essentially a convolutional neural network model. In this way, it is possible to efficiently extract features that are crucial to understanding student behavior, providing support for subsequent behavior analysis and real-time feedback.

[0028] In the embodiment of the present application, the behavior operation semantic understanding module 130 is used to input the sequence of the behavior operation feature vectors into the behavior operation context semantic understanding encoding module to obtain the sequence of the behavior operation context semantic understanding feature vectors. Specifically, in the embodiment of the present application, the behavior operation semantic understanding module is used to: input the sequence of the behavior operation feature vectors into the behavior operation context semantic understanding encoding module based on the bidirectional long short-term memory neural network model to obtain the sequence of the behavior operation context semantic understanding feature vectors. Accordingly, considering that there are different degrees of semantic associations between the behavior operations between different time point spans in the sequence of the behavior operation feature vectors, the bidirectional long short-term memory neural network model is a neural network model that is particularly suitable for processing time series data, which can capture the semantic dependencies between different time spans in the behavior operation feature sequence. Based on this, in the technical solution of the present application, the sequence of the behavior operation feature vectors is input into the behavior operation context semantic understanding encoding module based on the bidirectional long short-term memory neural network model to obtain the sequence of the behavior operation context semantic understanding feature vectors. It should be understood that the essence of the behavior operation context semantic understanding encoding module based on the bidirectional long short-term memory neural network model is the bidirectional long short-term memory neural network model. Unlike traditional LSTM, the bidirectional long short-term memory neural network model can process past and future information at the same time, which helps to more comprehensively understand the contextual semantics of behavioral operations. At the same time, through its gating mechanism (including input gate, forget gate and output gate), it effectively solves the long-term dependency problem encountered by traditional neural networks when processing long sequence data.

[0029] In an embodiment of the present application, the behavior operation context semantic understanding differential change calculation module 140 is used to calculate the behavior operation context semantic understanding differential change at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vector to obtain a sequence of behavior operation change context semantic understanding feature vectors. Specifically, in an embodiment of the present application, the behavior operation context semantic understanding differential change calculation module is used to: calculate the positional difference of the behavior operation context semantic understanding feature vector at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vector to obtain the sequence of the behavior operation change context semantic understanding feature vector. Furthermore, in order to understand and consider the changing trend of each behavior operation feature in a continuous time, so as to evaluate the continuous dynamic change trend of the behavior operation, in the technical solution of the present application, the positional difference of the behavior operation context semantic understanding feature vector at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vector is calculated to obtain the sequence of the behavior operation change context semantic understanding feature vector. That is, through differential calculation, the changes of behavioral operations at continuous time points can be captured, which helps to understand the dynamic progress and real-time performance of students' skills. In addition, differentials can highlight the changing parts of the feature vector, which can enhance the model's sensitivity to behavioral changes, thereby improving the accuracy of behavior recognition and prediction.

[0030] In an embodiment of the present application, the operation change full-time domain feature fusion module 150 is used to fuse the sequence of the behavior operation change context semantic understanding feature vectors to obtain the behavior operation change context semantic understanding full-time domain feature vector. Accordingly, in order to be able to integrate the feature information of the student's behavior operation during the entire training process to provide a comprehensive view and thus more comprehensively understand the student's behavior operation process, in the technical solution of the present application, the sequence of the behavior operation change context semantic understanding feature vectors is fused to obtain the behavior operation change context semantic understanding full-time domain feature vector. In this way, through feature fusion based on the full time domain, the training management system can more accurately capture and understand the student's behavior patterns and skill development, thereby providing more effective teaching support and real-time feedback.

[0031] In the embodiment of the present application, the judgment result generation module 160 is used to understand the full-time domain feature vector based on the context semantics of the behavior operation change, obtain a judgment result, and generate a real-time feedback instruction based on the judgment result. Figure 3 is a block diagram of a judgment result generation module in a training management system based on virtual reality technology according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 3As shown, the judgment result generating module 160 includes: a behavior operation change context semantic similarity learning compensation unit 161, which is used to perform similarity learning compensation on the behavior operation change context semantic understanding full-time domain feature vector based on category recognition weight to obtain an optimized behavior operation change context semantic understanding full-time domain feature vector; a student operation step judging unit 162, which is used to input the optimized behavior operation change context semantic understanding full-time domain feature vector into a classifier-based operation behavior judgement to obtain a judgment result of whether the student's operation conforms to the correct steps of the experiment; a feedback instruction generating unit 163, which is used to generate real-time feedback instructions based on the fact that the student's operation does not conform to the correct steps of the experiment.

[0032] Specifically, the behavior operation change context semantic similarity learning compensation unit 161 is used to perform similarity learning compensation based on category recognition weights on the behavior operation change context semantic understanding full-time domain feature vector to obtain an optimized behavior operation change context semantic understanding full-time domain feature vector. In particular, in the technical solution of the present application, it is considered that the sampling of key frames may not fully capture all subtle changes in behavior operations, especially when the operation is very fast or subtle, which may cause the feature vector to lose important information. Although convolutional neural networks perform outstandingly in image recognition, they may not fully capture all visual features related to behavior operations, especially when the operation involves complex hand-eye coordination. The bidirectional long short-term memory neural network (Bi-LSTM) model may be limited by training data when understanding the context semantics of behavior operations and cannot fully understand the complexity of all behaviors. Calculating the difference of feature vectors at adjacent time points may amplify small differences, resulting in excessive sensitivity to noise or irrelevant changes. This leads to local abnormal distribution of the full-time domain feature vector of the context semantic understanding of behavior operation changes. However, the full-time domain feature vector of the contextual semantic understanding of behavioral operation changes has local abnormal distribution. The classifier may not be able to correctly learn the feature distribution of normal operations during the training process, resulting in a bias towards outliers when judging, that is, category outlier shift. Category outlier shift will affect the prediction accuracy and generalization ability of the classifier, so that the judgment result may not accurately reflect the correctness of the student's operation, affecting the generation of real-time feedback instructions and the quality of teaching. In order to solve this problem, in the technical solution of the present application, the full-time domain feature vector of the contextual semantic understanding of the behavioral operation changes is compensated by similarity learning based on category recognition weights.

[0033] More specifically, the behavior operation change context semantic similarity learning compensation unit is used to: determine the category decoding weight matrix of the classifier-based operation behavior judge; perform matrix decoupling of the category decoding weight matrix in units of row vectors to obtain a set of category decoding weight vectors; calculate the behavior operation change metric learning loss information item between each category decoding weight vector in the set of category decoding weight vectors and the behavior operation change context semantic understanding full time domain feature vector to obtain a behavior operation change metric learning compensation representation vector; input the behavior operation change metric learning compensation representation vector An activation function is used to obtain a probabilistic behavior operation change metric learning compensation weight vector; the probabilistic behavior operation change metric learning compensation weight vector and the behavior operation change context semantic understanding full-time domain feature vector are calculated by positional point multiplication to obtain a point product feature vector, and a weighted sum of the point product feature vector and the behavior operation change context semantic understanding full-time domain feature vector is calculated to obtain the optimized behavior operation change context semantic understanding full-time domain feature vector.

[0034] In the embodiment of the present application, specifically, the following formula is used to calculate the class behavior operation change metric learning loss information item between each class decoding weight vector in the set of class decoding weight vectors and the behavior operation change context semantic understanding full time domain feature vector to obtain the behavior operation change metric learning compensation representation vector; wherein, the formula is: ;in, represents the first in the set of class decoding weight vectors The first class decoding weight vector The eigenvalues ​​at the positions, The first time domain feature vector representing the contextual semantic understanding of the behavior operation change The eigenvalues ​​at the positions, Indicated by natural constant The exponential function with base , represents the logarithmic function value with base 2, represents the length of the feature vector, The first The feature value at each position.

[0035] That is, the behavior operation change context semantic understanding full-time domain feature vector is compensated by similarity learning based on category recognition weights, which first determines the category decoding weight matrix of the classifier to understand the classifier distinguishing different category features. Subsequently, the category decoding weight matrix is ​​decoupled by row vectors to obtain a set of category decoding weight vectors to allow the model to independently analyze the weight of each category and provide more refined control for feature compensation. Next, the behavior operation change metric learning loss information item is calculated. This process involves quantitative analysis of the interaction between each category weight vector and the behavior operation change context semantic understanding full-time domain feature vector. Through the calculation of a specific loss function, the inconsistency between the feature vector and the category weight is captured, providing a quantitative indicator for compensation. Next, the behavior operation change metric learning compensation representation vector is input into the activation function, nonlinearly transformed, and converted into a probabilistic form. The application of the activation function introduces nonlinearity, enhances the model's ability to interpret the compensation weight, and makes the compensation weight more consistent with the actual distribution characteristics. Then, the behavior operation change metric learning compensation representation vector processed by the activation function is used to calculate the probabilistic behavior operation change metric learning compensation weight vector. This step provides probabilistic guidance for the adjustment of feature vectors in a probabilistic form, making the optimization process more consistent with statistical characteristics, thereby enhancing the accuracy of the model's feature adjustment. Finally, by calculating the dot product of the probabilistic behavior operation change metric learning compensation weight vector and the behavior operation change context semantic understanding full-time domain feature vector, and then calculating the weighted sum with the original behavior operation change context semantic understanding full-time domain feature vector, the optimized behavior operation change context semantic understanding full-time domain feature vector is obtained. This step combines the original feature information and compensation information to generate an adjusted feature vector, reducing the impact of abnormal distribution and improving the accuracy of generation.

[0036] Specifically, the student operation step judgment unit 162 is used to input the optimized behavior operation change context semantic understanding full time domain feature vector into the classifier-based operation behavior judgement to obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment; the feedback instruction generation unit 163 is used to generate real-time feedback instructions based on the fact that the student's operation does not conform to the correct steps of the experiment. That is, the optimized behavior operation change context semantic understanding full time domain feature vector is classified and processed, so as to intelligently obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment, and generate real-time feedback instructions. In this way, practical training can be carried out in a virtual space, no longer limited by the size and availability of the physical space, and no longer subject to a specific time and place, thereby solving the problem of limited practical training teaching space and improving the flexibility of time and place. At the same time, the system can interact with students more effectively according to their specific operation behaviors, provide instant guidance and personalized feedback, thereby improving learning efficiency and effectiveness.

[0037] In summary, the training management system 100 based on virtual reality technology according to the embodiment of the present application is explained, which obtains the behavior operation video of the student in the virtual training, and extracts the key frame of the behavior operation video, and adopts the video key frame analysis technology and processing algorithm based on deep learning to analyze the behavior characteristics of each behavior operation video key frame and the semantic association between the context, and the sequence of the behavior operation change context semantic understanding feature vector obtained by differential calculation of multiple key frames is fused based on the full time domain, so as to intelligently obtain the judgment result of whether the student's operation conforms to the correct steps of the experiment, and generate real-time feedback instructions. In this way, the training can be carried out in the virtual space, no longer limited by the size and availability of the physical space, and no longer subject to a specific time and place, thereby solving the problem of limited space for training teaching, and improving the flexibility of time and place. At the same time, the system can interact with students more effectively according to their specific operation behaviors, provide instant guidance and personalized feedback, so as to improve learning efficiency and effect.

[0038] Figure 4 Flowchart of the training management method based on virtual reality technology according to an embodiment of the present application. Figure 4As shown, according to the embodiment of the present application, the training management method based on virtual reality technology includes: S110, obtaining the behavior operation video of the students in the virtual training; S120, extracting the key frames of the behavior operation video and inputting the key frames into the behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; S130, inputting the sequence of behavior operation feature vectors into the behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; S140, calculating the differential change of the behavior operation context semantic understanding at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors; S150, fusing the sequence of the behavior operation change context semantic understanding feature vectors to obtain the behavior operation change context semantic understanding full-time domain feature vector; S160, obtaining a judgment result based on the behavior operation change context semantic understanding full-time domain feature vector, and generating a real-time feedback instruction based on the judgment result.

[0039] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned training management method based on virtual reality technology have been referred to above. Figures 1 to 3 The invention has been introduced in detail in the description of the training management system based on virtual reality technology, and therefore, its repeated description will be omitted.

[0040] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0041] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0042] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0043] In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware, or through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented in hardware, firmware, or a combination of them. When using software implementation, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a computer. Taking this as an example but not limited to: the computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition. Any connection can be appropriately a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless and microwave are included in the fixation of the medium. As used in the present invention, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically and discs use lasers to reproduce data optically. The above combinations should also be included in the scope of protection of computer-readable media.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. A training management system based on virtual reality technology, characterized in that: include: The behavior operation video acquisition module is used to acquire the behavior operation video of students in virtual training; A behavior operation key frame extraction feature capture module is used to extract key frames from the behavior operation video and then input the key frames into the behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; A behavior operation semantic understanding module, used for inputting the sequence of behavior operation feature vectors into a behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; A behavior operation context semantic understanding differential change calculation module, used to calculate the behavior operation context semantic understanding differential change of every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors; An operation change full-time domain feature fusion module, used to fuse the sequence of behavior operation change context semantic understanding feature vectors to obtain a behavior operation change context semantic understanding full-time domain feature vector; The judgment result generation module is used to understand the full-time domain feature vector based on the context semantics of the behavior operation change, obtain the judgment result, and generate a real-time feedback instruction based on the judgment result.

2. The training management system based on virtual reality technology according to claim 1 is characterized in that: The behavior operation key frame extraction feature capture module includes: A behavior operation video key frame sampling unit, used for performing key frame sampling on the behavior operation video to obtain a sequence of behavior operation image key frames; The behavior operation key frame convolution encoding unit is used to input the sequence of the behavior operation image key frames into the behavior operation feature extraction module based on the deep neural network to obtain the sequence of the behavior operation feature vectors.

3. The training management system based on virtual reality technology according to claim 2 is characterized in that: The behavior operation feature extraction module based on deep neural network is a behavior operation feature extraction module based on convolutional neural network.

4. The training management system based on virtual reality technology according to claim 3 is characterized in that: The behavior operation semantic understanding module is used to: The sequence of the behavior operation feature vectors is input into a behavior operation context semantic understanding encoding module based on a bidirectional long short-term memory neural network model to obtain the sequence of the behavior operation context semantic understanding feature vectors.

5. The training management system based on virtual reality technology according to claim 4 is characterized in that: The behavior operation context semantic understanding differential change calculation module is used to: The positional difference of the behavior operation context semantic understanding feature vectors at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors is calculated to obtain the sequence of the behavior operation change context semantic understanding feature vectors.

6. The training management system based on virtual reality technology according to claim 5 is characterized in that: The judgment result generating module comprises: A behavior operation change context semantic similarity learning compensation unit, used for performing similarity learning compensation based on category recognition weights on the behavior operation change context semantic understanding full-time domain feature vector to obtain an optimized behavior operation change context semantic understanding full-time domain feature vector; A student operation step judgment unit, used for inputting the optimized behavior operation change context semantic understanding full-time domain feature vector into a classifier-based operation behavior judgement unit to obtain a judgment result of whether the student's operation conforms to the correct steps of the experiment; The feedback instruction generating unit is used to generate real-time feedback instructions based on the fact that the student's operation does not conform to the correct steps of the experiment.

7. The training management system based on virtual reality technology according to claim 6 is characterized in that: The behavior operation change context semantic similarity learning compensation unit is used to: Determine a category decoding weight matrix of the classifier-based operation behavior determiner; Decoupling the class decoding weight matrix in units of row vectors to obtain a set of class decoding weight vectors; Calculate the behavior operation change metric learning loss information item between each category decoding weight vector in the set of category decoding weight vectors and the behavior operation change context semantic understanding full time domain feature vector to obtain a behavior operation change metric learning compensation representation vector; The behavior operation change metric learning compensation representation vector is input Activation function to obtain probabilistic behavior operation change measurement learning compensation weight vector; Calculate the positional point multiplication of the probabilistic behavior operation change metric learning compensation weight vector and the behavior operation change context semantic understanding full-time domain feature vector to obtain the point product feature vector, and calculate the weighted sum of the point product feature vector and the behavior operation change context semantic understanding full-time domain feature vector to obtain the optimized behavior operation change context semantic understanding full-time domain feature vector.

8. A training management method based on virtual reality technology, characterized in that: include: Obtain videos of students’ behaviors and operations in virtual training; Extracting key frames from the behavior operation video and inputting the key frames into a behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors; Inputting the sequence of behavior operation feature vectors into a behavior operation context semantic understanding encoding module to obtain a sequence of behavior operation context semantic understanding feature vectors; Calculating the difference change of the behavior operation context semantic understanding at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors; The sequence of the behavior operation change context semantic understanding feature vectors is fused to obtain the behavior operation change context semantic understanding full time domain feature vector; Based on the context semantics of the behavior operation change, the full-time domain feature vector is understood to obtain a judgment result, and based on the judgment result, a real-time feedback instruction is generated.

9. The training management method based on virtual reality technology according to claim 8 is characterized in that: After extracting key frames from the behavior operation video, the key frames are input into a behavior operation feature extraction module to obtain a sequence of behavior operation feature vectors, including: Performing key frame sampling on the behavior operation video to obtain a sequence of key frames of the behavior operation image; The sequence of the behavior operation image key frames is input into a behavior operation feature extraction module based on a deep neural network to obtain a sequence of the behavior operation feature vectors.

10. The training management method based on virtual reality technology according to claim 9 is characterized in that: Calculating the behavior operation context semantic understanding differential changes at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors to obtain a sequence of behavior operation change context semantic understanding feature vectors, including: The positional difference of the behavior operation context semantic understanding feature vectors at every two adjacent time points in the sequence of the behavior operation context semantic understanding feature vectors is calculated to obtain the sequence of the behavior operation change context semantic understanding feature vectors.

Images