VR experience system based on educational training

Abstract

The invention discloses a VR experience system based on educational training, which relates to the technical field of VR teaching systems and comprises VR playing equipment for playing teaching videos, a video capturing module for capturing and recording video clips of the teaching videos, a voice capturing module for capturing voice strip information and a data processing unit. The data processing unit divides the plurality of video clips into a plurality of different comparison groups according to the time information of the video clips, extracts the video frame data of each video clip in the comparison groups, and calculates the structural similarity between different video frames; and extracting words in the voice bar information, and analyzing the product of the word frequency and the inverse document frequency of each word. Through the calculation result of the video frame structure similarity, the time corresponding to the data frame with high similarity is used as the time for easily causing questions in the teaching video; words in the voice strip information are extracted, the product of the word frequency of each word and the inverse document frequency is calculated, and the word with the highest product value serves as a keyword to represent the focus in the questions asked by the students.

Description

Technical Field

[0001] The present invention relates to the technical field of VR teaching systems, and in particular to a VR experience system based on education and training. Background Art

[0002] At present, with the rapid development of information technology, various systems for training and teaching have emerged. Their fundamental purpose is to improve students' understanding of the course through immersive teaching experience.

[0003] However, the existing VR teaching system mainly relies on letting students watch videos for teaching, and for multiple student user terminals, it is unable to complete the real-time interaction function between the teacher and multiple student terminals. When different students have questions about the teaching videos, faced with a large number of different questions raised at different time points, it is difficult for teachers to identify the time when different questions are raised and the main content of the questions, as well as the time periods in the teaching videos when students are more likely to have problems. As a result, teachers are unable to solve students' doubts in a timely and accurate manner, affecting teaching effectiveness and learning quality. Summary of the invention

[0004] 1) Technical issues solved

[0005] The present invention provides a VR experience system based on education and training, so as to solve the problem that the existing VR experience system for teaching cannot identify the time of raising different questions and the type of question content.

[0006] 2) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a VR experience system based on education and training, comprising:

[0008] A VR playback device is used to play the teaching video and divide the teaching video into a plurality of continuous time periods;

[0009] A video capture module, used to capture and record video clips while playing the teaching video;

[0010] A voice capture module, used to capture voice information while capturing and recording the video clip;

[0011] A data processing unit divides a plurality of video clips into a plurality of different comparison groups according to the time information of the video clips, extracts the video frame data of each video clip in the comparison group, calculates the structural similarity between different video frames, and marks the corresponding time of the data frames with high structural similarity on the time axis of the teaching video; and extracts words from the voice bar information, calculates the product of the word frequency and the inverse document frequency of each word, and marks the words with the highest product value as keywords; wherein the time information specifically includes the distribution period and start and end time points of each video clip on the time axis based on the teaching video, as well as the duration of each video clip.

[0012] Furthermore, the data processing unit divides the plurality of video clips into a plurality of different comparison groups according to the time information of the video clips, including the following steps:

[0013] In the time period of the same teaching video, the video segment that overlaps with the time period is selected, and the relative duration of the video segment in the time period exceeds the first set value;

[0014] After selecting a plurality of video clips whose duration ratios exceed the first set value, a plurality of video clips whose duration ranges and start and end time points are close are taken as a comparison group.

[0015] Further, the data processing unit is in communication connection with the VR playback device, and the data processing unit extracts and calculates the structural similarity between different video frames in the comparison group;

[0016] According to the calculation results of the structural similarity between the different video frames, multiple video frames with higher structural similarity are selected as key frames;

[0017] Mark the time corresponding to each key frame on the timeline of the teaching video.

[0018] Furthermore, the data processing unit calculates the structural similarity between the video frames by calculating the SSIM scores between different video frames in each comparison group to calculate the structural similarity; specifically:

[0019] The SSIM formula is used to calculate the structural similarity between video frames; the calculation formula is as follows:

[0020]

[0021] Where x and y represent the pixel value distribution of the two frames respectively; μ x and μ y are the pixel means of x and y respectively; σ x 2 and σ y2 are the pixel variances of x and y, respectively; xy is the pixel covariance of x and y; C1 and C2 are constants;

[0022] Averaging the SSIM scores of the video frames in each comparison group to obtain an average structural similarity index between each of the video clips;

[0023] The average SSIM score is used as an indicator of the structural similarity between different video frames. The closer the average SSIM score is to 1, the more similar they are.

[0024] Furthermore, the data processing unit extracts words from the voice bar information and analyzes the product of the word frequency and the inverse document frequency of each word, specifically including the following steps:

[0025] Convert each of the speech pieces of information in each of the comparison groups into a corresponding separate document, extract multiple words in the document, and calculate the product of the word frequency and the inverse document frequency of each of the words in the document, where the product of the word frequency and the inverse document frequency is the TF-IDF value;

[0026] The words in each individual document in a comparison group are merged to find the words with the highest TF-IDF value within the time range covered by different video clips in the comparison group.

[0027] Furthermore, it also includes a start-stop module, which is used to control the start and stop of the video capture module to capture video clips; and

[0028] While the video capture module is working, the voice capture module is controlled to start and stop capturing voice bar information.

[0029] Furthermore, a wearable device is included, the start-stop module includes at least one sensor, and the sensor is located on the wearable device, and the data processing unit is communicatively connected with the start-stop module.

[0030] Furthermore, the wearable device includes two wearable gloves suitable for left and right hands, and the start-stop module includes pressure sensors respectively located on the wearable left glove and the wearable right glove.

[0031] Furthermore, the pressure sensors on the wearable left glove and the wearable right glove are each in a continuous pressure detection state, and the time interval for each detecting the continuous pressure is less than a set time value, the video capture module starts to capture video clips, and the sound capture module starts to capture sound bar information.

[0032] Furthermore, the pressure sensor on the wearable left glove or the wearable right glove stops the pressure detection state, and the video capture module or the sound capture module stops the capture work.

[0033] 3) Beneficial effects:

[0034] Compared with the prior art, the invention has the following beneficial effects:

[0035] The present invention captures different video clips and voice bar information for VR teaching videos, and analyzes and calculates the structural similarity of different video frames in the video clips. A high structural similarity between different video frames indicates that the video clips represented by these video frames have high content similarity. Multiple video frames with high structural similarity are used as key frames, and the corresponding time of the key frames on the timeline of the teaching video is marked, so as to show in which time period of the teaching video the content of students' learning questions appears.

[0036] While calculating the structural similarity based on the video frames extracted from different video clips to find the problem-focused segments, after extracting the words in the voice bar information corresponding to the different video clips captured by the student end, the product of the word frequency and the inverse document frequency of each word is calculated, and the words with the highest product values ​​of the word frequency and the inverse document frequency are used as keywords to represent the focus of the questions asked by the students. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A block diagram of the working principle of a VR experience system based on education and training provided by an embodiment of the present invention;

[0038] Figure 2 A flowchart of a data processing unit provided in an embodiment of the present invention for dividing a plurality of video clips into a plurality of different comparison groups;

[0039] Figure 3 A schematic diagram showing the distribution of duration intervals of different captured video clips in different time periods of a teaching video provided by an embodiment of the present invention;

[0040] Figure 4 A block diagram of the working principle of the VR experience system provided by an embodiment of the present invention;

[0041] Figure 5 A wearable left glove and a wearable right glove provided by an embodiment of the present invention, and a schematic diagram of the distribution of their respective sensors:

[0042] Figure 6A schematic diagram of an application scenario in which a first pressure sensor on a wearable left glove and a second pressure sensor on a wearable right glove provided in an embodiment of the present invention are both in a continuous pressure detection state, and a video capture module and a voice capture module are started;

[0043] Figure 7 Schematic diagram of an application scenario in which the first pressure sensor of the wearable left glove provided in an embodiment of the present invention is in a continuous pressure detection state, the second pressure sensor of the wearable right glove is not in a pressure detection state, the video capture module continues to work, and the voice capture module stops working. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be understood that the terms "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0046] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0047] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0048] During the implementation of the embodiments of the present invention, the inventors of the present invention found that:

[0049] When using VR equipment for teaching, such as when students wear wearable VR helmets and conduct VR experience courses for students, there is a disadvantage of not being able to accurately integrate student questions for multiple student user terminals.

[0050] Specifically, in virtual reality experience teaching, students may encounter multiple questions that need to be answered. However, for large-class teaching with multiple students, especially when the VR teaching video cannot be paused, the teacher cannot provide personalized support and answers to different questions raised by multiple students at different time periods, which affects the students' learning effect and experience.

[0051] Furthermore, in order to improve the defects existing in the above-mentioned prior art, the present invention provides a VR experience system based on education and training, which can integrate different questions raised by multiple students at different times, and accurately extract the time period and content of each question. After integration, it helps the teacher to understand the concentrated time period and question type of question points, which is helpful to solve student problems more targetedly.

[0052] First, an embodiment of the present invention provides a VR experience system based on education and training, including a VR playback device and a data processing unit communicated with the VR playback device, wherein the VR playback device includes but is not limited to a head-mounted display, a holographic projection device, and a mobile VR device, etc. The data processing unit communicated with the VR playback device is used to receive and process data from various sensors in the VR device to communicate with an external server or other devices, etc. The data processing unit includes but is not limited to a central processing unit, a sensor data processor, a data decoding and encoding processor, etc.

[0053] Specifically, the VR playback device is used to play the teaching video, and divide the teaching video as a whole into multiple continuous time periods. After the entire video is divided into several time periods, the complex video integration and screening tasks can be decomposed into smaller units, simplifying the operation process and reducing the complexity and difficulty of the operation. By dividing the teaching video into multiple continuous time periods, the subsequent screening and integration work is more efficient and accurate. Among them, as to how to divide the entire teaching video into several continuous time periods and the duration range of each time period, it can be based on the separability of the teaching video content or other, and the present invention does not limit the division method.

[0054] The VR experience system based on education and training provided in the embodiment of the present invention also includes a video capture module and a voice capture module. Figure 1 , Figure 1The working principle block diagram of a VR experience system based on education and training provided by an embodiment of the present invention. When a student watches a teaching video through a VR device and has questions about the teaching segment of the teaching video, the video capture module, without interfering with the normal playback of the teaching video, the student turns on the video capture module to capture and record the teaching video, thereby capturing a single recorded video segment. At the same time, the video capture module records the start and end time of the video segment relative to the recording of the teaching video, so as to facilitate subsequent time information analysis and processing.

[0055] On the basis of the above, the video capture module establishes a communication connection with the data processing unit so as to transmit the captured and recorded video clips and their time information to the data processing unit. After receiving the video clips and their time information, the data processing unit compares and analyzes them with the previously divided teaching video time periods. The data processing unit determines the specific interval position of the video clip in the teaching video time period based on the recorded video clip time information for subsequent integration and analysis.

[0056] During the process of capturing and recording the video clips by the video capture module, the voice capture module is started at the same time to capture the voice strips of questions that the student has about the video clips. This method of capturing video and voice information at the same time can help students record and express their learning needs and questions more comprehensively. The voice capture module is connected to the data processing unit in communication and inputs the question voice strips into the data processing unit for subsequent processing.

[0057] It should be noted that after the data processing unit receives the time information and voice information of the video clip, it integrates and associates these two types of information to ensure that each question voice item matches the corresponding video clip time information. By analyzing the correspondence between the question voice item and the video clip time information, the data processing unit can more accurately analyze what kind of questions are likely to gather in which time period of the teaching video.

[0058] For the VR experience system connected to multiple student terminals provided by an embodiment of the present invention, the data processing unit will receive multiple captured video clips with learning questions. Taking into account the accuracy of classification and integration of multiple video clips at time points and reducing system resource consumption, the multiple video clips will be divided into multiple different comparison groups according to time information, the video frame data of each video clip in the comparison group will be extracted, and the structural similarity between different video frames will be calculated.

[0059] In video processing, the structural similarity index mainly measures the brightness, contrast and structural information between two video frames. High structural similarity means that the two video frames have strong consistency in these features, so their similarity in content can be inferred. Among them, content usually refers to the objects, scenes, actions, etc. contained in the image. If two video frames are similar in structure, it means that they may show the same object or scene. In addition, in videos, the content between frames often has visual continuity. In video streams, the changes in content are usually gradual, so similar frames at different time nodes reflect similar visual content.

[0060] It is understandable that when dividing the video clips into different comparison groups, selecting clips of similar time periods helps to ensure that these clips contain similar content. After such grouping, the result of the structural similarity calculation can better reflect the relevance of the content.

[0061] On the basis of the above, if there are several video frames in the video clips of several different time periods with high structural similarity, it can be considered that the video clips represented by these video frames have high content similarity. That is to say, through the calculation results of the video frame structural similarity, it is possible to determine in which time period of the teaching video the content where students have learning questions appears based on the time corresponding to the data frames with high similarity.

[0062] In summary, it can be understood that by analyzing and calculating the structural similarity of different video frames of the video clip in each time period, the time points where problems exist in the teaching video can be more precisely located, which helps to more accurately identify the content parts that cause students' questions.

[0063] For the purpose of increasing the calculation accuracy, regarding how to divide multiple video clips into different comparison groups according to time information, in some feasible embodiments of the present invention, reference is made to Figure 2 , Figure 2 The work flow chart of the data processing unit provided in the embodiment of the present invention dividing multiple video clips into multiple different comparison groups, after the time period of the entire teaching video is divided, firstly, step 101 is performed: in the time period of the same teaching video, a video clip that overlaps with the time period is selected, and the relative duration of the video clip in the time period exceeds a first set value;

[0064] It is understandable that after dividing the time periods of the teaching videos, the video clips are first screened, and then the video clips captured within the time range are screened in different time periods in turn, that is, the video clips that coincide with the time period and meet the duration requirements are screened, so that the calculation of structural similarity for the video clips in each time period is more concentrated and targeted, thereby improving the efficiency of analyzing and processing video clips and avoiding unnecessary duplication of work.

[0065] After the initial screening is completed, step 102 is performed: after selecting multiple video clips whose duration ratio exceeds the first set value, multiple video clips with similar duration ranges and start and end time points are used as a comparison group.

[0066] It is understandable that in the same time period, video clips with similar duration range and time points are selected again as a comparison group; because video clips with similar time contain similar content and teaching focus, focusing on the content of the video clips in a certain period of time can better understand the focus and characteristics of teaching in that period of time.

[0067] For specific grouping methods, see Figure 3 In some feasible embodiments of the present invention, the entire teaching video is divided into three continuous and uniform time periods, namely, the first time period {0, t1}, the second time period {t1, t2} and the third time period {t2, T}, and different video clips are divided into comparison groups in the three time periods in turn.

[0068] First, the different video clips in the first time period {0, t1} are divided into comparison groups. In some feasible embodiments of the present invention, the initial screening and judgment procedure is: in the time period of the same teaching video, select the video clip that overlaps with the time period, and the relative duration of the video clip in the time period exceeds the first set value.

[0069] If the first setting value is set to 60%, the duration of the video segment falling within the time period must exceed 60% of its own duration. Figure 3 As shown, it can be understood that the video segments that overlap with the first time period include video segment A, video segment B, video segment C and video segment D, among which, the video segments that fall in this time period for a time length that exceeds 60% of their own length include video segment A, video segment B and video segment C.

[0070] In some feasible embodiments of the present invention, the judgment procedure for selecting the comparison group after the initial screening is: multiple video clips with similar duration ranges and start and end time points are used as a comparison group.

[0071] Combine again Figure 3 It can be understood that the duration range of video segment A and video segment B is close, and the start and end time points are close. Therefore, video segment A and video segment B can be used as a comparison group. After extracting the video frames of video segment A and video segment B in the comparison group, the structural similarity of the two video frames is calculated.

[0072] In summary, different video clips appearing in other time periods are also divided according to the above method steps and then the structural similarity of their video frames is calculated.

[0073] In addition, regarding the method of extracting video frames, it is possible but not limited to constructing an image processing library and framework (such as OpenCV, FFmpeg, etc.) to extract video frames, and the specific method is not limited in the present invention.

[0074] Regarding how to calculate the structural similarity between different video frames, in some feasible embodiments of the present invention, it is represented by calculating the SSIM score between the video frames in each comparison group.

[0075] Specifically, the SSIM formula is used to calculate the structural similarity between video frames; the SSIM formula is calculated as follows:

[0076]

[0077] Among them, x and y represent the pixel value distribution of the two frames respectively;

[0078] μ x and μ y are the pixel means of x and y respectively;

[0079] σ x 2 and σ y 2 are the pixel variances of x and y respectively;

[0080] σ xy is the pixel covariance of x and y;

[0081] C1 and C2 are constants used for stable calculations, usually C1 = (K1L) 2 and C2=(K2L) 2 , where L is the pixel value range (such as 255), K1=0.01 and K2=0.03 are constants.

[0082] First, for each video clip captured by each student, the structural similarity index (SSIM) between each pair of frames is calculated.

[0083] Assume that the video frame sequence is I1, I2, ..., I n , where n is the number of frames.

[0084] Still reference Figure 3 , for video segment A, assuming the frame sequence is I A1 , I A2 , ..., I An ; For video segment B, assume that the frame sequence is I B1 , I B2, ..., I Bn .

[0085] Inter-frame SSIM score between video clip A and video clip B:

[0086]

[0087] This method is used to evaluate the structural similarity between video clips captured at different time periods. This method takes into account the similarity between frames within the video clips and comprehensively considers the similarity between video clips captured by different student terminals.

[0088] The following is a Python example code in an embodiment of the present invention, demonstrating the use of the structural similarity index (SSIM) as a similarity metric. In this example, we still refer to Figure 3 The video segments A and B in the figure are assumed to be from the 12th to the 18th minute, and the video segment B is assumed to be from the 13th to the 16th minute. The OpenCV library is used to process the video frames and calculate the SSIM value.

[0089]

[0090]

[0091]

[0092] In summary, the average SSIM score is used as an indicator of similarity. The closer it is to 1, the higher the similarity. According to the calculation results of the structural similarity of different video frames, multiple video frames with high structural similarity are selected as key frames, and the timestamp of each key frame is marked on the timeline of the teaching video, so that the teacher can refer to find the time range where the problems are concentrated. In some embodiments of the present invention, as to how to define a high structural similarity, an SSIM score threshold can be selected, and the SSIM scores between video frames above the threshold are regarded as having a high structural similarity for subsequent processing. Regarding the size of the SSIM score threshold, reference can be made to the historical operation data of different application scenarios, and the present invention does not make specific limitations.

[0093] In combination with the above, while calculating the structural similarity based on the video frames extracted from different video clips and finding the time period when the problem occurred, the voice bar information corresponding to the different video clips raised by the student end is extracted, and the words in the different voice bar information are extracted, and then the keywords that best represent the questions asked by the students are found among multiple words.

[0094] Still reference Figure 3, suppose the voice message corresponding to video clip A is: "How to solve the matrix inversion problem in linear algebra?"; the voice message corresponding to video clip B is: "What is overfitting in machine learning and how to avoid it?"

[0095] The speech information corresponding to the video segments A and B is converted into corresponding separate documents, multiple words in the documents are extracted, and the TF-IDF value of each word in the document is calculated. The TF-IDF value is the product of the word frequency and the inverse document frequency.

[0096] Combine the words and their corresponding TF-IDF values ​​in each individual document. It should be noted that for the same words, their TF-IDF values ​​can be added together to obtain the comprehensive importance over the entire time range. The following is a Python sample code in an embodiment of the present invention, demonstrating how to calculate the TF-IDF value.

[0097]

[0098]

[0099] According to the merged words and their comprehensive TF-IDF values, the words with the highest TF-IDF value in the covered time range are found. In some feasible embodiments of the present invention, the words with the highest TF-IDF values ​​are used as keywords, and the keywords represent the expressions of the core topics that the students are concerned about as a whole.

[0100] The words for the problem corresponding to video clip A: linear algebra, matrix, inversion;

[0101] The corresponding TF-IDF values ​​are set to 0.05, 0.03, and 0.07 respectively.

[0102] The words corresponding to the problem in video clip B: machine learning, overfitting, avoidance;

[0103] The corresponding TF-IDF values ​​are set to 0.06, 0.04, and 0.08 respectively.

[0104] Then the merged words and their TF-IDF values ​​are: linear algebra (0.05), matrix (0.03), inversion (0.07), machine learning (0.06), overfitting (0.08), avoidance (0.04).

[0105] It is understandable that the keywords in the entire time range can be found based on the size of the TF-IDF value: "overfitting" (TF-IDF value is 0.08);

[0106] This shows that students paid special attention to the issue of "how to avoid overfitting" during this period.

[0107] On the basis of the above, consider how to facilitate the student end to control the start and stop of the video capture module to capture video clips, and control the start or stop of the voice capture module to capture voice information while the video capture module starts or stops.

[0108] In some practicable embodiments of the present invention, reference is made to Figures 4 to 7 ,in, Figure 5 The wearable devices provided in the embodiments of the present invention include a wearable left glove 100 and a wearable right glove 200. A first pressure sensor 110 is arranged near the palm of the hand on the wearable left glove 100, and a second pressure sensor 210 is arranged near the palm of the hand on the wearable right glove 200. It should be noted that the first pressure sensor 110 and the second pressure sensor 210 are arranged at positions convenient for the thumb to bend and touch.

[0109] As mentioned above, the models of the first pressure sensor 110 and the second pressure sensor 210 are not limited in the present invention, and they only need to meet the pressure range that can detect the bending of the finger pressing the palm.

[0110] like Figure 4 As shown, the start-stop module that is connected to the video capture module and the voice capture module and controls their operation includes a first pressure sensor 110 and a second pressure sensor 210. The first pressure sensor 110 is used to control the operation of the video capture module, and the second pressure sensor 210 is used to control the operation of the voice capture module.

[0111] More specifically, refer to Figure 6 , Figure 6 The first pressure sensor 110 on the wearable left glove 100 and the second pressure sensor 210 on the wearable right glove 200 provided in the embodiment of the present invention are both in a continuous pressure detection state, and a schematic diagram of an application scenario in which the video capture module and the voice capture module are started. Considering the possibility of accidentally touching the sensor, in some feasible embodiments of the present invention, the pressure sensors on the wearable left glove 100 and the wearable right glove 200 are each in a continuous pressure detection state, and the time interval between two continuous pressures is detected to be less than a set time value, for example, the set time value is 2s, that is, the time interval between two continuous pressures is detected to be less than 2s, the video capture module starts to capture video clips, and the voice capture module starts to capture voice bar information. The advantage of such a setting is to prevent the captured data from being cluttered due to accidental touches.

[0112] For information on how to stop the voice capture module from working, refer to Figure 7 , Figure 7A schematic diagram of an application scenario in which the first pressure sensor 110 of the wearable left glove 100 provided in an embodiment of the present invention is in a continuous pressure detection state, the second pressure sensor 210 of the wearable right glove 200 is not in a pressure detection state, the video capture module continues to work, and the voice capture module stops working. That is, when the voice capture of the student end is completed, but it is still necessary to continue to capture the video clip, it is only necessary to stop pressing the second pressure sensor 210 with the right thumb and continue pressing the first pressure sensor 110 with the left thumb, that is, the recording of the voice bar can be stopped while the video clip is being recorded.

[0113] When the video clip is captured, refer to Figure 5 When the left and right thumbs no longer press the pressure sensor, the video capture module also stops capturing and recording the video.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. The patent protection scope of the present invention shall be based on the claims. All equivalent structural changes made using the contents of the description and drawings of the present invention should also be included in the protection scope of the present invention.

Claims

1. A VR experience system based on education and training, characterized in that: include: A VR playback device is used to play the teaching video and divide the teaching video into a plurality of continuous time periods; A video capture module, used to capture and record video clips while playing the teaching video; A voice capture module, used to capture voice information while capturing and recording the video clip; A data processing unit, which divides the plurality of video clips into a plurality of different comparison groups according to the time information of the video clips, extracts the video frame data of each video clip in the comparison group, calculates the structural similarity between different video frames, and marks the corresponding time of the data frames with high structural similarity on the time axis of the teaching video; And extract the words in the voice bar information, calculate the product of the word frequency and the inverse document frequency of each word, and mark the word with the highest product value as a keyword; wherein the time information is specifically the distribution period and start and end time points of each video clip on the timeline based on the teaching video, as well as the duration of each video clip.

2. The VR experience system based on education and training according to claim 1, characterized in that: The data processing unit divides the plurality of video clips into a plurality of different comparison groups according to the time information of the video clips, including the following steps: In the time period of the same teaching video, the video segment that overlaps with the time period is selected, and the relative duration of the video segment in the time period exceeds the first set value; After selecting a plurality of video clips whose duration ratios exceed the first set value, a plurality of video clips whose duration ranges and start and end time points are close are taken as a comparison group.

3. A VR experience system based on education and training according to claim 2, characterized in that: The data processing unit is in communication with the VR playback device, and the data processing unit extracts and calculates the structural similarity between different video frames in the comparison group; According to the calculation results of the structural similarity between the different video frames, multiple video frames with higher structural similarity are selected as key frames; Mark the time corresponding to each key frame on the timeline of the teaching video.

4. The VR experience system based on education and training according to claim 3, characterized in that: The data processing unit calculates the structural similarity between the video frames by calculating the SSIM scores between different video frames in each comparison group; specifically: The SSIM formula is used to calculate the structural similarity between video frames; the calculation formula is as follows: Where x and y represent the pixel value distribution of the two frames respectively; μ x and μ y are the pixel means of x and y respectively; σ x 2 and σ y 2 are the pixel variances of x and y, respectively; xy is the pixel covariance of x and y; C1 and C2 are constants; Averaging the SSIM scores of the video frames in each comparison group to obtain an average structural similarity index between each of the video clips; The average SSIM score is used as an indicator of the structural similarity between different video frames. The closer the average SSIM score is to 1, the more similar they are.

5. The VR experience system based on education and training according to claim 1, characterized in that: The data processing unit extracts words from the voice bar information and analyzes the product of the word frequency and the inverse document frequency of each word, specifically including the following steps: Convert each of the speech pieces of information in each of the comparison groups into a corresponding separate document, extract multiple words in the document, and calculate the product of the word frequency and the inverse document frequency of each of the words in the document, where the product of the word frequency and the inverse document frequency is the TF-IDF value; The words in each individual document in a comparison group are merged to find the words with the highest TF-IDF value within the time range covered by different video clips in the comparison group, and the words with the highest TF-IDF value are marked as keywords.

6. The VR experience system based on education and training according to claim 1, characterized in that: Also included is a start-stop module, the start-stop module is used to control the start and stop of the video capture module capturing video clips; and While the video capture module is working, the voice capture module is controlled to start and stop capturing voice bar information.

7. The VR experience system based on education and training according to claim 6, characterized in that: A wearable device is also included, the start-stop module includes at least one sensor, and the sensor is located on the wearable device, and the data processing unit is communicatively connected with the start-stop module.

8. The VR experience system based on education and training according to claim 7, characterized in that: The wearable device includes two wearable gloves suitable for left and right hands, and the start-stop module includes pressure sensors respectively located on the wearable left glove and the wearable right glove.

9. The VR experience system based on education and training according to claim 8, characterized in that: The pressure sensors on the wearable left glove and the wearable right glove are each in a continuous pressure detection state, and the time interval for each detecting the continuous pressure is less than the set time value, the video capture module starts to capture the video clip, and the sound capture module starts to capture the sound bar information.

10. A VR experience system based on education and training according to claim 9, characterized in that: The pressure sensor on the wearable left glove or the wearable right glove stops the pressure detection state, and the video capture module or the sound capture module stops capturing.

Images