A new form of teaching material learning system based on digital twins

Abstract

The present invention relates to artificial intelligence technology, and discloses a new form of teaching material learning system based on digital twins, including: performing field prediction on the teacher speech audio converted from text, obtaining the speech field corresponding to the teacher speech audio, and selecting the target offline teaching material consistent with the speech field in the offline teaching material library. Identify the teaching material content data corresponding to the speech request in the target offline teaching material, identify the data type of the teaching material content data, generate virtual dialogue videos or stereoscopic images according to different data types, and add the virtual dialogue videos and stereoscopic images to the online teaching materials respectively, to obtain optimized teaching materials, perform teaching material writing on the teaching material content data and the optimized online teaching materials, obtain new form teaching materials and conduct classroom teaching. The present invention also proposes a new form teaching material learning device, electronic device and computer-readable storage medium based on digital twins. The present invention can solve the problem of low accuracy of teaching material optimization.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a new form of teaching material learning system based on digital twins, and more specifically, to a new form of teaching material learning method based on digital twins. Background Art

[0002] With the development of science and technology, online teaching has become an important way to supplement classroom teaching. The current online teaching is simply a way for teachers to conduct live teaching and students to learn. The more important part of online teaching is the production of online teaching materials. The boring online teaching materials cannot mobilize the enthusiasm of students in online teaching. At the same time, the current online teaching materials lack the combination with students' offline teaching materials, which leads to low students' learning initiative. Therefore, it is necessary to propose a teaching material optimization method.

[0003] Existing textbook optimization methods usually directly copy and paste the corresponding content from offline textbooks into online textbooks. The types of resources that digital textbooks under existing technologies support for insertion into textbooks are limited. In addition to the main text of the textbooks, they are restricted to the use of documents, pictures, audio and video, etc. to present them. They cannot help learners better understand the textbook content from the perspective of interaction or operational experience. Therefore, this method cannot improve the fun and accuracy of textbook optimization. Therefore, it is urgent to propose a more accurate textbook optimization method. Summary of the invention

[0004] The present invention provides a new form of teaching material learning system based on digital twins, the main purpose of which is to solve the problem of low accuracy of teaching material optimization.

[0005] To achieve the above purpose, the present invention provides a new form of teaching material learning method based on digital twins, comprising:

[0006] Obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio;

[0007] Selecting an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identifying teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio;

[0008] Identify the data type of the teaching material content data, and when the teaching material content data is of a dialogue type, iteratively optimize a preset 3D face model using pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video;

[0009] Adding the virtual conversation video to a pre-built online teaching material to obtain an optimized online teaching material;

[0010] When the teaching material content data is of a non-dialog type, performing named entity recognition on the teaching material content data, and retrieving a related two-dimensional plane image according to the entity name obtained by the named entity recognition;

[0011] Calculating a depth map of the two-dimensional plane image, generating a stereoscopic image based on a Markov random field model and the depth map, and adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material;

[0012] The teaching material content data is stored in a preset database, and a preset teaching material editor is used to edit the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material;

[0013] When a teaching preparation request is received, the new form teaching material is called to conduct classroom teaching.

[0014] Optionally, calculating the depth map of the two-dimensional plane image includes:

[0015] Dividing the two-dimensional plane image into a preset number of plane rectangular blocks;

[0016] Performing filtering processing on a preset number of planar rectangular blocks respectively to obtain a plurality of filtered rectangular blocks;

[0017] Calculating the absolute depths of the plurality of filtering rectangular blocks using a preset depth calculation formula to obtain a depth feature;

[0018] A depth map of the two-dimensional plane image is generated according to the depth feature.

[0019] Optionally, generating a stereoscopic image based on the Markov random field model and the depth map includes:

[0020] Performing super-resolution processing and denoising on the depth map to obtain a denoised depth map;

[0021] The two-dimensional plane image is defined as a right viewpoint image, a disparity map between the right viewpoint image and the denoised depth map is established using a Markov random field model, and a stereoscopic image is generated according to the disparity map.

[0022] Optionally, performing named entity recognition on the teaching material content data includes:

[0023] Dividing the textbook content data into a plurality of textbook segmentation words, and respectively vectorizing the plurality of textbook segmentation words to obtain a plurality of segmentation vectors;

[0024] Inputting the plurality of word segmentation vectors into a long short-term memory network for feature extraction to obtain a feature sequence representation;

[0025] The feature sequence representation is input into the CRF layer for decoding to obtain the sequence corresponding to each textbook word segmentation;

[0026] The corresponding entity name is determined according to the sequence corresponding to the textbook segmentation words.

[0027] Optionally, the iterative optimization of a preset 3D face model using the pre-acquired conversation audio data to obtain a standard face model includes:

[0028] Splitting the conversation audio data into phonemes, and performing vectorization processing on the phonemes to obtain phoneme sequence vectors;

[0029] Using a bidirectional long short-term memory network in a preset 3D face model to perform a first sequence transformation process on the phoneme sequence vector to obtain a mouth shape feature sequence;

[0030] Performing a second sequence transformation process on the mouth shape feature sequence to obtain facial expression parameters of the human face;

[0031] The 3D face model is iteratively optimized according to the face expression parameters to obtain a standard face model.

[0032] Optionally, splitting the conversation audio data into phonemes comprises:

[0033] Converting the conversation audio data into text to obtain an initial conversation text;

[0034] Removing punctuation marks from the initial dialogue text to obtain a standard dialogue text;

[0035] The phoneme corresponding to each character in the standard dialogue text is marked using a preset phonetic symbol rule.

[0036] Optionally, the identifying the teaching material content data in the target offline teaching material corresponding to the speech request in the teacher speech audio includes:

[0037] Extracting a request action in a speech request in the teacher's speech audio, and locating the target offline teaching material to a corresponding page number according to the request action;

[0038] A text extraction tool is used to extract text content from pages of the target offline teaching material corresponding to the page number, and the text content is used as teaching material content data.

[0039] Optionally, before using the trained domain prediction model to perform domain prediction on the teacher's speech text, the method further includes:

[0040] Perform word frequency analysis on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix;

[0041] Based on a plurality of preset field labels, the word frequency matrix is ​​randomly numbered to obtain a field number corresponding to the word frequency matrix;

[0042] Using the domain number as an input value of the domain prediction model, and calculating the Gibbs sampling value and the Dirichlet parameter estimation value of the domain prediction model;

[0043] The Gibbs sampling value and the Dirichlet parameter estimation value are used as model parameters of the domain prediction model to obtain a trained domain prediction model.

[0044] Optionally, performing word frequency analysis on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix includes:

[0045] Performing word segmentation processing on the training speeches in the training speech set to obtain a training word segmentation set;

[0046] Filtering the training segmented words in the training segmented word set that are consistent with the keywords in the preset keyword library as training keywords, and summarizing the training keywords to obtain a training keyword set;

[0047] The keyword identifier and keyword frequency corresponding to each keyword in the training keyword set are obtained, and a corresponding frequency matrix is ​​constructed according to the keyword identifier and the keyword frequency.

[0048] In order to solve the above problems, the present invention also provides a new form of teaching material learning device based on digital twins, the device comprising:

[0049] A domain recognition module is used to obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, and use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio;

[0050] A content extraction module, configured to select an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identify teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio;

[0051] A first teaching material optimization module is used to identify the data type of the teaching material content data. When the teaching material content data is of a dialogue type, iteratively optimize the preset 3D face model using the pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video, and add the virtual dialogue video to the pre-built online teaching material to obtain an optimized online teaching material;

[0052] The second teaching material optimization module is used for performing named entity recognition on the teaching material content data when the teaching material content data is of a non-dialog type, and retrieving relevant two-dimensional plane images according to the entity names obtained by the named entity recognition, calculating the depth map of the two-dimensional plane image, and generating a stereoscopic image based on a Markov random field model and the depth map, adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material, storing the teaching material content data in a preset database, and using a preset teaching material editor to compile the teaching material content data in the preset database and the optimized online teaching material to obtain a new form teaching material, and when a teaching preparation request is received, calling the new form teaching material for classroom teaching.

[0053] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0054] a memory storing at least one instruction; and

[0055] A processor executes instructions stored in the memory to implement the new form of teaching material learning method based on digital twins as described above.

[0056] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned new form of teaching material learning method based on digital twins.

[0057] In an embodiment of the present invention, by performing field prediction on the teacher's speech audio converted into text, the speech field corresponding to the teacher's speech audio is obtained, and the target offline teaching material consistent with the speech field is selected in the offline teaching material library, so as to achieve accurate selection. The teaching material content data corresponding to the speech request in the target offline teaching material is identified, and the data type of the teaching material content data is identified, virtual dialogue video generation or stereoscopic image generation is performed according to different data types, and the virtual dialogue video and the stereoscopic image are added to the online teaching material to obtain an optimized teaching material, the teaching material content data is stored in a preset database, and the teaching material content data in the preset database and the optimized online teaching material are compiled using a preset teaching material editor to obtain a new form teaching material, and when a teaching preparation request is received, the new form teaching material is called for classroom teaching, so as to achieve the optimization processing of the online teaching material. Therefore, the new form teaching material learning method, device, electronic device and computer-readable storage medium based on digital twins proposed in the present invention can solve the problem of low accuracy of teaching material optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a new form of teaching material learning method based on digital twins provided in one embodiment of the present invention;

[0059] Figure 2 A functional module diagram of a new form of teaching material learning device based on digital twins provided in one embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of an electronic device for implementing the new form of teaching material learning method based on digital twins provided in one embodiment of the present invention.

[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0063] The embodiment of the present application provides a new form of teaching material learning method based on digital twins. The execution subject of the new form of teaching material learning method based on digital twins includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the new form of teaching material learning method based on digital twins can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0064] Reference Figure 1 As shown, it is a flow chart of a new form of teaching material learning method based on digital twins provided by an embodiment of the present invention. In this embodiment, the new form of teaching material learning method based on digital twins includes:

[0065] S1. Obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text, and obtain the speech domain corresponding to the teacher's speech audio.

[0066] In an embodiment of the present invention, the teacher speech audio refers to the command voice issued by the teacher in the teacher's classroom. Since the obtained teacher speech audio may not be accurate in terms of hearing, it is necessary to convert the teacher speech audio into text and perform subsequent data processing based on the obtained teacher speech text.

[0067] Preferably, the teacher's speech audio can be converted into text according to speech-to-text technology, wherein the speech-to-text technology (Automatic Speech Recognition) takes speech as the research object, and allows the machine to automatically recognize and understand human spoken language through speech signal processing and pattern recognition. Speech recognition technology is a high technology that allows the machine to convert speech signals into corresponding text or commands through the recognition and understanding process. Speech recognition is a cross-disciplinary subject with a wide range of implications. It has a very close relationship with subjects such as acoustics, phonetics, linguistics, information theory, pattern recognition theory, and neurobiology.

[0068] Specifically, before using the trained domain prediction model to perform domain prediction on the teacher's speech text, the method further includes:

[0069] Perform word frequency analysis on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix;

[0070] Based on a plurality of preset field labels, the word frequency matrix is ​​randomly numbered to obtain a field number corresponding to the word frequency matrix;

[0071] Using the domain number as an input value of the domain prediction model, and calculating the Gibbs sampling value and the Dirichlet parameter estimation value of the domain prediction model;

[0072] The Gibbs sampling value and the Dirichlet parameter estimation value are used as model parameters of the domain prediction model to obtain a trained domain prediction model.

[0073] Furthermore, the word frequency analysis is performed on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix, including:

[0074] Performing word segmentation processing on the training speeches in the training speech set to obtain a training word segmentation set;

[0075] Filtering the training segmented words in the training segmented word set that are consistent with the keywords in the preset keyword library as training keywords, and summarizing the training keywords to obtain a training keyword set;

[0076] The keyword identifier and keyword frequency corresponding to each keyword in the training keyword set are obtained, and a corresponding frequency matrix is ​​constructed according to the keyword identifier and the keyword frequency.

[0077] In detail, the word segmentation processing can be performed by a probability-based statistical word segmentation model, or the word segmentation processing can be performed by a jieba word segmentation model. The preset keyword library contains multiple topic-related keywords. For example, the corresponding keywords under the interest topic are "astronomy", "swimming", "taekwondo" and "running". The multiple keywords are randomly sorted, and the ID values ​​corresponding to each keyword are assigned in the order of random sorting. For example, the word ID corresponding to "astronomy" is 0 and the word frequency is 8 (the word ID corresponding to "astronomy" is 0 and the word frequency is 8, forming the matrix element (0, 8) corresponding to "astronomy"), the word ID corresponding to "swimming" is 1 and the word frequency is 5, the word ID corresponding to "taekwondo" is 2 and the word frequency is 4, and the word ID corresponding to "running" is 3 and the word frequency is 3. Then the word frequency matrix corresponding to the keyword set is [(0, 8)(1, 5), (2, 4), (3, 3)...].

[0078] S2. Select an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identify teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio.

[0079] In an embodiment of the present invention, the preset offline teaching material library includes multiple offline teaching materials in different fields, wherein the multiple offline teaching materials in different fields may be popular science books related to astronomy, popular science data related to military affairs, cooking data related to cooking, etc. When the speech field is the astronomy field, popular science books related to astronomy are selected from the offline teaching material library.

[0080] Specifically, the identifying the teaching material content data in the target offline teaching material corresponding to the speech request in the teacher speech audio includes:

[0081] Extracting a request action in a speech request in the teacher's speech audio, and locating the target offline teaching material to a corresponding page number according to the request action;

[0082] A text extraction tool is used to extract text content from pages of the target offline teaching material corresponding to the page number, and the text content is used as teaching material content data.

[0083] In detail, the teacher's speech audio usually contains many aspects of content, such as calling a classmate's name first, or containing some interactive words, but these contents have no actual meaning. Therefore, it is necessary to extract the request action in the speech request in the teacher's speech audio and make corresponding responses according to the request action.

[0084] For example, the teacher's speech audio is "Hello, students, please turn to page 23." The extraction request action is "turn to page 23". According to the request action, the target offline teaching material is located to page 23, and the text content on page 23 is identified as teaching material content data using a text extraction tool.

[0085] Preferably, the text extraction tool may be a text extraction applet or the like.

[0086] S3. Identify the data type of the teaching material content data. When the teaching material content data is of a dialogue type, iteratively optimize the preset 3D face model using the pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video.

[0087] In an embodiment of the present invention, the teaching material content data may be of different types. For example, the teaching material content data may be a conversation between two people, in which case the teaching material content data is of a conversation type. Alternatively, the teaching material content data may be popular science text, in which case the teaching material content data is of a non-conversation type.

[0088] Specifically, the iterative optimization of the preset 3D face model using the pre-acquired conversation audio data to obtain the standard face model includes:

[0089] Splitting the conversation audio data into phonemes, and performing vectorization processing on the phonemes to obtain phoneme sequence vectors;

[0090] Using a bidirectional long short-term memory network in a preset 3D face model to perform a first sequence transformation process on the phoneme sequence vector to obtain a mouth shape feature sequence;

[0091] Performing a second sequence transformation process on the mouth shape feature sequence to obtain facial expression parameters of the human face;

[0092] The 3D face model is iteratively optimized according to the face expression parameters to obtain a standard face model.

[0093] Further, splitting the conversation audio data into phonemes includes:

[0094] Perform text conversion on the dialogue audio data to obtain an initial dialogue text;

[0095] Remove punctuation marks from the initial dialogue text to obtain a standard dialogue text;

[0096] Mark the phoneme corresponding to each character in the standard dialogue text using a preset phonetic symbol rule.

[0097] Specifically, in the embodiment of the present application, a text conversion software is used to perform text conversion on the dialogue audio data to obtain an initial dialogue text. Since the text conversion software will mark punctuation marks according to the pauses in the speech in the dialogue audio data, there are still multiple punctuation marks in the initial dialogue text. In order to facilitate subsequent splitting of the text into phonemes, it is necessary to remove the punctuation marks from the initial dialogue text.

[0098] Further, according to the one-to-one correspondence between characters and phonemes in the preset phonetic symbol rule, each character in the standard text is marked to obtain phonemes.

[0099] For example, in the embodiment of the present application, the preset phonetic symbol rule is the International Phonetic Alphabet rule. When marking the character "啊", the corresponding phoneme is a, and the obtained phoneme is [a].

[0100] Further, in another embodiment of the present application, the vectorization process is performed on the phonemes to obtain a phoneme sequence vector, including:

[0101] Use a preset unsupervised model to perform vectorization processing on the phonemes to obtain a phoneme sequence vector.

[0102] Further, the preset unsupervised model can be word2vec, doc2vec, fasttext, etc. In the embodiment of the present application, the preset unsupervised model is the Skip-gram model in word2vec. Among them, word2vec is a type of neural network model. Based on this model, a semantic vector, that is, a word vector, can be generated for a single word segmentation in a corpus.

[0103] In the embodiment of the present application, the preset 3D face model can be composed of two bidirectional LSTM networks (Bi-directional long Short-Term Memory). The bidirectional long short-term memory network is a type of time-recurrent neural network, including: an input gate, a forget gate, and an output gate.

[0104] In the embodiment of the present application, the iterative optimization of the 3D face model according to the facial expression parameters of the face to obtain a standard face model includes:

[0105] Calculating the loss value between the facial expression parameters of the human face and the preset real facial parameters;

[0106] When the loss value is less than a preset face threshold, outputting the 3D face model as an initial face model;

[0107] When the loss value is greater than or equal to the preset face threshold, the internal parameters of the 3D face model are adjusted, and the step of performing the first sequence transformation processing on the phoneme sequence vector using the bidirectional long short-term memory network in the preset 3D face model is returned until the loss value is less than the preset face threshold, and the output is a standard face model.

[0108] The internal parameter may be a gradient parameter or a weight parameter of the model.

[0109] Specifically, the teaching material content data is input into the standard face model to obtain a virtual dialogue video, which can present the teaching material content data in the form of a voice broadcast by a virtual anchor, which has a richer presentation form and is easier to understand in class.

[0110] S4. Add the virtual dialogue video to the pre-built online teaching material to obtain the optimized online teaching material.

[0111] In an embodiment of the present invention, the online teaching material may be a PPT, and the virtual dialogue video is inserted into the online teaching material, i.e., the PPT, to obtain an optimized online teaching material. The optimized online teaching material can help students better understand the content presented in the teaching material and increase the fun of the class.

[0112] S5. When the teaching material content data is of a non-dialog type, named entity recognition is performed on the teaching material content data, and a related two-dimensional plane image is retrieved according to the entity name obtained by the named entity recognition.

[0113] In an embodiment of the present invention, when the teaching material content data is of a non-dialog type, it indicates that the teaching material content data mostly contains large paragraphs of text content. Therefore, named entities in the teaching material content data can be identified and processed to enrich the online teaching materials.

[0114] Specifically, the performing named entity recognition on the teaching material content data includes:

[0115] Dividing the textbook content data into a plurality of textbook segmentation words, and respectively vectorizing the plurality of textbook segmentation words to obtain a plurality of segmentation vectors;

[0116] Inputting the plurality of word segmentation vectors into a long short-term memory network for feature extraction to obtain a feature sequence representation;

[0117] The feature sequence representation is input into the CRF layer for decoding to obtain the sequence corresponding to each textbook word segmentation;

[0118] The corresponding entity name is determined according to the sequence corresponding to the textbook segmentation words.

[0119] In detail, the LSTM (Long Short-Term Memory) network is a time recurrent neural network. The CRF layer refers to a conditional random field, which can constrain the output data of the LSTM network so that the final entity name is more accurate.

[0120] The entity names for named entity recognition may be proper nouns in the military field, such as aircraft carriers, tanks, and airplanes.

[0121] Furthermore, by searching on the Internet based on the entity names obtained by named entity recognition, multiple related two-dimensional plane images can be obtained, wherein the two-dimensional plane images are taken from different angles or different scenes.

[0122] For example, the two-dimensional plane image may be a front view of an aircraft carrier, or a side view of an aircraft carrier, etc.

[0123] S6. Calculate the depth map of the two-dimensional plane image, generate a stereoscopic image based on the Markov random field model and the depth map, and add the stereoscopic image to the online teaching material to obtain an optimized online teaching material.

[0124] In the embodiment of the present invention, a Gaussian probability model in a Markov random field (MRF) model is used as a priori probability model to describe the relationship between the depth feature of an image and a depth map.

[0125] Specifically, calculating the depth map of the two-dimensional plane image includes:

[0126] Dividing the two-dimensional plane image into a preset number of plane rectangular blocks;

[0127] Performing filtering processing on a preset number of planar rectangular blocks respectively to obtain a plurality of filtered rectangular blocks;

[0128] Calculating the absolute depths of the plurality of filtering rectangular blocks using a preset depth calculation formula to obtain a depth feature;

[0129] A depth map of the two-dimensional plane image is generated according to the depth feature.

[0130] In detail, the preset number may be nine. The purpose of dividing the two-dimensional plane image into a preset number of plane rectangular blocks is to divide the image into small rectangular blocks so that each rectangular block corresponds to a depth feature. An edge detection filter may be used to filter the plane rectangular blocks to obtain a plurality of filtered rectangular blocks.

[0131] Furthermore, the calculating the absolute depths of the plurality of filtered rectangular blocks by using a preset depth calculation formula includes:

[0132] The preset depth calculation formula is:

[0133] E i (n) = ∑ (x,y)∈S |I(x,y) * F n (x,y)| q

[0134] Among them, E i (n) is the deep feature, q∈{1,,2}, I(x,y) * is the sum of the absolute energies of the filtered rectangular blocks, F n (x, y) is the absolute energy of the filtered rectangular block.

[0135] Specifically, the generating of a stereoscopic image based on the Markov random field model and the depth map includes:

[0136] Performing super-resolution processing and denoising on the depth map to obtain a denoised depth map;

[0137] The two-dimensional plane image is defined as a right viewpoint image, a disparity map between the right viewpoint image and the denoised depth map is established using a Markov random field model, and a stereoscopic image is generated according to the disparity map.

[0138] Specifically, unlike super-resolution of ordinary images, depth maps can be processed by super-resolution based on the information contained in the original image. Since the depth map after super-resolution processing has the same resolution and clear object contours as the two-dimensional plane image, but at the same time introduces noise, denoising is required. This solution uses the histogram statistics method to find noise points in the depth map and achieve denoising.

[0139] Preferably, the generated stereogram can enable the viewer to perceive the depth information of the objects and scenes, and appreciate the spatial positional relationship between the objects in the image in terms of front and back, up and down, and orientation, thus bringing a more realistic and vivid visual experience.

[0140] Furthermore, the stereoscopic image is added to the online teaching material, and the stereoscopic image can enrich the online teaching material, so that the original plane image is transformed into a stereoscopic image and added to the online teaching material, so that teachers can introduce and interact with students more accurately, thereby ensuring the optimization of the online teaching material.

[0141] S7. The teaching material content data is stored in a preset database, and a preset teaching material editor is used to compile the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material.

[0142] In an embodiment of the present invention, the textbook content data is stored in a preset database, which can ensure that the textbook content data can be called from the preset database during subsequent data processing, thereby improving the efficiency of textbook content data search. The preset textbook editor is used to compile the textbook content data in the preset database and the optimized online textbook, wherein the preset textbook editor is a dedicated textbook editor that can complete the production of new forms of electronic textbooks. Through the textbook editor, a variety of template styles are inserted into the textbook; digital resources are combined with integrated media textbooks to form modern digital integrated media textbooks for online, offline, and online and offline combined teaching, learning and other scenarios. The rich template styles include title templates, paragraph templates, question and answer templates, etc., which allow teachers to quickly complete digital templates through convenient operations, making it easier to update and iterate textbooks.

[0143] Specifically, the obtained new form of teaching materials is uploaded to the server for storage. The new form of teaching materials supports pictures, audio, video, two-dimensional animation, three-dimensional animation, virtual interactive models, etc. When uploaded to the server, the information and data involved in the new form of teaching materials will also be stored in the database.

[0144] S8. When a teaching preparation request is received, the new form teaching material is called to conduct classroom teaching.

[0145] In an embodiment of the present invention, the teaching preparation request refers to an instruction that the teacher is ready to perform teaching actions in the teaching scene. The teacher will call the required teaching materials according to his own teaching scene, and the students can install the new form teaching materials into the reader in the device to read the teaching materials. Students use the built-in components of the reader to view various resources in the teaching materials. In addition to conventional pictures, audio and video resources, they can also zoom in and out, split and other operations on 3D models, so as to understand the content learned more intuitively and deeply. At the same time, students use the reader installed in the device to read the teaching materials. Students realize immersive and interactive learning through structured resources and digital teaching materials, complete different learning scenes and complete tasks such as teacher homework through the reader, improve students' learning interest and improve students' learning quality. In detail, this solution realizes a quick jump from paper teaching materials to the corresponding content of digital teaching materials through a variety of interactive forms, and supports a variety of interactive methods such as audio and video playback, 3D model interaction, virtual simulation training, etc. Achieve a teaching effect of virtual-real interaction and the combination of theory and practice. Through this innovative learning model, the way of talent cultivation is upgraded from one-way indoctrination to perceptual interaction, and the form of learning experience is upgraded from 2D plane to 3D stereoscopic interaction, so as to achieve structured integration and most efficient use of digital resources, greatly improving the effect of knowledge learning and skills training.

[0146] In an embodiment of the present invention, by performing field prediction on the teacher's speech audio after text conversion, the speech field corresponding to the teacher's speech audio is obtained, and the target offline teaching material consistent with the speech field is selected in the offline teaching material library, so as to achieve accurate selection. The teaching material content data corresponding to the speech request in the target offline teaching material is identified, and the data type of the teaching material content data is identified, virtual dialogue video generation or stereoscopic image generation is performed according to different data types, and the virtual dialogue video and the stereoscopic image are added to the online teaching material to obtain an optimized teaching material, the teaching material content data is stored in a preset database, and the teaching material content data in the preset database and the optimized online teaching material are compiled using a preset teaching material editor to obtain a new form teaching material, and when a teaching preparation request is received, the new form teaching material is called for classroom teaching, so as to achieve the optimization processing of the online teaching material. Therefore, the new form teaching material learning method based on digital twins proposed in the present invention can solve the problem of low accuracy of teaching material optimization.

[0147] like Figure 2 As shown, it is a functional module diagram of a new form of teaching material learning device based on digital twins provided by one embodiment of the present invention.

[0148] The new form of teaching material learning device 100 based on digital twins of the present invention can be installed in an electronic device. According to the functions implemented, the new form of teaching material learning device 100 based on digital twins can include a domain identification module 101, a content extraction module 102, a first teaching material optimization module 103 and a second teaching material optimization module 104. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0149] In this embodiment, the functions of each module / unit are as follows:

[0150] The domain identification module 101 is used to obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, and use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio;

[0151] The content extraction module 102 is used to select an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identify teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio;

[0152] The first teaching material optimization module 103 is used to identify the data type of the teaching material content data. When the teaching material content data is of a dialogue type, the pre-acquired dialogue audio data is used to iteratively optimize the preset 3D face model to obtain a standard face model, and the teaching material content data is input into the standard face model to obtain a virtual dialogue video, and the virtual dialogue video is added to the pre-built online teaching material to obtain an optimized online teaching material;

[0153] The second teaching material optimization module 104 is used to perform named entity recognition on the teaching material content data when the teaching material content data is of a non-dialog type, and retrieve relevant two-dimensional plane images according to the entity names obtained by the named entity recognition, calculate the depth map of the two-dimensional plane image, and generate a stereoscopic image based on the Markov random field model and the depth map, add the stereoscopic image to the online teaching material to obtain an optimized online teaching material, store the teaching material content data in a preset database, and use a preset teaching material editor to compile the teaching material content data in the preset database and the optimized online teaching material to obtain a new form teaching material, and when a teaching preparation request is received, call the new form teaching material for classroom teaching.

[0154] In detail, the specific implementation of each module of the new form teaching material learning device 100 based on digital twin is as follows:

[0155] Step 1: obtain the teacher's speech audio, convert the teacher's speech audio into text to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text, and obtain the speech domain corresponding to the teacher's speech audio.

[0156] In an embodiment of the present invention, the teacher speech audio refers to the command voice issued by the teacher in the teacher's classroom. Since the obtained teacher speech audio may not be accurate in terms of hearing, it is necessary to convert the teacher speech audio into text and perform subsequent data processing based on the obtained teacher speech text.

[0157] Preferably, the teacher's speech audio can be converted into text according to speech-to-text technology, wherein the speech-to-text technology (Automatic Speech Recognition) takes speech as the research object, and allows the machine to automatically recognize and understand human spoken language through speech signal processing and pattern recognition. Speech recognition technology is a high technology that allows the machine to convert speech signals into corresponding text or commands through the recognition and understanding process. Speech recognition is a cross-disciplinary subject with a wide range of implications. It has a very close relationship with subjects such as acoustics, phonetics, linguistics, information theory, pattern recognition theory, and neurobiology.

[0158] Specifically, before using the trained domain prediction model to perform domain prediction on the teacher's speech text, the method further includes:

[0159] Perform word frequency analysis on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix;

[0160] Based on a plurality of preset field labels, the word frequency matrix is ​​randomly numbered to obtain a field number corresponding to the word frequency matrix;

[0161] Using the domain number as an input value of the domain prediction model, and calculating the Gibbs sampling value and the Dirichlet parameter estimation value of the domain prediction model;

[0162] The Gibbs sampling value and the Dirichlet parameter estimation value are used as model parameters of the domain prediction model to obtain a trained domain prediction model.

[0163] Furthermore, the word frequency analysis is performed on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix, including:

[0164] Performing word segmentation processing on the training speeches in the training speech set to obtain a training word segmentation set;

[0165] Filtering the training segmented words in the training segmented word set that are consistent with the keywords in the preset keyword library as training keywords, and summarizing the training keywords to obtain a training keyword set;

[0166] The keyword identifier and keyword frequency corresponding to each keyword in the training keyword set are obtained, and a corresponding frequency matrix is ​​constructed according to the keyword identifier and the keyword frequency.

[0167] In detail, the word segmentation processing can be performed by a probability-based statistical word segmentation model, or the word segmentation processing can be performed by a jieba word segmentation model. The preset keyword library contains multiple topic-related keywords. For example, the corresponding keywords under the interest topic are "astronomy", "swimming", "taekwondo" and "running". The multiple keywords are randomly sorted, and the ID values ​​corresponding to each keyword are assigned in the order of random sorting. For example, the word ID corresponding to "astronomy" is 0 and the word frequency is 8 (the word ID corresponding to "astronomy" is 0 and the word frequency is 8, forming the matrix element (0, 8) corresponding to "astronomy"), the word ID corresponding to "swimming" is 1 and the word frequency is 5, the word ID corresponding to "taekwondo" is 2 and the word frequency is 4, and the word ID corresponding to "running" is 3 and the word frequency is 3. Then the word frequency matrix corresponding to the keyword set is [(0, 8)(1, 5), (2, 4), (3, 3)...].

[0168] Step 2: Select an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identify teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio.

[0169] In an embodiment of the present invention, the preset offline teaching material library includes multiple offline teaching materials in different fields, wherein the multiple offline teaching materials in different fields may be popular science books related to astronomy, popular science data related to military affairs, cooking data related to cooking, etc. When the speech field is the astronomy field, popular science books related to astronomy are selected from the offline teaching material library.

[0170] Specifically, the identifying the teaching material content data in the target offline teaching material corresponding to the speech request in the teacher speech audio includes:

[0171] Extracting a request action in a speech request in the teacher's speech audio, and locating the target offline teaching material to a corresponding page number according to the request action;

[0172] A text extraction tool is used to extract text content from pages of the target offline teaching material corresponding to the page number, and the text content is used as teaching material content data.

[0173] In detail, the teacher's speech audio usually contains many aspects of content, such as calling a classmate's name first, or containing some interactive words, but these contents have no actual meaning. Therefore, it is necessary to extract the request action in the speech request in the teacher's speech audio and make corresponding responses according to the request action.

[0174] For example, the teacher's speech audio is "Hello, students, please turn to page 23." The extraction request action is "turn to page 23". According to the request action, the target offline teaching material is located to page 23, and the text content on page 23 is identified as teaching material content data using a text extraction tool.

[0175] Preferably, the text extraction tool may be a text extraction applet or the like.

[0176] Step three, identify the data type of the teaching material content data. When the teaching material content data is of a dialogue type, use the pre-acquired dialogue audio data to iteratively optimize the preset 3D face model to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video.

[0177] In an embodiment of the present invention, the teaching material content data may be of different types. For example, the teaching material content data may be a conversation between two people, in which case the teaching material content data is of a conversation type. Alternatively, the teaching material content data may be popular science text, in which case the teaching material content data is of a non-conversation type.

[0178] Specifically, the iterative optimization of the preset 3D face model using the pre-acquired conversation audio data to obtain the standard face model includes:

[0179] Splitting the conversation audio data into phonemes, and performing vectorization processing on the phonemes to obtain phoneme sequence vectors;

[0180] Using a bidirectional long short-term memory network in a preset 3D face model to perform a first sequence transformation process on the phoneme sequence vector to obtain a mouth shape feature sequence;

[0181] Performing a second sequence transformation process on the mouth shape feature sequence to obtain facial expression parameters of the human face;

[0182] The 3D face model is iteratively optimized according to the face expression parameters to obtain a standard face model.

[0183] Further, splitting the conversation audio data into phonemes includes:

[0184] Converting the conversation audio data into text to obtain an initial conversation text;

[0185] Remove the punctuation marks from the initial dialogue text to obtain the standard dialogue text;

[0186] Mark the phoneme corresponding to each character in the standard dialogue text using a preset phonetic symbol rule.

[0187] Specifically, in the embodiment of the present application, text conversion software is used to perform text conversion on the dialogue audio data to obtain the initial dialogue text. Since the text conversion software will mark punctuation marks according to the pauses in the speech in the dialogue audio data, there are still multiple punctuation marks in the initial dialogue text. To facilitate subsequent splitting of the text into phonemes, it is necessary to remove the punctuation marks from the initial dialogue text.

[0188] Further, according to the one-to-one correspondence between characters and phonemes in the preset phonetic symbol rule, each character in the standard text is marked to obtain phonemes.

[0189] For example, in the embodiment of the present application, the preset phonetic symbol rule is the International Phonetic Alphabet rule. When marking the character "啊", the corresponding phoneme is a, and the obtained phoneme is [a].

[0190] Further, in another embodiment of the present application, the vectorization process is performed on the phonemes to obtain a phoneme sequence vector, including:

[0191] Use a preset unsupervised model to perform vectorization processing on the phonemes to obtain a phoneme sequence vector.

[0192] Further, the preset unsupervised model can be word2vec, doc2vec, fasttext, etc. In the embodiment of the present application, the preset unsupervised model is the Skip-gram model in word2vec. Among them, word2vec is a type of neural network model. Based on this model, a vector that can express semantics, that is, a word vector, can be generated for a single word segmentation in a corpus.

[0193] In the embodiment of the present application, the preset 3D face model can be composed of two bidirectional LSTM networks (Bi-directional long Short-Term Memory, bidirectional long short-term memory network). The bidirectional long short-term memory network is a type of time-recurrent neural network, including: an input gate, a forget gate, and an output gate.

[0194] In the embodiment of the present application, the iterative optimization of the 3D face model according to the facial expression parameters of the face to obtain a standard face model includes:

[0195] Calculating the loss value between the facial expression parameters of the human face and the preset real facial parameters;

[0196] When the loss value is less than a preset face threshold, outputting the 3D face model as an initial face model;

[0197] When the loss value is greater than or equal to the preset face threshold, the internal parameters of the 3D face model are adjusted, and the step of performing the first sequence transformation processing on the phoneme sequence vector using the bidirectional long short-term memory network in the preset 3D face model is returned until the loss value is less than the preset face threshold, and the output is a standard face model.

[0198] The internal parameter may be a gradient parameter or a weight parameter of the model.

[0199] Specifically, the teaching material content data is input into the standard face model to obtain a virtual dialogue video, which can present the teaching material content data in the form of a voice broadcast by a virtual anchor, which has a richer presentation form and is easier to understand in class.

[0200] Step 4: Add the virtual conversation video to the pre-built online teaching material to obtain the optimized online teaching material.

[0201] In an embodiment of the present invention, the online teaching material may be a PPT, and the virtual dialogue video is inserted into the online teaching material, i.e., the PPT, to obtain an optimized online teaching material. The optimized online teaching material can help students better understand the content presented in the teaching material and increase the fun of the class.

[0202] Step 5: When the teaching material content data is of a non-dialog type, named entity recognition is performed on the teaching material content data, and relevant two-dimensional plane images are retrieved according to the entity names obtained by the named entity recognition.

[0203] In an embodiment of the present invention, when the teaching material content data is of a non-dialog type, it indicates that the teaching material content data mostly contains large paragraphs of text content. Therefore, named entities in the teaching material content data can be identified and processed to enrich the online teaching materials.

[0204] Specifically, the performing named entity recognition on the teaching material content data includes:

[0205] Dividing the textbook content data into a plurality of textbook segmentation words, and respectively vectorizing the plurality of textbook segmentation words to obtain a plurality of segmentation vectors;

[0206] Inputting the plurality of word segmentation vectors into a long short-term memory network for feature extraction to obtain a feature sequence representation;

[0207] The feature sequence representation is input into the CRF layer for decoding to obtain the sequence corresponding to each textbook word segmentation;

[0208] The corresponding entity name is determined according to the sequence corresponding to the textbook segmentation words.

[0209] In detail, the LSTM (Long Short-Term Memory) network is a time recurrent neural network. The CRF layer refers to a conditional random field, which can constrain the output data of the LSTM network so that the final entity name is more accurate.

[0210] The entity names for named entity recognition may be proper nouns in the military field, such as aircraft carriers, tanks, and airplanes.

[0211] Furthermore, by searching on the Internet based on the entity names obtained by named entity recognition, multiple related two-dimensional plane images can be obtained, wherein the two-dimensional plane images are taken from different angles or different scenes.

[0212] For example, the two-dimensional plane image may be a front view of an aircraft carrier, or a side view of an aircraft carrier, etc.

[0213] Step six: calculate the depth map of the two-dimensional plane image, and generate a stereoscopic image based on the Markov random field model and the depth map, and add the stereoscopic image to the online teaching material to obtain an optimized online teaching material.

[0214] In the embodiment of the present invention, a Gaussian probability model in a Markov random field (MRF) model is used as a priori probability model to describe the relationship between the depth feature of an image and a depth map.

[0215] Specifically, calculating the depth map of the two-dimensional plane image includes:

[0216] Dividing the two-dimensional plane image into a preset number of plane rectangular blocks;

[0217] Performing filtering processing on a preset number of planar rectangular blocks respectively to obtain a plurality of filtered rectangular blocks;

[0218] Calculating the absolute depths of the plurality of filtering rectangular blocks using a preset depth calculation formula to obtain a depth feature;

[0219] A depth map of the two-dimensional plane image is generated according to the depth feature.

[0220] In detail, the preset number may be nine. The purpose of dividing the two-dimensional plane image into a preset number of plane rectangular blocks is to divide the image into small rectangular blocks so that each rectangular block corresponds to a depth feature. An edge detection filter may be used to filter the plane rectangular blocks to obtain a plurality of filtered rectangular blocks.

[0221] Furthermore, the calculating the absolute depths of the plurality of filtered rectangular blocks by using a preset depth calculation formula includes:

[0222] The preset depth calculation formula is:

[0223] E i (n) = ∑ (x,y)∈S |I(x,y) * F n (x,y)| q

[0224] Among them, E i (n) is the deep feature, q∈{1,,2}, I(x,y) * is the sum of the absolute energies of the filtered rectangular blocks, F n (x, y) is the absolute energy of the filtered rectangular block.

[0225] Specifically, the generating of a stereoscopic image based on the Markov random field model and the depth map includes:

[0226] Performing super-resolution processing and denoising on the depth map to obtain a denoised depth map;

[0227] The two-dimensional plane image is defined as a right viewpoint image, a disparity map between the right viewpoint image and the denoised depth map is established using a Markov random field model, and a stereoscopic image is generated according to the disparity map.

[0228] Specifically, unlike super-resolution of ordinary images, depth maps can be processed by super-resolution based on the information contained in the original image. Since the depth map after super-resolution processing has the same resolution and clear object contours as the two-dimensional plane image, but at the same time introduces noise, denoising is required. This solution uses the histogram statistics method to find noise points in the depth map and achieve denoising.

[0229] Preferably, the generated stereogram can enable the viewer to perceive the depth information of the objects and scenes, and appreciate the spatial positional relationship between the objects in the image in terms of front and back, up and down, and orientation, thus bringing a more realistic and vivid visual experience.

[0230] Furthermore, the stereoscopic image is added to the online teaching material, and the stereoscopic image can enrich the online teaching material, so that the original plane image is transformed into a stereoscopic image and added to the online teaching material, so that teachers can introduce and interact with students more accurately, thereby ensuring the optimization of the online teaching material.

[0231] Step seven: store the teaching material content data in a preset database, and use a preset teaching material editor to compile the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material.

[0232] In an embodiment of the present invention, the textbook content data is stored in a preset database, which can ensure that the textbook content data can be called from the preset database during subsequent data processing, thereby improving the efficiency of textbook content data search. The preset textbook editor is used to compile the textbook content data in the preset database and the optimized online textbook, wherein the preset textbook editor is a dedicated textbook editor that can complete the production of new forms of electronic textbooks. Through the textbook editor, a variety of template styles are inserted into the textbook; digital resources are combined with integrated media textbooks to form modern digital integrated media textbooks for online, offline, and online and offline combined teaching, learning and other scenarios. The rich template styles include title templates, paragraph templates, question and answer templates, etc., which allow teachers to quickly complete digital templates through convenient operations, making it easier to update and iterate textbooks.

[0233] Specifically, the obtained new form of teaching materials is uploaded to the server for storage. The new form of teaching materials supports pictures, audio, video, two-dimensional animation, three-dimensional animation, virtual interactive models, etc. When uploaded to the server, the information and data involved in the new form of teaching materials will also be stored in the database.

[0234] Step 8: When a teaching preparation request is received, the new form teaching material is called for classroom teaching.

[0235] In an embodiment of the present invention, the teaching preparation request refers to an instruction that the teacher is ready to perform teaching actions in the teaching scene. The teacher will call the required teaching materials according to his own teaching scene, and the students can install the new form teaching materials into the reader in the device to read the teaching materials. Students use the built-in components of the reader to view various resources in the teaching materials. In addition to conventional pictures, audio and video resources, they can also zoom in and out, split and other operations on 3D models, so as to understand the content learned more intuitively and deeply. At the same time, students use the reader installed in the device to read the teaching materials. Students realize immersive and interactive learning through structured resources and digital teaching materials, complete different learning scenes and complete tasks such as teacher homework through the reader, improve students' learning interest and improve students' learning quality. In detail, this solution realizes a quick jump from paper teaching materials to the corresponding content of digital teaching materials through a variety of interactive forms, and supports a variety of interactive methods such as audio and video playback, 3D model interaction, virtual simulation training, etc. Achieve a teaching effect of virtual-real interaction and the combination of theory and practice. Through this innovative learning model, the way of talent cultivation is upgraded from one-way indoctrination to perceptual interaction, and the form of learning experience is upgraded from 2D plane to 3D stereoscopic interaction, so as to achieve structured integration and most efficient use of digital resources, greatly improving the effect of knowledge learning and skills training.

[0236] In an embodiment of the present invention, by performing field prediction on the teacher's speech audio after text conversion, the speech field corresponding to the teacher's speech audio is obtained, and the target offline teaching material consistent with the speech field is selected in the offline teaching material library, so as to achieve accurate selection. The teaching material content data corresponding to the speech request in the target offline teaching material is identified, and the data type of the teaching material content data is identified, virtual dialogue video or stereoscopic image generation is performed according to different data types, and the virtual dialogue video and the stereoscopic image are added to the online teaching material to obtain an optimized teaching material, the teaching material content data is stored in a preset database, and the teaching material content data in the preset database and the optimized online teaching material are compiled using a preset teaching material editor to obtain a new form teaching material, and when a teaching preparation request is received, the new form teaching material is called for classroom teaching, so as to achieve the optimization processing of the online teaching material. Therefore, the new form teaching material learning device based on digital twins proposed in the present invention can solve the problem of low accuracy of teaching material optimization.

[0237] like Figure 3 As shown, it is a structural schematic diagram of an electronic device for implementing a new form of teaching material learning method based on digital twins provided by one embodiment of the present invention.

[0238] The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a new form of teaching material learning program based on digital twins.

[0239] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of a new form of teaching material learning program based on digital twins, but also can be used to temporarily store data that has been output or is to be output.

[0240] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which connects the various components of the entire electronic device using various interfaces and lines, and executes or executes programs or modules stored in the memory 11 (such as new forms of teaching materials learning programs based on digital twins, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0241] The communication interface 12 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0242] The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 13 may be divided into an address bus, a data bus, a control bus, etc. The bus 13 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0243] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0244] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0245] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0246] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0247] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0248] The new form of teaching material learning program based on digital twins stored in the memory 11 in the electronic device is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0249] Obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio;

[0250] Selecting an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identifying teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio;

[0251] Identify the data type of the teaching material content data, and when the teaching material content data is of a dialogue type, iteratively optimize a preset 3D face model using pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video;

[0252] Adding the virtual conversation video to a pre-built online teaching material to obtain an optimized online teaching material;

[0253] When the teaching material content data is of a non-dialog type, performing named entity recognition on the teaching material content data, and retrieving a related two-dimensional plane image according to the entity name obtained by the named entity recognition;

[0254] Calculating a depth map of the two-dimensional plane image, generating a stereoscopic image based on a Markov random field model and the depth map, and adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material;

[0255] The teaching material content data is stored in a preset database, and a preset teaching material editor is used to edit the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material;

[0256] When a teaching preparation request is received, the new form teaching material is called to conduct classroom teaching.

[0257] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0258] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0259] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0260] Obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio;

[0261] Selecting an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identifying teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio;

[0262] Identify the data type of the teaching material content data, and when the teaching material content data is of a dialogue type, iteratively optimize a preset 3D face model using pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video;

[0263] Adding the virtual conversation video to a pre-built online teaching material to obtain an optimized online teaching material;

[0264] When the teaching material content data is of a non-dialog type, performing named entity recognition on the teaching material content data, and retrieving a related two-dimensional plane image according to the entity name obtained by the named entity recognition;

[0265] Calculating a depth map of the two-dimensional plane image, generating a stereoscopic image based on a Markov random field model and the depth map, and adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material;

[0266] The teaching material content data is stored in a preset database, and a preset teaching material editor is used to edit the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material;

[0267] When a teaching preparation request is received, the new form teaching material is called to conduct classroom teaching.

[0268] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0269] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0270] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0271] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0272] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0273] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0274] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A new form of teaching material learning method based on digital twins, characterized in that: The method comprises: Obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio; Selecting an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identifying teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio; Identify the data type of the teaching material content data, and when the teaching material content data is of a dialogue type, iteratively optimize a preset 3D face model using pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video; Adding the virtual conversation video to a pre-built online teaching material to obtain an optimized online teaching material; When the teaching material content data is of a non-dialog type, performing named entity recognition on the teaching material content data, and retrieving a related two-dimensional plane image according to the entity name obtained by the named entity recognition; Calculating a depth map of the two-dimensional plane image, generating a stereoscopic image based on a Markov random field model and the depth map, and adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material; The teaching material content data is stored in a preset database, and a preset teaching material editor is used to edit the teaching material content data in the preset database and the optimized online teaching material to obtain a new form of teaching material; When a teaching preparation request is received, the new form teaching material is called to conduct classroom teaching.

2. The new form of teaching material learning method based on digital twins as claimed in claim 1 is characterized in that: The step of calculating the depth map of the two-dimensional plane image comprises: Dividing the two-dimensional plane image into a preset number of plane rectangular blocks; Performing filtering processing on a preset number of planar rectangular blocks respectively to obtain a plurality of filtered rectangular blocks; Calculating the absolute depths of the plurality of filtering rectangular blocks using a preset depth calculation formula to obtain a depth feature; A depth map of the two-dimensional plane image is generated according to the depth feature.

3. The new form of teaching material learning method based on digital twins as claimed in claim 1 is characterized in that: The generating of a stereoscopic image based on the Markov random field model and the depth map comprises: Performing super-resolution processing and denoising on the depth map to obtain a denoised depth map; The two-dimensional plane image is defined as a right viewpoint image, a disparity map between the right viewpoint image and the denoised depth map is established using a Markov random field model, and a stereoscopic image is generated according to the disparity map.

4. The new form of teaching material learning method based on digital twins as claimed in claim 1 is characterized in that: The performing named entity recognition on the teaching material content data includes: Dividing the textbook content data into a plurality of textbook segmentation words, and respectively vectorizing the plurality of textbook segmentation words to obtain a plurality of segmentation vectors; Inputting the plurality of word segmentation vectors into a long short-term memory network for feature extraction to obtain a feature sequence representation; The feature sequence representation is input into the CRF layer for decoding to obtain the sequence corresponding to each textbook word segmentation; The corresponding entity name is determined according to the sequence corresponding to the textbook segmentation words.

5. The new form of teaching material learning method based on digital twins according to claim 1 is characterized in that: The method of iteratively optimizing the preset 3D face model using the pre-acquired conversation audio data to obtain a standard face model includes: Splitting the conversation audio data into phonemes, and performing vectorization processing on the phonemes to obtain phoneme sequence vectors; Using a bidirectional long short-term memory network in a preset 3D face model to perform a first sequence transformation process on the phoneme sequence vector to obtain a mouth shape feature sequence; Performing a second sequence transformation process on the mouth shape feature sequence to obtain facial expression parameters of the human face; The 3D face model is iteratively optimized according to the face expression parameters to obtain a standard face model.

6. The new form of teaching material learning method based on digital twins as claimed in claim 5, characterized in that: The step of splitting the conversation audio data into phonemes comprises: Converting the conversation audio data into text to obtain an initial conversation text; Removing punctuation marks from the initial dialogue text to obtain a standard dialogue text; The phoneme corresponding to each character in the standard dialogue text is marked using a preset phonetic symbol rule.

7. The new form of teaching material learning method based on digital twins as claimed in claim 1, characterized in that: The identifying teaching material content data in the target offline teaching material corresponding to the speech request in the teacher speech audio includes: Extracting a request action in a speech request in the teacher's speech audio, and locating the target offline teaching material to a corresponding page number according to the request action; A text extraction tool is used to extract text content from pages of the target offline teaching material corresponding to the page number, and the text content is used as teaching material content data.

8. The new form of teaching material learning method based on digital twins as claimed in claim 1, characterized in that: Before using the trained domain prediction model to perform domain prediction on the teacher's speech text, the method further includes: Perform word frequency analysis on the training speeches in the pre-acquired training speech set to obtain a word frequency matrix; Based on a plurality of preset field labels, the word frequency matrix is ​​randomly numbered to obtain a field number corresponding to the word frequency matrix; Using the domain number as an input value of the domain prediction model, and calculating the Gibbs sampling value and the Dirichlet parameter estimation value of the domain prediction model; The Gibbs sampling value and the Dirichlet parameter estimation value are used as model parameters of the domain prediction model to obtain a trained domain prediction model.

9. The new form of teaching material learning method based on digital twins as claimed in claim 8, characterized in that: The word frequency analysis is performed on the training speech in the pre-acquired training speech set to obtain a word frequency matrix, including: Performing word segmentation processing on the training speeches in the training speech set to obtain a training word segmentation set; Filtering the training segmented words in the training segmented word set that are consistent with the keywords in the preset keyword library as training keywords, and summarizing the training keywords to obtain a training keyword set; The keyword identifier and the keyword frequency corresponding to each keyword in the training keyword set are obtained, and a corresponding frequency matrix is ​​constructed according to the keyword identifier and the keyword frequency.

10. A new form of teaching material learning device based on digital twins, characterized in that: The device comprises: A domain recognition module is used to obtain the teacher's speech audio, perform text conversion on the teacher's speech audio to obtain the teacher's speech text, and use the trained domain prediction model to perform domain prediction on the teacher's speech text to obtain the speech domain corresponding to the teacher's speech audio; A content extraction module, configured to select an offline teaching material consistent with the speech field from a preset offline teaching material library as a target offline teaching material, and identify teaching material content data in the target offline teaching material corresponding to the speech request in the teacher's speech audio; A first teaching material optimization module is used to identify the data type of the teaching material content data. When the teaching material content data is of a dialogue type, iteratively optimize the preset 3D face model using the pre-acquired dialogue audio data to obtain a standard face model, and input the teaching material content data into the standard face model to obtain a virtual dialogue video, and add the virtual dialogue video to the pre-built online teaching material to obtain an optimized online teaching material; The second teaching material optimization module is used for performing named entity recognition on the teaching material content data when the teaching material content data is of a non-dialog type, and retrieving relevant two-dimensional plane images according to the entity names obtained by the named entity recognition, calculating the depth map of the two-dimensional plane image, and generating a stereoscopic image based on a Markov random field model and the depth map, adding the stereoscopic image to the online teaching material to obtain an optimized online teaching material, storing the teaching material content data in a preset database, and using a preset teaching material editor to compile the teaching material content data in the preset database and the optimized online teaching material to obtain a new form teaching material, and when a teaching preparation request is received, calling the new form teaching material for classroom teaching.

Images