A Multimodal Digital Textbook Content Review Method and Device

Abstract

This invention discloses a method and apparatus for reviewing multimodal digital teaching materials, relating to the field of digital education technology. The method includes: acquiring digital teaching materials from the internet to construct digital teaching material resources; annotating the digital teaching material resources to obtain annotated digital teaching material resources; classifying the annotated digital teaching material resources into qualified and unqualified resources to construct multimodal training samples; processing the multimodal training samples to obtain multiple unimodal feature vectors; fusing them using a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors; constructing an initial multimodal classification model; training the multimodal feature vectors using a neural network-based training method to obtain a trained multimodal classification model; and inputting the acquired digital teaching materials to be reviewed into the trained multimodal classification model for review to obtain review results. Using this invention can improve the accuracy of the review results.

Description

Technical Field

[0001] This invention relates to the field of digital education technology, and in particular to a multimodal digital teaching material content review method and apparatus. Background Technology

[0002] Digital textbooks are interactive teaching materials that utilize multimedia technology to deeply integrate textual content with rich digital resources, ultimately transforming it into materials suitable for various electronic terminals.

[0003] In a narrow sense, digital teaching materials refer to digital textbooks. In a broader sense, digital teaching materials include not only digital textbooks but also supplementary learning materials such as reference books, multimedia content, and hyperlinks to other learning resources. In terms of their format, digital teaching materials can take the form of electronic text, images, audio, and video. In terms of their carriers, they include various digital educational applications, such as educational apps, mini-programs, and cloud platforms.

[0004] Digital textbooks offer a rich variety of presentation formats, providing students with engaging and interesting learning content. Their unlimited scalability and flexible adaptability make student learning more cutting-edge and personalized, compensating for the shortcomings of paper textbooks in adapting to the information age.

[0005] However, the development of digital textbooks has also brought challenges to content review. Ensuring the standardization and scientific accuracy of these textbooks has become an urgent issue. This is mainly reflected in:

[0006] First, the sheer volume of digital textbook products makes manual review difficult. While these products and applications expand the concepts, boundaries, and teaching services of the traditional textbook system and provide rich educational content, some textbooks or links contain inappropriate content such as exceeding teaching standards and vulgar information. The sheer volume of teaching resources makes it difficult to review their content manually.

[0007] Secondly, the diversification of formats necessitates new technological approaches. Currently, students have access to digital teaching resources that include not only text but also audiovisual and interactive media. This richness in digital resources also presents new challenges to content moderation technologies and methods. Simple rule-based moderation systems struggle to understand multimodal data. Conversely, single-modal moderation systems are unable to address the challenges posed by multimodal data.

[0008] Third, user-generated content is difficult to identify in a timely manner. UGC (User-Generated Content) or AIGC content is characterized by its real-time nature, but it also carries inherent risks. If harmful content cannot be identified and filtered in a timely manner, the negative impact of its spread among students will be difficult to eliminate.

[0009] In response to the enormous challenges brought about by digital textbooks, there is a need for efficient, rapid, and accurate methods and means for reviewing digital textbook content that can adapt to the new situation. Summary of the Invention

[0010] To address the technical problems of existing technologies, such as the difficulty of manually reviewing the massive amounts of teaching resources and the inability of existing technologies to review multimodal data, this invention provides a method and apparatus for reviewing multimodal digital teaching material content. The technical solution is as follows:

[0011] On the one hand, a multimodal digital textbook content review method is provided, which is implemented by a multimodal digital textbook content review device. The method includes:

[0012] S1. Obtain digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, construct digital textbook resources that include text, images, audio, and video;

[0013] S2. Label the digital teaching materials resources to obtain labeled digital teaching materials resources; divide the labeled digital teaching materials resources into qualified resources and unqualified resources, and construct multimodal training samples;

[0014] S3. Process the multimodal training samples to obtain multiple single-modal feature vectors;

[0015] S4. Based on multiple single-modal feature vectors, a three-layer cross-attention fusion mechanism is used to perform multimodal fusion to obtain multimodal feature vectors;

[0016] S5. Construct an initial multimodal classification model; Based on the multimodal feature vectors, use a neural network-based training method to train the initial multimodal classification model to obtain a trained multimodal classification model;

[0017] S6. Obtain the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into the trained multimodal classification model for review, and obtain the review results.

[0018] Optionally, step S3 processes the multimodal training samples to obtain multiple single-modal feature vectors, including:

[0019] S31. Use a Transformer-based pre-trained language model to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0020] S32. Use wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors;

[0021] S33. Use the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; use the Vision Transformer model to extract the human frame features of the image as supplementary feature vectors of the image modality.

[0022] Optionally, the plurality of single-modal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for image modality.

[0023] Optionally, step S33 involves using the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors, including:

[0024] S331. Divide the input image into a grid of fixed-size blocks; flatten each image block into a one-dimensional vector.

[0025] S332. Input each image block vector into a linear layer for processing to obtain the embedding sequence corresponding to each image block;

[0026] S333. Add position encoding to the embedding sequence corresponding to each image block and input it into the standard Transformer encoder. Extract local and global information in the image through the self-attention mechanism and output the image modal feature vector.

[0027] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0028] The first layer is used to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing both audio and text.

[0029] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0030] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0031] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0032] (1)

[0033] (2)

[0034] (3)

[0035] (4)

[0036] (5)

[0037] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0038] On the other hand, a multimodal digital textbook content review device is provided, which is applied to the multimodal digital textbook content review method. The device includes:

[0039] The digital textbook acquisition unit is used to acquire digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, it constructs digital textbook resources that include text, images, audio, and video.

[0040] The digital textbook annotation unit is used to annotate the digital textbook resources to obtain annotated digital textbook resources; the annotated digital textbook resources are divided into qualified resources and unqualified resources to construct multimodal training samples;

[0041] The processing unit is used to process multimodal training samples to obtain multiple single-modal feature vectors;

[0042] The fusion unit is used to perform multimodal fusion based on multiple single-modal feature vectors and adopt a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors;

[0043] The training unit is used to build an initial multimodal classification model; based on the multimodal feature vectors, a neural network-based training method is used to train the initial multimodal classification model to obtain a trained multimodal classification model.

[0044] The review unit is used to acquire digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into a trained multimodal classification model for review, and obtain the review results.

[0045] Optionally, the processing unit is configured to:

[0046] A Transformer-based pre-trained language model is used to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0047] We used wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors.

[0048] The Vision Transformer model is used to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; the Vision Transformer model is also used to extract the human frame features of the image as supplementary feature vectors for the image modality.

[0049] Optionally, the plurality of single-modal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for image modality.

[0050] Optionally, the step of using the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors includes:

[0051] The input image is divided into a grid of fixed-size blocks; each image block is flattened into a one-dimensional vector.

[0052] Each image patch vector is input into a linear layer for processing to obtain the embedding sequence corresponding to each image patch.

[0053] Positional encoding is added to the embedding sequence corresponding to each image patch and input into a standard Transformer encoder. Local and global information in the image is extracted through a self-attention mechanism, and the image modal feature vector is output.

[0054] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0055] The first layer is used to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing both audio and text.

[0056] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0057] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0058] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0059] (1)

[0060] (2)

[0061] (3)

[0062] (4)

[0063] (5)

[0064] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0065] On the other hand, a multimodal digital textbook content review device is provided, the multimodal digital textbook content review device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for multimodal digital textbook content review.

[0066] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the methods in the above-described multimodal digital textbook content review methods.

[0067] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0068] This invention first collects electronic textbooks from various educational publishers, as well as digital textbooks in text, image, audio, and video formats from the internet. These digital textbooks are then manually labeled and categorized as either qualified or unqualified, forming multimodal training samples. Single-modal features are extracted from the multimodal samples, and a cross-attention mechanism is used to fuse these features into a unified feature representation. A deep learning algorithm is then used to train a multimodal classification model. The multimodal classification model is tested, and optimized based on the test results. Finally, the multimodal classification model is used to review the content of the digital textbooks.

[0069] The embodiments of the present invention can not only handle multimodal digital teaching materials and integrate multimodal features to better understand context and adopt a unified processing approach for digital teaching materials of different modalities, but also combine different review models as needed by adding or removing modalities to adapt to the diverse review requirements of digital teaching materials, thereby improving the flexibility, timeliness of response, and accuracy of review results in the review of digital teaching material content. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of a multimodal digital teaching material content review method provided by an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram illustrating the embedded representation of input text in a text modality provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of an image feature extraction algorithm provided in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of key points of the human skeleton provided in an embodiment of the present invention;

[0075] Figure 5 This is a schematic diagram of a cross-attention unit provided in an embodiment of the present invention;

[0076] Figure 6 This is a network architecture diagram of a content moderation model provided in an embodiment of the present invention;

[0077] Figure 7This is a block diagram of a multimodal digital teaching material content review device provided in an embodiment of the present invention;

[0078] Figure 8 This is a schematic diagram of the structure of a multimodal digital textbook content review device provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0080] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0081] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0082] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0083] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0084] This invention provides a method for reviewing the content of multimodal digital teaching materials. This method can be implemented using a multimodal digital teaching material content review device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates the multimodal digital textbook content review method. The processing flow of this method may include the following steps:

[0085] S1. Obtain digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, construct digital textbook resources that include text, images, audio and video.

[0086] The scope of digital textbook collection includes electronic versions of textbooks published by authoritative national educational institutions, digital textbooks from various educational publishers, and electronic textbooks on the Internet.

[0087] In one feasible implementation, the data collection method of this invention includes: communicating with the digital textbook publisher via a communication protocol to obtain content, downloading digital textbooks using web crawling technology, and receiving digital textbooks uploaded by users. Specifically, for digital textbooks presented in text form, after collecting the content, link analysis is performed on the content, and web crawling technology is used to download linked content until the content no longer contains any links. The content of each link is processed as independent text content.

[0088] S2. Label the digital teaching materials to obtain labeled digital teaching materials; divide the labeled digital teaching materials into qualified and unqualified resources, and construct multimodal training samples.

[0089] The annotation methods used in the embodiments of the present invention include: automatic machine annotation, manual annotation, and expert review.

[0090] In one feasible implementation, the text is annotated sentence by sentence, and each sentence is marked as qualified or unqualified based on its content.

[0091] In one feasible implementation, the specific process of image annotation includes: uniformly cropping the image to a fixed size and annotating the image; wherein, when the image contains text, it needs to be extracted separately and annotated with text.

[0092] In one feasible implementation, audio content is annotated sentence by sentence.

[0093] In one feasible implementation, video annotation includes two methods: one is to annotate the entire video; the other is to process the video into fixed-size frames and annotate the image of each frame as well as the audio and text associated with that frame.

[0094] S3. Process the multimodal training samples to obtain multiple single-modal feature vectors.

[0095] Optionally, the multiple unimodal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for the image modality.

[0096] Optionally, the specific implementation process of S3 includes S31-S33:

[0097] S31. Use a Transformer-based pre-trained language model to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0098] in, Figure 2This is a schematic diagram of the embedding representation of input text in a text modality provided by an embodiment of the present invention. In one feasible implementation, the input layer of the pre-trained language model based on Transformer adds special start markers "[CLS]" and "[SEP]" to the text. The input information is composed of word vectors, segment vectors, and position vectors superimposed. The input layer can be a single sentence or a pair of sentences; a sentence consists of multiple tokens; word vectors are the vector representation of tokens, with each word being converted into a fixed-length vector; segment vectors represent the segmentation of the sentence; if the input layer is a pair of sentences, the segment vectors corresponding to all tokens in the first sentence are 0, and the segment vectors corresponding to all tokens in the second sentence are 1; position vectors ensure the order of words, adding positional information to the input sequence.

[0099] S32. Use wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors;

[0100] Among them, wav2vec2.0 is an audio pre-training model based on self-supervised learning. It learns deep representations of audio through training on large-scale unlabeled audio data. Its core idea is to transform audio signals into a discrete latent representation that contains both the temporal information of the audio and can capture the semantic content of the audio.

[0101] S33. Use the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; use the Vision Transformer model to extract the human frame features of the image as supplementary feature vectors of the image modality.

[0102] Specifically, an image feature extraction method based on Vision Transformer (ViT) is employed to extract features as a representation of the image. ViT is an application of Transformer in the vision field. ViT treats images as sequences of smaller patches, thereby enabling it to capture global relationships and long-range dependencies in visual data.

[0103] Optionally, the specific implementation process of S33 includes S331-S333:

[0104] S331. Divide the input image into a grid of fixed-size blocks; flatten each image block into a one-dimensional vector.

[0105] Each image block is 16x16 pixels in size. Each image block is flattened into a one-dimensional vector, thus effectively converting the two-dimensional image into a series of block vectors.

[0106] S332. Input each image block vector into a linear layer for processing to obtain the embedding sequence corresponding to each image block;

[0107] In one feasible implementation, each image patch vector is processed through a linear layer and projected into a high-dimensional space. The projection process creates an embedding sequence, with one embedding corresponding to each patch.

[0108] S333. Add position encoding to the embedding sequence corresponding to each image block and input it into the standard Transformer encoder. Extract local and global information in the image through the self-attention mechanism and output the image modal feature vector.

[0109] Among them, positional encoding injects information about the spatial location of each image patch, helping the model preserve the structure of the original image.

[0110] In one feasible implementation, image patches are input into a standard Transformer encoder. This encoder consists of multiple layers of multi-head self-attention and feedforward neural networks. Through the self-attention mechanism, the model learns the relationships between different image patches, thereby capturing local and global patterns in the image.

[0111] In one feasible implementation, a special learnable tag is added to the front of the block embedding sequence. The learnable tag is a classification tag labeled "Class". After processing by the Transformer encoder, the output corresponding to the classification tag is used as the overall representation of the image for task classification.

[0112] Among them, such as Figure 3 The diagram illustrates an image feature extraction algorithm provided by an embodiment of the present invention. The process of further extracting the frame features of a person based on image modalities includes: extracting the position information of 18 key points of the human body using OpenPose for the entire image; wherein, the 18 key points can outline the frame of the person, such as... Figure 4 The figure shown is a schematic diagram of key points of a human skeleton provided by an embodiment of the present invention; an image feature extraction method based on ViT is used to extract features as the feature representation of the image; specifically, the image is divided into multiple small blocks of 16x16 pixels in size; for each small block, it is determined whether there are key points, and if not, the block is kept unchanged; for each small block containing key points, the middle pixel of the key point is determined, which is represented by the following formula (1):

[0113] (1)

[0114] in, Represents the x-coordinate of the middle pixel. The ordinate represents the middle pixel, and n represents the number of keypoints. This represents the x-coordinate of the j-th key point. This represents the ordinate of the j-th key point;

[0115] The Manhattan distance method is used to calculate the distance between each other pixel in the block and the midpoint, which is then used as the new pixel value of the block, expressed by the following formula (2):

[0116] (2)

[0117] in, Represents two pixels and The distance between them;

[0118] In this process, each block is filled with new pixel values ​​to form a new smaller block. For each smaller block, image features are extracted using a ViT-based image feature extraction method to obtain the frame feature vector of the person. OpenPose can extract the keypoint positions of multiple people in the image; however, this embodiment of the invention only selects the keypoint positions of one person for calculation.

[0119] In one feasible implementation, for the video modality, the video frames are divided into multiple segments based on the speaker's utterance. Each segment corresponds to a single sentence from the speaker, with a length between 1 and 10 seconds, and the corresponding utterance is converted into text. Simultaneously, each video segment is extracted into individual image frames, and the corresponding audio for each frame is saved. The processing of the video modality can involve combining the processing of text, images, and audio.

[0120] S4. Based on multiple single-modal feature vectors, a three-layer cross-attention fusion mechanism is used to perform multimodal fusion to obtain multimodal feature vectors.

[0121] Multimodal fusion represents data from different modalities in a unified form. This invention employs a feature-level multimodal data fusion method based on cross-attention. Cross-attention is an attention mechanism used in multimodal analysis to fuse information from different modalities. It can cross-calculate attention scores between modalities such as images and text to extract shared features and enhance the representational power of each modality. After processing data from different modalities separately, their respective feature representations are obtained. Semantic fusion of these features enables in-depth interaction and feature fusion across modalities. This feature-level data fusion method provides a flexible framework, unrestricted by the number of single-modal inputs. In the fusion model, adding or removing modalities does not require considering the interaction effects of previous modalities; only cross-attention calculation is performed. This characteristic gives the feature fusion model better scalability and flexibility, making it more convenient to adapt to different numbers of single-modal inputs.

[0122] Cross attention establishes an attention relationship between two different sequences; what makes cross attention unique is that the sources of the query Q, key K, and value V are different.

[0123] In one feasible implementation, the calculation process for cross-modal attention from text modality to audio modality is represented by the following formulas (3)-(5):

[0124] (3)

[0125] (4)

[0126] (5)

[0127] Where t represents the text modality; a represents the audio modality; and v represents the video modality; This indicates a query originating from modality t; Represents the text modal feature vector; This represents a bond originating from mode a; Represents the audio modal feature vector; This represents the value from mode a; , , , This represents the weight matrix of the linear transformation of the first text feature. This represents the weight matrix of the linear transformation of the second text features. The weight matrices representing the linear transformations of the third text features are all obtained through training. The dimension of the feature vector representing the text modality. The dimension of the feature vector of the audio modality.

[0128] The attention weights are represented by the following formula (6):

[0129] (6)

[0130] in, Indicator key The size of the dimension, express The transpose of .

[0131] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0132] Each cross-attention layer consists of two parallel cross-attention units; where, for example... Figure 5 The diagram shown is a schematic of a cross-attention unit provided in an embodiment of the present invention; for the cross-attention unit, Q comes from one mode, and K and V come from another mode.

[0133] The first layer is used to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing both audio and text.

[0134] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (7)-(11):

[0135] (7)

[0136] (8)

[0137] (9)

[0138] (10)

[0139] (11)

[0140] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0141] In this embodiment of the invention, the cross-attention layer has three layers. This indicates the result after the first layer of cross-attention fusion, while the result of the third layer corresponds to the result in the embodiment of this invention. Figure 6 The result of the fusion vector layer processing. Among them, formula (7) and formula (8) are the calculation of cross attention units for mode A and mode T. The calculation process of formula (7) corresponds to the process of multi-head attention, residual connection and layer normalization. Formula (8) corresponds to the process of feedforward neural network, residual connection and layer normalization. Among them, formula (9) and formula (10) are the calculation of cross attention units for mode T and mode A. Formula (9) corresponds to the process of multi-head attention, residual connection and layer normalization. Formula (10) corresponds to the process of feedforward neural network, residual connection and layer normalization. Among them, formula (11) means that a new feature vector is obtained by concatenating the feature vectors of the two modes.

[0142] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0143] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0144] S5. Construct an initial multimodal classification model; based on the multimodal feature vectors, use a neural network-based training method to train the initial multimodal classification model to obtain a trained multimodal classification model.

[0145] Among them, such as Figure 6 The diagram shown is a network architecture diagram of a content moderation model provided by an embodiment of the present invention; wherein, the multimodal classification model includes: a fusion vector layer, multiple convolutional layers, a pooling layer, and a fully connected layer.

[0146] In one feasible implementation, a deep learning neural network algorithm is used to train the model. In this embodiment of the invention, for multimodal digital teaching materials, after single-modal feature extraction, a one-dimensional convolutional network is used to align the internal features of the three modalities: text, audio, and image. A cross-modal cross-attention mechanism is used to realize the interaction between text-audio, text-image, and audio-image, thereby achieving intermodal information interaction and feature enhancement, generating richer multimodal features after the fusion of text, audio, and images. An average pooling strategy is adopted for dimensionality reduction, and finally, a fully connected layer is used for classification, with a softmax layer converting the output into probabilities.

[0147] S6. Obtain the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into the trained multimodal classification model for review, and obtain the review results.

[0148] In one feasible implementation, the present invention first employs an optimization method for a single-modal classification model, and then optimizes the classification model constructed in the present invention by employing an optimization method for a multimodal classification model.

[0149] Among them, the method of optimizing multimodal classification models can flexibly obtain classification models containing different modalities by adding or reducing modalities according to the characteristics of the data and application needs.

[0150] In the optimization process, this embodiment of the invention introduces hyperparameter search algorithms, such as grid search or Bayesian optimization, to systematically explore the model architecture space, including parameters such as the number of attention heads, the depth of the fusion layer, and the dropout rate, thereby improving the robustness and generalization performance of the model. For single-modality models, a five-fold cross-validation method is used during the training phase to obtain several sets of optimal model parameters, mainly including parameters such as the number of convolutional layers and the size of the convolutional kernel. In addition, for the optimization of the text modality, this embodiment of the invention combines learning rate decay strategies, such as cosine annealing scheduling and L2 regularization, to prevent overfitting; for audio and image modalities, batch normalization and data augmentation techniques, such as random cropping and noise injection, are integrated to improve the stability of feature extraction. Through single-modality optimization, several sets of optimal model parameters for different modalities are obtained. The common parameter values ​​of the same parameter in different modalities are selected as the optimal candidate parameter values ​​for the combined modality, and the five-fold cross-validation method is also used to select the optimal parameters.

[0151] In the multimodal fusion process, this embodiment of the invention employs a gradient pruning algorithm to avoid the gradient explosion problem and uses the AdamW optimizer combined with weight decay to further refine the training dynamics of the cross-attention mechanism, ensuring the effective integration of multimodal feature vectors. This embodiment of the invention can also improve the model's generalization ability by increasing the number of training samples. Specifically, it uses generative adversarial networks or diffusion models to synthesize additional multimodal samples, thereby expanding the dataset size and mitigating the class imbalance problem.

[0152] This invention first collects electronic textbooks from various educational publishers, as well as digital textbooks in text, image, audio, and video formats from the internet. These digital textbooks are then manually labeled and categorized as either qualified or unqualified, forming multimodal training samples. Single-modal features are extracted from the multimodal samples, and a cross-attention mechanism is used to fuse these features into a unified feature representation. A deep learning algorithm is then used to train a multimodal classification model. The multimodal classification model is tested, and optimized based on the test results. Finally, the multimodal classification model is used to review the content of the digital textbooks.

[0153] The embodiments of the present invention can not only handle multimodal digital teaching materials and integrate multimodal features to better understand context and adopt a unified processing approach for digital teaching materials of different modalities, but also combine different review models as needed by adding or removing modalities to adapt to the diverse review requirements of digital teaching materials, thereby improving the flexibility, timeliness of response, and accuracy of review results in the review of digital teaching material content.

[0154] Figure 7 This is a block diagram of a multimodal digital textbook content review device provided in an embodiment of the present invention. This device is used for a multimodal digital textbook content review method. (Refer to...) Figure 7 The device includes a digital textbook acquisition unit 710, a digital textbook annotation unit 720, a processing unit 730, a fusion unit 740, a training unit 750, and a review unit 760. Among them:

[0155] The digital textbook acquisition unit 710 is used to acquire digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, it constructs digital textbook resources that include text, images, audio, and video.

[0156] The digital textbook annotation unit 720 is used to annotate the digital textbook resources to obtain annotated digital textbook resources; the annotated digital textbook resources are divided into qualified resources and unqualified resources to construct multimodal training samples;

[0157] The processing unit 730 is used to process multimodal training samples to obtain multiple single-modal feature vectors;

[0158] The fusion unit 740 is used to perform multimodal fusion based on multiple single-modal feature vectors and adopt a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors.

[0159] Training unit 750 is used to build an initial multimodal classification model; based on the multimodal feature vectors, a neural network-based training method is used to train the initial multimodal classification model to obtain a trained multimodal classification model;

[0160] The review unit 760 is used to acquire the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into a trained multimodal classification model for review, and obtain the review results.

[0161] Optionally, the processing unit 730 is configured to:

[0162] A Transformer-based pre-trained language model is used to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0163] We used wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors.

[0164] The Vision Transformer model is used to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; the Vision Transformer model is also used to extract the human frame features of the image as supplementary feature vectors for the image modality.

[0165] Optionally, the plurality of single-modal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for image modality.

[0166] Optionally, the step of using the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors includes:

[0167] The input image is divided into a grid of fixed-size blocks; each image block is flattened into a one-dimensional vector.

[0168] Each image patch vector is input into a linear layer for processing to obtain the embedding sequence corresponding to each image patch.

[0169] Positional encoding is added to the embedding sequence corresponding to each image patch and input into a standard Transformer encoder. Local and global information in the image is extracted through a self-attention mechanism, and the image modal feature vector is output.

[0170] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0171] The first layer is used to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing both audio and text.

[0172] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0173] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0174] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0175] (1)

[0176] (2)

[0177] (3)

[0178] (4)

[0179] (5)

[0180] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0181] This invention first collects electronic textbooks from various educational publishers, as well as digital textbooks in text, image, audio, and video formats from the internet. These digital textbooks are then manually labeled and categorized as either qualified or unqualified, forming multimodal training samples. Single-modal features are extracted from the multimodal samples, and a cross-attention mechanism is used to fuse these features into a unified feature representation. A deep learning algorithm is then used to train a multimodal classification model. The multimodal classification model is tested, and optimized based on the test results. Finally, the multimodal classification model is used to review the content of the digital textbooks.

[0182] The embodiments of the present invention can not only handle multimodal digital teaching materials and integrate multimodal features to better understand context and adopt a unified processing approach for digital teaching materials of different modalities, but also combine different review models as needed by adding or removing modalities to adapt to the diverse review requirements of digital teaching materials, thereby improving the flexibility, timeliness of response, and accuracy of review results in the review of digital teaching material content.

[0183] Figure 8 This is a schematic diagram of the structure of a multimodal digital textbook content review device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the multimodal digital textbook content review equipment may include the above-mentioned... Figure 7 The multimodal digital textbook content review device shown. Optionally, the multimodal digital textbook content review device 810 may include a first processor 2001.

[0184] Optionally, the multimodal digital textbook content review device 810 may also include a memory 2002 and a transceiver 2003.

[0185] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0186] The following is combined Figure 8 A detailed introduction to each component of the multimodal digital textbook content review device 810:

[0187] The first processor 2001 is the control center of the multimodal digital textbook content review device 810. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0188] Optionally, the first processor 2001 can perform various functions of the multimodal digital textbook content review device 810 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0189] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram.

[0190] In a specific implementation, as one example, the multimodal digital textbook content review device 810 may also include multiple processors, for example... Figure 8 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0191] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0192] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the multimodal digital textbook content review device 810. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0193] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0194] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 8 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0195] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the multimodal digital textbook content review device 810. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0196] It should be noted that, Figure 8 The structure of the multimodal digital textbook content review device 810 shown in the diagram does not constitute a limitation on the router. Actual multimodal digital textbook content review devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0197] Furthermore, the technical effects of the multimodal digital textbook content review device 810 can be referenced from the technical effects of the multimodal digital textbook content review method described in the above method embodiments, and will not be repeated here.

[0198] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0199] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0200] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0201] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0202] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0203] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0204] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0205] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0206] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0209] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal digital textbook content review method, characterized in that, The method includes: S1. Obtain digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, construct digital textbook resources that include text, images, audio, and video; S2. The digital teaching materials resources are labeled by processing the video into fixed-size frames, labeling the image of each frame and the associated audio and text to obtain labeled digital teaching materials resources; the labeled digital teaching materials resources are divided into qualified resources and unqualified resources to construct multimodal training samples. S3. Process the multimodal training samples to obtain multiple single-modal feature vectors; The plurality of single-modal feature vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector of image modality; wherein, the Vision Transformer model is used to extract the human frame feature vector of the image as the supplementary feature vector of image modality; S4. Based on multiple single-modal feature vectors, a three-layer cross-attention fusion mechanism is used to perform multimodal fusion to obtain multimodal feature vectors; The three-layer cross-attention fusion mechanism includes three layers of cross-attention; The first layer is used to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing both audio and text. The process of fusing the audio modal feature vector and the text modal feature vector is represented by the following formulas (1)-(5): （1） （2） （3） （4） （5） Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention; The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images. The third layer is used to fuse the feature vectors containing audio, text, and images with the supplementary feature vectors of the image modality to obtain feature vectors containing audio, text, images, and supplementary image modalities. S5. Construct an initial multimodal classification model; Based on the multimodal feature vectors, use a neural network-based training method to train the initial multimodal classification model to obtain a trained multimodal classification model; The multimodal classification model includes: a fusion vector layer, multiple convolutional layers, pooling layers, and fully connected layers. S6. Obtain the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into the trained multimodal classification model for review, and obtain the review results.

2. The multimodal digital textbook content review method according to claim 1, characterized in that, S3 processes the multimodal training samples to obtain multiple single-modal feature vectors, including: S31. Use a Transformer-based pre-trained language model to process the text data in the multimodal training samples to obtain text modality feature vectors. S32. Use wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors; S33. Use the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modal feature vectors.

3. The multimodal digital textbook content review method according to claim 2, characterized in that, S33 employs the Vision Transformer model to extract features from image data in multimodal training samples, obtaining image modality feature vectors, including: S331. Divide the input image into a grid of fixed-size blocks; flatten each image block into a one-dimensional vector. S332. Input each image block vector into a linear layer for processing to obtain the embedding sequence corresponding to each image block; S333. Add position encoding to the embedding sequence corresponding to each image block and input it into the standard Transformer encoder. Extract local and global information in the image through the self-attention mechanism and output the image modal feature vector.

4. A multimodal digital textbook content review device, wherein the multimodal digital textbook content review device is used to implement the multimodal digital textbook content review method as described in any one of claims 1-3, characterized in that, The device includes: The digital textbook acquisition unit is used to acquire digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, it constructs digital textbook resources that include text, images, audio, and video. The digital textbook annotation unit is used to annotate the digital textbook resources to obtain annotated digital textbook resources; the annotated digital textbook resources are divided into qualified resources and unqualified resources to construct multimodal training samples; The processing unit is used to process multimodal training samples to obtain multiple single-modal feature vectors; The fusion unit is used to perform multimodal fusion based on multiple single-modal feature vectors and adopt a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors; The training unit is used to build an initial multimodal classification model; based on the multimodal feature vectors, a neural network-based training method is used to train the initial multimodal classification model to obtain a trained multimodal classification model. The review unit is used to acquire digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into a trained multimodal classification model for review, and obtain the review results.

5. The multimodal digital textbook content review device according to claim 4, characterized in that, The processing unit is used for: A Transformer-based pre-trained language model is used to process the text data in the multimodal training samples to obtain text modality feature vectors. We used wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors. The Vision Transformer model is used to extract features from the image data in the multimodal training samples to obtain image modality feature vectors.

6. A multimodal digital textbook content review device, characterized in that, The multimodal digital textbook content review equipment includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 3.

Images