Multi-modal digital textbook content examination method and device

Abstract

The invention discloses a multi-mode digital textbook content examination method and device, and relates to the technical field of digital education. The method comprises the following steps: acquiring digital textbooks on the Internet, and constructing digital textbook resources; marking the digital textbook resources to obtain marked digital textbook resources; dividing the marked digital textbook resources into qualified resources and unqualified resources, and constructing a multi-modal training sample; processing the multi-modal training sample to obtain a plurality of single-modal feature vectors; carrying out fusion by adopting a three-layer cross attention fusion mechanism to obtain a multi-modal feature vector; constructing an initial multi-modal classification model; based on the multi-modal feature vector, a training method based on a neural network is adopted for training, and a trained multi-modal classification model is obtained; and inputting the obtained digital textbook to be examined into the trained multi-modal classification model for examination to obtain an examination result. According to the invention, the accuracy of the examination result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital education, in particular to a multi-modal digital teaching material content review method and device. BACKGROUND

[0002] Digital teaching material is a kind of interactive teaching material that uses multimedia technology to deeply integrate text content and rich digital resources, and finally converts it into a form suitable for various electronic terminals.

[0003] In a narrow sense, digital teaching material refers to digital textbooks. In a broad sense, digital teaching material not only includes digital textbooks, but also includes supplementary learning materials such as reference books, hyperlinks to multimedia content and other learning resources. In terms of its form of expression, digital teaching material has the forms of electronic text, image, audio and video. In terms of its carrier, there are various digital education applications such as education APP, small program and cloud platform.

[0004] The rich forms of digital teaching materials can bring students lively and interesting teaching content; its unlimited expansion and flexible adaptability make students' learning more cutting-edge and personalized, making up for the disadvantages of paper teaching materials that cannot adapt to the information age.

[0005] However, the development of digital teaching materials also brings challenges to content review. How to ensure the standardization and scientificity of the content of these teaching materials, as well as the correctness of the ideological attributes, has become an urgent problem. The main manifestations are:

[0006] First, the massive digital teaching material products are difficult to rely on manual review. Massive digital teaching material products and application examples expand the concept, boundary and teaching service of traditional teaching material system, providing rich educational content, while the teaching material or teaching material link also hides pornographic, violent and other harmful information, and there are excessive teaching and vulgar information and other illegal content. Massive teaching resources are difficult to rely on manpower to review their content.

[0007] Second, new technical means are needed for rich forms. The digital teaching resources that students can access now not only contain text, but also audio-visual media, interactive media form content. The rich forms of digital resources also put forward new requirements for the technology and means of content review. A simple rule-based review system cannot understand multi-modal data. A single-modal review system cannot cope with the challenges brought by multi-modal data.

[0008] Third, the content generated by users in real time is difficult to identify in time. UGC (User-Generated Content) or AIGC generated content has the characteristics of strong real-time, but its content implies risks. If harmful content cannot be identified and filtered in time, the adverse effects of its spread among students are difficult to eliminate.

[0009] In the face of the huge challenge brought by digital teaching materials, efficient, fast and accurate digital teaching material content review methods and means that can adapt to the new situation are needed. SUMMARY

[0010] In order to solve the technical problems that the existing technology cannot review the content of the massive teaching resources by manpower, and the existing technology cannot review the multi-modal data, the embodiments of the present application provide a multi-modal digital teaching material content review method and device. The technical solution is as follows:

[0011] On the one hand, a multi-modal digital teaching material content review method is provided, which is implemented by a multi-modal digital teaching material content review device, and the method comprises:

[0012] S1, obtaining digital teaching materials published by various educational publishing houses and digital teaching materials on the Internet; based on the teaching materials, constructing digital teaching material resources containing text, images, audio and video;

[0013] S2, labeling the digital teaching material resources to obtain labeled digital teaching material resources; dividing the labeled digital teaching material resources into qualified resources and unqualified resources to construct multi-modal training samples;

[0014] S3, processing the multi-modal training samples to obtain a plurality of single-modal feature vectors;

[0015] S4, based on the plurality of single-modal feature vectors, using a three-layer cross-attention fusion mechanism for multi-modal fusion to obtain a multi-modal feature vector;

[0016] S5, constructing an initial multi-modal classification model; based on the multi-modal feature vector, using a neural network-based training method to train the initial multi-modal classification model to obtain a trained multi-modal classification model;

[0017] S6, obtaining a digital teaching material to be reviewed; inputting the digital teaching material to be reviewed into the trained multi-modal classification model for review to obtain a review result.

[0018] Optionally, the S3 of processing the multi-modal training samples to obtain a plurality of single-modal feature vectors comprises:

[0019] S31, using a Transformer-based pre-training language model to process the text data in the multi-modal training samples to obtain a text modal feature vector;

[0020] S32, using wav2vec2.0 to extract features from the audio data in the multi-modal training samples to obtain an audio modal feature vector;

[0021] S33, performing feature extraction on image data in the multi-modal training sample by using a Vision Transformer model to obtain an image modality feature vector; and performing feature extraction on the image by using the Vision Transformer model to obtain a supplementary feature vector of the image modality.

[0022] Optionally, the plurality of single-modality feature representation vectors include: a text modality representation feature vector, an audio modality representation feature vector, an image modality feature vector, and a supplementary feature vector of the image modality.

[0023] Optionally, the S33 of performing feature extraction on image data in the multi-modal training sample by using a Vision Transformer model to obtain an image modality feature vector includes:

[0024] S331, dividing an input image into a grid composed of small blocks of a fixed size; and flattening each image block into a one-dimensional vector;

[0025] S332, inputting each image block vector into a linear layer for processing to obtain an embedding sequence corresponding to each image block;

[0026] S333, adding position encoding to the embedding sequence corresponding to each image block, and inputting the embedding sequence into a standard Transformer encoder to extract local information and global information in the image by using a self-attention mechanism, and outputting an image modality feature vector.

[0027] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention layers.

[0028] The first layer is configured to fuse the audio modality feature vector and the text modality feature vector to obtain a feature vector containing audio and text.

[0029] The second layer is configured to fuse the feature vector containing audio and text and the image modality feature vector to obtain a feature vector containing audio, text, and image.

[0030] The third layer is configured to fuse the feature vector containing audio, text, and image and the image frame feature vector to obtain a feature vector containing audio, text, image, and image frame.

[0031] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0032] (1)

[0033] (2)

[0034] (3)

[0035] (4)

[0036] (5)

[0037] wherein, A represents an audio modality feature vector; T represents a text modality feature vector; MA represents a multi-head attention calculation; MA(A, T, T) represents a cross-attention calculation of the modality A feature vector and the modality T feature vector; FFN represents a feed-forward neural network calculation; MA(T, A, A) represents a cross-attention calculation of the modality T feature vector and the modality A feature vector; represents a sum of the feature vector A and the multi-head attention vector; represents a new vector obtained after the audio modality vector A and the text modality vector T pass through a cross-attention unit calculation; represents a sum of the feature vector T and the multi-head attention vector; represents a new vector obtained after the text modality vector T and the audio modality vector A pass through a cross-attention unit calculation; represents a final result feature vector obtained after A and T pass through cross-attention fusion.

[0038] In another aspect, a multi-modal digital textbook content review device is provided, which is applied to a multi-modal digital textbook content review method, and the device comprises:

[0039] A digital textbook collection unit is configured to acquire digital textbooks published by various educational publishing houses and digital textbooks on the Internet, and construct digital textbook resources containing text, images, audio, and video based on the textbooks.

[0040] A digital textbook labeling unit is configured to label the digital textbook resources to obtain labeled digital textbook resources, divide the labeled digital textbook resources into qualified resources and unqualified resources, and construct multi-modal training samples.

[0041] A processing unit is configured to process the multi-modal training samples to obtain a plurality of single-modality feature vectors.

[0042] A fusion unit is configured to perform multi-modal fusion based on the plurality of single-modality feature vectors using a three-layer cross-attention fusion mechanism to obtain a multi-modal feature vector.

[0043] A training unit is configured to construct an initial multi-modal classification model, and train the initial multi-modal classification model based on the multi-modal feature vector using a neural network-based training method to obtain a trained multi-modal classification model.

[0044] The review unit is configured to obtain a digital teaching material to be reviewed, and input the digital teaching material to be reviewed into the trained multi-modal classification model to obtain a review result.

[0045] Optionally, the processing unit is configured to:

[0046] The text data in the multi-modal training sample is processed by using a pre-training language model based on a Transformer to obtain a text modal feature vector.

[0047] The audio data in the multi-modal training sample is subjected to feature extraction by using wav2vec2.0 to obtain an audio modal feature vector.

[0048] The image data in the multi-modal training sample is subjected to feature extraction by using a Vision Transformer model to obtain an image modal feature vector.

[0049] Optionally, the plurality of single-modal feature representation vectors include a text modal representation feature vector, an audio modal representation feature vector, an image modal feature vector, and a supplementary feature vector of the image modal.

[0050] Optionally, the feature extraction of the image data in the multi-modal training sample by using the Vision Transformer model to obtain the image modal feature vector includes:

[0051] The input image is divided into a grid composed of small blocks of a fixed size, and each image block is flattened into a one-dimensional vector.

[0052] Each image block vector is input into a linear layer for processing to obtain an embedding sequence corresponding to each image block.

[0053] The embedding sequence corresponding to each image block is added with position encoding and input into a standard Transformer encoder to extract local information and global information in the image by using a self-attention mechanism, and an image modal feature vector is output.

[0054] Optionally, the three-layer cross-attention fusion mechanism includes three cross-attention layers.

[0055] The first layer is configured to fuse the audio modal feature vector and the text modal feature vector to obtain a feature vector containing audio and text.

[0056] The second layer is configured to fuse the feature vector containing audio and text and the image modal feature vector to obtain a feature vector containing audio, text, and image.

[0057] The third layer is configured to fuse the feature vectors containing audio, text and image and the image frame feature vectors to obtain feature vectors containing audio, text, image and image frame.

[0058] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0059] (1)

[0060] (2)

[0061] (3)

[0062] (4)

[0063] (5)

[0064] wherein A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention calculation; MA(A, T, T) represents cross-attention calculation of the modality A feature vector and the modality T feature vector; FFN represents feed-forward neural network calculation; MA(T, A, A) represents cross-attention calculation of the modality T feature vector and the modality A feature vector. represents the sum of the feature vector A and the multi-head attention vector; represents a new vector obtained by calculating the audio modality vector A and the text modality vector T through a cross-attention unit; represents the sum of the feature vector T and the multi-head attention vector; represents a new vector obtained by calculating the text modality vector T and the audio modality vector A through a cross-attention unit; represents the final result feature vector obtained by fusing A and T through cross-attention.

[0065] In another aspect, a multi-modal digital textbook content review device is provided, which comprises a processor and a memory having computer readable instructions stored thereon, wherein the computer readable instructions are executed by the processor to implement any one of the above multi-modal digital textbook content review methods.

[0066] In another aspect, a computer readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above multi-modal digital textbook content review methods.

[0067] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0068] The embodiment of the present application firstly collects electronic teaching materials of various educational publishing houses and text, image, audio and video digital teaching materials on the Internet; the digital teaching materials such as text, image, audio and video are manually marked and divided into qualified teaching materials and unqualified teaching materials to form multi-modal training samples; single-modal feature extraction is performed on the multi-modal samples, multi-modal fusion is performed on the single-modal features by using a cross attention mechanism to form a unified feature representation form; a multi-modal classification model is trained by using a deep learning algorithm; the multi-modal classification model is tested, and the multi-modal classification model is optimized according to the test result; and the multi-modal classification model is used for content review of digital teaching materials.

[0069] The embodiment of the present application can not only process multi-modal digital teaching materials and fuse multi-modal features, but also better perform context perception, adopt a unified processing mode for digital teaching materials of different modes, and simultaneously combine different review models by increasing and reducing modes as needed, adapt to diversified review requirements of digital teaching materials, improve the flexibility of digital teaching material content review, response timeliness and accuracy of review results. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0071] Figure 1 It is a multi-modal digital teaching material content review method flowchart provided by the embodiment of the present application;

[0072] Figure 2 It is an embedded representation of input text of text mode provided by the embodiment of the present application;

[0073] Figure 3 It is an image feature extraction algorithm schematic diagram provided by the embodiment of the present application;

[0074] Figure 4 It is a human skeleton key point schematic diagram provided by the embodiment of the present application;

[0075] Figure 5 It is a cross attention unit schematic diagram provided by the embodiment of the present application;

[0076] Figure 6 It is a content review model network architecture diagram provided by the embodiment of the present application;

[0077] Figure 7This is a block diagram of a multimodal digital teaching material content review device provided in an embodiment of the present invention;

[0078] Figure 8 This is a schematic diagram of the structure of a multimodal digital textbook content review device provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0080] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0081] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0082] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0083] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0084] This invention provides a method for reviewing the content of multimodal digital teaching materials. This method can be implemented using a multimodal digital teaching material content review device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates the multimodal digital textbook content review method. The processing flow of this method may include the following steps:

[0085] S1. Obtain digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, construct digital textbook resources that include text, images, audio and video.

[0086] The scope of digital textbook collection includes electronic versions of textbooks published by authoritative national educational institutions, digital textbooks from various educational publishers, and electronic textbooks on the Internet.

[0087] In one feasible implementation, the data collection method of this invention includes: communicating with the digital textbook publisher via a communication protocol to obtain content, downloading digital textbooks using web crawling technology, and receiving digital textbooks uploaded by users. Specifically, for digital textbooks presented in text form, after collecting the content, link analysis is performed on the content, and web crawling technology is used to download linked content until the content no longer contains any links. The content of each link is processed as independent text content.

[0088] S2. Label the digital teaching materials to obtain labeled digital teaching materials; divide the labeled digital teaching materials into qualified and unqualified resources, and construct multimodal training samples.

[0089] The annotation methods used in the embodiments of the present invention include: automatic machine annotation, manual annotation, and expert review.

[0090] In one feasible implementation, the text is annotated sentence by sentence, and each sentence is marked as qualified or unqualified based on its content.

[0091] In one feasible implementation, the specific process of image annotation includes: uniformly cropping the image to a fixed size and annotating the image; wherein, when the image contains text, it needs to be extracted separately and annotated with text.

[0092] In one feasible implementation, audio content is annotated sentence by sentence.

[0093] In one feasible implementation, video annotation includes two methods: one is to annotate the entire video; the other is to process the video into fixed-size frames and annotate the image of each frame as well as the audio and text associated with that frame.

[0094] S3. Process the multimodal training samples to obtain multiple single-modal feature vectors.

[0095] Optionally, the multiple unimodal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for the image modality.

[0096] Optionally, the specific implementation process of S3 includes S31-S33:

[0097] S31. Use a Transformer-based pre-trained language model to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0098] in, Figure 2This is a schematic diagram of the embedding representation of input text in a text modality provided by an embodiment of the present invention. In one feasible implementation, the input layer of the pre-trained language model based on Transformer adds special start markers "[CLS]" and "[SEP]" to the text. The input information is composed of word vectors, segment vectors, and position vectors superimposed. The input layer can be a single sentence or a pair of sentences; a sentence consists of multiple tokens; word vectors are the vector representation of tokens, with each word being converted into a fixed-length vector; segment vectors represent the segmentation of the sentence; if the input layer is a pair of sentences, the segment vectors corresponding to all tokens in the first sentence are 0, and the segment vectors corresponding to all tokens in the second sentence are 1; position vectors ensure the order of words, adding positional information to the input sequence.

[0099] S32. Use wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors;

[0100] Among them, wav2vec2.0 is an audio pre-training model based on self-supervised learning. It learns deep representations of audio through training on large-scale unlabeled audio data. Its core idea is to transform audio signals into a discrete latent representation that contains both the temporal information of the audio and can capture the semantic content of the audio.

[0101] S33. Use the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; use the Vision Transformer model to extract the human frame features of the image as supplementary feature vectors of the image modality.

[0102] Specifically, an image feature extraction method based on Vision Transformer (ViT) is employed to extract features as a representation of the image. ViT is an application of Transformer in the vision field. ViT treats images as sequences of smaller patches, thereby enabling it to capture global relationships and long-range dependencies in visual data.

[0103] Optionally, the specific implementation process of S33 includes S331-S333:

[0104] S331. Divide the input image into a grid of fixed-size blocks; flatten each image block into a one-dimensional vector.

[0105] Each image block is 16x16 pixels in size. Each image block is flattened into a one-dimensional vector, thus effectively converting the two-dimensional image into a series of block vectors.

[0106] S332. Input each image block vector into a linear layer for processing to obtain the embedding sequence corresponding to each image block;

[0107] In one feasible implementation, each image patch vector is processed through a linear layer and projected into a high-dimensional space. The projection process creates an embedding sequence, with one embedding corresponding to each patch.

[0108] S333. Add position encoding to the embedding sequence corresponding to each image block and input it into the standard Transformer encoder. Extract local and global information in the image through the self-attention mechanism and output the image modal feature vector.

[0109] Among them, positional encoding injects information about the spatial location of each image patch, helping the model preserve the structure of the original image.

[0110] In one feasible implementation, image patches are input into a standard Transformer encoder. This encoder consists of multiple layers of multi-head self-attention and feedforward neural networks. Through the self-attention mechanism, the model learns the relationships between different image patches, thereby capturing local and global patterns in the image.

[0111] In one feasible implementation, a special learnable tag is added to the front of the block embedding sequence. The learnable tag is a classification tag labeled "Class". After processing by the Transformer encoder, the output corresponding to the classification tag is used as the overall representation of the image for task classification.

[0112] Among them, such as Figure 3 The diagram illustrates an image feature extraction algorithm provided by an embodiment of the present invention. The process of further extracting the frame features of a person based on image modalities includes: extracting the position information of 18 key points of the human body using OpenPose for the entire image; wherein, the 18 key points can outline the frame of the person, such as... Figure 4 The figure shown is a schematic diagram of key points of a human skeleton provided by an embodiment of the present invention; an image feature extraction method based on ViT is used to extract features as the feature representation of the image; specifically, the image is divided into multiple small blocks of 16x16 pixels in size; for each small block, it is determined whether there are key points, and if not, the block is kept unchanged; for each small block containing key points, the middle pixel of the key point is determined, which is represented by the following formula (1):

[0113] (1)

[0114] in, Represents the x-coordinate of the middle pixel. The ordinate represents the middle pixel, and n represents the number of keypoints. This represents the x-coordinate of the j-th key point. This represents the ordinate of the j-th key point;

[0115] The Manhattan distance method is used to calculate the distance between each other pixel in the block and the midpoint, which is then used as the new pixel value of the block, expressed by the following formula (2):

[0116] (2)

[0117] in, Represents two pixels and The distance between them;

[0118] In this process, each block is filled with new pixel values ​​to form a new smaller block. For each smaller block, image features are extracted using a ViT-based image feature extraction method to obtain the frame feature vector of the person. OpenPose can extract the keypoint positions of multiple people in the image; however, this embodiment of the invention only selects the keypoint positions of one person for calculation.

[0119] In one feasible implementation, for the video modality, the video frames are divided into multiple segments based on the speaker's utterance. Each segment corresponds to a single sentence from the speaker, with a length between 1 and 10 seconds, and the corresponding utterance is converted into text. Simultaneously, each video segment is extracted into individual image frames, and the corresponding audio for each frame is saved. The processing of the video modality can involve combining the processing of text, images, and audio.

[0120] S4. Based on multiple single-modal feature vectors, a three-layer cross-attention fusion mechanism is used to perform multimodal fusion to obtain multimodal feature vectors.

[0121] Multimodal fusion represents data from different modalities in a unified form. This invention employs a feature-level multimodal data fusion method based on cross-attention. Cross-attention is an attention mechanism used in multimodal analysis to fuse information from different modalities. It can cross-calculate attention scores between modalities such as images and text to extract shared features and enhance the representational power of each modality. After processing data from different modalities separately, their respective feature representations are obtained. Semantic fusion of these features enables in-depth interaction and feature fusion across modalities. This feature-level data fusion method provides a flexible framework, unrestricted by the number of single-modal inputs. In the fusion model, adding or removing modalities does not require considering the interaction effects of previous modalities; only cross-attention calculation is performed. This characteristic gives the feature fusion model better scalability and flexibility, making it more convenient to adapt to different numbers of single-modal inputs.

[0122] Cross attention establishes an attention relationship between two different sequences; what makes cross attention unique is that the sources of the query Q, key K, and value V are different.

[0123] In one feasible implementation, the calculation process for cross-modal attention from text modality to audio modality is represented by the following formulas (3)-(5):

[0124] (3)

[0125] (4)

[0126] (5)

[0127] Where t represents the text modality; a represents the audio modality; and v represents the video modality; This indicates a query originating from modality t; Represents the text modal feature vector; This represents a bond originating from mode a; Represents the audio modal feature vector; This represents the value from mode a; , , , This represents the weight matrix of the linear transformation of the first text feature. This represents the weight matrix of the linear transformation of the second text features. The weight matrices representing the linear transformations of the third text features are all obtained through training. The dimension of the feature vector representing the text modality. The dimension of the feature vector of the audio modality.

[0128] The attention weights are represented by the following formula (6):

[0129] (6)

[0130] in, Indicator key The size of the dimension, express The transpose of .

[0131] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0132] Each cross-attention layer consists of two parallel cross-attention units; where, for example... Figure 5 The diagram shown is a schematic of a cross-attention unit provided in an embodiment of the present invention; for the cross-attention unit, Q comes from one mode, and K and V come from another mode.

[0133] The first layer is used to fuse the audio modal feature vector and the text modal feature vector to obtain a feature vector containing both audio and text.

[0134] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (7)-(11):

[0135] (7)

[0136] (8)

[0137] (9)

[0138] (10)

[0139] (11)

[0140] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0141] In this embodiment of the invention, the cross-attention layer has three layers. This indicates the result after the first layer of cross-attention fusion, while the result of the third layer corresponds to the result in the embodiment of this invention. Figure 6 The result of the fusion vector layer processing. Among them, formula (7) and formula (8) are the calculation of cross attention units for mode A and mode T. The calculation process of formula (7) corresponds to the process of multi-head attention, residual connection and layer normalization. Formula (8) corresponds to the process of feedforward neural network, residual connection and layer normalization. Among them, formula (9) and formula (10) are the calculation of cross attention units for mode T and mode A. Formula (9) corresponds to the process of multi-head attention, residual connection and layer normalization. Formula (10) corresponds to the process of feedforward neural network, residual connection and layer normalization. Among them, formula (11) means that a new feature vector is obtained by concatenating the feature vectors of the two modes.

[0142] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0143] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0144] S5. Construct an initial multimodal classification model; based on the multimodal feature vectors, use a neural network-based training method to train the initial multimodal classification model to obtain a trained multimodal classification model.

[0145] Among them, such as Figure 6 The diagram shown is a network architecture diagram of a content moderation model provided by an embodiment of the present invention; wherein, the multimodal classification model includes: a fusion vector layer, multiple convolutional layers, a pooling layer, and a fully connected layer.

[0146] In one feasible implementation, a deep learning neural network algorithm is used to train the model. In this embodiment of the invention, for multimodal digital teaching materials, after single-modal feature extraction, a one-dimensional convolutional network is used to align the internal features of the three modalities: text, audio, and image. A cross-modal cross-attention mechanism is used to realize the interaction between text-audio, text-image, and audio-image, thereby achieving intermodal information interaction and feature enhancement, generating richer multimodal features after the fusion of text, audio, and images. An average pooling strategy is adopted for dimensionality reduction, and finally, a fully connected layer is used for classification, with a softmax layer converting the output into probabilities.

[0147] S6. Obtain the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into the trained multimodal classification model for review, and obtain the review results.

[0148] In one feasible implementation, the present invention first employs an optimization method for a single-modal classification model, and then optimizes the classification model constructed in the present invention by employing an optimization method for a multimodal classification model.

[0149] Among them, the method of optimizing multimodal classification models can flexibly obtain classification models containing different modalities by adding or reducing modalities according to the characteristics of the data and application needs.

[0150] In the optimization process, this embodiment of the invention introduces hyperparameter search algorithms, such as grid search or Bayesian optimization, to systematically explore the model architecture space, including parameters such as the number of attention heads, the depth of the fusion layer, and the dropout rate, thereby improving the robustness and generalization performance of the model. For single-modality models, a five-fold cross-validation method is used during the training phase to obtain several sets of optimal model parameters, mainly including parameters such as the number of convolutional layers and the size of the convolutional kernel. In addition, for the optimization of the text modality, this embodiment of the invention combines learning rate decay strategies, such as cosine annealing scheduling and L2 regularization, to prevent overfitting; for audio and image modalities, batch normalization and data augmentation techniques, such as random cropping and noise injection, are integrated to improve the stability of feature extraction. Through single-modality optimization, several sets of optimal model parameters for different modalities are obtained. The common parameter values ​​of the same parameter in different modalities are selected as the optimal candidate parameter values ​​for the combined modality, and the five-fold cross-validation method is also used to select the optimal parameters.

[0151] In the multimodal fusion process, this embodiment of the invention employs a gradient pruning algorithm to avoid the gradient explosion problem and uses the AdamW optimizer combined with weight decay to further refine the training dynamics of the cross-attention mechanism, ensuring the effective integration of multimodal feature vectors. This embodiment of the invention can also improve the model's generalization ability by increasing the number of training samples. Specifically, it uses generative adversarial networks or diffusion models to synthesize additional multimodal samples, thereby expanding the dataset size and mitigating the class imbalance problem.

[0152] This invention first collects electronic textbooks from various educational publishers, as well as digital textbooks in text, image, audio, and video formats from the internet. These digital textbooks are then manually labeled and categorized as either qualified or unqualified, forming multimodal training samples. Single-modal features are extracted from the multimodal samples, and a cross-attention mechanism is used to fuse these features into a unified feature representation. A deep learning algorithm is then used to train a multimodal classification model. The multimodal classification model is tested, and optimized based on the test results. Finally, the multimodal classification model is used to review the content of the digital textbooks.

[0153] The embodiments of the present invention can not only handle multimodal digital teaching materials and integrate multimodal features to better understand context and adopt a unified processing approach for digital teaching materials of different modalities, but also combine different review models as needed by adding or removing modalities to adapt to the diverse review requirements of digital teaching materials, thereby improving the flexibility, timeliness of response, and accuracy of review results in the review of digital teaching material content.

[0154] Figure 7 This is a block diagram of a multimodal digital textbook content review device provided in an embodiment of the present invention. This device is used for a multimodal digital textbook content review method. (Refer to...) Figure 7 The device includes a digital textbook acquisition unit 710, a digital textbook annotation unit 720, a processing unit 730, a fusion unit 740, a training unit 750, and a review unit 760. Among them:

[0155] The digital textbook acquisition unit 710 is used to acquire digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, it constructs digital textbook resources that include text, images, audio, and video.

[0156] The digital textbook annotation unit 720 is used to annotate the digital textbook resources to obtain annotated digital textbook resources; the annotated digital textbook resources are divided into qualified resources and unqualified resources to construct multimodal training samples;

[0157] The processing unit 730 is used to process multimodal training samples to obtain multiple single-modal feature vectors;

[0158] The fusion unit 740 is used to perform multimodal fusion based on multiple single-modal feature vectors and adopt a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors.

[0159] Training unit 750 is used to build an initial multimodal classification model; based on the multimodal feature vectors, a neural network-based training method is used to train the initial multimodal classification model to obtain a trained multimodal classification model;

[0160] The review unit 760 is used to acquire the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into a trained multimodal classification model for review, and obtain the review results.

[0161] Optionally, the processing unit 730 is configured to:

[0162] A Transformer-based pre-trained language model is used to process the text data in the multimodal training samples to obtain text modality feature vectors.

[0163] We used wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors.

[0164] The Vision Transformer model is used to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; the Vision Transformer model is also used to extract the human frame features of the image as supplementary feature vectors for the image modality.

[0165] Optionally, the plurality of single-modal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for image modality.

[0166] Optionally, the step of using the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors includes:

[0167] The input image is divided into a grid of fixed-size blocks; each image block is flattened into a one-dimensional vector.

[0168] Each image patch vector is input into a linear layer for processing to obtain the embedding sequence corresponding to each image patch.

[0169] Positional encoding is added to the embedding sequence corresponding to each image patch and input into a standard Transformer encoder. Local and global information in the image is extracted through a self-attention mechanism, and the image modal feature vector is output.

[0170] Optionally, the three-layer cross-attention fusion mechanism includes three layers of cross-attention;

[0171] The first layer is used to fuse the audio modal feature vector and the text modal feature vector to obtain a feature vector containing both audio and text.

[0172] The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images.

[0173] The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

[0174] Optionally, the process of fusing the audio modality feature vector and the text modality feature vector is represented by the following formulas (1)-(5):

[0175] (1)

[0176] (2)

[0177] (3)

[0178] (4)

[0179] (5)

[0180] Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

[0181] This invention first collects electronic textbooks from various educational publishers, as well as digital textbooks in text, image, audio, and video formats from the internet. These digital textbooks are then manually labeled and categorized as either qualified or unqualified, forming multimodal training samples. Single-modal features are extracted from the multimodal samples, and a cross-attention mechanism is used to fuse these features into a unified feature representation. A deep learning algorithm is then used to train a multimodal classification model. The multimodal classification model is tested, and optimized based on the test results. Finally, the multimodal classification model is used to review the content of the digital textbooks.

[0182] The embodiments of the present invention can not only handle multimodal digital teaching materials and integrate multimodal features to better understand context and adopt a unified processing approach for digital teaching materials of different modalities, but also combine different review models as needed by adding or removing modalities to adapt to the diverse review requirements of digital teaching materials, thereby improving the flexibility, timeliness of response, and accuracy of review results in the review of digital teaching material content.

[0183] Figure 8 This is a schematic diagram of the structure of a multimodal digital textbook content review device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the multimodal digital textbook content review equipment may include the above-mentioned... Figure 7 The multimodal digital textbook content review device shown. Optionally, the multimodal digital textbook content review device 810 may include a first processor 2001.

[0184] Optionally, the multimodal digital textbook content review device 810 may also include a memory 2002 and a transceiver 2003.

[0185] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0186] The following is combined with Figure 8 A detailed introduction to each component of the multimodal digital textbook content review device 810:

[0187] The first processor 2001 is the control center of the multimodal digital textbook content review device 810. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0188] Optionally, the first processor 2001 can perform various functions of the multimodal digital textbook content review device 810 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0189] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram.

[0190] In a specific implementation, as one example, the multimodal digital textbook content review device 810 may also include multiple processors, for example... Figure 8 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0191] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0192] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the multimodal digital textbook content review device 810. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0193] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0194] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 8 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0195] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the multimodal digital textbook content review device 810. Figure 8 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0196] It should be noted that, Figure 8 The structure of the multimodal digital textbook content review device 810 shown in the diagram does not constitute a limitation on the router. Actual multimodal digital textbook content review devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0197] Furthermore, the technical effects of the multimodal digital textbook content review device 810 can be referenced from the technical effects of the multimodal digital textbook content review method described in the above method embodiments, and will not be repeated here.

[0198] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0199] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0200] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0201] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0202] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0203] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0204] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0205] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0206] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0209] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal digital textbook content review method, characterized in that, The method includes: S1. Obtain digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, construct digital textbook resources that include text, images, audio, and video; S2. Label the digital teaching materials resources to obtain labeled digital teaching materials resources; divide the labeled digital teaching materials resources into qualified resources and unqualified resources, and construct multimodal training samples; S3. Process the multimodal training samples to obtain multiple single-modal feature vectors; S4. Based on multiple single-modal feature vectors, a three-layer cross-attention fusion mechanism is used to perform multimodal fusion to obtain multimodal feature vectors; S5. Construct an initial multimodal classification model; Based on the multimodal feature vectors, use a neural network-based training method to train the initial multimodal classification model to obtain a trained multimodal classification model; S6. Obtain the digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into the trained multimodal classification model for review, and obtain the review results.

2. The multimodal digital textbook content review method according to claim 1, characterized in that, S3 processes the multimodal training samples to obtain multiple single-modal feature vectors, including: S31. Use a Transformer-based pre-trained language model to process the text data in the multimodal training samples to obtain text modality feature vectors. S32. Use wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors; S33. Use the Vision Transformer model to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; use the Vision Transformer model to extract the human frame features of the image as supplementary feature vectors of the image modality.

3. The multimodal digital textbook content review method according to claim 1, characterized in that, The plurality of single-modal feature representation vectors include: text modality representation feature vector, audio modality representation feature vector, image modality feature vector, and supplementary feature vector for image modality.

4. The multimodal digital textbook content review method according to claim 2, characterized in that, S33 employs the Vision Transformer model to extract features from image data in multimodal training samples, obtaining image modality feature vectors, including: S331. Divide the input image into a grid of fixed-size blocks; flatten each image block into a one-dimensional vector. S332. Input each image block vector into a linear layer for processing to obtain the embedding sequence corresponding to each image block; S333. Add position encoding to the embedding sequence corresponding to each image block and input it into the standard Transformer encoder. Extract local and global information in the image through the self-attention mechanism and output the image modal feature vector.

5. The multimodal digital textbook content review method according to claim 1, characterized in that, The three-layer cross-attention fusion mechanism includes three layers of cross-attention; The first layer is used to fuse the audio modal feature vector and the text modal feature vector to obtain a feature vector containing both audio and text. The second layer is used to fuse the feature vectors of audio and text with the feature vectors of image modalities to obtain feature vectors that include audio, text and images. The third layer is used to fuse the feature vectors containing audio, text, and images with the feature vectors of the image boxes to obtain feature vectors containing audio, text, images, and image boxes.

6. The multimodal digital textbook content review method according to claim 5, characterized in that, The process of fusing the audio modal feature vector and the text modal feature vector is represented by the following formulas (1)-(5): （1） （2） （3） （4） （5） Where A represents the audio modality feature vector; T represents the text modality feature vector; MA represents multi-head attention computation; MA(A,T,T) represents calculating the cross-attention between the feature vectors of modality A and modality T; FFN represents feedforward neural network computation; MA(T,A,A) represents calculating the cross-attention between the feature vectors of modality T and modality A. This represents the sum of feature vector A and the multi-head attention vector; This indicates that the audio modality vector A and the text modality vector T are processed by a cross-attention unit to obtain a new vector; This represents the sum of the feature vector T and the multi-head attention vector; This indicates that the text modality vector T and the audio modality vector A are processed by a cross-attention unit to obtain a new vector; This represents the final feature vector obtained after A and T are fused through cross-attention.

7. A multimodal digital textbook content review device, wherein the multimodal digital textbook content review device is used to implement the multimodal digital textbook content review method as described in any one of claims 1-6, characterized in that, The device includes: The digital textbook acquisition unit is used to acquire digital textbooks published by various educational publishers and digital textbooks on the Internet; based on the textbooks, it constructs digital textbook resources that include text, images, audio, and video. The digital textbook annotation unit is used to annotate the digital textbook resources to obtain annotated digital textbook resources; the annotated digital textbook resources are divided into qualified resources and unqualified resources to construct multimodal training samples; The processing unit is used to process multimodal training samples to obtain multiple single-modal feature vectors; The fusion unit is used to perform multimodal fusion based on multiple single-modal feature vectors and adopt a three-layer cross-attention fusion mechanism to obtain multimodal feature vectors; The training unit is used to build an initial multimodal classification model; based on the multimodal feature vectors, a neural network-based training method is used to train the initial multimodal classification model to obtain a trained multimodal classification model. The review unit is used to acquire digital teaching materials to be reviewed; input the digital teaching materials to be reviewed into a trained multimodal classification model for review, and obtain the review results.

8. The multimodal digital textbook content review device according to claim 7, characterized in that, The processing unit is used for: A Transformer-based pre-trained language model is used to process the text data in the multimodal training samples to obtain text modality feature vectors. We used wav2vec2.0 to extract features from the audio data in the multimodal training samples to obtain audio modality feature vectors. The Vision Transformer model is used to extract features from the image data in the multimodal training samples to obtain image modality feature vectors; the Vision Transformer model is also used to extract the human frame features of the image as supplementary feature vectors for the image modality.

9. A multimodal digital textbook content review device, characterized in that, The multimodal digital textbook content review equipment includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images