Educational resource management method and system based on multi-modal semantic analysis

Abstract

The invention relates to the technical field of educational resource management, and discloses an educational resource management method and system based on multi-modal semantic analysis, and the method comprises the steps: obtaining educational resource data and user-related data, and inputting the data into a preset text feature model, an image feature model, an audio feature model and a video feature model respectively, text features, image features, audio features and video features are obtained, then text feature vectors, image feature vectors, audio feature vectors and video feature vectors are obtained through calculation according to the text features, the image features, the audio feature vectors and the video feature vectors, and attention weights among the text feature vectors, the image feature vectors, the audio feature vectors and the video feature vectors are calculated based on a multi-modal semantic fusion model; and according to the attention weights, performing weighted averaging on the attention weights to obtain a resource representation vector, inputting the resource representation vector into a clustering library, then scoring the educational resources retrieved by the user, and outputting the educational resources of which the scores are greater than an optimal score threshold. The method has the following effect that the educational resource management system can perform accurate resource search and recommendation.

Description

Technical Field

[0001] The present invention relates to the technical field of educational resource management, and in particular to an educational resource management method and system based on multimodal semantic analysis. Background Art

[0002] At present, with the development and popularization of Internet technology, more and more learning resources are pouring into the educational resource system. Traditional educational resource management methods are mostly based on classification and keyword search, but this method can only retrieve text information and cannot meet the retrieval needs of multimedia resources such as video and audio. In one existing technology, deep learning-based image recognition technology is used to automatically process image resources; text analysis technology is used to identify and segment long text resources; audio recognition technology is used to process audio resources, etc. These methods have improved the accuracy and efficiency of resource management to a certain extent, but there are also some disadvantages: image recognition has a better effect on high-quality images, but due to differences in image quality and resolution, the recognition accuracy is not stable; text analysis technology has certain requirements on the quality and format of text, and it is difficult to process complex or non-uniform information; audio recognition technology faces similar problems to image recognition, and the quality and format of audio affect the accuracy of recognition.

[0003] There is a problem in the existing technology that the educational resource management system is unable to perform accurate resource search and recommendation. Summary of the invention

[0004] The present invention provides an educational resource management method and system based on multimodal semantic analysis to solve the problem that the educational resource management system cannot perform accurate resource search and recommendation.

[0005] In a first aspect, in order to solve the above technical problems, the present invention provides an educational resource management method based on multimodal semantic analysis, comprising: Acquire educational resource data and user-related data; wherein the user-related data is the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; Inputting the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model respectively to obtain text features, image features, audio features, and video features; According to the text features, the image features, the audio features and the video features, respectively calculating a text feature vector, an image feature vector, an audio feature vector and a video feature vector; Based on a multimodal semantic fusion model, calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector; According to the attention weight, weighted average the text feature vector, the image feature vector, the audio feature vector, and the video feature vector is obtained to obtain a resource representation vector; Inputting the resource representation vector into a clustering library, and outputting all resources whose distances to the resource representation vector are less than a preset target distance as educational resources retrieved by the user; All educational resources retrieved by the user are scored, and educational resources with scores exceeding a pre-configured optimal score threshold are output.

[0006] In an optional implementation, the training process of the text feature model, the image feature model, the audio feature model, and the video feature model includes: Acquire historical text data, historical image data, historical audio data, and historical video data; Performing data preprocessing on the historical text data to obtain text standardization data; Taking the text normalization data as input data, constructing an initial text feature model based on the Glove algorithm, and training the initial text feature model according to the sample text data to obtain a text feature model; Performing data preprocessing on the historical image data to obtain image standardized data; Using the image normalization data as input data, using a deep learning VGG network to build an initial text feature model, and training the initial text feature model according to sample image data to obtain an image feature model; Performing data preprocessing on the historical audio data to obtain audio standardization data; Using the audio normalization data as input data, constructing an initial audio feature model based on a convolutional neural network, and training the initial audio feature model according to the sample audio data to obtain an audio feature model; Performing data preprocessing on the historical video data to obtain video standardized data; The video normalization data is used as input data, an initial video feature model is constructed using a deep learning VGG network, and the initial video feature model is trained according to sample video data to obtain a video feature model.

[0007] In an optional implementation, the text features, the image features, the audio features, and the video features are respectively calculated to obtain a text feature vector, an image feature vector, an audio feature vector, and a video feature vector, including: The text feature vector is calculated using the following formula: The image feature vector is calculated by the following formula: The audio feature vector is calculated by the following formula: The video feature vector is calculated using the following formula: in, Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Respectively represent resource data In the Layer text features, image features, audio features, and video features; Represents the number of modalities for text, image, audio, and video respectively.

[0008] In an optional implementation, the calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on the multimodal semantic fusion model includes: The multimodal semantic fusion model is constructed by the following formula: The attention weight between the text feature vector and the image feature vector is calculated by the following formula: The attention weight of the video feature vector is calculated by the following formula: The attention weight of the audio feature vector is calculated by the following formula: in, Represent the attention weights of text data, image data, video data, and audio data respectively; represents the activation function; Represents a learnable weight matrix for bilinear transformation; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent the dimensions of text feature vector and image feature vector respectively; Represents the dimension of the video feature vector; Represents the dimension of the audio feature vector; and Represents a learnable weight matrix for multi-head attention transformation.

[0009] In an optional implementation, the text feature vector, the image feature vector, the audio feature vector, and the video feature vector are weighted averaged according to the attention weight to obtain a resource representation vector, including: The resource representation vector is calculated using the following formula: in, Represents a resource representation vector; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively.

[0010] In an optional implementation, the step of inputting the resource representation vector into a clustering library and outputting all resources whose distances to the resource representation vector are less than a preset target distance as educational resources retrieved by the user includes: The resource representation vectors are clustered using a clustering algorithm, and randomly selected data points as the initial cluster centers; Initialize the parameters of the clustering algorithm and set the initial cluster center as the first clustering feature; Iterate the first clustering feature to obtain a second clustering feature, and record the number of iterations; Determine whether the second clustering feature is better than the first clustering feature, if so, update the second clustering feature to the first clustering feature; if not, discard the second clustering feature; Continue to the next iteration, when the number of iterations is greater than or equal to the preset number of iterations, output the current cluster center value as the optimal cluster center value; Calculating the distance between the resource representation vector and the optimal cluster center value as the target distance; All resources whose distance to the resource representation vector is less than the target distance are output as educational resources retrieved by the user.

[0011] In an optional implementation, the step of scoring all educational resources retrieved by the user and outputting educational resources with scores exceeding a pre-configured optimal score threshold comprises: The educational resources retrieved by the user are scored using the following formula: like , then output the educational resources; like , then the educational resources are not output; in, Indicates the rating; represents the activation function; and represents the parameters of the scoring model; represents a constant; Represents the optimal scoring threshold.

[0012] In an optional implementation, the configuration process of the optimal scoring threshold includes: Get historical rating data; Preprocessing the historical rating data to obtain non-overlapping subsets; Dividing all the subsets into training set data and validation set data; The training set data is used as input data, an initial threshold model is constructed using a threshold regression algorithm, the initial threshold model is trained according to the validation set data, and the output of the trained initial threshold model is used as the optimal scoring threshold.

[0013] In a second aspect, the present invention provides an educational resource management system based on multimodal semantic analysis, comprising: The data acquisition module is used to acquire educational resource data and user-related data; wherein the user-related data is the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; A feature acquisition module, used to input the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model, respectively, to obtain text features, image features, audio features, and video features; A feature vector calculation module, used to calculate a text feature vector, an image feature vector, an audio feature vector and a video feature vector according to the text feature, the image feature, the audio feature and the video feature; An attention weight calculation module, used to calculate the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on a multimodal semantic fusion model; A resource representation vector calculation module, configured to obtain a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector, and the video feature vector according to the attention weight; An educational resource retrieval module, used for inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; The scoring module is used to score all educational resources retrieved by the user and output the educational resources whose scores exceed a pre-configured optimal scoring threshold.

[0014] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements any one of the educational resource management methods based on multimodal semantic analysis described above.

[0015] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the educational resource management methods based on multimodal semantic analysis described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses an educational resource management method based on multimodal semantic analysis, comprising acquiring educational resource data and user-related data; inputting the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model and a video feature model respectively to obtain text features, image features, audio features and video features; calculating a text feature vector, an image feature vector, an audio feature vector and a video feature vector respectively according to the text features, the image features, the audio features and the video features; calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector and the video feature vector based on a multimodal semantic fusion model; obtaining a resource representation vector after weighted averaging the text feature vector, the image feature vector, the audio feature vector and the video feature vector according to the attention weights; inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; scoring all educational resources retrieved by the user, and performing an output operation on educational resources whose scores exceed a pre-configured optimal scoring threshold. In summary, this method has the following effects: it enables the educational resource management system to perform accurate resource search and recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an educational resource management method based on multimodal semantic analysis provided by the first embodiment of the present invention; Figure 2 It is a structural diagram of an educational resource management system based on multimodal semantic analysis provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Reference Figure 1 The first embodiment of the present invention provides an educational resource management method based on multimodal semantic analysis, comprising the following steps: S11, obtaining educational resource data and user-related data; wherein the user-related data includes the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; S12, inputting the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model respectively to obtain text features, image features, audio features, and video features; S13, calculating a text feature vector, an image feature vector, an audio feature vector, and a video feature vector according to the text feature, the image feature, the audio feature, and the video feature, respectively; S14, calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on the multimodal semantic fusion model; S15, according to the attention weight, performing weighted averaging on the text feature vector, the image feature vector, the audio feature vector, and the video feature vector to obtain a resource representation vector; S16, inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; S17, scoring all educational resources retrieved by the user, and outputting educational resources with scores exceeding a pre-configured optimal score threshold.

[0020] In step S11, educational resource data and user-related data are obtained; wherein the user-related data includes the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource.

[0021] It should be noted that educational resource data refers to the title, knowledge points, labels and text information of the title corresponding to the educational resources. Educational resource data can be obtained by accessing EBSCO's Education Source education database and journals and using Google Scholar and Baidu academic search engines.

[0022] User-related data can be obtained by using Python crawlers to simulate browser behavior, automatically crawl, analyze and collect data from the Internet, obtain web page content by sending HTTP requests using libraries such as requests and selenium, and use the Beautiful Soup library to parse the web page and extract the required data, namely the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource.

[0023] In step S12, the educational resource data and the user-related data are respectively input into a preset text feature model, an image feature model, an audio feature model and a video feature model to obtain text features, image features, audio features and video features.

[0024] Acquire historical text data, historical image data, historical audio data, and historical video data; Performing data preprocessing on the historical text data to obtain text standardization data; Taking the text normalization data as input data, constructing an initial text feature model based on the Glove algorithm, and training the initial text feature model according to the sample text data to obtain a text feature model; Performing data preprocessing on the historical image data to obtain image standardized data; Using the image normalization data as input data, using a deep learning VGG network to build an initial text feature model, and training the initial text feature model according to sample image data to obtain an image feature model; Performing data preprocessing on the historical audio data to obtain audio standardization data; Using the audio normalization data as input data, constructing an initial audio feature model based on a convolutional neural network, and training the initial audio feature model according to the sample audio data to obtain an audio feature model; Performing data preprocessing on the historical video data to obtain video standardized data; The video normalization data is used as input data, an initial video feature model is constructed using a deep learning VGG network, and the initial video feature model is trained according to sample video data to obtain a video feature model.

[0025] It should be noted that the Glove algorithm is an unsupervised learning method for learning word embedding. Its core idea is to capture the semantic relationship between words by modeling the co-occurrence frequency between words. The Glove algorithm combines the method based on global statistical information and the method based on local windows, and directly uses the statistical information in the entire corpus to train word vectors, rather than predicting surrounding words. The Glove algorithm can count the number of co-occurrences of all word pairs in the corpus to form a word co-occurrence matrix. The number of co-occurrences reflects the symbiotic relationship between words and is an important basis for constructing word vectors.

[0026] The significance of the Glove algorithm is that it can capture the semantic relationship between words and obtain word vectors through global statistical information, which makes it more effective when processing large-scale data sets. Compared with Word2Vec, Glove not only relies on local statistical information, but also combines global statistical information to obtain word vectors, which helps to better capture the semantic relationship between words.

[0027] The VGG network is a classic convolutional neural network architecture. The VGG network is very deep. The basic models have 16 layers (VGG16) and 19 layers (VGG19). This deep structure enables the network to learn more features and improve the model's ability to represent images.

[0028] In step S13, a text feature vector, an image feature vector, an audio feature vector and a video feature vector are calculated respectively according to the text feature, the image feature, the audio feature and the video feature.

[0029] The text feature vector is calculated using the following formula: The image feature vector is calculated by the following formula: The audio feature vector is calculated by the following formula: The video feature vector is calculated using the following formula: in, Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Respectively represent resource data In the Layer text features, image features, audio features, and video features; Represents the number of modalities for text, image, audio, and video respectively.

[0030] It should be noted that the modal number refers to the number of natural vibration characteristics of the structural system. Each mode contains a modal frequency, modal vibration shape, modal damping, modal stiffness and modal mass. The modal number can be obtained through modal analysis. Modal analysis is a method for studying the dynamic characteristics of structures. It includes analytical (theoretical) modal analysis, experimental modal analysis and operational modal analysis. The modal number can also be obtained through calculation methods such as finite element analysis, solving the eigenvector or eigenvalue corresponding to the vibration characteristic equation of the structure to obtain the modal parameters.

[0031] In step S14, based on the multimodal semantic fusion model, the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector are calculated.

[0032] The multimodal semantic fusion model is constructed by the following formula: The attention weight between the text feature vector and the image feature vector is calculated by the following formula: The attention weight of the video feature vector is calculated by the following formula: The attention weight of the audio feature vector is calculated by the following formula: in, Represent the attention weights of text data, image data, video data, and audio data respectively; represents the activation function; Represents a learnable weight matrix for bilinear transformation; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent the dimensions of text feature vector and image feature vector respectively; Represents the dimension of the video feature vector; Represents the dimension of the audio feature vector; and Represents a learnable weight matrix for multi-head attention transformation.

[0033] It should be noted that the multimodal semantic fusion model is a model that integrates information from different modalities, such as text, images, audio, video, etc., to obtain richer and more accurate semantic understanding. This model can make full use of the complementarity between different modalities and integrate the information extracted from different modalities into a stable multimodal representation. The multimodal semantic fusion model can improve decision accuracy. It combines data from different modalities to provide more comprehensive information, allowing the multimodal semantic fusion model to make more accurate decisions.

[0034] Attention weight is the core concept in the attention mechanism, which represents the importance that the model attaches to different input information when processing a specific task. Attention weight is calculated by query ( ) and a set of keys ( ), which determines the weight coefficient of each value when aggregating information. These weight coefficients reflect the importance of different input information for the current task. The attention weight can be calculated by attention score functions, such as dot product, scaled dot product, etc. and similarity or correlation; attention weights can also be obtained by applying Function to normalize: the calculated score is passed through Function processing is performed to obtain a normalized weight distribution so that the sum of all weights is 1, forming a valid probability distribution.

[0035] The calculated attention weights allow the model to focus on the most important information for the current task and ignore less important information. The attention weight mechanism also allows the model to dynamically focus on different parts when processing sequence data, thereby improving the performance and accuracy of the model.

[0036] The bilinear transform is a special conformal mapping, which can also be called the Möbius transform. It can transform the transfer function of the linear non-time-varying system filter in the continuous time domain into the transfer function of the linear and translation-invariant filter in the discrete time domain. This transformation is done by The position in the plane is The points of the axis are mapped to the unit circle in the complex plane , thus realizing the conversion between continuous time and discrete time systems. The bilinear transform can maintain the stability of the continuous time filter into the discrete time filter, that is, if the continuous time filter is stable, then the discrete time filter obtained by the bilinear transform is also stable.

[0037] The bilinear transform can maintain the stability of the continuous-time filter into the discrete-time filter, that is, if the continuous-time filter is stable, then the discrete-time filter obtained by the bilinear transform is also stable. The multi-head attention transform extends the traditional attention mechanism. Its core idea is to learn the correlation of different subspaces of data in parallel through multiple different "heads", thereby improving the expressiveness of the model. To obtain the multi-head attention transform, the data input can be first mapped to multiple subspaces through different weight matrices, and each subspace calculates an independent attention score, which is then obtained through the scaled dot product attention operation.

[0038] Multi-head attention projects data through multiple heads, allowing the model to focus on different aspects of the data. In a specific application scenario, for example, in a sentence, one head focuses on the relationship between the subject and the predicate, and the other head will focus on the tense consistency of the context.

[0039] In step S15, the text feature vector, the image feature vector, the audio feature vector and the video feature vector are weighted averaged according to the attention weight to obtain a resource representation vector.

[0040] The resource representation vector is calculated using the following formula: in, Represents a resource representation vector; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively.

[0041] It should be noted that the resource representation vector is a vector used in the field of machine learning and information retrieval. It is used to represent the semantic features of text, images, audio, video or other types of resources. These vectors are extracted by deep learning models, which can capture the meaning of resources and represent the semantic relationship between resources in numerical form. Resource representation vectors can convert abstract semantic features into numerical forms that can be processed by computers or in high-dimensional space. Semantically similar resources will be mapped to adjacent areas, allowing computers to recognize and process the similarities between them.

[0042] In step S16, the resource representation vector is input into a clustering library, and all resources whose distance to the resource representation vector is less than a preset target distance are output as educational resources retrieved by the user.

[0043] Clustering the resource representation vectors using a clustering algorithm, and randomly selecting data points as initial cluster centers; Initialize the parameters of the clustering algorithm and set the initial cluster center as the first clustering feature; Iterate the first clustering feature to obtain a second clustering feature, and record the number of iterations; Determine whether the second clustering feature is better than the first clustering feature, if so, update the second clustering feature to the first clustering feature; if not, discard the second clustering feature; Continue to the next iteration, when the number of iterations is greater than or equal to the preset number of iterations, output the current cluster center value as the optimal cluster center value; Calculating the distance between the resource representation vector and the optimal cluster center value as the target distance; All resources whose distance to the resource representation vector is less than the target distance are output as educational resources retrieved by the user.

[0044] It should be noted that the clustering algorithm is an unsupervised learning method used to divide a set of data points into several clusters so that the data points in the same cluster are similar to each other, while the data points in different clusters are quite different. The clustering algorithm can be obtained through the K-means algorithm, which randomly selects k center points, assigns each data point to the nearest center point, and then calculates the average value of each cluster as the new center point, and repeats this process until the center point no longer changes.

[0045] Clustering algorithms can divide objects in a data set into multiple groups or clusters based on similarity, thereby organizing and classifying data; and reduce the dimension of data and simplify problems through clustering.

[0046] Resource representation vectors are vectors used in the field of machine learning and information retrieval to represent the semantic features of text, images, audio, video or other types of resources. These vectors are extracted by deep learning models, can capture the meaning of resources, and represent the semantic relationship between resources in numerical form. Resource representation vectors can convert abstract semantic features into numerical forms that can be processed by computers or in high-dimensional space. Semantically similar resources will be mapped to adjacent areas, allowing computers to recognize and process the similarities between them.

[0047] In step S17, all educational resources retrieved by the user are scored, and educational resources with scores exceeding a pre-configured optimal score threshold are output. The educational resources retrieved by the user are scored using the following formula: like , then output the educational resources; like , then the educational resources are not output; in, Indicates the rating; represents the activation function; and represents the parameters of the scoring model; represents a constant; Represents the optimal scoring threshold.

[0048] It should be noted that the activation function is a key component in artificial neural networks, which is used to introduce nonlinear factors into each neuron of the neural network. Without the activation function, no matter how many layers the neural network has, it can only perform linear transformations and cannot solve nonlinear problems. The activation function enables the neural network to learn and simulate nonlinear relationships and complex functions and decide whether a neuron should be activated, thereby enhancing the selectivity of the network.

[0049] Furthermore, the configuration process of the optimal scoring threshold includes: Get historical rating data; Preprocessing the historical rating data to obtain non-overlapping subsets; Dividing all the subsets into training set data and validation set data; The training set data is used as input data, an initial threshold model is constructed using a threshold regression algorithm, the initial threshold model is trained according to the validation set data, and the output of the trained initial threshold model is used as the optimal scoring threshold.

[0050] It should be noted that the training set data and the validation set data are two parts of the data set used to evaluate model performance: the training set is the data set used to train the model, that is, the model learns and adjusts parameters through these data to minimize the prediction error; the validation set is the data set used to verify the generalization ability of the model, that is, to evaluate the performance of the model on unseen data to prevent the model from overfitting. The training set is used to train the model to help the model learn how to map from input to output; the validation set is used to evaluate the generalization ability of the model, that is, the model's ability to predict new data.

[0051] The following describes the working process of the present invention using a common scenario as an example. Figure 1Schematic diagram of the working scenario of the method.

[0052] In a language learning center of a university, staff use an intelligent recommendation system to help students find educational resources suitable for them, by obtaining educational resource data and student user-related data; inputting the educational resource data and the student user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model respectively to obtain text features, image features, audio features, and video features; calculating text feature vectors, image feature vectors, audio feature vectors, and video feature vectors respectively based on the text features, the image features, the audio features, and the video features; based on a multimodal semantic fusion model, Calculate the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector; according to the attention weights, obtain a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector, and the video feature vector; input the resource representation vector into a clustering library, and output all resources whose distance to the resource representation vector is less than a target distance as educational resources retrieved by student users; score all educational resources retrieved by the student users, and output educational resources with scores greater than an optimal score threshold, thereby helping students find educational resources suitable for them.

[0053] In summary, the present invention discloses an educational resource management method based on multimodal semantic analysis, including obtaining educational resource data and user-related data; inputting the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model and a video feature model respectively to obtain text features, image features, audio features and video features; calculating a text feature vector, an image feature vector, an audio feature vector and a video feature vector respectively according to the text features, the image features, the audio features and the video features; calculating the attention weights between the text feature vector, the image feature vector, the audio feature vector and the video feature vector based on a multimodal semantic fusion model; according to the attention weights, obtaining a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector and the video feature vector; inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; scoring all educational resources retrieved by the user, and performing an output operation on educational resources whose scores exceed a pre-configured optimal scoring threshold. In summary, this method has the following effects: it enables the educational resource management system to perform accurate resource search and recommendation.

[0054] Reference Figure 2The second embodiment of the present invention provides an educational resource management system based on multimodal semantic analysis, comprising: The data acquisition module is used to acquire educational resource data and user-related data; wherein the user-related data is the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; A feature acquisition module, used to input the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model, respectively, to obtain text features, image features, audio features, and video features; A feature vector calculation module, used to calculate a text feature vector, an image feature vector, an audio feature vector and a video feature vector according to the text feature, the image feature, the audio feature and the video feature; An attention weight calculation module, used to calculate the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on a multimodal semantic fusion model; A resource representation vector calculation module, configured to obtain a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector, and the video feature vector according to the attention weight; An educational resource retrieval module, used for inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; The scoring module is used to score all educational resources retrieved by the user and output the educational resources whose scores exceed a pre-configured optimal scoring threshold.

[0055] It should be noted that the educational resource management system based on multimodal semantic analysis provided in an embodiment of the present invention is used to execute all the process steps of an educational resource management method based on multimodal semantic analysis in the above-mentioned embodiment. The working principles and beneficial effects of the two correspond one to one, and therefore will not be repeated here.

[0056] The embodiment of the present invention also provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data processing terminal program. When the processor executes the computer program, the steps in the above-mentioned method embodiment of the educational resource management method based on multimodal semantic analysis are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are realized, such as the scoring module.

[0057] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0058] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0059] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0060] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0061] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0062] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0063] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An educational resource management method based on multimodal semantic analysis, characterized in that: Executed by a computer, including: Acquire educational resource data and user-related data; wherein the user-related data is the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; According to the educational resource data and the user-related data, the educational resource data and the user-related data are respectively input into a preset text feature model, an image feature model, an audio feature model and a video feature model to obtain text features, image features, audio features and video features; According to the text features, the image features, the audio features and the video features, respectively calculating a text feature vector, an image feature vector, an audio feature vector and a video feature vector; Based on a multimodal semantic fusion model, calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector; According to the attention weight, weighted average the text feature vector, the image feature vector, the audio feature vector, and the video feature vector is obtained to obtain a resource representation vector; Inputting the resource representation vector into a clustering library, and outputting all resources whose distances to the resource representation vector are less than a preset target distance as educational resources retrieved by the user; All educational resources retrieved by the user are scored, and educational resources with scores exceeding a pre-configured optimal score threshold are output.

2. The educational resource management method based on multimodal semantic analysis according to claim 1 is characterized in that: The training process of the text feature model, the image feature model, the audio feature model and the video feature model includes: Acquire historical text data, historical image data, historical audio data, and historical video data; Performing data preprocessing on the historical text data to obtain text standardization data; Taking the text normalization data as input data, constructing an initial text feature model based on the Glove algorithm, and training the initial text feature model according to the sample text data to obtain a text feature model; Performing data preprocessing on the historical image data to obtain image standardized data; Using the image normalization data as input data, using a deep learning VGG network to build an initial text feature model, and training the initial text feature model according to sample image data to obtain an image feature model; Performing data preprocessing on the historical audio data to obtain audio standardization data; Using the audio normalization data as input data, constructing an initial audio feature model based on a convolutional neural network, and training the initial audio feature model according to the sample audio data to obtain an audio feature model; Performing data preprocessing on the historical video data to obtain video standardized data; The video normalization data is used as input data, an initial video feature model is constructed using a deep learning VGG network, and the initial video feature model is trained according to sample video data to obtain a video feature model.

3. The educational resource management method based on multimodal semantic analysis according to claim 2 is characterized in that: According to the text features, the image features, the audio features and the video features, a text feature vector, an image feature vector, an audio feature vector and a video feature vector are calculated respectively, including: The text feature vector is calculated using the following formula: The image feature vector is calculated by the following formula: The audio feature vector is calculated by the following formula: The video feature vector is calculated using the following formula: in, Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Respectively represent resource data In the Layer text features, image features, audio features, and video features; Represents the number of modalities for text, image, audio, and video respectively.

4. The educational resource management method based on multimodal semantic analysis according to claim 1 is characterized in that: The step of calculating the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on the multimodal semantic fusion model includes: The multimodal semantic fusion model is constructed by the following formula: The attention weight between the text feature vector and the image feature vector is calculated by the following formula: The attention weight of the video feature vector is calculated by the following formula: The attention weight of the audio feature vector is calculated by the following formula: in, Respectively represent the attention weight between text data and image data, the attention weight of video data, and the attention weight of audio data; represents the activation function; Represents a learnable weight matrix for bilinear transformation; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent the dimensions of text feature vector and image feature vector respectively; Represents the dimension of the video feature vector; Represents the dimension of the audio feature vector; and Represents a learnable weight matrix for multi-head attention transformation.

5. The educational resource management method based on multimodal semantic analysis according to claim 4 is characterized in that: The step of obtaining a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector, and the video feature vector according to the attention weights includes: The resource representation vector is calculated using the following formula: in, Represents a resource representation vector; Representation Resources The attention weights between the corresponding text feature vector and image feature vector, Representation Resources The attention weight of the corresponding video feature vector, Representation Resources The attention weight of the corresponding audio feature vector; Represent text feature vector, image feature vector, audio feature vector and video feature vector respectively.

6. The educational resource management method based on multimodal semantic analysis according to claim 5 is characterized in that: The step of inputting the resource representation vector into a clustering library and outputting all resources whose distances from the resource representation vector are less than a preset target distance as educational resources retrieved by the user includes: Clustering the resource representation vectors using a clustering algorithm, and randomly selecting data points as initial cluster centers; Initialize the parameters of the clustering algorithm and set the initial cluster center as the first clustering feature; Iterate the first clustering feature to obtain a second clustering feature, and record the number of iterations; Determine whether the second clustering feature is better than the first clustering feature, if so, update the second clustering feature to the first clustering feature; if not, discard the second clustering feature; Continue to the next iteration, when the number of iterations is greater than or equal to the preset number of iterations, output the current cluster center value as the optimal cluster center value; Calculating the distance between the resource representation vector and the optimal cluster center value as the target distance; All resources whose distance to the resource representation vector is less than the target distance are output as educational resources retrieved by the user.

7. The educational resource management method based on multimodal semantic analysis according to claim 6 is characterized in that: The step of scoring all educational resources retrieved by the user and outputting educational resources with scores exceeding a pre-configured optimal score threshold comprises: The educational resources retrieved by the user are scored using the following formula: like , then output the educational resources; like , then the educational resources are not output; in, Indicates the rating; represents the activation function; and represents the parameters of the scoring model; represents a constant; Represents the optimal scoring threshold.

8. The educational resource management method based on multimodal semantic analysis according to claim 7 is characterized in that: The configuration process of the optimal scoring threshold includes: Get historical rating data; Preprocessing the historical rating data to obtain non-overlapping subsets; Dividing all the subsets into training set data and validation set data; The training set data is used as input data, an initial threshold model is constructed using a threshold regression algorithm, the initial threshold model is trained according to the validation set data, and the output of the trained initial threshold model is used as the optimal scoring threshold.

9. An educational resource management system based on multimodal semantic analysis, characterized in that: include: The data acquisition module is used to acquire educational resource data and user-related data; wherein the user-related data is the resource name clicked by the user, the knowledge point corresponding to the resource, and the tag data corresponding to the resource; A feature acquisition module, used to input the educational resource data and the user-related data into a preset text feature model, an image feature model, an audio feature model, and a video feature model, respectively, to obtain text features, image features, audio features, and video features; A feature vector calculation module, used to calculate a text feature vector, an image feature vector, an audio feature vector and a video feature vector according to the text feature, the image feature, the audio feature and the video feature; An attention weight calculation module, used to calculate the attention weights among the text feature vector, the image feature vector, the audio feature vector, and the video feature vector based on a multimodal semantic fusion model; A resource representation vector calculation module, configured to obtain a resource representation vector by weighted averaging the text feature vector, the image feature vector, the audio feature vector, and the video feature vector according to the attention weight; An educational resource retrieval module, used for inputting the resource representation vector into a clustering library, and outputting all resources whose distance to the resource representation vector is less than a preset target distance as educational resources retrieved by the user; The scoring module is used to score all educational resources retrieved by the user and output the educational resources whose scores exceed a pre-configured optimal scoring threshold.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the educational resource management method based on multimodal semantic analysis as described in any one of claims 1 to 8.

Images