A method and system for automatically generating multimodal standardized knowledge graphs

Abstract

The present invention discloses a method and system for automatically generating a multimodal standardized knowledge graph, which specifically relates to the technical field of automatic generation of educational knowledge graphs, and is used to solve the problem of inaccurate knowledge graph generation. The present invention preprocesses and selects features of multimodal data, performs text segmentation, video key frame extraction, and image grayscale conversion, so that different modal data are standardized to eliminate differences between each other, and then screens the key features of each modality through information gain, and performs normalization processing to ensure that the feature vector has a uniform numerical range, laying a foundation for subsequent feature fusion and knowledge graph generation. The weight is calculated through a modal credibility function, and the importance of each modality is considered in the feature fusion process to generate a unified feature vector, avoid hierarchical conflicts, and automatically update according to dynamic changes in data, dynamically adjust knowledge points and their hierarchical labels, thereby improving the intelligence and adaptability of the graph and the efficiency of knowledge management.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic generation of educational knowledge graphs, and more specifically, to a method and system for automatic generation of multimodal standardized knowledge graphs. Background Art

[0002] Multimodal standardized knowledge graph is a technology that integrates information of different modalities (such as text, images, audio, video, etc.) into a unified knowledge structure. It aims to improve the presentation and understanding of knowledge through the association, fusion and analysis of cross-modal data. In the field of education, student learning resources are usually presented in various forms, such as text explanations in textbooks, video explanations, interactive charts, etc. Information in a single modality is often difficult to meet different learning needs. Through the generation of multimodal knowledge graphs, the system can automatically integrate and present educational resources of different modalities, so that students can obtain a more comprehensive knowledge structure in the learning process and enhance learning effects.

[0003] Deficiencies in existing technologies: There are deficiencies in processing the abstract level of knowledge points and the hierarchical correspondence between modalities. The abstract level classification of knowledge points often relies on the analysis of a single modality (such as text), and lacks a deep understanding of other modalities such as images, videos, and audio. As a result, the abstract level division of multimodal information is not accurate enough, and it is unable to fully reflect the complexity and abstractness of knowledge points in different modalities. Secondly, there are limitations in the generation of hierarchical correspondences between modalities. Most of the existing cross-modal mapping algorithms are based on simple semantic similarity calculations, which make it difficult to capture the subtle differences in the levels of knowledge points in each modality, resulting in inaccurate modal alignment. When detecting and processing hierarchical conflicts between modalities, there is a lack of effective mechanisms to resolve the differences between levels, which can easily lead to inconsistent levels in the learning path, resulting in inconsistent levels of information obtained by students in different modalities, which in turn affects the coherence and in-depth understanding of knowledge. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a method and system for automatically generating a multimodal standardized knowledge graph to solve the problem of inaccurate knowledge graph generation in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for automatically generating a multimodal standardized knowledge graph comprises the following steps:

[0007] Collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and preprocess the collected data;

[0008] Perform feature selection on each modality data, determine key features through information gain, and normalize feature vectors of text, video, and image to unify the numerical range of features;

[0009] The credibility values ​​of different modes are calculated according to the modal credibility function, and the modal weights are assigned according to the credibility values. The features of each mode are fused to generate a unified feature vector representation.

[0010] The knowledge points are hierarchically divided according to the fused feature vectors, the corresponding hierarchical labels are generated, and the consistency between modalities is processed to generate a multimodal standardized knowledge graph and automatically update it.

[0011] In a preferred embodiment, multimodal data related to knowledge points are collected from data sources of text modality, video modality and image modality, and the collected data are preprocessed. The specific process is as follows:

[0012] The text modality extracts paragraphs and chapters from electronic textbooks, documents, and online course materials; the video modality extracts clips from video lectures or classroom recordings; the image modality extracts from teaching charts, schematic diagrams, or demonstration images;

[0013] Associate the knowledge points of each modality data and establish a unified knowledge point mapping table between modalities;

[0014] The text data is segmented, stop words are removed, and stemming is performed to generate cleaned text. The frame sequence in the video is obtained, and the frame images are filtered, denoised, and enhanced. The images are grayscaled, edge detected, and enhanced, and the text, video, and image data features are extracted.

[0015] In a preferred embodiment, feature selection is performed on each modality data, key features are determined by information gain, and feature vectors of text, video, and image are normalized to unify the numerical range of features. The specific process is as follows:

[0016] For text, video, and image features, information gain is used to determine the contribution of features to knowledge point classification, and each modality feature is screened based on information gain;

[0017] Sort the features of each modality by information gain, obtain the top features based on the sorting, and form a new feature vector;

[0018] The feature vectors of text, video and image are normalized to the maximum and minimum values ​​to generate the feature vectors of text, video and image.

[0019] In a preferred embodiment, the credibility values ​​of different modalities are calculated according to the modal credibility function, and the modal weights are assigned according to the credibility values. The features of each modality are fused to generate a unified feature vector representation. The specific process is as follows:

[0020] The construction of historical information datasets involves extracting information of different dimensions from multiple data sources and constructing a joint dataset;

[0021] According to the result of the modal credibility function, the weight factor of each modality is calculated, and the text, video, and image modalities are fused to obtain a fused feature vector;

[0022] The abstract level of the fused feature vector is measured, the abstract degree is obtained according to the abstract degree function of the knowledge point, and basic knowledge and high abstraction are distinguished according to the obtained abstract degree.

[0023] In a preferred embodiment, knowledge points are hierarchically divided according to the fused feature vectors, corresponding hierarchical labels are generated, and consistency processing between modalities is performed to generate a multimodal standardized knowledge graph and automatically update it. The specific process is as follows:

[0024] Perform knowledge point hierarchical division and label generation, divide knowledge points into basic level, intermediate level and advanced level, define hierarchical division threshold, and generate hierarchical labels for knowledge points according to hierarchical division threshold;

[0025] After the levels are divided, the hierarchical matching of knowledge points in different modalities is determined through modal consistency verification;

[0026] If the consistency verification function is less than or equal to the consistency verification threshold, it is considered consistent, otherwise the hierarchical expression between modalities is readjusted.

[0027] In a preferred embodiment, a multimodal standardized knowledge graph is generated and automatically updated, and the specific process is as follows:

[0028] Perform modal feature distribution mapping, map the features of each modality to a unified hierarchical space, and compare and correspond the features of different modalities in the same space;

[0029] A mapping function is defined for the features of each modality and mapped to a multi-dimensional hierarchical space. A similarity function between modalities is defined based on the Euclidean distance, and the similarity of knowledge points under different modalities is measured based on the similarity function.

[0030] Define a similarity threshold and generate a hierarchical correspondence between modalities. If the similarity function is greater than or equal to the similarity threshold, the hierarchies of different modalities are consistent, and a unified hierarchical label is generated for the knowledge point. If the similarity function is less than the similarity threshold, the hierarchy between modalities is adjusted.

[0031] After completing the generation of hierarchical correspondence between modalities, the multimodal standardized knowledge graph is automatically generated.

[0032] In a preferred embodiment, if the similarity function is less than the similarity threshold, the inter-modality hierarchy is adjusted, and the specific steps are as follows:

[0033] When the similarity function between the modes is less than the similarity threshold, it is adjusted through the interpolation function, and the transition level is automatically generated to adjust the modal features according to the level difference between the modes;

[0034] After the hierarchy adjustment is completed, the modal hierarchy labels are refitted using the minimized modal hierarchy according to the interpolated hierarchy;

[0035] After generating the final unified hierarchical labels, modal hierarchical consistency verification and feedback are performed based on the squared difference between levels.

[0036] In a preferred embodiment, after the generation of the hierarchical correspondence between modalities is completed, the multimodal standardized knowledge graph is automatically generated, and the specific steps are as follows:

[0037] Map knowledge points to nodes in the knowledge graph according to their features, and determine the associations between nodes;

[0038] According to the association similarity function, edge weights are assigned to the association relationship of each pair of knowledge points;

[0039] Construct a knowledge graph based on the generated nodes and generated edge weights.

[0040] A multimodal standardized knowledge graph automatic generation system, used to implement the above-mentioned multimodal standardized knowledge graph automatic generation method, comprising:

[0041] A modality data processing module is used to collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and pre-process the collected data;

[0042] The feature fusion module is used to select features for each modality data, determine key features through information gain, and normalize the feature vectors of text, video, and image to unify the numerical range of features. The reliability values ​​of different modalities are calculated according to the modality reliability function, and the modality weights are assigned according to the reliability values. The features of each modality are fused to generate a unified feature vector representation.

[0043] The graph generation module is used to divide the knowledge points into levels according to the fused feature vectors, generate corresponding hierarchical labels, perform consistency processing between modalities, generate a multimodal standardized knowledge graph and automatically update it.

[0044] Technical effects and advantages of the present invention:

[0045] The present invention preprocesses and selects features of multimodal data through specific preprocessing methods, such as text segmentation, video key frame extraction, image grayscale, etc., so that different modal data can be extracted under a unified processing framework, eliminating the differences between modalities. The feature selection step selects the key features of each modality through information gain calculation, and then combines normalization processing to unify the numerical range of feature vectors of different modalities, thereby ensuring the accuracy and consistency of subsequent feature fusion;

[0046] The credibility value of each modality is calculated through the modal credibility function, and weights are assigned according to the credibility values. In this way, in the process of multimodal feature fusion, the modality with a higher weight can play a greater role in the generation of feature vectors, thereby improving the representation effect of knowledge points. The fused unified feature vector is used for the hierarchical division of knowledge points, which not only ensures the accuracy of the hierarchical expression of knowledge points, but also avoids hierarchical conflicts between different modalities through inter-modal consistency processing. It can dynamically adjust knowledge points and their hierarchies as multimodal data changes, realizing more intelligent and dynamic knowledge graph management. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of a method for automatically generating a multimodal standardized knowledge graph according to the present invention.

[0048] Figure 2 This is a structural schematic diagram of a multimodal standardized knowledge graph automatic generation system of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example 1: Figure 1 As shown, a method for automatically generating a multimodal standardized knowledge graph includes the following steps:

[0051] Collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and preprocess the collected data;

[0052] Perform feature selection on each modality data, determine key features through information gain, and normalize feature vectors of text, video, and image to unify the numerical range of features;

[0053] The credibility values ​​of different modes are calculated according to the modal credibility function, and the modal weights are assigned according to the credibility values. The features of each mode are fused to generate a unified feature vector representation.

[0054] The knowledge points are hierarchically divided according to the fused feature vectors, the corresponding hierarchical labels are generated, and the consistency between modalities is processed to generate a multimodal standardized knowledge graph and automatically update it.

[0055] Data acquisition and preprocessing of knowledge points. Multimodal data acquisition is to collect multimodal learning data from different data sources. Assume that the knowledge dataset contains three modalities: text modality T, video modality V and image modality I, which are represented as datasets ,Each modality dataset contains different information sources related to the knowledge points;

[0056] The text modality extracts paragraphs and chapters from electronic textbooks, documents, and online course materials, represented as ,in, is the nth text segment;

[0057] The video modality extracts clips from video lectures or classroom recordings, represented as ,in, is the nth video clip;

[0058] Image modalities are extracted from teaching diagrams, schematics, or demonstration images and are represented as in, is the nth image;

[0059] The data of each modality needs to be associated with the corresponding knowledge points to establish a unified knowledge point mapping table between modalities Among them, each Represents the i-th knowledge point. At this time, each knowledge point Contains a set of multimodal data , collect text, video, image and other modal data related to knowledge points from multiple different data sources, and establish cross-modal data mapping for each knowledge point to ensure that the basic data source for subsequent processing is rich and structured.

[0060] After collecting multimodal data, data preprocessing is performed to ensure that the format of each modality is consistent. This step standardizes and filters the data of different modalities. The specific processing includes:

[0061] The text data is segmented, stop words are removed, and stemming is performed to generate cleaned text; the video is segmented and key frames are extracted to obtain the frame sequence in the video, and the frame image is filtered, denoised, and enhanced; the image is grayed, edge detected, and image enhanced; the text data is processed by standard methods such as segmentation, stop word removal, and stemming to ensure the uniformity and analyzability of the text, the video data is processed by extracting key frames and denoising to retain its main feature information, and the image data is processed by graying and edge detection to extract its core features;

[0062] For each modality feature, information gain is used to measure the contribution of the feature to the classification of knowledge points. Represents the jth feature in the modality and defines the information gain of the feature for: ,in, Indicates the category to which the knowledge point belongs. For the characteristics If there is, the probability that the knowledge point belongs to category c is calculated by , filter the features with the highest information gain;

[0063] The features of each modality are sorted by information gain to obtain the top d most representative features of each modality. For the features of text, video, and image modalities, the top d features are selected respectively. features to form a new feature vector, such as the new text feature vector: ;

[0064] The feature selection process is further optimized, and information entropy constraints are used for marginalization processing to ensure that the selected features not only have high information gain, but also reduce feature redundancy between modalities. Through information entropy, modal feature redundancy is controlled and features with large differences are retained, thereby avoiding information overlap and conflict between modalities.

[0065] After feature selection, the text, video, and image features are standardized, and the features of each modality are normalized to the maximum and minimum;

[0066] To ensure that the features of data from different modalities can be represented in a unified semantic space, the feature vectors of each modality are standardized. First, the feature vectors of text, video, and image are normalized to have the same numerical range. The text feature vector of the text modality is The maximum and minimum normalization is performed, and the formula is: ,in, For the cleaned text, the same normalization process is performed on the video frames and images to generate normalized features, namely, video feature vectors and image feature vectors.

[0067] Assign weights to different modes and introduce modal credibility function To measure the reliability of each mode. The modal credibility function is calculated by evaluating the complexity of the mode, the amount of information, and the difference between the mode and other modes. The formula of the credibility value function is: ,in, represents the characteristic complexity of mode i, is the adjustment coefficient, which is used to adjust the change range of the reliability value. The purpose of this formula is to adaptively adjust the reliability weight according to the complexity of the modality to ensure that the data of the complex modality occupies a reasonable proportion during fusion;

[0068] According to the result of the modal credibility function, the weight factor of each mode is calculated as follows: The modal weight value of the modal with higher modal reliability is larger, and the modal weight value of the modal with lower reliability is smaller. The fusion feature vector is obtained by fusing the modalities such as text, video, and image. ;

[0069] The fused feature vector after feature fusion It is necessary to measure the abstract level and define the abstract function of the knowledge point. The function is based on information density and complexity. The expression formula of abstract degree is: ,in, Represents the jth and kth components in the feature vector. The larger the value of the abstraction function, the more abstract the knowledge point level is, thus distinguishing basic knowledge from highly abstract concepts.

[0070] Perform hierarchical division and label generation to divide knowledge points into basic levels , Intermediate level and advanced level , define the hierarchical division threshold , generate hierarchical labels using the following rules: , here, The dynamic threshold is automatically learned based on the overall distribution of knowledge point features and is determined according to the actual demand scenario;

[0071] Perform modal consistency verification of hierarchical labels. After dividing the levels, ensure that the knowledge points of different modalities match each other through modal consistency verification. Define the consistency verification function: ,in, They are the hierarchical labels of text, video and image modalities respectively. Through the consistency verification function, it is judged whether there are significant differences in the hierarchy between modalities. If the consistency verification function is less than or equal to the consistency verification threshold, it is considered consistent. Otherwise, the hierarchical expression between modalities needs to be readjusted. If the consistency verification fails, the hierarchical division will be further adjusted.

[0072] In the process of generating a multimodal standardized knowledge graph, the generation of hierarchical correspondence between modalities aims to ensure that the hierarchical expression of knowledge points under different modalities remains consistent and that the abstract levels of each modality can be effectively aligned.

[0073] According to the classification and annotation of the abstract level of knowledge points, the generated knowledge point level is matched and aligned with the features under different modalities. Through feature mapping, hierarchical relationship generation and inconsistency processing between modalities, the consistency of information in each modality at the abstract level is ensured to avoid conflicts or deviations in the hierarchical expression of knowledge points under different modalities. The specific steps are as follows:

[0074] Step 1: Map modal features to a unified hierarchical space; perform modal feature distribution mapping to map the features of each modality to a unified hierarchical space so that features under different modalities can be compared and corresponded in the same space. For each modal feature, first define a mapping function M to map it to a multidimensional hierarchical space S. The mapping formula is as follows: ,in, is the feature vector of modality M (such as text features, video features, image features), and U is a mapping matrix that maps the features of each modality to a unified hierarchical space S. The generation process of this matrix is ​​based on the semantic distribution of modality features and is obtained by performing principal component analysis (PCA) or singular value decomposition (SVD) on the features of different modalities.

[0075] The feature vector is embedded in a unified hierarchical space. By embedding the features of each modality into a unified hierarchical space, the knowledge point levels of all modalities can be directly compared through this space. In the unified hierarchical space, the levels of knowledge points are represented by a multidimensional vector. Representation, that is, representing the different abstract levels of knowledge points corresponding to each dimension; by mapping the features of each modality to a unified hierarchical space, it is ensured that the features of different modalities can be compared in this space. By introducing the mapping matrix, the features of different modalities can share a common representation form, avoiding the problem of inconsistent levels due to differences between modalities;

[0076] After the features are mapped to the unified hierarchical space, the inter-modal hierarchical similarity function is defined to measure the similarity of knowledge points under different modalities. The similarity calculation is based on the high-order generalization of the Euclidean distance. Considering the distribution differences between different modal features in the multidimensional space, the similarity function is defined as: , where M, N∈|T, V, I|, means that the values ​​of M, N are selected from text features, video features, and image features; the similarity function calculates the distance difference between different modalities and weights them in the form of a cubic function to ensure that modal differences with large distances are significantly punished.

[0077] Step 2: Generate the hierarchical correspondence between modalities by defining the similarity threshold. If the similarity function is greater than or equal to the similarity threshold, the hierarchies of different modalities are considered to be consistent, and a unified hierarchical label is generated for the knowledge point; if the similarity function is less than the similarity threshold, the hierarchy between modalities needs to be adjusted.

[0078] Step 3: Automatic adjustment of modal hierarchy differences. When the similarity function between modalities is less than the similarity threshold, the modal hierarchy with large differences is automatically adjusted. The adjustment process is performed through the interpolation function This function automatically generates transition levels according to the level difference between modes and adjusts the modal features with large differences. The formula of the interpolation function is: ,in, Indicates the modal level to be adjusted, is the median value of the overall hierarchical label, is the average level of the mode, is the adjustment factor to control the smoothness of the interpolation process;

[0079] After the level adjustment is completed, the level labels are refitted according to the interpolated level to ensure that the adjusted modal level meets the unified level standard. The level refitting process is completed by minimizing the level difference between the modes. By minimizing the difference after the modal level adjustment, the fitting formula is: , generate the final fitted unified hierarchical labels, so that the hierarchical expression of each knowledge point in different modalities remains consistent.

[0080] Step 4: Perform hierarchical consistency verification and feedback. After generating the final unified hierarchical label, check whether there is deviation in the hierarchies under different modes through the consistency verification function. The verification function measures the modal consistency based on the square difference between the hierarchies. The formula is: ,Through this formula, the hierarchical deviation values ​​of different modes can be calculated.,If the deviation exceeds the set threshold, it returns to the previous step and readjusts,until the difference in the modal hierarchy converges to the set range.

[0081] For example, in the educational scenario, one of the knowledge points involved in the knowledge graph is "Newton's Second Law", which has different expressions in different modes:

[0082] Text modality: extract descriptive text about “Newton’s Second Law” from the e-textbook, for example: “Newton’s Second Law states that the acceleration of an object is directly proportional to the net force exerted on the object and inversely proportional to the mass of the object.”

[0083] Video mode: Contains a 2-minute video clip of a teacher explaining Newton's second law. The video includes explanations and blackboard demonstrations.

[0084] Graphical modality: Provides a diagram showing the different ways an object moves when different forces are applied.

[0085] First, the features of each modality are extracted and mapped to a unified hierarchical space. Let the text modality, video modality and image modality be the feature representations of the knowledge point respectively. The feature vector of the text modality consists of key terms and syntactic structures. The following features are extracted: Keywords: "force", "acceleration", "mass", syntactic structure: causal relationship ("proportional", "inversely proportional"), and the feature vector is {force, acceleration, mass}''

[0086] The feature extraction of the video modality comes from the key frames and speech transcription content in the video, and the feature vector obtained is {formula F=ma, object motion example, concept of force};

[0087] Image modality extracts visual features in the image, such as the direction arrow of the force, the symbol of the acceleration, and the schematic diagram of the force on the object. The extracted feature vector is {arrow direction, acceleration symbol};

[0088] The feature vector of each modality is mapped to a unified hierarchical space. The text, video, and image features are standardized through the mapping matrix so that the text, video, and image features can be compared in the same semantic space. The mapped features are expressed as: , , ;

[0089] The hierarchical similarity between text, video and image modalities is calculated through the similarity function. The weighted cosine similarity formula is used to substitute the eigenvalue to obtain a similarity value of 0.94. The similarity value of 0.94 indicates that the levels between the modalities are relatively close, that is, the similarity threshold is set to 0.9. Therefore, it is determined that the "Newton's Second Law" in the text, video and image modalities has a consistent hierarchical expression and no further hierarchical adjustment is required. This shows that under multimodality, the hierarchical label of the knowledge point "Newton's Second Law" is consistent, and it is expressed as a medium-to-high-level knowledge point.

[0090] After completing the generation of hierarchical correspondences between modalities, the multimodal standardized knowledge graph is automatically generated to organize each knowledge point and its association into a structured graph so that the knowledge points under different modalities are expressed in a unified form. The following are the specific steps for generating the knowledge graph:

[0091] After the hierarchical correspondence between modalities is generated, each knowledge point has a unified hierarchical label and feature representation. According to these features, the knowledge point is mapped to a node in the knowledge graph, and the node generation function is defined ,in Represents knowledge point i and adds the following information to the node: hierarchical label It is the abstract level of knowledge points, such as basic, intermediate or advanced, modal features Unified feature representation after mapping from different modalities, semantic categories According to the knowledge category generated by feature representation and semantic label (for example, physical laws, mathematical formulas, etc.), the node generation formula is: ;

[0092] To ensure the uniqueness of knowledge point nodes in the knowledge graph, the unique identifier of each node is bound to the knowledge point name and hierarchical label to avoid the same knowledge point appearing repeatedly in the graph;

[0093] Generate knowledge point relationships. After generating knowledge point nodes, calculate the association relationship between nodes. The association relationship between knowledge points is based on feature similarity and semantic relevance. Define the association similarity function to measure the knowledge point The semantic relevance of is as follows: ,in, Representing knowledge points The nth eigencomponent of Representing knowledge points The nth characteristic component of ;

[0094] According to the association similarity function, a weight is assigned to the association relationship of each pair of knowledge points. The higher the weight, the stronger the association between the knowledge points, that is, the larger the edge weight. The edge weight function can be defined as;

[0095] The level difference When included in the calculation, the association weights between knowledge points with large hierarchical differences will be reduced to ensure the hierarchy of the association relationship in the graph;

[0096] Construct the knowledge graph. According to the generated nodes and edge weights, start to construct the knowledge graph and define the graph generation function G(V, E), where V represents the knowledge point node set and E represents the edge set between knowledge points. The graph generation formula is as follows:

[0097] , the generated graph G contains all knowledge point nodes and the relationship edges between each node.

[0098] The knowledge graph is not a static structure and needs to support dynamic updating of the graph. When new knowledge points are added or the characteristics of existing knowledge points change, the knowledge graph should be automatically updated and adjusted.

[0099] Through knowledge graph generation, educational content can be presented in a highly structured manner. For knowledge points such as "Newton's Second Law", the system will place it at the core of the knowledge graph, and link it with other physical laws (such as Newton's First Law, Newton's Third Law), mathematical formulas and other knowledge points through multimodal association. Through this structured display, students can not only learn the core knowledge points, but also clearly see the connection between these knowledge points through the nodes and edges of the graph. This display method helps students establish systematic cognition, especially in interdisciplinary learning. The graph can help students better understand the intersection and application of knowledge. The knowledge graph organizes scattered knowledge points into a complete knowledge network, helping students understand complex knowledge systems and thus improve learning efficiency.

[0100] It should be noted that the thresholds involved in the embodiments may be determined according to specific scenarios and requirements, and are usually adjusted and optimized according to changes in factors such as real-time input data and user requirements.

[0101] The present invention preprocesses and selects features of multimodal data through specific preprocessing methods, such as text segmentation, video key frame extraction, image grayscale, etc., so that different modal data can be extracted under a unified processing framework, eliminating the differences between modalities. The feature selection step selects the key features of each modality through information gain calculation, and then combines normalization processing to unify the numerical range of feature vectors of different modalities, thereby ensuring the accuracy and consistency of subsequent feature fusion;

[0102] The credibility value of each modality is calculated through the modal credibility function, and weights are assigned according to the credibility values. In this way, in the process of multimodal feature fusion, the modality with a higher weight can play a greater role in the generation of feature vectors, thereby improving the representation effect of knowledge points. The fused unified feature vector is used for the hierarchical division of knowledge points, which not only ensures the accuracy of the hierarchical expression of knowledge points, but also avoids hierarchical conflicts between different modalities through inter-modal consistency processing. It can dynamically adjust knowledge points and their hierarchies as multimodal data changes, realizing more intelligent and dynamic knowledge graph management.

[0103] Embodiment 2: A multimodal standardized knowledge graph automatic generation system, such as Figure 2 As shown, specifically including:

[0104] A modality data processing module is used to collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and pre-process the collected data;

[0105] The feature fusion module is used to select features for each modality data, determine key features through information gain, and normalize the feature vectors of text, video, and image to unify the numerical range of features. The reliability values ​​of different modalities are calculated according to the modality reliability function, and the modality weights are assigned according to the reliability values. The features of each modality are fused to generate a unified feature vector representation.

[0106] The graph generation module is used to divide the knowledge points into levels according to the fused feature vectors, generate corresponding hierarchical labels, perform consistency processing between modalities, generate a multimodal standardized knowledge graph and automatically update it.

[0107] The above formulas are all dimensionless and calculated numerically. Specific dimension removal can be achieved by various means such as standardization, which will not be elaborated here. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0108] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, an ATA hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state ATA hard disk.

[0109] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for automatically generating a multimodal standardized knowledge graph, characterized in that: The steps include: Collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and preprocess the collected data; Perform feature selection on each modality data, determine key features through information gain, and normalize feature vectors of text, video, and image to unify the numerical range of features; The credibility values ​​of different modes are calculated according to the modal credibility function, and the modal weights are assigned according to the credibility values. The features of each mode are fused to generate a unified feature vector representation. Divide the knowledge points into layers according to the fused feature vectors, generate corresponding layer labels, and perform consistency processing between modalities to generate a multimodal standardized knowledge graph and automatically update it. Perform feature selection on each modal data, determine key features through information gain, and normalize the feature vectors of text, video, and image to unify the numerical range of features. The specific process is as follows: For text, video, and image features, information gain is used to determine the contribution of features to knowledge point classification, and each modality feature is screened based on information gain; Sort the features of each modality by information gain, obtain the top features based on the sorting, and form a new feature vector; Perform maximum and minimum normalization processing on the feature vectors of text, video and image to generate text, video and image feature vectors; The credibility values ​​of different modes are calculated according to the modal credibility function, and the modal weights are assigned according to the credibility values. The features of each mode are fused to generate a unified feature vector representation. The specific process is as follows: The construction of historical information datasets involves extracting information of different dimensions from multiple data sources and constructing a joint dataset; According to the result of the modal credibility function, the weight factor of each modality is calculated, and the text, video, and image modalities are fused to obtain a fused feature vector; The abstract level of the fused feature vector is measured, the abstract degree is obtained according to the abstract degree function of the knowledge point, and basic knowledge and high abstraction are distinguished according to the obtained abstract degree; According to the fused feature vector, the knowledge points are divided into levels, the corresponding level labels are generated, and the consistency between the modalities is processed to generate a multimodal standardized knowledge graph and automatically update it. The specific process is as follows: Perform knowledge point hierarchical division and label generation, divide knowledge points into basic level, intermediate level and advanced level, define hierarchical division threshold, and generate hierarchical labels for knowledge points according to hierarchical division threshold; After the levels are divided, the hierarchical matching of knowledge points in different modalities is determined through modal consistency verification; If the consistency verification function is less than or equal to the consistency verification threshold, it is considered consistent, otherwise the hierarchical expression between modalities is readjusted.

2. The method for automatically generating a multimodal standardized knowledge graph according to claim 1, characterized in that: Collect multimodal data related to knowledge points from data sources of text modality, video modality, and image modality, and preprocess the collected data. The specific process is as follows: The text modality extracts paragraphs and chapters from electronic textbooks, documents, and online course materials; the video modality extracts clips from video lectures or classroom recordings; the image modality extracts from teaching charts, schematic diagrams, or demonstration images; Associate the knowledge points of each modality data and establish a unified knowledge point mapping table between modalities; The text data is segmented, stop words are removed, and stemming is performed to generate cleaned text. The frame sequence in the video is obtained, and the frame images are filtered, denoised, and enhanced. The images are grayscaled, edge detected, and enhanced, and the text, video, and image data features are extracted.

3. The method for automatically generating a multimodal standardized knowledge graph according to claim 2, characterized in that: Generate a multimodal standardized knowledge graph and automatically update it. The specific process is as follows: Perform modal feature distribution mapping, map the features of each modality to a unified hierarchical space, and compare and correspond the features of different modalities in the same space; A mapping function is defined for the features of each modality and mapped to a multi-dimensional hierarchical space. A similarity function between modalities is defined based on the Euclidean distance, and the similarity of knowledge points under different modalities is measured based on the similarity function. Define a similarity threshold and generate a hierarchical correspondence between modalities. If the similarity function is greater than or equal to the similarity threshold, the hierarchies of different modalities are consistent, and a unified hierarchical label is generated for the knowledge point. If the similarity function is less than the similarity threshold, the hierarchy between modalities is adjusted. After completing the generation of hierarchical correspondence between modalities, the multimodal standardized knowledge graph is automatically generated.

4. The method for automatically generating a multimodal standardized knowledge graph according to claim 3, characterized in that: If the similarity function is less than the similarity threshold, the inter-modal hierarchy is adjusted. The specific steps are as follows: When the similarity function between the modes is less than the similarity threshold, it is adjusted through the interpolation function, and the transition level is automatically generated to adjust the modal features according to the level difference between the modes; After the hierarchy adjustment is completed, the modal hierarchy labels are refitted using the minimized modal hierarchy according to the interpolated hierarchy; After generating the final unified hierarchical labels, modal hierarchical consistency verification and feedback are performed based on the squared difference between levels.

5. The method for automatically generating a multimodal standardized knowledge graph according to claim 4, characterized in that: After completing the generation of hierarchical correspondences between modalities, the multimodal standardized knowledge graph is automatically generated. The specific steps are as follows: Map the knowledge points to nodes in the knowledge graph according to their features, and determine the associations between the nodes; According to the association similarity function, edge weights are assigned to the association relationship of each pair of knowledge points; Construct a knowledge graph based on the generated nodes and generated edge weights.

6. A multimodal standardized knowledge graph automatic generation system, used to implement a multimodal standardized knowledge graph automatic generation method according to any one of claims 1 to 5, characterized in that: include: A modality data processing module is used to collect multimodal data related to knowledge points from data sources of text modality, video modality and image modality, and pre-process the collected data; The feature fusion module is used to select features for each modality data, determine key features through information gain, and normalize the feature vectors of text, video, and image to unify the numerical range of features. The reliability values ​​of different modalities are calculated according to the modality reliability function, and the modality weights are assigned according to the reliability values. The features of each modality are fused to generate a unified feature vector representation. The graph generation module is used to divide the knowledge points into levels according to the fused feature vectors, generate corresponding hierarchical labels, perform consistency processing between modalities, generate a multimodal standardized knowledge graph and automatically update it.

Images