Subject knowledge point multi-modal resource association generation method based on knowledge graph

Abstract

The embodiment of the invention discloses a subject knowledge point multi-modal resource association generation method based on a knowledge graph, and the method comprises the steps: obtaining knowledge data of different modals, and the different modals comprise at least one of a text, a video, an audio and an image; according to the knowledge points, associating the knowledge data of different modalities; and in a knowledge graph taking the knowledge points as nodes, displaying the mutually associated modal knowledge data as multimedia resource links of the knowledge points to which the modal knowledge data belong. According to the embodiment of the invention, automatic association of the multi-modal resources in the knowledge graph can be realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method for generating multimodal resource associations of subject knowledge points based on a knowledge graph. Background Art

[0002] Knowledge points based on knowledge graphs can bring significant significance to education, scientific research and other fields. It displays the knowledge structure graphically, makes the learning context clear, and promotes personalized learning and interdisciplinary connections. At the same time, knowledge graphs can integrate diverse resources such as videos, audio, and pictures to enrich the learning experience. Teachers can monitor students' mastery in real time, improve teaching efficiency, and adjust teaching strategies as needed. In addition, the vivid graph display can also stimulate students' interest in learning, enhance their learning motivation, and improve the overall teaching quality.

[0003] However, traditional knowledge graph construction usually relies on manual annotation and expert knowledge, and has limitations such as high labor costs, lagging knowledge updates, and poor scalability. Especially in terms of associating multimodal resources (such as video, audio, and pictures), there are more technical difficulties such as data heterogeneity, annotation difficulties, information redundancy, architectural complexity, and computing resource consumption, which makes the construction process of knowledge graphs more complicated and resource-intensive, limiting its widespread application.

[0004] Therefore, there is an urgent need for a method that can automatically associate multimodal resources in a knowledge graph to reduce labor costs and improve the speed and efficiency of knowledge update and expansion. Patent CN112015955A discloses a multimodal data association method and device, and patent CN117035081A discloses a multivariate multimodal knowledge graph construction method and device, but neither of them can solve the above problems well. Summary of the invention

[0005] An embodiment of the present invention provides a method for generating multimodal resource associations of subject knowledge points based on a knowledge graph to solve the above-mentioned technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating multimodal resource associations of subject knowledge points based on a knowledge graph, comprising:

[0007] Acquire knowledge data in different modalities, wherein the different modalities include at least one of text, video, audio, and image;

[0008] According to the knowledge points they belong to, the knowledge data of different modalities are associated;

[0009] In the knowledge graph with knowledge points as nodes, the interrelated knowledge data of each modality are displayed as multimedia resource links to the corresponding knowledge points.

[0010] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0011] one or more processors;

[0012] a memory for storing one or more programs,

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating multimodal resource associations of subject knowledge points based on knowledge graphs as described in any embodiment.

[0014] In a third aspect, an embodiment of the present invention further provides a subject knowledge point multimodal resource association generation system based on a knowledge graph, comprising:

[0015] A data acquisition module, used to acquire and collect multimodal data;

[0016] The knowledge extraction module is used to extract knowledge points and knowledge data of different modalities from the collected multimodal data;

[0017] A knowledge graph construction module is used to construct a knowledge graph with knowledge points as nodes according to the knowledge points and knowledge data of different modalities, and to associate knowledge data of different modalities according to the knowledge points;

[0018] The user interface module is used to display the knowledge graph in a graphical manner and display the interrelated modal knowledge data as multimedia resource links of the corresponding knowledge points.

[0019] To sum up, an embodiment of the present invention provides a method for generating multimodal resource associations for subject knowledge points based on a knowledge graph, which can automatically associate multimodal knowledge data belonging to the same knowledge point and display them in the knowledge graph, providing multimedia resource links for each knowledge point. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a collaboration flow chart between modules of a subject knowledge point multimodal resource association generation system based on a knowledge graph provided by an embodiment of the present invention;

[0022] Figure 2It is a flow chart of a method for generating associations of multimodal resources of subject knowledge points based on a knowledge graph provided by an embodiment of the present invention;

[0023] Figure 3 It is a structural schematic diagram of a knowledge association model provided by an embodiment of the present invention;

[0024] Figure 4 It is a structural diagram of an encoder and decoder model in the training phase provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0027] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0028] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] The embodiment of the present invention provides a method for generating multimodal resource associations of subject knowledge points based on a knowledge graph. In order to illustrate the method, a system for generating multimodal resource associations of subject knowledge points based on a knowledge graph that supports the implementation of the method is first described. The system adopts a modular structure, including a data acquisition module, a knowledge extraction module, a knowledge graph construction module, and a user interface module. Each module is independent in function and is easy to maintain and upgrade. At the same time, the system uses a distributed database to store knowledge graphs and related resources to ensure high availability and scalability of data.

[0030] Among them, the data acquisition module is mainly used to realize the following functions:

[0031] Multi-source data acquisition: Automatically collect educational resources in various forms such as teaching materials, documents, videos, audio and pictures through API interfaces, crawler technology and online platforms.

[0032] Automated data capture and processing: Use automated scripts and scheduling tools to regularly capture updated data and convert it into a standard format for subsequent processing.

[0033] The knowledge extraction module is mainly used to achieve the following functions:

[0034] Natural language processing: Use large language models to analyze text data, automatically extract key knowledge points, definitions and concepts, and identify contextual relationships.

[0035] Video analysis: Use a large multimodal model to process video content, automatically extract visual information and voice content related to knowledge points, and identify important scenes and key points.

[0036] Audio analysis: The same large model technology is used for audio processing to extract key knowledge information from explanations and discussions to ensure comprehensive integration of knowledge.

[0037] Image processing: Use large models to perform image classification and feature extraction, identify relevant knowledge points in images, and achieve effective integration of multimodal information.

[0038] The knowledge graph construction module is mainly used to achieve the following functions:

[0039] Automated construction: Combine the results of knowledge extraction and use algorithms to automatically generate a knowledge graph to form a structure of nodes and edges.

[0040] Dynamic update mechanism: Set periodic update and real-time update functions to ensure that new knowledge and relationships are incorporated into the knowledge graph in a timely manner, automatically detect data changes and trigger updates.

[0041] The user interface module is mainly used to implement the following functions:

[0042] Visual display: Design an intuitive user interface to display the knowledge graph in a graphical way, so that users can browse and interact easily.

[0043] Personalized learning path recommendation: Based on the user's learning history and preferences, provide personalized recommended knowledge points and learning resources to improve learning outcomes.

[0044] Multimedia resource links: Provide relevant video, audio and picture links for each knowledge point, which users can access directly to enhance the learning experience.

[0045] At the same time, an evaluation and feedback mechanism has been established in the system, which is mainly used to achieve the following functions:

[0046] Automated feedback collection: Design a feedback mechanism so that users can evaluate the accuracy and practicality of the knowledge graph, and the system automatically records and analyzes the feedback data.

[0047] Effect evaluation: Regularly analyze system usage data, evaluate the effect of knowledge extraction and user satisfaction, and use automated analysis tools to continuously optimize system performance.

[0048] In summary, the system adopts a modular design to ensure efficient collaboration between various functional modules. Figure 1 The collaborative process between the modules of the system is shown. The system combines automation technology and advanced large model technology to achieve visual expression of subject knowledge points, and can automatically identify related knowledge points and resources from multiple resources (such as videos, audios and pictures). This integration not only improves the efficiency of knowledge graph construction, but also solves the problems of manual annotation and knowledge update lag in traditional methods. Through this system, users can obtain a dynamic and intelligent knowledge graph platform, provide personalized learning paths and rich multimedia resources, and promote more efficient learning and teaching experience.

[0049] After understanding the above system, Figure 2 1 is a flow chart of a method for generating multimodal resource associations of subject knowledge points based on a knowledge graph provided by an embodiment of the present invention. The method can be executed by the various modules in the above system, or by other electronic devices independent of the above system. Figure 2 As shown, the method specifically includes:

[0050] S110. Acquire knowledge data in different modalities, where the different modalities include at least one of text, video, audio, and image.

[0051] The knowledge data here can be understood as every piece of knowledge data associated with a certain knowledge point.

[0052] Optionally, multiple pieces of knowledge data can be collected through API interfaces, crawler technology and online platforms, and content segmentation can be performed when necessary. Among them, the knowledge points associated with text data can be predetermined through natural language processing technology, and the knowledge points associated with knowledge data of other modalities are unknown.

[0053] S120. Associate knowledge data of different modalities according to the knowledge points to which they belong.

[0054] This step uses a knowledge point association model based on deep learning to associate knowledge data belonging to the same knowledge point. Figure 3 is a schematic diagram of the structure of a knowledge association model provided by an embodiment of the present invention, such as Figure 3 As shown, the knowledge point association model includes encoders of different modes and association calculation branches under different reference modes. The encoders of each mode are used to encode the knowledge data of each mode respectively; the association calculation branches under each reference mode include a conversion matrix, a bias matrix and a similarity calculation layer. The conversion matrix and the bias matrix are used to uniformly convert the encoding features of knowledge data of different modes to the encoding space of the reference mode, and the similarity calculation layer is used to calculate the similarity between the converted features of knowledge data of different modes.

[0055] In particular, in this embodiment, according to the amount of information of different modal data, a higher feature dimension is set for the modality with a larger amount of information, so as to represent the effective information in the high-information modality with as rich a dimension as possible. Optionally, the encoding feature dimension of the video modality>the encoding feature dimension of the audio modality>the encoding feature dimension of the image modality>the encoding feature dimension of the text modality.

[0056] Based on the above model, S120 may include the following steps:

[0057] Step 1: Encode the knowledge data of different modalities through encoders of different modalities to obtain the encoding features of the knowledge data of each modality. Figure 3 , multiple pieces of knowledge data in four modes, text, video, audio, and image, can be input into the corresponding encoders respectively to obtain the encoding features corresponding to the four modes. , , and .

[0058] Step 2: Select the reference mode of the current learning topic according to the maximized representation form of the learning topic. Since different modes correspond to different encoding feature dimensions, this embodiment selects a reference mode according to the learning topic. This mode corresponds to the best knowledge form of the current learning topic and can accurately display the knowledge content under the current learning topic. In this embodiment, the best knowledge form is also referred to as the maximized representation form, where maximization refers to the maximization of information accuracy.

[0059] Optionally, if the current learning topic is descriptive knowledge, the maximum representation form of descriptive knowledge, "text", can be used as the reference mode of the current learning topic. Exemplarily, if the current learning topic is literature, and the maximum representation form of literature is text, text is used as the reference mode of literature.

[0060] If the current learning topic is static visual knowledge, the image of the maximum representation form of static visual knowledge is selected as the reference mode of the current learning topic. For example, if the current learning topic is fine arts, and the maximum representation form of fine arts is image, the image is selected as the reference mode of fine arts.

[0061] If the current learning topic is dynamic visual knowledge, the video of the maximum representation form of dynamic visual knowledge is selected as the reference mode of the current learning topic. For example, if the current learning topic is animation design, and the maximum representation form of animation design is video, the video is selected as the reference mode of animation design.

[0062] If the current learning topic is auditory knowledge, the audio, which is the maximal representation form of auditory knowledge, is selected as the reference mode of the current learning topic. For example, if the current learning topic is music, and the maximal representation form of music is audio, audio is selected as the reference mode of music.

[0063] Step 3: uniformly convert the encoding features of each modal knowledge data under the current learning topic into the encoding space of the reference modality. Figure 3 After selecting the reference modality, the encoding features of any modality knowledge data under the current learning topic can be converted into the encoding space of the reference modality by the following formula:

[0064]

[0065] Where n represents the index of each modality type, n=1,2,3,4…, Represents the encoding features of the knowledge data of the modality type with index n; S represents the index of the reference modality, and S is one of all modality type indices. express The features after switching to the encoding space of the reference mode, and denote the transformation matrix and the bias matrix from the encoding space of the modality type indexed as n to the encoding space of the reference modality indexed as S. For example, if the feature dimension of the encoding space indexed as n is a and the feature dimension of the encoding space of the reference modality is b, then is a column vector of dimension = a, is a matrix of dimension = b´a, and are all column vectors of dimension = b, and The elements in are all trained in advance.

[0066] Specifically, Figure 3 For example, and They respectively represent the conversion matrix and bias matrix from the encoding space of the video modality to the encoding space of the reference modality when the text modality is used as the reference modality; and They respectively represent the conversion matrix and bias matrix from the coding space of the audio mode to the coding space of the reference mode when the text mode is used as the reference mode; and They respectively represent the conversion matrix and bias matrix from the encoding space of the image modality to the encoding space of the reference modality when the text modality is used as the reference modality; , and They represent the features of the encoded features of the video modality, audio modality, and image modality after being converted to the encoding space of the reference modality when the text modality is used as the reference modality. Similarly, and They respectively represent the conversion matrix and bias matrix from the encoding space of the text modality to the encoding space of the reference modality when the video modality is used as the reference modality; and They respectively represent the conversion matrix and bias matrix from the coding space of the audio mode to the coding space of the reference mode when the video mode is used as the reference mode; and They respectively represent the conversion matrix and bias matrix from the coding space of the image modality to the coding space of the reference modality when the video modality is used as the reference modality; , and They respectively represent the features of the encoded features of the text modality, audio modality and image modality after being converted to the encoding space of the reference modality when the video modality is used as the reference modality. The meanings of the remaining variable symbols are similar and will not be repeated one by one.

[0067] Step 4: Determine the multimodal data belonging to the same knowledge point based on the encoded features of each modal knowledge data after conversion. Specifically, each piece of knowledge data in each modality after conversion corresponds to a , calculate every two features of different modes (Right now and , ), and the features with similarity higher than the set threshold correspond to the multimodal data belonging to the same knowledge point.

[0068] In practical applications, the structure of each modal encoder can be flexibly selected as needed, and if necessary, the knowledge data input to the encoder can be preprocessed to meet the input requirements of the encoder. Exemplarily, the text encoder can use the Glove structure or other text vectorization models; the audio encoder can use the VGGish structure, the audio knowledge data can be first converted to the frequency domain to generate a log-Mel spectrum diagram, and then the log-Mel spectrum diagram is converted to a coding vector; the video encoder can use the ResNet structure, and for the video knowledge data, a few frames of video images can be extracted from it, and each frame of the image can be adjusted to the specified size of the model input, and the BGR format of each frame is converted to RGB format through cv2.cvtColor, and then input into the video encoder; the image encoder can also use image processing structures such as ResNet.

[0069] Further, in a specific embodiment, in order to make Figure 3 The model shown implements the above-mentioned knowledge point association function, and the model can be trained in the following ways:

[0070] First, based on Figure 4 The network structure shown in the figure connects the corresponding decoder to the encoder of each modality, and trains the encoder and decoder as a whole. Specifically, for the encoder-decoder model of any modality, after the original knowledge data is input into the encoder, the encoder extracts the deep features (i.e., encoding features) of the original knowledge data, and the decoder restores the original knowledge data according to the deep features. The encoder and decoder can be trained as a whole by minimizing the difference between the restored knowledge data and the original knowledge data. Optionally, the following loss function can be set:

[0071]

[0072] Where i represents the sample index, and Represent the original knowledge data and the restored knowledge data in the sample respectively. It should be noted that if the data input to the encoder is the preprocessed knowledge data, then and They also correspond to the preprocessed data. By minimizing L1, it can be ensured that the encoder extracts the most important information dimension in the knowledge data and minimizes the information loss in the encoding process.

[0073] Then, remove each decoder and connect the association calculation branch under each reference mode after each modal encoder to obtain the following: Figure 3 The model structure shown. At the same time, a sample set is constructed for the model, and the multimodal knowledge data belonging to the same knowledge point under various learning topics with each modality as the benchmark modality is used as a positive sample, and the multimodal knowledge data belonging to different knowledge points is used as a negative sample to train the knowledge point association model. During the training process, the parameters of each encoder remain unchanged, and only the parameters in the conversion matrix and the bias matrix under each modality are updated. Optionally, the model training can be completed by constraining the final similarity of the multimodal knowledge data belonging to the same knowledge point to be maximized, and the final similarity of the multimodal knowledge data belonging to different knowledge points to be minimized. For example, the following loss function is set:

[0074]

[0075] Among them, p and q represent knowledge point indexes, ; m and n represent the modal index, ; Represents the encoded features after the knowledge data of modality n belonging to knowledge point p is converted, and The meaning is similar to; Representation characteristics and The similarity of The meaning is similar to; and Respectively represent the weights of the loss terms. By minimizing L2, it can be ensured that after the encoded features of each modality are converted to the vector space of the reference modality, the interrelated knowledge data can be identified in the space.

[0076] S130. In a knowledge graph with knowledge points as nodes, mutually related modal knowledge data are displayed as multimedia resource links of the corresponding knowledge points.

[0077] As shown in the user interface module in the above system, text data can be displayed as the name or attribute of the associated knowledge point in the knowledge graph, and other modal data associated with the text data can be displayed as video, audio and picture links of the knowledge point, which users can directly access to enhance the learning experience.

[0078] Furthermore, if there is more than one knowledge point associated with a certain text data, one knowledge point can be selected as the main knowledge point, and the data of each modality can be associated according to the main knowledge point; each knowledge point can also be encoded by the encoder of the text modality, and the encoded features are converted to the reference modal space, and then the projection of the encoded features after the conversion of each modal knowledge data on the encoded features after the conversion of the specific knowledge point is calculated, and the similarity of each modal knowledge data is calculated according to the projection vector. The similarity at this time is the similarity of the multimodal knowledge data with respect to the specific knowledge point. Correspondingly, the multimodal knowledge data with high similarity should be displayed as the relevant attributes or links of the specific knowledge point in the knowledge graph. In practical applications, the above processing method can be flexibly selected according to the knowledge granularity and scale in the knowledge graph, and this embodiment does not make specific restrictions.

[0079] In summary, this embodiment provides a method for generating multimodal resource associations of subject knowledge points based on knowledge graphs, which can automatically associate multimodal knowledge data belonging to the same knowledge point and display them in the knowledge graph, providing multimedia resource links for each knowledge point. Specifically, this embodiment sets different encoding dimensions for each modality according to the amount of information included in different modalities, and through the model training of the encoder-decoder structure, ensures that each encoder can extract key features under each modality and minimize information loss; at the same time, this embodiment selects the most accurate information modality for expressing the learning content as the reference modality according to the maximization representation form of the learning subject, and converts each modality data into the vector space of the reference modality respectively, and calculates the similarity of each modality data through the converted encoding vector, thereby determining the interrelated multimodal data. The reference modality reflects the dimensions that different learning subjects focus on, and converts other modality data into the encoding space that the learning subject focuses on, which can avoid losing too much information on the dimensions that the learning subject focuses on, and ensure the accuracy of knowledge association as much as possible.

[0080] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 5 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0081] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the subject knowledge point multimodal resource association generation method based on the knowledge graph in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned subject knowledge point multimodal resource association generation method based on the knowledge graph.

[0082] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0084] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating multimodal resource associations of subject knowledge points based on a knowledge graph in any embodiment is implemented.

[0085] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0086] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0087] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0088] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating multimodal resource associations of subject knowledge points based on knowledge graph, characterized in that: include: Acquire knowledge data in different modalities, wherein the different modalities include at least one of text, video, audio, and image; According to the knowledge points they belong to, the knowledge data of different modalities are associated; In the knowledge graph with knowledge points as nodes, the interrelated knowledge data of each modality are displayed as multimedia resource links to the corresponding knowledge points.

2. The method according to claim 1, characterized in that The step of associating knowledge data of different modes according to the knowledge points to which they belong includes: Through encoders of different modes, knowledge data of different modes are encoded respectively to obtain encoding features of knowledge data of each mode, wherein the mode with greater information content has a larger feature dimension. According to the maximal representation form of the learning topic, select the benchmark mode of the current learning topic; The encoding features of the knowledge data of each modality under the current learning topic are uniformly converted into the encoding space of the reference modality; According to the encoded features after conversion of each modal knowledge data, the multimodal data belonging to the same knowledge point is determined.

3. The method according to claim 2, characterized in that The size relationship of the encoding feature dimensions of the different modalities is: video>audio>image>text.

4. The method according to claim 2, characterized in that: The step of selecting the reference mode of the current learning topic according to the maximized representation form of the learning topic includes: If the current learning topic is descriptive knowledge, a text that maximizes the representation of descriptive knowledge is selected as the reference mode of the current learning topic; If the current learning topic is static visual knowledge, a morphological image that maximizes the representation of static visual knowledge is selected as the reference modality of the current learning topic; If the current learning topic is dynamic visual knowledge, a video of the maximum representation form of dynamic visual knowledge is selected as the reference mode of the current learning topic; If the current learning topic is auditory knowledge, the audio mode that maximizes the representation of auditory knowledge is selected as the reference mode of the current learning topic.

5. The method according to claim 2, characterized in that: The step of uniformly converting the encoding features of the knowledge data of each modality under the current learning topic into the encoding space of the reference modality includes: The encoding features of any modal knowledge data under the current learning topic are converted into the encoding space of the reference modality through the following formula: in, Represents the knowledge data encoding features of the modality type indexed as n, and They represent the conversion matrix and bias matrix from the encoding space of the modal type indexed by n to the encoding space of the reference modality, respectively. express Features after transformation to the encoding space of the reference modality.

6. The method according to claim 2, characterized in that The step of determining multimodal data belonging to the same knowledge point according to the encoded features after conversion of each modal knowledge data includes: For each two pieces of knowledge data of different modalities, the similarity of the encoded features after conversion of each two pieces of knowledge data is calculated respectively; Multimodal knowledge data with a similarity greater than a set threshold are determined as multimodal knowledge data belonging to the same knowledge point.

7. The method according to claim 5, characterized in that Before encoding the knowledge data of different modalities by encoders of different modalities to obtain encoding features of the knowledge data of each modality, the method further includes: The corresponding decoders are connected to the encoders of each modality, each encoder is used to encode the original knowledge data, and each decoder is used to restore each knowledge data according to each encoding feature; The knowledge data of each modality is used to train each encoder and decoder. During the training, the difference between the knowledge data after constraint restoration and the original knowledge data is minimized to update the parameters. Each decoder is removed, and the association calculation branch under each modality is connected after each modality encoder. Each association calculation branch includes a conversion matrix, a bias matrix and a similarity calculation layer. The conversion matrix and the bias matrix are used to uniformly convert the encoding features of knowledge data of different modalities to the encoding space of the reference modality. The similarity calculation layer is used to calculate the similarity of the features after the conversion of multi-modal knowledge data; The parameters of each encoder are fixed, and the transformation matrix and bias matrix are trained using multimodal knowledge data under various learning topics. During the training, the parameters are updated by maximizing the final similarity of multimodal knowledge data belonging to the same knowledge point and minimizing the final similarity of multimodal knowledge data belonging to different knowledge points.

8. The method according to claim 1, characterized in that The step of associating knowledge data of different modes according to the knowledge points to which they belong includes: Use large language models to analyze text data, automatically extract key knowledge points, definitions and concepts, and identify contextual relationships; Use a large multimodal model to process video content, automatically extract visual information and voice content related to knowledge points, and identify important scenes and key points. Use large model technology to process audio and extract key knowledge information from explanations and discussions; Use large models to perform image classification and feature extraction to identify relevant knowledge points in images; According to the results of extraction and recognition, the knowledge data of different modalities belonging to the same knowledge point are associated.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating multimodal resource associations of subject knowledge points based on knowledge graphs as described in any one of claims 1-8.

10. A knowledge graph-based subject knowledge point multimodal resource association generation system, characterized in that: include: A data acquisition module, used to acquire and collect multimodal data; The knowledge extraction module is used to extract knowledge points and knowledge data of different modalities from the collected multimodal data; A knowledge graph construction module is used to construct a knowledge graph with knowledge points as nodes according to the knowledge points and knowledge data of different modalities, and to associate knowledge data of different modalities according to the knowledge points; The user interface module is used to display the knowledge graph in a graphical manner and display the interrelated modal knowledge data as multimedia resource links of the corresponding knowledge points.

Images