Knowledge graph construction method and system, electronic equipment and storage medium

Abstract

The invention relates to a knowledge graph construction method, and relates to the field of intelligent teaching, and the method comprises the steps: obtaining course data of at least two modes, carrying out the feature extraction of the course data, obtaining the data features of each mode, carrying out the correlation fusion of the data features based on a preset multilayer neural network, and obtaining the fusion features, feature labeling is conducted on the fused features, combined feature representation and collaborative feature representation are obtained, the combined feature representation is used for mapping information of all modalities into the same feature space, the collaborative feature representation is used for mapping information of all modalities, and a knowledge graph corresponding to the course data is constructed based on the combined feature representation and the collaborative feature representation. Through the method and the device, the problem of inaccurate classroom knowledge graph construction is solved. And a multi-modal fusion labeling technology is adopted, so that the representation capability of the data is improved, the expression of the same knowledge point in different modal data is consistent, and the inaccuracy caused by insufficient data fusion is reduced.

Description

Technical Field

[0001] The present application relates to the field of intelligent teaching, and in particular to a knowledge graph construction method, system, electronic device and storage medium. Background Art

[0002] Using classroom teaching data to automatically construct a knowledge graph of a course is conducive to better understanding, management and organization of the knowledge domain of the course.

[0003] Classroom teaching data includes multimodal data such as audio, video, images, and text, and some courses involve some abstract concepts and logical reasoning. For example, mathematics widely uses various symbols and formulas. This leads to huge differences in the expression of the same content in different modal data, making it difficult to construct a correct knowledge graph.

[0004] Currently, no effective solution has been proposed for the problem of inaccurate construction of classroom knowledge graphs in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a knowledge graph construction method, system, electronic device and storage medium to at least solve the problem of inaccurate classroom knowledge graph construction in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a knowledge graph, the method comprising: Obtaining course data of at least two modes, performing feature extraction on the course data, and obtaining data features of each mode; Based on a preset multi-layer neural network, correlation fusion is performed on the data features to obtain fusion features; Performing feature annotation on the fused features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map each modality information into the same feature space, and the collaborative feature representation is used to map the information of each modality respectively; Based on the joint feature representation and the collaborative feature representation, a corresponding relationship is established between the course data and the corresponding knowledge points to construct a course knowledge graph.

[0007] In some embodiments, the feature labeling of the fused features to obtain the joint feature representation and the collaborative feature representation includes: Performing dimensionality reduction processing on the fusion features based on principal component analysis and linear discriminant analysis; Inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain joint prediction results and collaborative prediction results; According to the joint prediction result and the collaborative prediction result, feature annotation is performed on the fusion feature to obtain the joint feature representation and the collaborative feature representation.

[0008] In some embodiments, inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain a joint prediction result includes: Based on the cross-modal similarity method, a fusion loss function is constructed; With the goal of minimizing the fusion loss function value, the data features of each modality are combined to obtain the joint prediction result.

[0009] In some embodiments, the fusion loss function includes:

[0010] Among them, t and v are feature samples randomly selected from data features of two different modalities, α is the margin, S(.) is the similarity measurement function, t - is a feature sample in t that does not match v, v - are the feature samples in v that do not match t.

[0011] In some embodiments, combining the data features of each modality to obtain the combined prediction result includes: Based on the joint model, the data features of each modality are converted into a shared subspace to obtain the joint prediction result. The combined model includes:

[0012] Among them, z represents the joint prediction result, n represents the number of modal types, the subscripts represent different modalities, v represents the data features of each modality, w represents the weight corresponding to each modality, and f (.) represents the mapping of the semantic transformation of each modality to the shared subspace.

[0013] In some embodiments, combining the data features of each modality to obtain the combined prediction result includes: Based on the multiplication joint model, the data features of each modality are fused into a unified tensor to obtain the joint prediction result. The multiplication joint model includes:

[0014] Wherein, z represents the joint prediction result, n represents the number of modal types, superscripts represent different modalities, and v represents the data features of each modality. represents the outer product operator.

[0015] In some embodiments, the step of inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain a collaborative prediction result includes: Based on the preset classifier weights, classifier decision fusion is performed to obtain the prediction probability after fusing multiple classifier information under the current mode; Modal information fusion is performed based on preset modal weights, and the collaborative prediction result is obtained according to the modal information fusion result and the prediction probability.

[0016] In some of the embodiments, the course data in at least two modalities are selected from text data, image data, audio data and video data.

[0017] In a second aspect, an embodiment of the present application provides a knowledge graph construction system, the system comprising: A feature extraction module is used to obtain course data of at least two modes, perform feature extraction on the course data, and obtain data features of each mode; A feature fusion module, used to perform correlation fusion on the data features based on a preset multi-layer neural network to obtain fusion features; A feature annotation module, used to perform feature annotation on the fusion features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately; A graph construction module is used to construct a knowledge graph corresponding to the course data based on the joint feature representation, the collaborative feature representation and the fusion feature.

[0018] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the knowledge graph construction method as described in the first aspect above is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge graph construction method as described in the first aspect above.

[0020] Compared with the related art, the knowledge graph construction method provided in the embodiment of the present application obtains course data of at least two modalities, extracts features from the course data, obtains data features of each modality, performs correlation fusion on the data features based on a preset multi-layer neural network, obtains fusion features, and performs feature annotation on the fusion features to obtain joint feature representation and collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately, and the knowledge graph corresponding to the course data is constructed based on the joint feature representation and the collaborative feature representation, which solves the problem of inaccurate construction of classroom knowledge graphs. The multimodal fusion annotation technology is used to improve the representation ability of data, so that the expression of the same knowledge point in different modal data is consistent, and the inaccuracy caused by insufficient data fusion is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flowchart of a knowledge graph construction method according to an embodiment of the present application; Figure 2 It is a structural block diagram of a knowledge graph construction system according to an embodiment of the present application; Figure 3 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0025] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0026] This embodiment provides a method for constructing a knowledge graph. Figure 1 is a flowchart of a method for constructing a knowledge graph according to an embodiment of the present application. Figure 1 As shown, the process includes the following steps: Step S101, obtaining course data of at least two modes, performing feature extraction on the course data, and obtaining data features of each mode.

[0027] In some of the embodiments, the course data in at least two modalities are selected from text data, image data, audio data, and video data.

[0028] In this embodiment, information from two or more modalities is combined to predict the knowledge points corresponding to the course data. In the prediction process, the information contained in a single modality usually cannot obtain accurate prediction results. The multimodal fusion process combines information from two or more modalities to achieve information supplementation, broaden the coverage of information contained in the input data, improve the accuracy of the prediction results, and improve the robustness of the prediction model.

[0029] Some course data (such as mathematics course content) contains abstract content (such as mathematical concepts and formulas) presented in multiple modalities such as text, images, audio and video. Through multimodal fusion, the mathematics course content information from multiple modalities is integrated to improve the comprehensiveness and accuracy of data features.

[0030] Since the original data of each modality of text, image, audio, and video in the course content have different data characteristics, feature-level fusion is adopted, that is, the feature representation is first extracted from each modality, and then the features of various modalities are fused.

[0031] The text data T, image data G, audio data A, and video data V in the course data are segmented respectively to obtain the corresponding text data segment t, image data segment g, audio data segment a, and video data segment v: (t , g , a , v)=partition(T , G , A , V) After the mathematics course content is split, the data segments t, g, a, and v are extracted respectively to obtain the text data segment feature f t , image data features f g , audio data features f a , video data features fv: (f t , f g , f a , f v )=feature(t , g , a , v) Step S102, performing correlation fusion on the data features based on a preset multi-layer neural network to obtain fused features.

[0032] The above four data features f t 、f g 、f a 、f vConnect to get the fused feature f U : f U =concatenate(f t , f g , f a , f v ) Step S103 , feature labeling is performed on the fused features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately.

[0033] In order to match data features with mathematics course knowledge points in order to build and utilize knowledge graphs, it is necessary to label the features of mathematics course content accordingly. This labeling is performed during the model training phase.

[0034] The results of multimodal fusion annotation are represented by features, including joint representations and coordinated representations. Joint representations are used to map the information of each modality into the same feature space, while coordinated representations are used to map the information of each modality separately, and it is necessary to ensure that there are certain constraints between each modality after mapping so that they enter the coordinated space.

[0035] In some embodiments, step S103 specifically includes: Step S1031, performing dimension reduction processing on the fusion features based on principal component analysis and linear discriminant analysis.

[0036] The fusion features generated after the transformation and scaling of each modal feature usually have a high dimension. The fusion features are reduced in dimension using principal component analysis and linear discriminant analysis to obtain the reduced-dimensional fusion feature f U ': f U =PCA(f U ) Step S1032, input the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain joint prediction results and collaborative prediction results.

[0037] Step S1033: perform feature labeling on the fusion features according to the joint prediction results and the collaborative prediction results to obtain joint feature representation and collaborative feature representation.

[0038] The multimodal data of mathematics courses after dimensionality reduction is fused with features f U ' Input the multi-layer perceptron MLP and get the prediction result y. y=mlp(f U ') The predicted result y will be labeled and thus associated with the knowledge graph of the mathematics course content.

[0039] In some embodiments, in step S1032, the fusion features after dimension reduction are input into a pre-built multi-layer perceptron to obtain a joint prediction result including: Step S201, constructing a fusion loss function based on the cross-modal similarity method.

[0040] The cross-modal similarity method maintains the similarity structure between and within modalities under the constraint of similarity measurement, so that the cross-modal similarity distance of the same semantics or related objects is as small as possible, and the distance of different semantics is as large as possible. The inter-modal ranking method is used to complete the visual and text fusion task, and the matching embedding vectors of vision and text are represented as (v, t)∈D. The fusion objective function is represented by the loss function f. This method maintains the similarity structure between and within modalities, while achieving mutual synergy between modalities.

[0041] In some embodiments, the fusion loss function includes:

[0042] Among them, t and v are feature samples randomly selected from data features of two different modalities, α is the margin, S(.) is the similarity measurement function, t - is a feature sample in t that does not match v, v - are the feature samples in v that do not match t.

[0043] Step S202, with the goal of minimizing the fusion loss function value, the data features of each modality are combined to obtain a joint prediction result.

[0044] In some of the embodiments, the combination is performed by an additive combination method, and step S202 specifically includes: based on an additive combination model, converting the data features of each modality into a shared subspace to obtain a joint prediction result.

[0045] The combined model includes:

[0046] Among them, z represents the joint prediction result (the output result in the shared semantic subspace), n represents the number of modal types, the subscripts represent different modalities, v represents the data features of each modality, w represents the weight corresponding to each modality, and f (.) represents the mapping of the semantic transformation of each modality to the shared subspace.

[0047] The key to the multimodal joint architecture is to realize the "joint" of features. A simpler method is to directly connect, that is, the "addition" joint method. This method realizes the shared semantic subspace in different hidden layers, combines the semantics of each converted single-modal feature vector together, and thus realizes multimodal fusion.

[0048] In some of the embodiments, the combination is performed through a multiplication combination method, and step S202 specifically includes: based on a multiplication combination model, fusing the data features of each modality into a unified tensor to obtain a joint prediction result.

[0049] The multiplication joint model includes:

[0050] Among them, z represents the joint prediction result, n represents the number of modal types, the superscript represents different modalities, and v represents the data characteristics of each modality. represents the outer product operator.

[0051] Another multimodal joint method is the "multiplication" joint method, which fuses modalities such as language, video, and audio into a unified tensor, which is composed of the output product of all single-modal feature vectors.

[0052] In some embodiments, in step S1032, the fusion features after dimension reduction are input into a pre-built multi-layer perceptron, and the collaborative prediction results obtained include: Step S301, classifier decision fusion is performed based on preset classifier weights to obtain the prediction probability after fusing multiple classifier information under the current mode.

[0053] Step S302, performing modal information fusion based on preset modal weights, and obtaining a collaborative prediction result according to the modal information fusion result and the prediction probability.

[0054] First, weights are assigned to each classifier and each modality (pictures, texts, audios, PPT titles, etc. corresponding to the course knowledge), and the first round of fusion is performed at the classifier level. Then, the second round of fusion is performed at the modality level, and classifier decision fusion and modality information fusion are realized in turn.

[0055] Classifier decision fusion is a weighted summation of the prediction probability of each classifier:

[0056] Among them, P i m Indicates the predicted probability value of each classifier for the predicted sample under single mode, w i represents the weight of each classifier, P m is the predicted probability after fusing multiple classifier information under the current modality.

[0057] Step S104, based on the joint feature representation and the collaborative feature representation, establish a corresponding relationship between the course data and the corresponding knowledge points to construct a course knowledge graph.

[0058] Through multimodal fusion annotation, we can establish an understanding and mutual correspondence of multimodal data, so that in the process of knowledge graph construction, we can conduct collaborative predictions between multimodal data, improve the accuracy of the correspondence between knowledge points and course content, and realize the automatic and accurate construction of the corresponding knowledge graph based on the multimodal data of mathematics classroom.

[0059] In view of the formulas, symbols and logical reasoning unique to mathematics courses, feature fusion is achieved by constructing a two-layer neural network. The input layer is a concatenation of traditional features and deep features. The fused features are obtained by learning the weights of the hidden layer. The number of nodes in the hidden layer is the dimension of the fused features. Then, the traditional features and deep multimodal features of data of different modes of mathematical knowledge are fused with a multi-layer neural network.

[0060] For image data, the Scale Invariant Feature Transform (SIFT) is used to detect significant and stable feature points in the image, and multidimensional features are generated based on the local neighborhood of the detected pixel points. The multidimensional features represent the description of the current feature points; the feature operator of the local texture of the image is described by the Local Binary Pattern (LBP). This operator has significant advantages such as rotation invariance and grayscale invariance (not afraid of illumination changes). LBP is calculated as follows:

[0061] Among them, P and R represent the number of pixels in the field and the field radius respectively. c Refers to the brightness of the central pixel, p i Refers to the brightness of the i-th pixel in the field. The function S(.) is expressed as follows:

[0062] Then, we select the multimodal feature fusion vector, use the long short-term memory network LSTM to study the changing trend and relationship of the feature vectors in each period, and use the bidirectional long short-term memory model to predict, determine the sequence characteristics of the data, and determine its prediction results. After using the collaborative prediction of multimodal feature fusion to determine the knowledge points corresponding to the multimodal data, we can establish a corresponding relationship between the classroom multimodal data and the corresponding knowledge points, and finally realize the construction of the corresponding knowledge graph.

[0063] Through the above steps, course data of at least two modalities are obtained, feature extraction is performed on the course data to obtain data features of each modality, correlation fusion is performed on the data features based on a preset multi-layer neural network to obtain fused features, feature annotation is performed on the fused features to obtain joint feature representation and collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately, and a knowledge graph corresponding to the course data is constructed based on the joint feature representation and the collaborative feature representation, thereby solving the problem of inaccurate construction of classroom knowledge graphs.

[0064] Taking into account the characteristics and importance of each modal data, multimodal fusion annotation technology is adopted to improve the data representation ability, make the expression of the same knowledge point in different modal data consistent, and reduce the inaccuracy caused by insufficient data fusion.

[0065] It specifically processes the formulas, symbols and logical reasoning unique to mathematics courses, designs feature extraction and fusion methods suitable for the characteristics of mathematics subjects, and improves the ability to recognize and associate abstract concepts and logical relationships.

[0066] By selecting and optimizing appropriate deep learning models, the generalization ability and accuracy of the models can be enhanced to ensure efficient processing and prediction of multimodal data, especially for complex data expressions in classroom teaching.

[0067] Build a unified and standardized multimodal data annotation method to improve the consistency and interoperability between different modal data, ensure the accurate annotation of knowledge points, and provide high-quality annotation data support for subsequent feature extraction and model training.

[0068] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] This embodiment also provides a knowledge graph construction system, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0070] Figure 2 is a structural block diagram of a knowledge graph construction system according to an embodiment of the present application, such as Figure 2 As shown, the system includes: The feature extraction module 21 is used to obtain course data of at least two modes, perform feature extraction on the course data, and obtain data features of each mode.

[0071] The feature fusion module 22 is used to perform correlation fusion on data features based on a preset multi-layer neural network to obtain fused features.

[0072] The feature annotation module 23 performs feature annotation on the fused features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately.

[0073] The graph construction module 24 constructs a knowledge graph corresponding to the course data based on joint feature representation, collaborative feature representation and fusion features.

[0074] In some embodiments, the feature annotation module 23 includes: The dimension reduction module is used to reduce the dimension of fusion features based on principal component analysis and linear discriminant analysis.

[0075] The prediction module inputs the fusion features after dimension reduction into the pre-built multi-layer perceptron to obtain the joint prediction results and the collaborative prediction results.

[0076] The labeling module labels the fusion features according to the joint prediction results and the collaborative prediction results to obtain the joint feature representation and the collaborative feature representation.

[0077] In some of the embodiments, the prediction module includes: a joint prediction module, which is used to construct a fusion loss function based on a cross-modal similarity method, and to combine the data features of each modality to obtain a joint prediction result with the goal of minimizing the value of the fusion loss function.

[0078] In some embodiments, the fusion loss function includes:

[0079] Among them, t and v are feature samples randomly selected from data features of two different modalities, α is the margin, S(.) is the similarity measurement function, t - is a feature sample in t that does not match v, v - are the feature samples in v that do not match t.

[0080] In some of the embodiments, the joint prediction module includes: an addition joint module, which is used to convert the data features of each modality into a shared subspace based on the addition joint model to obtain a joint prediction result.

[0081] The combined model includes:

[0082] Among them, z represents the joint prediction result, n represents the number of modal types, the subscripts represent different modalities, v represents the data features of each modality, w represents the weight corresponding to each modality, and f (.) represents the mapping of the semantic transformation of each modality to the shared subspace.

[0083] In some of the embodiments, the joint prediction module includes: a multiplication joint module, which is used to fuse the data features of each modality into a unified tensor based on a multiplication joint model to obtain a joint prediction result.

[0084] The multiplication joint model includes:

[0085] Among them, z represents the joint prediction result, n represents the number of modal types, the superscript represents different modalities, and v represents the data characteristics of each modality. represents the outer product operator.

[0086] In some of the embodiments, the prediction module includes: a collaborative prediction module, which is used to perform classifier decision fusion based on preset classifier weights to obtain the prediction probability after fusing multiple classifier information under the current modality; perform modal information fusion based on preset modal weights, and obtain a collaborative prediction result based on the modal information fusion result and the prediction probability.

[0087] In some of the embodiments, the course data in at least two modalities are selected from text data, image data, audio data, and video data.

[0088] Through the above collaboration, the feature extraction module 21 obtains course data of at least two modalities, extracts features from the course data, and obtains data features of each modality. The feature fusion module 22 performs correlation fusion on the data features based on a preset multi-layer neural network to obtain fusion features. The feature annotation module 23 performs feature annotation on the fusion features to obtain joint feature representation and collaborative feature representation, wherein the joint feature representation is used to map the information of each modality to the same feature space, and the collaborative feature representation is used to map the information of each modality separately. The graph construction module 24 constructs the knowledge graph corresponding to the course data based on the joint feature representation, collaborative feature representation and fusion features, solving the problem of inaccurate construction of classroom knowledge graphs. The multimodal fusion annotation technology is used to improve the representation ability of data, so that the expression of the same knowledge point in different modal data is consistent, and the inaccuracy caused by insufficient data fusion is reduced.

[0089] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0090] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0091] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0092] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program: S1, obtaining course data of at least two modes, performing feature extraction on the course data, and obtaining data features of each mode.

[0093] S2, based on the preset multi-layer neural network, the data features are correlated and fused to obtain fused features.

[0094] S3, feature labeling is performed on the fused features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately.

[0095] S4, based on joint feature representation and collaborative feature representation, establish a correspondence between course data and corresponding knowledge points to construct a course knowledge graph.

[0096] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0097] In one embodiment, Figure 3 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 3 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 3As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a knowledge graph construction method is implemented.

[0098] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0100] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A knowledge graph construction method, characterized in that: The method comprises: Obtaining course data of at least two modes, performing feature extraction on the course data, and obtaining data features of each mode; Based on a preset multi-layer neural network, correlation fusion is performed on the data features to obtain fusion features; Performing feature annotation on the fused features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map each modality information into the same feature space, and the collaborative feature representation is used to map the information of each modality respectively; Based on the joint feature representation and the collaborative feature representation, a corresponding relationship is established between the course data and the corresponding knowledge points to construct a course knowledge graph.

2. The method according to claim 1, characterized in that The feature labeling of the fused features to obtain the joint feature representation and the collaborative feature representation includes: Performing dimensionality reduction processing on the fusion features based on principal component analysis and linear discriminant analysis; Inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain joint prediction results and collaborative prediction results; According to the joint prediction result and the collaborative prediction result, feature annotation is performed on the fusion feature to obtain the joint feature representation and the collaborative feature representation.

3. The method according to claim 2, characterized in that The step of inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain a joint prediction result includes: Based on the cross-modal similarity method, a fusion loss function is constructed; With the goal of minimizing the fusion loss function value, the data features of each modality are combined to obtain the joint prediction result.

4. The method according to claim 3, characterized in that The fusion loss function includes: Among them, t and v are feature samples randomly selected from data features of two different modalities, α is the margin, S(.) is the similarity measurement function, t - is a feature sample in t that does not match v, v - are the feature samples in v that do not match t.

5. The method according to claim 3, characterized in that: The combining of the data features of each modality to obtain the combined prediction result includes: Based on the joint model, the data features of each modality are converted into a shared subspace to obtain the joint prediction result. The combined model includes: Among them, z represents the joint prediction result, n represents the number of modal types, the subscripts represent different modalities, v represents the data features of each modality, w represents the weight corresponding to each modality, and f (.) represents the mapping of the semantic transformation of each modality to the shared subspace.

6. The method according to claim 3, characterized in that The combining of the data features of each modality to obtain the combined prediction result includes: Based on the multiplication joint model, the data features of each modality are fused into a unified tensor to obtain the joint prediction result. The multiplication joint model includes: Wherein, z represents the joint prediction result, n represents the number of modal types, superscripts represent different modalities, and v represents the data features of each modality. represents the outer product operator.

7. The method according to claim 2, characterized in that The step of inputting the fusion features after dimension reduction into a pre-built multi-layer perceptron to obtain a collaborative prediction result includes: Based on the preset classifier weights, classifier decision fusion is performed to obtain the prediction probability after fusing multiple classifier information under the current mode; Modal information fusion is performed based on preset modal weights, and the collaborative prediction result is obtained according to the modal information fusion result and the prediction probability.

8. The method according to claim 1, characterized in that The course data of at least two modalities are selected from text data, image data, audio data and video data.

9. A knowledge graph construction system, characterized in that: The system comprises: A feature extraction module is used to obtain course data of at least two modes, perform feature extraction on the course data, and obtain data features of each mode; A feature fusion module, used to perform correlation fusion on the data features based on a preset multi-layer neural network to obtain fusion features; A feature annotation module, used to perform feature annotation on the fusion features to obtain a joint feature representation and a collaborative feature representation, wherein the joint feature representation is used to map the information of each modality into the same feature space, and the collaborative feature representation is used to map the information of each modality separately; A graph construction module is used to construct a knowledge graph corresponding to the course data based on the joint feature representation, the collaborative feature representation and the fusion feature.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the knowledge graph construction method as described in any one of claims 1 to 8.

Images