A method and device for constructing a knowledge graph

Abstract

The application provides a knowledge graph construction method and device. The method comprises the following steps: determining the modal category of first data, wherein the first data refers to single-modal data extracted from original data used for constructing a knowledge graph; if the modal category comprises multiple categories, obtaining the representation vector of the first data corresponding to each modal category; constructing a first knowledge graph, wherein the first knowledge graph refers to a knowledge graph constructed by extracted knowledge after knowledge extraction based on all the representation vectors; and constructing a second knowledge graph, wherein the second knowledge graph refers to a knowledge graph obtained by knowledge fusion and knowledge completion on the knowledge contained in the first knowledge graph. The knowledge graph constructed by the method fully contains information associated with various modalities, does not split the association relationship between various modal information, has higher accuracy, and has better applicability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for constructing a knowledge graph. Background Technology

[0002] Typically, the construction of a knowledge graph includes processes such as knowledge extraction, knowledge fusion, and knowledge completion. Furthermore, the raw data used to construct a knowledge graph can include data from multiple modalities; for example, raw data can include text modal data and image modal data.

[0003] In existing technologies, during the construction of knowledge graphs, each modality of the original data needs to be extracted separately, resulting in independent extraction results. Then, the extracted results are fused and supplemented to construct the knowledge graph corresponding to the original data. This construction method results in the extraction of knowledge from different modalities being independent of each other, severing the connections between them and failing to effectively represent the knowledge information contained in the original data, leading to low accuracy of the knowledge graph. Summary of the Invention

[0004] This application provides a method and apparatus for constructing a knowledge graph to solve the problem that existing methods for constructing knowledge graphs often sever the connections between multiple modalities and result in low accuracy.

[0005] In a first aspect, this application provides a method for constructing a knowledge graph, the method comprising: determining the modality category of first data, wherein the first data refers to single-modality data extracted from the original data used to construct the knowledge graph; if the modality category includes multiple categories, obtaining the representation vectors of the first data corresponding to each modality category; constructing a first knowledge graph, wherein the first knowledge graph refers to a knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the representation vectors; and constructing a second knowledge graph, wherein the second knowledge graph refers to a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the first knowledge graph.

[0006] In one possible implementation, the modality categories include text modality and image modality; obtaining the representation vector of the first data corresponding to each modality category includes: obtaining the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

[0007] In one possible implementation, constructing the first knowledge graph includes: generating a first representation vector, which is a representation vector obtained by concatenating the text representation vector and the image representation vector; generating a second representation vector, which is a representation vector obtained by encoding the first representation vector using a fully connected layer; generating a third representation vector, which is a representation vector obtained by convolving the second representation vector; and constructing the first knowledge graph based on the normalization result of the third representation vector.

[0008] In one possible implementation, the method further includes: if the modality category includes a category, constructing a first entity graph, wherein the first entity graph refers to a knowledge graph constructed from the extracted knowledge after extracting entity knowledge from the first data using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model corresponding to the modality category; constructing a first attribute graph, wherein the first attribute graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; and constructing a first relation graph, wherein the first relation graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. The knowledge graph obtained by the knowledge extraction model trained by the representation learning model is used to extract relational knowledge from the first entity graph; a first event graph is constructed, which is a knowledge graph obtained by extracting event knowledge from the first entity graph using the knowledge extraction model trained based on the pre-trained model and / or the representation learning model; a third knowledge graph is constructed, which is a knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relation graph, and the first event graph; and a fourth knowledge graph is constructed, which is a knowledge graph obtained by performing knowledge fusion and knowledge completion on the knowledge contained in the third knowledge graph.

[0009] In one possible implementation, the method further includes: acquiring a first training corpus, wherein the first training corpus refers to labeled data obtained by automatically labeling second data according to existing knowledge and / or general linguistic rules corresponding to the modality category; the second data is the original data used to generate the training corpus; generating an initial extraction model, wherein the initial extraction model refers to a model used to extract knowledge by training a pre-trained model corresponding to the modality category using the first training corpus; generating a second training corpus, wherein the second training corpus refers to labeled data obtained by automatically labeling the extracted knowledge after knowledge extraction from third data using the initial extraction model; the third data is the original data used to generate the training corpus; and training the initial extraction model based on the second training corpus to obtain the knowledge extraction model.

[0010] Secondly, this application provides a knowledge graph construction apparatus, comprising: a determining module for determining the modality category of first data, wherein the first data refers to single-modality data extracted from the original data used to construct the knowledge graph; a first acquiring module for acquiring, if the modality category includes multiple categories, the representation vectors of the first data corresponding to each modality category; a first construction module for constructing a first knowledge graph, wherein the first knowledge graph refers to a knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the representation vectors; and a second construction module for constructing a second knowledge graph, wherein the second knowledge graph refers to a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the first knowledge graph.

[0011] In one possible implementation, the modality categories include text modality and image modality; the first acquisition module is used to acquire the representation vector of the first data corresponding to each modality category, specifically: the first acquisition module is used to acquire the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

[0012] In one possible implementation, the first construction module is used to construct a first knowledge graph, specifically: the first construction module is used to: generate a first representation vector, which is a representation vector obtained by concatenating the text representation vector and the image representation vector; generate a second representation vector, which is a representation vector obtained by encoding the first representation vector with a fully connected layer; generate a third representation vector, which is a representation vector obtained by convolution processing the second representation vector; and construct the first knowledge graph based on the normalization result of the third representation vector.

[0013] In one possible implementation, the apparatus further includes: a third construction module, configured to construct a first entity graph if the modality category includes a category, wherein the first entity graph refers to a knowledge graph constructed from the extracted knowledge after extracting entity knowledge from the first data using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model corresponding to the modality category; a fourth construction module, configured to construct a first attribute graph, wherein the first attribute graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; and a fifth construction module, configured to construct a first relation graph, wherein the first relation graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; and a fifth construction module, configured to construct a first relation graph, wherein the first relation graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. The sixth construction module is used to construct a first event graph, which is a knowledge graph obtained by extracting relational knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; the seventh construction module is used to construct a third knowledge graph, which is a knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relation graph, and the first event graph; the eighth construction module is used to construct a fourth knowledge graph, which is a knowledge graph obtained by performing knowledge fusion and knowledge completion on the knowledge contained in the third knowledge graph.

[0014] In one possible implementation, the apparatus further includes: a second acquisition module, configured to acquire a first training corpus, wherein the first training corpus refers to labeled data obtained by automatically labeling second data according to existing knowledge and / or general linguistic rules corresponding to the modality category; the second data is the original data used to generate the training corpus; a first generation module, configured to generate an initial extraction model, wherein the initial extraction model refers to a model used to extract knowledge by training a pre-trained model corresponding to the modality category using the first training corpus; a second generation module, configured to generate a second training corpus, wherein the second training corpus refers to labeled data obtained by automatically labeling the extracted knowledge after knowledge extraction from third data using the initial extraction model; the third data is the original data used to generate the training corpus; and a training module, configured to train the initial extraction model based on the second training corpus to obtain the knowledge extraction model.

[0015] Thirdly, this application provides a network device including a processor, a memory, and a transceiver; the transceiver is used to receive or transmit signals; the memory is used to store computer programs or instructions; and the processor is used to execute the computer programs or instructions stored in the memory to cause the network device to perform the method described in the first aspect.

[0016] Fourthly, this application provides a computer storage medium for storing computer programs or instructions that, when executed, cause the method described in the first aspect to be implemented.

[0017] Fifthly, this application provides a computer program product including a computer program or instructions, which, when executed, causes the method described in the first aspect to be implemented.

[0018] To address the issue of low accuracy in knowledge graphs constructed using existing methods due to the fragmentation of connections between multiple modalities, this application provides a method and apparatus for constructing a knowledge graph. In this method, when the first raw data used to construct the knowledge graph includes single-modal data of multiple modal categories, each single-modal data point can be extracted from the first raw data. Then, the representation vectors of each single-modal data point are obtained separately. These representation vectors are then concatenated into a single representation vector. Knowledge extraction is performed based on this concatenated representation vector. The extracted entity knowledge, attribute knowledge, relational knowledge, and attribute knowledge all integrate information from multiple modalities. This ensures that the knowledge graph constructed based on the extracted knowledge fully contains information related to various modalities, without severing the connections between different modalities, resulting in higher accuracy and better applicability. Attached Figure Description

[0019] Figure 1 A schematic diagram illustrating a scenario for knowledge graph construction provided in this application;

[0020] Figure 2 A schematic diagram of another knowledge graph construction scenario provided for this application;

[0021] Figure 3 A flowchart illustrating one implementation of the knowledge graph construction method provided in this application;

[0022] Figure 4 A flowchart illustrating one implementation method for constructing a first knowledge graph provided in this application;

[0023] Figure 5 A flowchart illustrating another implementation of the knowledge graph construction method provided in this application;

[0024] Figure 6A flowchart illustrating another implementation of the knowledge graph construction method provided in this application;

[0025] Figure 7 A structural block diagram of one embodiment of the knowledge graph construction apparatus provided in this application. Detailed Implementation

[0026] The technical solutions provided in this application will now be described by way of example with reference to the accompanying drawings.

[0027] First, it should be noted that the embodiments provided in this application are only a part of the embodiments of this application, not all of the embodiments of this application, and this application is not limited to the embodiments described below.

[0028] Furthermore, in the exemplary description of this application, " / " signifies "or," for example, A / B can mean either A or B. The "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, "at least one" refers to one or more, and "multiple" refers to two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0029] This application can be applied to knowledge graph construction scenarios. For example, see... Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario for knowledge graph construction provided in this application. Combined with... Figure 1 It can be seen that the construction of a knowledge graph can include steps such as knowledge extraction, knowledge fusion, and knowledge completion.

[0030] The knowledge extraction process involves several stages, including entity extraction (also known as entity knowledge extraction), relation extraction (also known as relation knowledge extraction), attribute extraction (also known as attribute knowledge extraction), and event extraction (also known as event knowledge extraction). Therefore, knowledge extraction refers to extracting fundamental knowledge such as entities, relations, attributes, and events from the original data used to construct a knowledge graph, and then building a knowledge graph based on this extracted fundamental knowledge. In other words, it represents the extracted fundamental knowledge in the form of a knowledge graph.

[0031] The knowledge fusion process involves entity alignment, relation fusion, and attribute fusion. Therefore, knowledge fusion refers to processing the knowledge contained in the knowledge graph constructed based on the extracted basic knowledge by performing entity alignment, relation fusion, and attribute fusion, thereby removing redundant representations of knowledge and making the knowledge graph more concise and clear.

[0032] Knowledge completion refers to supplementing the missing knowledge in the knowledge graph obtained after knowledge fusion, such as supplementing the relationships between entities, so that the knowledge information represented by the final knowledge graph is more complete.

[0033] Furthermore, the raw data used to construct a knowledge graph typically includes data from one or more modalities. For example, the raw data may include text modal data, image modal data, graph modal data, video modal data, speech modal data, and spatiotemporal modal data. In constructing the knowledge graph corresponding to the raw data, it is first necessary to perform modal segmentation on the raw data and extract single-modal data from it. For example, text modal data and image modal data can be extracted from the raw data. This application does not limit the specific implementation methods for extracting single-modal data.

[0034] Currently, knowledge graphs are typically constructed based on text and / or image modal data. For audio modal data, a speech recognition model is needed to convert the speech into text modal data, and then the knowledge graph is constructed based on this text modal data. For video modal data, keyframes are typically obtained to acquire the image modal data, and the speech data within the video modal data is converted into text modal data. The knowledge graph is then constructed based on both the image and text modal data.

[0035] Based on this, the following section uses the example of raw data containing both text modal data and image modal data to illustrate the process of constructing a knowledge graph.

[0036] See Figure 2 , Figure 2 A schematic diagram illustrating another scenario for knowledge graph construction provided in this application. Combined with... Figure 2 It can be seen that, in this application scenario, the knowledge graph construction process may include:

[0037] The entity extraction model for text modality is used to extract entities from text modality data to obtain an entity map for text modality; and the entity extraction model for image modality is used to extract entities from image modality data to obtain an entity map for image modality.

[0038] The entity graph of the text modality is extracted using attribute extraction, relation extraction, and event extraction models, respectively, to obtain the attribute graph, relation graph, and event graph of the text modality; and the entity graph of the image modality is extracted using attribute extraction, relation extraction, and event extraction models, respectively, to obtain the attribute graph, relation graph, and event graph of the image modality.

[0039] Knowledge graphs are obtained by fusing and completing the attribute graphs, relation graphs, and event graphs of the text modality and the attribute graphs, relation graphs, and event graphs of the image modality.

[0040] It is evident that adopting Figure 2 The method described requires using separate network models tailored to each modality category and extraction task to extract knowledge from each modality's data. This necessitates training a large number of network models, resulting in poor applicability. Furthermore, during knowledge extraction, entity extraction, attribute extraction, relation extraction, and event extraction are performed separately for each modality category. The extraction results between different modalities are independent and unrelated, severing the connections between modal information in the original data. Consequently, the constructed knowledge graph cannot accurately represent the relationships between modal knowledge, resulting in low accuracy.

[0041] To address the aforementioned technical problems, this application provides a method for constructing a knowledge graph. The method for constructing a knowledge graph provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] See Figure 3 , Figure 3 This is a flowchart illustrating one implementation of the knowledge graph construction method provided in this application. The method can be applied to network devices that can be used to construct knowledge graphs, and the method may include the following steps:

[0043] Step S101: Determine the modality category of the first data.

[0044] As can be seen from the foregoing embodiments, before constructing a knowledge graph based on the original data used to construct the knowledge graph (hereinafter referred to as the first original data), it is necessary to perform modal segmentation on the first original data and extract single-modal data from the first original data. In this application, the single-modal data extracted from the first original data is defined as the first data.

[0045] The first raw data can include data with one or more modalities, and the modality category corresponding to the first data can also include one or more categories. Therefore, after extracting the first data from the first raw data, it is necessary to determine the modality category of the first data, that is, to determine how many modalities the first data includes.

[0046] In one possible implementation, the first raw data contains only one modality of data; that is, the first raw data itself contains only single-modal data, such as text modal data or image modal data. In this application scenario, by extracting single-modal data from the first raw data, the resulting first data corresponds to only one modality, meaning that the modality category of the first data contains only one category.

[0047] In one possible implementation, the first raw data can contain data of multiple modalities, such as text modal data, image modal data, graph modal data, and spatiotemporal modal data. In this application scenario, by extracting single-modal data from the first raw data, multiple modal first data can be obtained; that is, the modality categories of the first data include multiple categories.

[0048] Step S102: If the modality category includes multiple categories, obtain the representation vector of the first data corresponding to each modality category.

[0049] If in step S101 it is determined that the modal category of the first data includes multiple categories, then a knowledge graph is constructed according to the processing method of multiple modal categories. For details, please refer to the content of the following embodiments.

[0050] As can be seen from the foregoing embodiments, knowledge graphs are typically constructed based on text modal data and / or image modal data. Furthermore, the implementation methods for constructing knowledge graphs based on two or more single-modal data are similar. Therefore, in the following embodiments, taking the example that the modal categories of the first data include text modality and image modality, the technical solution provided by this application will be described.

[0051] In one possible implementation, the modal categories of the first data include text modality and image modality. That is, the first data extracted from the first original data contains both text modality data and image modality data. Then, the representation vectors of the first data corresponding to each modality category are obtained, including: obtaining the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

[0052] In one possible implementation, obtaining the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality can be achieved as follows: using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model (e.g., a knowledge graph representation learning model for the text modality) to obtain the text representation vector of the first data corresponding to the text modality; using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model (e.g., a knowledge graph representation learning model for the image modality) to obtain the image representation vector of the first data corresponding to the image modality.

[0053] Optionally, a knowledge extraction model trained using a text modality-based pre-trained model and / or representation learning model is used to obtain text representation vectors. First, the task category of the text modality pre-trained model and representation learning model is set to obtain representation vectors. Then, the pre-trained model and / or representation learning model with the task category set are trained to obtain a knowledge extraction model (which can be denoted as a text representation vector extraction model). Then, the text representation vector of the first data corresponding to the text modality is obtained through the trained knowledge extraction model (i.e., the text representation vector extraction model).

[0054] Similarly, when using a knowledge extraction model trained with an image modality-based pre-trained model and / or representation learning model to obtain image representation vectors, the task category of the image modality pre-trained model and representation learning model is first set to obtain representation vectors. Then, the pre-trained model and / or representation learning model with the task category set are trained to obtain the knowledge extraction model (which can be denoted as the image representation vector extraction model). Then, the image representation vector of the first data corresponding to the image modality is obtained through the trained knowledge extraction model (i.e., the image representation vector extraction model).

[0055] Step S103: Construct the first knowledge graph.

[0056] The first knowledge graph refers to the knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the aforementioned representation vectors.

[0057] In one possible implementation, the representation vector of the first data obtained in step S102 includes a text representation vector and an image representation vector. Based on this, the specific implementation process for constructing the first knowledge graph can be found in [reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating one implementation method for constructing a first knowledge graph provided in this application. (Combined with...) Figure 4 It can be seen that the implementation of constructing the first knowledge graph can include the following steps:

[0058] Step S201: Generate a first representation vector, which is a representation vector obtained by concatenating a text representation vector and an image representation vector.

[0059] Optionally, both text representation vectors and image representation vectors are represented as vector sequences. Concatenating text representation vectors and image representation vectors means combining the vector sequence representing the text representation vectors with the vector sequence representing the image representation vectors into a single vector sequence.

[0060] Step S202: Generate a second representation vector, which is the representation vector obtained by encoding the first representation vector using a fully connected layer.

[0061] After obtaining the first representation vector, which is formed by concatenating the text representation vector and the image representation vector, the first representation vector is input into a pre-set fully connected network model (also known as a fully connected layer). After the first representation vector is encoded by the fully connected layer, the second representation vector is output.

[0062] Step S203: Generate a third representation vector, which is the representation vector obtained by convolving the second representation vector.

[0063] After obtaining the second representation vector output by the fully connected network model, the second representation vector is input into a pre-set convolutional network model (also known as a convolutional layer). After performing convolution processing on the second representation vector, the third representation vector is output.

[0064] Step S204: Construct the first knowledge graph based on the normalization result of the third representation vector.

[0065] After obtaining the third representation vector, the third representation vector is input into the Softmax layer (also known as the normalization layer or normalized network model) which has multiple classification tasks. Through the multiple classification tasks set by the Softmax layer, entity knowledge, attribute knowledge, relation knowledge and event knowledge are obtained from the third representation vector. Then, based on the obtained entity knowledge, attribute knowledge, relation knowledge and event knowledge, a knowledge graph (denoted as the first knowledge graph) is constructed.

[0066] The third representation vector is generated by concatenating the text representation vector and the image representation vector. When performing classification prediction based on the third representation vector, classification prediction is performed simultaneously based on the text representation vector and the image representation vector. The resulting entity knowledge, attribute knowledge, relationship knowledge, and event knowledge fully contain the information related to the text modality and the image modality. The generated knowledge graph can more fully represent the text information and image information of the first original data without severing the relationship between the text information and the image information, resulting in higher accuracy.

[0067] Step S104: Construct the second knowledge graph.

[0068] After obtaining the first knowledge graph, knowledge fusion and knowledge completion are performed sequentially on the first knowledge graph to obtain the knowledge graph corresponding to the first original data (denoted as the second knowledge graph). Therefore, the second knowledge graph refers to the knowledge graph obtained by fusion and knowledge completion of the knowledge contained in the first knowledge graph.

[0069] It should be noted that, Figure 3 and Figure 4The illustrated embodiment uses the example of the first data's modality category including text and image modalities. In other application scenarios, the first data's modality category can also include other modalities, such as graph modalities and spatiotemporal modalities. In this application scenario, in step S102, it is also necessary to obtain the representation vectors of the first data corresponding to the graph modalities and spatiotemporal modalities (hereinafter referred to as graph representation vectors and spatiotemporal representation vectors). Then, in step S201, when generating the first representation vector, the text representation vector, image representation vector, graph representation vector, and spatiotemporal representation vector are concatenated to form the first representation vector.

[0070] Furthermore, the first raw data may also include video modal data and / or audio modal data. In this application scenario, it is only necessary to convert the audio modal data into text modal data, obtain the keyframes of the video modal data to obtain the image modal data of the video modal data, and convert the speech data in the video modal data into text modal data. Then, based on the image modal data and text modal data, according to... Figure 3 and Figure 4 As shown, the second knowledge graph can be constructed.

[0071] In the knowledge graph construction method provided in this application, when the original data (first original data) used to construct the knowledge graph includes single-modal data of multiple modal categories, each single-modal data can be extracted from the first original data first, and then the representation vector of each single-modal data can be obtained separately. Then, the representation vectors of all modalities can be concatenated into a single representation vector. Knowledge extraction is performed based on the concatenated representation vector. The extracted entity knowledge, attribute knowledge, relation knowledge, and attribute knowledge all integrate information from multiple modalities, so that the knowledge graph constructed based on the extracted knowledge fully contains information related to various modalities, does not sever the relationship between various modal information, and has higher accuracy and better applicability.

[0072] In some other optional application scenarios, the initial raw data may also include only one modality of data; that is, the modality category corresponding to the initial data includes only one category. For the implementation method of constructing the knowledge graph corresponding to the raw data in this application scenario, please refer to [link to relevant documentation]. Figure 5 , Figure 5 A flowchart illustrating another implementation of the knowledge graph construction method provided in this application, as shown below. Figure 5 As shown, the method may include the following steps:

[0073] Step S301: Determine the modality category of the first data.

[0074] The specific implementation of step S301 can be found in step S101 of the aforementioned embodiment, and will not be repeated here.

[0075] Step S302: If the modal category includes a category, construct the first entity graph.

[0076] The first entity graph refers to the knowledge graph constructed from the extracted knowledge after the knowledge extraction model, trained using a pre-trained model and / or representation learning model based on the modality category of the first data, extracts entity knowledge from the first data.

[0077] Optionally, in the process of constructing the first entity graph, it is necessary to first set the task category of the pre-trained model and the representation learning model corresponding to the modality category of the first data as entity extraction, and then train the pre-trained model and / or representation learning model with the task category set to obtain a trained knowledge extraction model (which can be referred to as the first entity extraction model). After that, the first data is input into the aforementioned trained knowledge extraction model (i.e., the first entity extraction model) to extract entity knowledge from the first data. Then, a knowledge graph (hereinafter referred to as the first entity graph) can be constructed based on the extracted knowledge.

[0078] Step S303: Construct the first attribute graph.

[0079] The first attribute graph refers to the knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained with a pre-trained model and / or a representation learning model based on the modality category of the first data.

[0080] Optionally, in the process of constructing the first attribute graph, it is necessary to first set the task category of the pre-trained model and the representation learning model corresponding to the modality category of the first data as attribute extraction, and then train the pre-trained model and / or representation learning model with the task category set to obtain the trained knowledge extraction model (this knowledge extraction model can be denoted as the first attribute extraction model). Then, the first entity graph is input into the aforementioned trained knowledge extraction model (i.e., the first attribute extraction model) to extract attribute knowledge from the first entity graph. Then, the knowledge graph (hereinafter referred to as the first attribute graph) can be constructed based on the extracted knowledge. It can be seen that the first attribute graph can not only represent the information of entities and attributes in the first data, but also represent the relationship between entities and attributes in the first data.

[0081] Step S304: Construct the first relational graph.

[0082] The first relation graph refers to the knowledge graph obtained by extracting relational knowledge from the first entity graph using a knowledge extraction model trained by a pre-trained model and / or a representation learning model based on the modality category of the first data.

[0083] Similarly, optionally, in the process of constructing the first relation graph, it is necessary to first set the task category of the pre-trained model and the representation learning model corresponding to the modality category of the first data as relation extraction, and then train the pre-trained model and / or representation learning model with the task category set to obtain the trained knowledge extraction model (this knowledge extraction model can be denoted as the first relation extraction model). After that, the first entity graph is input into the aforementioned trained knowledge extraction model (i.e., the first relation extraction model) to extract relation knowledge from the first entity graph. Then, a knowledge graph (hereinafter referred to as the first relation graph) can be constructed based on the extracted knowledge. It can be seen that the first relation graph can not only represent the information of entities and relations in the first data, but also represent the association relationship between entities and relations in the first data.

[0084] Step S305: Construct the first event graph.

[0085] The first event graph refers to the knowledge graph obtained by extracting event knowledge from the first entity graph using a pre-trained model and / or a representation learning model based on the modality category of the first data.

[0086] Similarly, optionally, in the process of constructing the first event graph, it is necessary to first set the task category of the pre-trained model and the representation learning model corresponding to the modality category of the first data as event extraction, and then train the pre-trained model and / or representation learning model with the task category set to obtain the trained knowledge extraction model (this knowledge extraction model can be denoted as the first event extraction model). After that, the first entity graph is input into the aforementioned trained knowledge extraction model (i.e., the first event extraction model) to extract event knowledge from the first entity graph. Then, a knowledge graph (hereinafter referred to as the first event graph) can be constructed based on the extracted knowledge. It can be seen that the first event graph can not only represent the information of entities and events in the first data, but also represent the relationship between entities and events in the first data.

[0087] Step S306: Construct the third knowledge graph.

[0088] The third knowledge graph refers to the knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relation graph, and the first event graph. In other words, the third knowledge graph is the knowledge graph obtained by merging (taking the union) the knowledge contained in the first entity graph, the first attribute graph, the first relation graph, and the first event graph.

[0089] Step S307: Construct the fourth knowledge graph.

[0090] The fourth knowledge graph refers to the knowledge graph obtained by fusing and completing the knowledge contained in the third knowledge graph. The specific details of knowledge fusion and completion can be found in the aforementioned embodiments and will not be repeated here.

[0091] It should be noted that this application does not restrict the execution order of steps S303, S304 and S305. Steps S303, S304 and S305 can be executed in any order, or they can be executed simultaneously.

[0092] In the knowledge graph construction method provided in this application, when the original data used to construct the knowledge graph includes only one modality, entity knowledge extraction, attribute knowledge extraction, relation knowledge extraction, and event knowledge extraction of the first data can be achieved by changing the task category of the same pre-trained model. Then, based on the extracted knowledge, the first entity graph, first attribute graph, first relation graph, and first event graph are constructed respectively, and the third and fourth knowledge graphs are constructed, thus obtaining the knowledge graph corresponding to the original data. It is evident that the method provided in this application eliminates the need to train multiple network models; by continuously changing the task category of the same pre-trained model, knowledge extraction for different tasks can be achieved, thus enabling a simpler and faster construction of the knowledge graph corresponding to the original data, and offering better applicability.

[0093] Furthermore, in the embodiments provided in this application, the knowledge extraction model needs to be trained based on a pre-trained model. Therefore, in another optional embodiment of this application, see also [the following text is missing from the original extract]. Figure 6 , Figure 6 A flowchart illustrating another implementation of the knowledge graph construction method provided in this application, combined with... Figure 6 As can be seen, the knowledge graph construction method provided in this application, in the above... Figures 3 to 5 Based on the illustrated embodiment, the following steps may also be included:

[0094] Step S401: Obtain the first training corpus.

[0095] The first training corpus refers to the labeled data obtained by automatically annotating the second data based on existing knowledge and / or general linguistic rules corresponding to the modality categories of the first data. The second data is the original data used to generate the training corpus. Typically, both the first training corpus and the second data are relatively small in size.

[0096] Step S402: Generate the initial extraction model.

[0097] The initial extraction model refers to the model used for knowledge extraction, which is trained using the first training corpus to train the pre-trained model corresponding to the modality categories of the first data. The pre-trained model can be a Prompt pre-trained model.

[0098] Step S403: Generate the second training corpus.

[0099] The second training corpus refers to the labeled data obtained by automatically annotating the extracted knowledge from the third data after knowledge extraction using the initial extraction model. The third data is the original data used to generate the training corpus. The amount of data in the third data is significantly larger than that in the second data, and the amount of data in the second training corpus is also significantly larger than that in the first training corpus. Therefore, the initial extraction model can automatically expand the training corpus, eliminating the need for manual acquisition of large amounts of training data, greatly improving the efficiency of training corpus acquisition, and enhancing its applicability.

[0100] Step S404: Train the initial extraction model based on the second training corpus to obtain the knowledge extraction model.

[0101] In the knowledge graph construction method provided in this application, the knowledge extraction model for any modality and any task category can be constructed according to the above method. Figure 6 The method shown is used for training. It should be noted that the knowledge extraction model can also be used to obtain new training data from the original data used to generate the training corpus and update the knowledge extraction model. This application does not impose any restrictions on this.

[0102] according to Figure 6 The method shown only requires a small amount of training data based on existing knowledge and / or general linguistic rules. Subsequently, a large amount of training data can be extracted by a knowledge extraction model trained on the small amount of training data. There is no need to obtain a large amount of training data through manual labeling. The model training is simpler, more efficient, and has better applicability.

[0103] The various method embodiments described herein can be independent solutions or combinations based on internal logic, and all such solutions fall within the protection scope of this application.

[0104] It is understood that the methods and operations implemented by the network device in the above-described method embodiments can also be implemented by components (such as chips or circuits) that can be used in the network device.

[0105] The above embodiments describe the method for constructing the knowledge graph provided in this application. It is understood that, in order to implement the above functions, the network device includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] This application embodiment can divide the network device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0107] The above, combined with Figures 1 to 6 The methods provided in the embodiments of this application are described in detail below. Figure 7 The apparatus provided in the embodiments of this application is described in detail. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, for content not described in detail, please refer to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0108] See Figure 7 , Figure 7 This is a structural block diagram of one embodiment of the knowledge graph construction apparatus provided in this application. Figure 7 As shown, the device 700 may include: a determining module 701, a first acquiring module 702, a first constructing module 703, and a second constructing module 704. The device 700 can perform the above-described... Figures 3 to 6 The operations performed by the network device in the method embodiment shown.

[0109] For example, in an optional embodiment of this application, the determining module 701 can be used to determine the modality category of the first data, which refers to single-modality data extracted from the original data used to construct the knowledge graph.

[0110] The first acquisition module 702 can be used to acquire the representation vector of the first data corresponding to each modality category if the modality category includes multiple categories.

[0111] The first construction module 703 can be used to construct a first knowledge graph, which refers to a knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the representation vectors.

[0112] The second construction module 704 can be used to construct a second knowledge graph, which refers to a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the first knowledge graph.

[0113] In one possible implementation, the modality categories include text modality and image modality; the first acquisition module 702 is used to acquire the representation vector of the first data corresponding to each modality category, specifically: the first acquisition module 702 is used to acquire the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

[0114] In one possible implementation, the first construction module 703 is used to construct a first knowledge graph, specifically: the first construction module 703 is used to: generate a first representation vector, which is a representation vector obtained by concatenating the text representation vector and the image representation vector; generate a second representation vector, which is a representation vector obtained by encoding the first representation vector with a fully connected layer; generate a third representation vector, which is a representation vector obtained by convolution processing the second representation vector; and construct the first knowledge graph based on the normalization result of the third representation vector.

[0115] In one possible implementation, the device 700 may further include: a third construction module, configured to construct a first entity graph if the modality category includes a category, wherein the first entity graph refers to a knowledge graph constructed from the extracted knowledge after extracting entity knowledge from the first data using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model corresponding to the modality category; a fourth construction module, configured to construct a first attribute graph, wherein the first attribute graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; and a fifth construction module, configured to construct a first relation graph, wherein the first relation graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model; and a fifth construction module, configured to construct a first relation graph, wherein the first relation graph refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. / or the knowledge graph obtained by extracting relational knowledge from the first entity graph using the knowledge extraction model trained by the representation learning model; the sixth construction module is used to construct a first event graph, which refers to the knowledge graph obtained by extracting event knowledge from the first entity graph using the knowledge extraction model trained based on the pre-trained model and / or the representation learning model; the seventh construction module is used to construct a third knowledge graph, which refers to the knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relation graph, and the first event graph; the eighth construction module is used to construct a fourth knowledge graph, which refers to the knowledge graph obtained by performing knowledge fusion and knowledge completion on the knowledge contained in the third knowledge graph.

[0116] In one possible implementation, the device 700 may further include: a second acquisition module, configured to acquire a first training corpus, wherein the first training corpus refers to labeled data obtained by automatically labeling second data according to existing knowledge and / or general linguistic rules corresponding to the modality category; the second data is the original data used to generate the training corpus; a first generation module, configured to generate an initial extraction model, wherein the initial extraction model refers to a model used to extract knowledge by training a pre-trained model corresponding to the modality category using the first training corpus; a second generation module, configured to generate a second training corpus, wherein the second training corpus refers to labeled data obtained by automatically labeling the extracted knowledge after the initial extraction model has processed the third data; the third data is the original data used to generate the training corpus; and a training module, configured to train the initial extraction model based on the second training corpus to obtain the knowledge extraction model.

[0117] In other words, the device 700 can achieve the corresponding Figures 3 to 6 The steps or processes performed by the network device in the illustrated method embodiment may be included in the apparatus 700 for execution. Figures 3 to 6 The method embodiments shown are modules of the method executed by the network device. It should be understood that the specific process of each module executing the corresponding steps described above has been explained in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0118] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0121] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit.

[0123] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The knowledge graph construction apparatus provided in the above-described embodiments of this application is used to execute the method described above. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects corresponding to the method described above, and will not be repeated here.

[0125] It should be understood that in the various embodiments of this application, the execution order of each step should be determined by its function and internal logic, and the size of each step number does not mean the order of execution, and does not constitute a limitation on the implementation process of the embodiments.

[0126] The various parts of this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments of the knowledge graph construction apparatus are basically similar to the method embodiments, so the description is relatively simple; relevant details can be found in the descriptions within the method embodiments.

[0127] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0128] The embodiments described above do not constitute a limitation on the scope of protection of this application.

Claims

1. A method for constructing a knowledge graph, characterized in that, The method includes: Determine the modality category of the first data, which refers to the single-modal data extracted from the original data used to construct the knowledge graph; If the modality category includes multiple categories, obtain the representation vector of the first data corresponding to each modality category; Construct a first knowledge graph, which is a knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the representation vectors; Construct a second knowledge graph, which is a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the first knowledge graph; The method further includes: If the modality category includes a category, a first entity graph is constructed. The first entity graph refers to a knowledge graph constructed from the extracted knowledge after the first data is extracted by a knowledge extraction model trained based on a pre-trained model and / or a representation learning model corresponding to the modality category. Construct a first attribute graph, which refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. Construct a first relation graph, which refers to a knowledge graph obtained by extracting relational knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. Construct a first event graph, which refers to a knowledge graph obtained by extracting event knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. Construct a third knowledge graph, which is a knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relationship graph, and the first event graph; Construct a fourth knowledge graph, which is a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the third knowledge graph.

2. The method according to claim 1, characterized in that, The modal categories include text modality and image modality; The step of obtaining the representation vector of the first data corresponding to each modality category includes: Obtain the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

3. The method according to claim 2, characterized in that, The construction of the first knowledge graph includes: Generate a first representation vector, which is a representation vector obtained by concatenating the text representation vector and the image representation vector; Generate a second representation vector, which is the representation vector obtained by encoding the first representation vector using a fully connected layer; Generate a third representation vector, which is the representation vector obtained by convolving the second representation vector; The first knowledge graph is constructed based on the normalization result of the third representation vector.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the first training corpus, which refers to the labeled data obtained by automatically labeling the second data according to the existing knowledge and / or general linguistic rules corresponding to the modality category; the second data is the original data used to generate the training corpus. Generate an initial extraction model, which refers to a model for extracting knowledge obtained by training a pre-trained model corresponding to the modality category using the first training corpus. Generate a second training corpus. The second training corpus refers to the labeled data obtained by automatically labeling the extracted knowledge after knowledge extraction from the third data through the initial extraction model. The third data is the original data used to generate the training corpus. The initial extraction model is trained based on the second training corpus to obtain the knowledge extraction model.

5. A knowledge graph construction apparatus, said apparatus being applicable to the method of any one of claims 1 to 4, characterized in that, The device includes: The determination module is used to determine the modality category of the first data, which refers to the single-modality data extracted from the original data used to construct the knowledge graph; The first acquisition module is used to acquire the representation vector of the first data corresponding to each modality category if the modality category includes multiple categories; The first construction module is used to construct a first knowledge graph, which refers to a knowledge graph constructed from the extracted knowledge after knowledge extraction based on all the representation vectors. The second construction module is used to construct a second knowledge graph, which refers to the knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the first knowledge graph.

6. The apparatus according to claim 5, characterized in that, The modal categories include text modality and image modality; The first acquisition module is used to acquire the representation vector of the first data corresponding to each modality category, specifically: The first acquisition module is used to acquire the text representation vector of the first data corresponding to the text modality and the image representation vector of the first data corresponding to the image modality.

7. The apparatus according to claim 6, characterized in that, The first construction module is used to construct the first knowledge graph, specifically: The first building module is used for: Generate a first representation vector, which is a representation vector obtained by concatenating the text representation vector and the image representation vector; Generate a second representation vector, which is the representation vector obtained by encoding the first representation vector using a fully connected layer; Generate a third representation vector, which is the representation vector obtained by convolving the second representation vector; The first knowledge graph is constructed based on the normalization result of the third representation vector.

8. The apparatus according to claim 5, characterized in that, The device further includes: The third construction module is used to construct a first entity graph if the modality category includes a category. The first entity graph refers to a knowledge graph constructed from the extracted knowledge after extracting entity knowledge from the first data using a knowledge extraction model trained based on a pre-trained model and / or a representation learning model corresponding to the modality category. The fourth construction module is used to construct a first attribute graph, which refers to a knowledge graph obtained by extracting attribute knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. The fifth construction module is used to construct a first relation graph, which refers to a knowledge graph obtained by extracting relational knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. The sixth construction module is used to construct a first event graph, which refers to a knowledge graph obtained by extracting event knowledge from the first entity graph using a knowledge extraction model trained based on the pre-trained model and / or the representation learning model. The seventh construction module is used to construct a third knowledge graph, which is a knowledge graph obtained by merging the first entity graph, the first attribute graph, the first relationship graph, and the first event graph. The eighth construction module is used to construct the fourth knowledge graph, which is a knowledge graph obtained by knowledge fusion and knowledge completion of the knowledge contained in the third knowledge graph.

9. The apparatus according to claim 8, characterized in that, The device further includes: The second acquisition module is used to acquire the first training corpus, which refers to the labeled data obtained by automatically labeling the second data according to the existing knowledge and / or general linguistic rules corresponding to the modality category; the second data is the original data used to generate the training corpus. The first generation module is used to generate an initial extraction model, which refers to a model for extracting knowledge obtained by training the pre-trained model corresponding to the modality category using the first training corpus. The second generation module is used to generate the second training corpus. The second training corpus refers to the labeled data obtained by automatically labeling the extracted knowledge after the knowledge is extracted from the third data through the initial extraction model. The third data is the original data used to generate the training corpus. The training module is used to train the initial extraction model based on the second training corpus to obtain the knowledge extraction model.

Images