Knowledge graph construction method, electronic device and computer-readable storage medium

Abstract

The present application provides a knowledge graph construction method, an electronic device and a computer-readable storage medium, the method comprising obtaining target modal information; wherein the modal information comprises text information and / or image information; identifying the target modal information through a multimodal large language model to obtain a representation vector of target triple information of the target modal information; determining whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph; wherein the learned knowledge graph is obtained by learning multiple representation vectors; each representation vector comprises a representation vector corresponding to triple sample information and neighborhood information corresponding to each triple sample information, the neighborhood information representing a representation vector within a preset similarity range with the representation vector of the triple sample information.

Description

Technical Field

[0001] The present application relates to the field of knowledge graph technology, and more specifically, to a knowledge graph construction method, an electronic device, and a computer-readable storage medium. Background Art

[0002] At present, the expansion of knowledge graphs is generally carried out by learning the features of entities and relationships, and then expanding the knowledge graph based on the metric relationships between entities and relationships. However, this method can only expand the features and metric relationships of entities and relationships used in the learning process, which limits the generalization ability of entities and relationships, resulting in a relatively limited expansion of the knowledge graph. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a knowledge graph construction method, an electronic device and a computer-readable storage medium to solve the problem that the current knowledge graph expansion construction can only further expand the features and metric relationships of the entities and relationships used in the learning process, resulting in a relatively limited expansion of the knowledge graph.

[0004] In a first aspect, the present invention provides a method for constructing a knowledge graph, the method comprising obtaining target modal information; wherein the modal information comprises text information and / or image information; identifying the target modal information through a multimodal large language model to obtain a representation vector of a target triple information of the target modal information; determining whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph; wherein the learned knowledge graph is obtained by learning multiple representation vectors; each representation vector comprises a representation vector corresponding to the triple sample information and neighborhood information corresponding to each triple sample information, the neighborhood information representing a representation vector within a preset similarity range with the representation vector of the triple sample information.

[0005] The method for constructing the knowledge graph designed above obtains representation vectors of entities and relationships through a multimodal large language model, and then improves the generalization ability of the representation of entities and relationships by adding neighborhood noise on the basis of the representation vectors of entities and relationships obtained by the multimodal large language model. The knowledge graph is learned based on the representation vectors of entities and relationships with added neighborhood noise. In this way, when the knowledge graph is expanded, entities and relationships within the generalization ability range of entities and relationships can expand the knowledge graph, that is, entities and relationships within the representation vectors of entities and relationships and their neighborhood noise can expand the knowledge graph, thereby improving the expansion and generalization capabilities of the knowledge graph.

[0006] In an optional implementation of the first aspect, the target modal information is identified through a multimodal large language model to obtain a representation vector of the target triple information of the target modal information, including: identifying the target modal information through a multimodal large language model to obtain target triple information corresponding to the target modal information output by the multimodal large language model; and representing the target triple information through the multimodal large language model to obtain a representation vector of the target triple information.

[0007] In an optional implementation of the first aspect, based on the representation vector of the target triple information and the learned knowledge graph, it is determined whether to expand and construct the learned knowledge graph, including: judging whether the representation vector of the target triple information meets the expansion and construction conditions through the learned knowledge graph; if it is determined that the representation vector of the target triple information meets the expansion and construction conditions, then expanding and constructing the learned knowledge graph based on the representation vector of the target triple information.

[0008] In an optional implementation of the first aspect, the learned knowledge graph is used to determine whether the representation vector of the target triple information meets the expansion construction condition, including: determining whether there is triple information identical to the target triple information in the learned knowledge graph, and determining whether there is neighborhood information corresponding to the target triple information in the learned knowledge graph; if it is determined that there is triple information identical to the target triple information in the learned knowledge graph, or there is neighborhood information corresponding to the target triple information in the learned knowledge graph, then it is determined that the representation vector of the target triple information meets the expansion construction condition.

[0009] In an optional implementation of the first aspect, the learned knowledge graph is constructed in the following manner: obtaining multiple modal sample information; identifying the multiple modal sample information through a multimodal large language model to obtain a representation vector of the triple sample information of each modal sample information; obtaining neighborhood information; generating multiple learning representation vectors based on the representation vector and neighborhood information of each triple sample information; performing representation learning on the knowledge graph based on the multiple learning representation vectors until the loss value calculated by the representation learning reaches the target loss value, thereby obtaining the learned knowledge graph.

[0010] In an optional implementation of the first aspect, the multiple modal sample information are identified through a multimodal large language model to obtain a representation vector of triple sample information of each modal sample information, including: identifying the modal sample information through a multimodal large language model to obtain triple sample information of the modal sample information; characterizing the triple sample information of the modal sample information through the multimodal large language model to obtain an initial representation vector of the triple sample information of the modal sample information; and reducing the dimension of the initial representation vector of the triple sample information of the modal sample information to obtain a representation vector of the triple sample information of each modal sample information.

[0011] In an optional implementation of the first aspect, obtaining neighborhood information includes: obtaining a similarity range corresponding to each meta-information in a preset triple information; and generating domain information corresponding to each meta-information according to the similarity range corresponding to each meta-information and the target noise.

[0012] In an optional implementation of the first aspect, multiple learning representation vectors are generated based on the representation vector and neighborhood information of each triple sample information, including: calculating the sum of the representation vector of each meta-sample information in the representation vector of each triple sample information and the domain information corresponding to the corresponding meta-information, and obtaining the learning representation vector corresponding to the representation vector of each triple sample information.

[0013] In a second aspect, the present application provides a device for constructing a knowledge graph, the device comprising an acquisition module, an identification and representation module, and a determination module, the acquisition module being used to acquire target modal information; wherein the modal information comprises text information and / or image information; the identification and representation module being used to identify the target modal information through a multimodal large language model, and obtain a representation vector of target triple information of the target modal information; the determination module being used to determine whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph; wherein the learned knowledge graph is obtained by learning multiple representation vectors; each representation vector comprises a representation vector corresponding to the triple sample information and neighborhood information corresponding to each triple sample information, and the neighborhood information represents a representation vector within a preset similarity range with the representation vector of the triple sample information.

[0014] The device for constructing the knowledge graph designed above, this scheme obtains representation vectors of entities and relationships through a multimodal large language model, and then improves the representation generalization ability of entities and relationships by adding neighborhood noise on the basis of the representation vectors of entities and relationships obtained by the multimodal large language model, and learns the knowledge graph based on the representation vectors of entities and relationships with added neighborhood noise. In this way, when the knowledge graph is expanded, entities and relationships within the generalization ability range of entities and relationships can expand the knowledge graph, that is, entities and relationships within the representation vectors of entities and relationships and their neighborhood noise can expand the knowledge graph, thereby improving the expansion ability and generalization ability of the knowledge graph.

[0015] In an optional implementation of the second aspect, the recognition and representation module is specifically used to recognize the target modal information through a multimodal large language model, and obtain target triple information corresponding to the target modal information output by the multimodal large language model; and represent the target triple information through the multimodal large language model to obtain a representation vector of the target triple information.

[0016] In an optional implementation of the second aspect, the determination module is specifically used to determine whether the representation vector of the target triple information meets the expansion construction conditions through the learned knowledge graph; if it is determined that the representation vector of the target triple information meets the expansion construction conditions, the learned knowledge graph is expanded and constructed according to the representation vector of the target triple information.

[0017] In an optional implementation of the second aspect, the determination module is also specifically used to determine whether there is triple information identical to the target triple information in the learned knowledge graph, and to determine whether there is neighborhood triple information corresponding to the target triple information in the learned knowledge graph; if it is determined that there is triple information identical to the target triple information in the learned knowledge graph, or there is neighborhood triple information corresponding to the target triple information in the learned knowledge graph, then it is determined that the representation vector of the target triple information meets the expansion construction condition.

[0018] In an optional implementation of the second aspect, the device also includes a learning construction module for obtaining multiple modal sample information; identifying multiple modal sample information through a multimodal large language model to obtain a representation vector of triple sample information of each modal sample information; obtaining neighborhood information; generating multiple learning representation vectors based on the representation vector and neighborhood information of each triple sample information; performing representation learning on the knowledge graph based on the multiple learning representation vectors until the loss value calculated by the representation learning reaches the target loss value, thereby obtaining a knowledge graph whose learning has been completed.

[0019] In an optional implementation of the second aspect, the learning construction module is specifically used to identify the modal sample information through a multimodal large language model to obtain triple sample information of the modal sample information; characterize the triple sample information of the modal sample information through the multimodal large language model to obtain an initial characterization vector of the triple sample information of the modal sample information; and perform dimensionality reduction on the initial characterization vector of the triple sample information of the modal sample information to obtain a characterization vector of the triple sample information of each modal sample information.

[0020] In an optional implementation of the second aspect, the learning construction module is further specifically used to obtain a similarity range corresponding to each meta-information in a preset triple information; and generate domain information corresponding to each meta-information according to the similarity range corresponding to each meta-information and the target noise.

[0021] In an optional implementation of the second aspect, the learning construction module is further specifically used to calculate the sum of the representation vector of each meta-sample information in the representation vector of each triple sample information and the domain information corresponding to the corresponding meta-information, and obtain the learning representation vector corresponding to the representation vector of each triple sample information.

[0022] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes any optional method described in the first aspect when executing the computer program.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs any optional method described in the first aspect.

[0024] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs the steps of any optional method described in the first aspect.

[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a process for constructing a knowledge graph provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the knowledge graph representation learning structure provided in the embodiment of the present application;

[0029] Figure 3 Entity and relationship domain loss diagram provided for the embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of a knowledge graph construction device provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] Icons: 400 - acquisition module; 410 - identification and characterization module; 420 - determination module; 430 - learning and construction module; 5 - electronic device; 501 - processor; 502 - memory; 503 - communication bus. DETAILED DESCRIPTION

[0033] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0035] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0038] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0039] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0040] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0041] In the prior art, the expansion of the knowledge graph is generally carried out by learning the features of entities and relationships, and then expanding the knowledge graph according to the metric relationships between the entities and relationships. However, this method can only expand the features and metric relationships of the entities and relationships used in the learning process, which limits the generalization ability of entities and relationships, resulting in a relatively limited expansion of the knowledge graph.

[0042] Based on the above problems, the present application designs a method for constructing a knowledge graph, an electronic device and a computer-readable storage medium, which obtains representation vectors of entities and relationships through a multimodal large language model, and then improves the generalization ability of the representation of entities and relationships by adding neighborhood noise on the basis of the representation vectors of entities and relationships obtained by the multimodal large language model. The knowledge graph is learned based on the representation vectors of entities and relationships with added neighborhood noise. In this way, when the knowledge graph is expanded, entities and relationships within the generalization ability range of entities and relationships can expand the knowledge graph, that is, entities and relationships within the representation vectors of entities and relationships and their neighborhood noise can expand the knowledge graph, thereby improving the expansion and generalization capabilities of the knowledge graph.

[0043] Based on the above ideas, this application first provides a method for constructing a knowledge graph, which can be applied to computing devices, including but not limited to computers, servers, computing chips, GPUs, CPUs, etc. Figure 1 As shown, the method for constructing the knowledge graph can be implemented in the following ways, including:

[0044] Step S100: Acquire target modality information.

[0045] Step S110: identifying the target modal information through a multimodal large language model to obtain a representation vector of the target triplet information of the target modal information.

[0046] Step S120: Determine whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph.

[0047] In the above implementation, the target modality information may include text information and / or image information, that is, the target modality information may be any one or both of text or image. The target modality information may be obtained through user input or by calling text or image stored in the computing device itself.

[0048] When the target modal information is obtained in the above manner, this solution recognizes the target modal information through a multimodal large language model to obtain a representation vector of the target triplet information of the target modal information.

[0049] As a possible implementation method, this solution can first identify the target modal information through a multimodal large language model, obtain the target triple information corresponding to the target modal information output by the multimodal large language model, and then characterize the obtained target triple information to obtain the representation vector of the target triple information. For example, this solution can input the following content to the multimodal large language model: "You are an expert in natural language processing and are very good at extracting entities and relationships from texts. Here is a text for you. You need to extract the triple information you think may exist from the text, that is, (head entity, relationship, tail entity). The text given to you is that Liu Bei and Sun Shangxiang are husband and wife." In this way, the multimodal large language model can obtain the triple information of Liu Bei, husband and wife, and Sun Shangxiang, and then characterize it to obtain the representation vector of the triple information.

[0050] Among them, the multimodal large language model described above can adopt any current multimodal large language model, for example, multimodal large language models such as LLAVA, qwen-vl, glm-4v, etc. can be adopted.

[0051] When the representation vector of the target triple information is obtained by the method described above, this solution determines whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph.

[0052] Among them, Figure 2 As shown in the figure, knowledge graphs can be represented and learned in the following ways:

[0053] First, multiple modal sample information is obtained, and then the multiple modal sample information is recognized through a multimodal large language model to obtain a representation vector of the triplet sample information of each modal sample information.

[0054] Specifically, the scheme can identify the modal sample information through the multimodal large language model to obtain the triple sample information of the modal sample information; then, the triple sample information of the modal sample information is represented by the multimodal large language model to obtain the initial representation vector of the triple sample information of the modal sample information, for example, the initial representation vector e of the head entity is obtained. h , the initial representation vector e of the relationship r , tail entity representation vector e t .

[0055] Specifically, it can be expressed as follows:

[0056] ; ;

[0057] Among them, e and r represent entities and relationships respectively. Respectively represent the text information and picture information of the relationship, the text information and picture information of the entity, For a multimodal large language model, are the obtained entity initial representation vector and relationship initial representation vector.

[0058] After the initial representation vector is obtained in the above manner, since the representation vectors obtained by the general multimodal large language model are all high-dimensional vectors, in this case, this solution can reduce the dimension of the initial representation vector of the triple sample information of the modal sample information to obtain the representation vector of the triple sample information of the modal sample information. Figure 2 As shown, this solution can reduce the dimension of high-dimensional vectors through multiple multi-layer perceptrons, thereby obtaining a representation vector of triple sample information. The multi-layer perceptrons used for relations and entities can be different. For example, the multi-layer perceptron used for entities can be MLP1, and the multi-layer perceptron used for relations can be MLP2.

[0059] Specifically, this solution can be expressed as follows:

[0060]

[0061] Among them, h, r, and t are the low-dimensional features of the head entity, relation, and tail entity, respectively.

[0062] In the above case, this solution obtains neighborhood information (neighborhood noise), and then generates multiple learning representation vectors based on the representation vector and neighborhood information of each triple sample information. Among them, obtaining the neighborhood information can first obtain the similarity range corresponding to each meta information in the preset triple information, and then generate the domain information corresponding to each meta information based on the similarity range corresponding to each meta information and the target noise; the generation of the learning representation vector can be obtained by calculating the sum of the representation vector of each meta sample information in the representation vector of each triple sample information and the domain information corresponding to the corresponding meta information.

[0063] Specifically, the above method can be illustrated by the following examples:

[0064] When low-dimensional features of the head entity, relationship, and tail entity are obtained, the scheme may first map the features of the entity and relationship to the same semantic space. For example, the scheme may map the low-dimensional features of the head entity, relationship, and tail entity obtained above to the same semantic space through the same multi-layer perceptron MLP:

[0065]

[0066] Then, Gaussian noise is introduced into the neighborhoods of the head entity, tail entity, and relationship to improve the generalization ability of entities and relationships, namely:

[0067] ;

[0068] ;

[0069] ;

[0070] in, represents Gaussian noise, , are the similarity ranges of the head entity, tail entity, and relationship, that is, the similarity range corresponding to each piece of information in the triple information, , as well as They are the neighborhood information of the head entity, tail entity and relationship respectively.

[0071] The difference between the learned representation vector with added neighborhood information and the original representation vector can be illustrated by the following example:

[0072] For example, in the field of interpersonal relationship recognition or user recognition, assuming that the representation vector of the original head entity is The representation vector representing the entity "Liu Bei", and the learning representation vector after adding neighborhood information It can represent the representation vector of entities that are similar to Liu Bei, such as "Liu Bei", "Liu Bei 1", "Liu Bei 2", etc.; for another example, in the field of disease recognition in the medical field, the representation vector of the original head entity is The representation vector representing the entity "Type I diabetes" and the learned representation vector after adding neighborhood information It can represent the representation vector of entities that are similar to the entity of diabetes, such as "type I diabetes" and "type II diabetes".

[0073] Based on the above, this solution performs representation learning on the knowledge graph based on multiple learning representation vectors until the loss value calculated by the representation learning reaches the target loss value, and then obtains the learned knowledge graph. That is, this solution calculates the loss function: , optimize the parameters in the network by checking whether the loss value meets the target loss value. Figure 3 , Figure 3 The comparison in shows the difference between the loss calculation of this scheme and the loss calculation of the existing knowledge graph representation learning. The loss calculation of the traditional knowledge graph representation learning only uses the method of h+r=t, that is, the low-dimensional features of the head entity, relationship, and tail entity, while this scheme uses the representation learning vector of the head entity, relationship, and tail entity obtained by introducing the neighborhood information, that is, = ,in, Figure 3The circle range in the figure can represent the recognition expansion range of the knowledge graph for each element of the triple, from Figure 3 It can be seen that The range of (dashed line range) is larger than the range of h, The range of (dashed line range) is larger than the range of r, The range of (dashed line range) is larger than the range of t, which makes the generalization ability of the knowledge graph representing learning stronger.

[0074] The learned knowledge graph can be obtained in the above manner. In this case, when the representation vector of the target triple information is obtained, the learned knowledge graph can determine whether the representation vector of the target triple information meets the expansion construction conditions; if it is determined that the representation vector of the target triple information meets the expansion construction conditions, the learned knowledge graph is expanded and constructed according to the representation vector of the target triple information; if it is determined that the representation vector of the triple information does not meet the expansion construction conditions, the learned knowledge graph is not expanded and constructed. Among them, the specific expansion construction process can adopt any current knowledge graph expansion construction method based on triple information, and this scheme does not limit this.

[0075] Specifically, this scheme can determine whether there is triple information identical to the target triple information in the learned knowledge graph, and determine whether there is neighborhood information corresponding to the target triple information in the learned knowledge graph; if it is determined that there is triple information identical to the target triple information in the learned knowledge graph, or there is neighborhood information corresponding to the target triple information in the learned knowledge graph, then it is determined that the representation vector of the target triple information meets the expansion construction conditions.

[0076] As a specific example, according to the description above, since the learned knowledge graph representation is obtained through the learning representation vector formed by the combination of the representation vector and the neighborhood information during the learning process, the generalization ability of the knowledge graph is relatively strong. For example, according to the description above, assuming that the head entity in the target triple information this time is "type II diabetes", and since the knowledge graph is obtained based on the learning representation vector formed by the combination of the representation vector of the head entity "type I diabetes" and its neighborhood information during the representation learning process, if the representation vector of the head entity of the target triple information whose head entity is "type II diabetes" is within the neighborhood information corresponding to the target triple information, therefore, the target triple information whose head entity is "type II diabetes" can be determined by the knowledge graph as meeting the expansion construction conditions, thereby expanding and constructing the learned knowledge graph according to the representation vector of the target triple information, so that the learned knowledge graph is not only expanded for the original entity and relationship representation in the learning process, but also for the entities and relationships in the neighborhood information with similar original entities and relationships, thereby improving the ability of the knowledge graph to expand and construct.

[0077] The method for constructing the knowledge graph designed above obtains representation vectors of entities and relationships through a multimodal large language model, and then improves the generalization ability of the representation of entities and relationships by adding neighborhood noise on the basis of the representation vectors of entities and relationships obtained by the multimodal large language model. The knowledge graph is learned based on the representation vectors of entities and relationships with added neighborhood noise. In this way, when the knowledge graph is expanded, entities and relationships within the generalization ability range of entities and relationships can expand the knowledge graph, that is, entities and relationships within the representation vectors of entities and relationships and their neighborhood noise can expand the knowledge graph, thereby improving the expansion and generalization capabilities of the knowledge graph.

[0078] Figure 4 The present application provides a schematic structural block diagram of a device. It should be understood that the device is applied to the electronic device described above. Figures 1 to 3The method embodiment executed in the embodiment corresponds to the method embodiment, and can execute the steps involved in the aforementioned method. The specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device. Specifically, the device includes: an acquisition module 400, an identification and representation module 410 and a determination module 420, wherein the acquisition module 400 is used to acquire target modal information; wherein the modal information includes text information and / or image information; the identification and representation module 410 is used to identify the target modal information through a multimodal large language model to obtain a representation vector of the target triple information of the target modal information; the determination module 420 is used to determine whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph; wherein the learned knowledge graph is obtained by learning multiple representation vectors; each representation vector includes a representation vector corresponding to the triple sample information and neighborhood information corresponding to each triple sample information, and the neighborhood information represents a representation vector within a preset similarity range with the representation vector of the triple sample information.

[0079] The device for constructing the knowledge graph designed above, this scheme obtains representation vectors of entities and relationships through a multimodal large language model, and then improves the representation generalization ability of entities and relationships by adding neighborhood noise on the basis of the representation vectors of entities and relationships obtained by the multimodal large language model, and learns the knowledge graph based on the representation vectors of entities and relationships with added neighborhood noise. In this way, when the knowledge graph is expanded, entities and relationships within the generalization ability range of entities and relationships can expand the knowledge graph, that is, entities and relationships within the representation vectors of entities and relationships and their neighborhood noise can expand the knowledge graph, thereby improving the expansion ability and generalization ability of the knowledge graph.

[0080] According to some embodiments of the present application, the recognition and representation module 410 is specifically used to recognize the target modal information through a multimodal large language model, and obtain target triple information corresponding to the target modal information output by the multimodal large language model; and to represent the target triple information through the multimodal large language model to obtain a representation vector of the target triple information.

[0081] According to some embodiments of the present application, the determination module 420 is specifically used to determine whether the representation vector of the target triple information meets the expansion construction conditions through the learned knowledge graph; if it is determined that the representation vector of the target triple information meets the expansion construction conditions, the learned knowledge graph is expanded and constructed according to the representation vector of the target triple information.

[0082] According to some embodiments of the present application, the determination module 420 is also specifically used to determine whether there is triple information identical to the target triple information in the learned knowledge graph, and to determine whether there is neighborhood triple information corresponding to the target triple information in the learned knowledge graph; if it is determined that there is triple information identical to the target triple information in the learned knowledge graph, or there is neighborhood triple information corresponding to the target triple information in the learned knowledge graph, then it is determined that the representation vector of the target triple information meets the expansion construction condition.

[0083] According to some embodiments of the present application, the device also includes a learning construction module 430, which is used to obtain multiple modal sample information; identify multiple modal sample information through a multimodal large language model to obtain a representation vector of the triple sample information of each modal sample information; obtain neighborhood information; generate multiple learning representation vectors based on the representation vector and neighborhood information of each triple sample information; perform representation learning on the knowledge graph based on the multiple learning representation vectors until the loss value calculated by the representation learning reaches the target loss value, thereby obtaining a knowledge graph whose learning has been completed.

[0084] According to some embodiments of the present application, the learning construction module 430 is specifically used to identify the modal sample information through a multimodal large language model to obtain triple sample information of the modal sample information; characterize the triple sample information of the modal sample information through the multimodal large language model to obtain an initial characterization vector of the triple sample information of the modal sample information; and reduce the dimension of the initial characterization vector of the triple sample information of the modal sample information to obtain a characterization vector of the triple sample information of each modal sample information.

[0085] According to some embodiments of the present application, the learning construction module 430 is also specifically used to obtain the similarity range corresponding to each meta-information in the preset triple information; and generate the domain information corresponding to each meta-information according to the similarity range corresponding to each meta-information and the target noise.

[0086] According to some embodiments of the present application, the learning construction module 430 is also specifically used to calculate the sum of the representation vector of each meta-sample information in the representation vector of each triple sample information and the domain information corresponding to the corresponding meta-information, and obtain the learning representation vector corresponding to the representation vector of each triple sample information.

[0087] According to some embodiments of the present application, Figure 5As shown, the present application provides an electronic device 5, including: a processor 501 and a memory 502, the processor 501 and the memory 502 are interconnected and communicate with each other through a communication bus 503 and / or other forms of connection mechanisms (not shown), the memory 502 stores a computer program executable by the processor 501, and when the computing device is running, the processor 501 executes the computer program to execute any optional implementation method, such as steps S100 to S120: obtaining target modal information; identifying the target modal information through a multimodal large language model to obtain a representation vector of the target triple information of the target modal information; determining whether to expand and construct the learned knowledge graph based on the representation vector of the target triple information and the learned knowledge graph.

[0088] The present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0089] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0090] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for constructing a knowledge graph, characterized in that: The method comprises: Acquire target modal information; wherein the modal information includes text information and / or image information; Recognizing the target modal information through a multimodal large language model to obtain a representation vector of the target triplet information of the target modal information; Determining whether to expand and construct the learned knowledge graph according to the representation vector of the target triple information and the learned knowledge graph; The learned knowledge graph is obtained by learning multiple representation vectors; Each representation vector includes a representation vector corresponding to the triple sample information and neighborhood information corresponding to each triple sample information, wherein the neighborhood information represents a representation vector within a preset similarity range with the representation vector of the triple sample information; The knowledge graph completed by learning is constructed in the following way: Get multiple modal sample information; Recognize the multiple modal sample information by using a multimodal large language model to obtain a representation vector of triplet sample information of each modal sample information; Get neighborhood information; Generate multiple learning representation vectors according to the representation vector of each triplet sample information and the neighborhood information; Representation learning is performed on the knowledge graph according to the multiple learning representation vectors until the loss value calculated by the representation learning reaches the target loss value, thereby obtaining a learned knowledge graph.

2. The method according to claim 1, characterized in that The step of identifying the target modal information by using a multimodal large language model to obtain a representation vector of target triplet information of the target modal information includes: The target modal information is identified by the multimodal large language model to obtain target triple information corresponding to the target modal information output by the multimodal large language model; The target triple information is represented by the multimodal large language model to obtain a representation vector of the target triple information.

3. The method according to claim 1, characterized in that The step of determining whether to expand and construct the learned knowledge graph according to the representation vector of the target triple information and the learned knowledge graph includes: By using the learned knowledge graph, determining whether the representation vector of the target triple information meets the expansion construction condition; If it is determined that the representation vector of the target triple information meets the expansion construction condition, the learned knowledge graph is expanded and constructed according to the representation vector of the target triple information.

4. The method according to claim 3, characterized in that The knowledge graph completed through the learning determines whether the representation vector of the target triple information meets the expansion construction condition, including: Determine whether there is triple information identical to the target triple information in the learned knowledge graph, and determine whether there is neighborhood information corresponding to the target triple information in the learned knowledge graph; If it is determined that there is triple information identical to the target triple information in the learned knowledge graph, or there is neighborhood information corresponding to the target triple information in the learned knowledge graph, then it is determined that the representation vector of the target triple information meets the expansion construction condition.

5. The method according to claim 1, characterized in that The identifying the plurality of modal sample information by using a multimodal large language model to obtain a representation vector of triplet sample information of each modal sample information includes: Recognizing the modal sample information by using a multimodal large language model to obtain triple sample information of the modal sample information; Characterizing the triple sample information of the modal sample information by a multimodal large language model to obtain an initial characterization vector of the triple sample information of the modal sample information; The initial characterization vector of the triplet sample information of the modal sample information is reduced in dimension to obtain a characterization vector of each triplet sample information of the modal sample information.

6. The method according to claim 5, characterized in that The obtaining of neighborhood information includes: Obtain the similarity range corresponding to each piece of information in the preset triple information; The domain information corresponding to each meta-information is generated according to the similarity range and target noise corresponding to each meta-information.

7. The method according to claim 6, characterized in that The step of generating a plurality of learning representation vectors according to the representation vector of each triplet sample information and the neighborhood information comprises: The sum of the representation vector of each meta-sample information in the representation vector of each triplet of sample information and the domain information corresponding to the corresponding meta-information is calculated to obtain a learning representation vector corresponding to the representation vector of each triplet of sample information.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images