Knowledge graph-based text drawing method, apparatus and device, medium and product
By acquiring and fusing multimodal features from text and entity information, and utilizing knowledge graphs to generate images that better align with human cognition, the problem of insufficient image accuracy in existing technologies is solved, achieving more refined and accurate image generation.
Patent Information
- Application Number
- CN202510284597.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-11
- Publication Date
- 2026-02-10
AI Technical Summary
Existing text-based graph models cannot fully utilize knowledge graphs, resulting in generated images that do not conform to human cognition and affect accuracy.
By acquiring entity information of the text to be processed, including entity text information, entity image information, and entity time information, feature extraction and fusion are performed using the first knowledge graph, and an image is generated by combining the feature processing model.
It improves the precision and accuracy of generated images to align with human cognition, and ensures the accuracy and real-time nature of information by updating the knowledge graph through large model agents.
Smart Images
Figure CN121502003A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text-to-image method and device based on a knowledge graph, equipment, media and products. BACKGROUND
[0002] Text-to-image is a technology that generates images that meet the description of the text by inputting text instructions. It is used in AIGC (Artificial Intelligence Generated Content) applications such as game modeling, advertising marketing, and media content generation. However, if the text instructions contain text content that can only be understood by experts in a specific field, the text-to-image model may not be able to understand the text content and generate images that do not conform to human understanding.
[0003] To solve the above problems, the current method mainly uses a knowledge graph to introduce entity knowledge and strengthen the understanding ability of the text-to-image model to generate images that are more consistent with human cognition. However, this method mainly focuses on the mining and knowledge representation of text information, and does not fully utilize image information, which may limit its effectiveness in certain application scenarios that require knowledge reasoning combined with image information and affect the accuracy of the generated images. SUMMARY
[0004] The present application provides a text-to-image method, device, equipment, media and products based on a knowledge graph to solve the problem of not being able to fully utilize the knowledge graph and affecting the accuracy of the generated images in the prior art.
[0005] To achieve the above-mentioned purpose, the present application provides a text-to-image method based on a knowledge graph, comprising:
[0006] Obtaining a text to be processed;
[0007] Extracting entity information of the text to be processed through a first knowledge graph; the entity information includes at least one of entity text information, entity image information and entity time information;
[0008] Performing feature extraction and fusion on the text to be processed and the entity information to obtain first fusion features of the text to be processed;
[0009] Generating an image of the text to be processed according to the first fusion features of the text to be processed.
[0010] As an improvement of the above-mentioned scheme, the generating an image of the text to be processed according to the first fusion features of the text to be processed comprises:
[0011] The first fusion feature of the text to be processed is input into the trained feature processing model to obtain the second fusion feature of the text to be processed; the trained feature processing model is obtained by comparing and learning positive fusion feature samples and negative fusion feature samples of text samples.
[0012] An image of the text to be processed is obtained based on the second fusion feature of the text to be processed.
[0013] As an improvement to the above scheme, the positive and negative fusion feature samples of the text samples are obtained through the following steps:
[0014] By using a pre-constructed second knowledge graph, positive entity information samples and negative entity information samples are extracted from the text samples;
[0015] Feature extraction and fusion are performed on the text sample and the positive entity information sample to obtain the positive fused feature sample of the text sample;
[0016] Feature extraction and fusion are performed on the text sample and the negative entity information sample to obtain the negative fused feature sample of the text sample.
[0017] As an improvement to the above scheme, during the contrastive learning process, the feature processing model performs contrastive learning based on the following loss function:
[0018]
[0019] Among them, L gen The loss function of the feature processing model. This is the first fusion feature sample of the text sample. These are positive fusion feature samples for text samples. Let be the i-th negative fusion feature sample of the text sample, k be the number of negative fusion feature samples of the text sample, τ be the temperature coefficient, exp be the exponential function with the natural base e, and log be the logarithmic function.
[0020] As an improvement to the above scheme, the step of extracting and fusing features from the text to be processed and the entity information to obtain the first fused feature of the text to be processed includes:
[0021] The text to be processed is input into a pre-trained first encoder to obtain the multimodal features of the text to be processed; wherein, the multimodal features of the text to be processed include at least one of the text structure features, text topic features, and text semantic features of the text to be processed; the pre-trained first encoder is obtained by machine learning training through multimodal feature samples of text samples;
[0022] The entity information is input into a pre-trained second encoder to obtain the multimodal features of the entity information; wherein, the pre-trained second encoder is obtained by machine learning training through multimodal feature samples of entity information samples of text samples;
[0023] The multimodal features of the text to be processed and the multimodal features of the entity information are combined to obtain the first fusion feature of the text to be processed.
[0024] As an improvement to the above solution, the step of extracting entity information of the text to be processed through the first knowledge graph includes:
[0025] Entity extraction is performed on the text to be processed to obtain the entities of the text to be processed;
[0026] The entity is queried in the first knowledge graph;
[0027] If the entity is found, the entity information is obtained;
[0028] If the entity is not found, the first knowledge graph is updated based on the webpage search results for the entity, and the entity is then queried in the updated first knowledge graph.
[0029] To achieve the above objectives, embodiments of this application also provide a text-to-graph method apparatus based on knowledge graphs, comprising:
[0030] The acquisition module is used to acquire the text to be processed;
[0031] An extraction module is used to extract entity information from the text to be processed using a first knowledge graph; the entity information includes at least one of entity text information, entity image information, and entity time information.
[0032] The fusion module is used to extract and fuse features from the text to be processed and the entity information to obtain the first fused feature of the text to be processed.
[0033] The generation module is used to generate an image of the text to be processed based on the first fusion feature of the text to be processed.
[0034] To achieve the above objectives, embodiments of this application also provide a knowledge graph-based text-to-graph method apparatus, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the knowledge graph-based text-to-graph method as described above.
[0035] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the knowledge graph-based text-to-graph method as described above.
[0036] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the knowledge graph-based text-to-graph method as described above.
[0037] Compared with existing technologies, the embodiments of this application provide a text-to-image method, apparatus, device, storage medium, and product based on a knowledge graph. This method extracts entity information from the text to be processed using a first knowledge graph. The entity information includes at least one of entity text information, entity image information, and entity time information. Features are extracted and fused from the text to be processed and the entity information to obtain a first fused feature of the text to be processed. Based on the first fused feature, an image of the text to be processed is generated. Therefore, the embodiments of this application not only combine the text features of the text to be processed but also at least one of the entity text information, entity image information, and entity time information, fully utilizing the text to be processed and the knowledge graph. This helps to generate more refined images that are more consistent with human cognition, improving the accuracy of the text-to-image result, i.e., the generated image. Attached Figure Description
[0038] Figure 1 This is a flowchart of a text-to-graph method based on knowledge graphs provided in an embodiment of this application;
[0039] Figure 2 This is a structural block diagram of a text-to-graph method device based on knowledge graphs provided in an embodiment of this application;
[0040] Figure 3 This is a structural block diagram of a text-to-graph method device based on knowledge graphs provided in an embodiment of this application. Detailed Implementation
[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0042] See Figure 1 , Figure 1This is a flowchart of a text-to-graph generation method based on knowledge graphs provided in an embodiment of this application. The text-to-graph generation method based on knowledge graphs includes:
[0043] Get the text to be processed;
[0044] The entity information of the text to be processed is extracted using the first knowledge graph; the entity information includes at least one of entity text information, entity image information, and entity time information.
[0045] Feature extraction and fusion are performed on the text to be processed and the entity information to obtain the first fused feature of the text to be processed.
[0046] An image of the text to be processed is generated based on the first fusion feature of the text to be processed.
[0047] It is understandable that the first knowledge graph is a domain knowledge graph. By introducing the domain knowledge graph, the generation of images that do not conform to human understanding is avoided. Furthermore, the embodiments of this application not only combine the text features of the text to be processed, but also combine at least one of the entity text information, entity image information, and entity time information of the text to be processed. This fully utilizes the text to be processed and the first knowledge graph, which helps to generate more refined images that are more in line with human cognition, thereby improving the accuracy of the text-to-image result, i.e., the generated image.
[0048] Optionally, the first knowledge graph is obtained by a large model agent based on web search results for the target entity.
[0049] It is worth noting that the method in this application embodiment is highly dependent on the quality of the first knowledge graph. If the first knowledge graph contains erroneous or outdated information, it may affect the overall effectiveness of the knowledge assistance. In traditional methods, knowledge graphs need to be regularly updated and maintained using key technologies such as data fusion, entity alignment, and relationship discovery to ensure the accuracy and reliability of the knowledge they provide. Thanks to the superior text understanding and reasoning capabilities of large-scale models, large-scale model agents demonstrate outstanding performance in natural language processing tasks such as web page information retrieval, entity recognition and disambiguation, and relationship extraction. Therefore, this application embodiment utilizes large-scale model agents to construct an efficient, accurate, and real-time updated knowledge graph based on the web page search results of the target sample, further improving the accuracy of the text-generated graph results.
[0050] The target entity can be either the entity of the text to be processed or the entity of the text sample, where the text sample is the text used as a sample.
[0051] In one optional embodiment, extracting entity information from the text to be processed using a first knowledge graph includes:
[0052] Entity extraction is performed on the text to be processed to obtain the entities of the text to be processed;
[0053] The entity is queried in the first knowledge graph;
[0054] If the entity is found, the entity information is obtained;
[0055] If the entity is not found, the first knowledge graph is updated based on the webpage search results for the entity, and the entity is then queried in the updated first knowledge graph.
[0056] This embodiment first extracts entities from the text to be processed. Then, it queries these entities in a first knowledge graph. If a corresponding entity is found, at least one of its textual, image, and temporal information is obtained. If no corresponding entity is found, the first knowledge graph is updated based on the web search results for that entity. Specifically, a graph construction command is input to the large model agent, and the entity information is obtained using a web search tool and a web parsing tool. An input tool is then called to insert the entity into the first knowledge graph, completing the update of the first knowledge graph. This ensures that the entity can be queried and its information obtained in the updated first knowledge graph.
[0057] In an optional embodiment, the step of extracting and fusing features from the text to be processed and the entity information to obtain a first fused feature of the text to be processed includes:
[0058] The text to be processed is input into a pre-trained first encoder to obtain the multimodal features of the text to be processed; wherein, the multimodal features of the text to be processed include at least one of the text structure features, text topic features, and text semantic features of the text to be processed; the pre-trained first encoder is obtained by machine learning training through multimodal feature samples of text samples;
[0059] The entity information is input into a pre-trained second encoder to obtain the multimodal features of the entity information; wherein, the pre-trained second encoder is obtained by machine learning training through multimodal feature samples of entity information samples of text samples;
[0060] The multimodal features of the text to be processed and the multimodal features of the entity information are combined to obtain the first fusion feature of the text to be processed.
[0061] It is worth noting that the embodiments of this application do not specifically limit the training methods of the first encoder and the second encoder, and they can be trained by any machine learning method.
[0062] The pre-trained first encoder includes one or more sub-encoders, each outputting a multimodal feature of the text to be processed. For example, it may include three sub-encoders, with the first sub-encoder being a TransformerEncoder. struct Second sub-encoder Transformer Encoder topic and the third sub-encoder TransformerEncoder text First sub-encoder (Transformer Encoder) struct The text structure features h used to output the text to be processed struct The second sub-encoder, Transformer Encoder topic The text topic feature h used to output the text to be processed topic The third sub-encoder, Transformer Encoder text Text semantic features h used to output the text to be processed text :
[0063] h struct =W struct *Transformer Encoder struct (h struct,1 ,h struct,2 ,...,h struct,t )+b struct ;
[0064] h topic =W topic *Transformer Encoder topic (h topic,1 ,h topic,2 ,...,h topic,t )+b topic ;
[0065] h text =W text *Transformer Encoder text (h text,1 ,h text,2 ,...,h text,t )+b text ;
[0066] Among them, W struct b struct These are the weights and biases of the text structure features, h, and so on. struct,t The text to be processed is in the first sub-encoder, Transformer Encoder struct The hidden vector at time t; Wtopic b topic These are the weights and biases of the text topic features, h, and so on. topic,t The text to be processed is in the second sub-encoder, Transformer Encoder topic The hidden vector at time t; W text b text These are the weights and biases of the text semantic features, h, respectively. text,t For the text to be processed in the third sub-encoder Transformer Encoder text The hidden vector at time t.
[0067] The entity information is input into the pre-trained second encoder, Transformer Encoder. entity The multimodal features h of the entity information are obtained. entity :
[0068] h entity =W entity *Transformer Encoder entity (h entity,1 ,h entity,2 ,...,h entity,t )+b entity ;
[0069] Among them W entity b entity These are the weights and biases of the features of entity information, respectively, h. entity,t For entity information in the second encoder, Transformer Encoder entity The hidden vector at time t. For example, if entity information includes entity text information, entity image information, and entity time information, then h entity,t =[h entity,t,文本 ,h entity,t,图像 ,h entity,t,时间 ], h entity,t,文本 The entity text information is represented in the second encoder, Transformer Encoder. entity The hidden vector at time t, h entity,t,图像 The entity image information is represented in the second encoder, Transformer Encoder. entity The hidden vector at time t, h entity,t,时间 The entity's time information is represented in the second encoder, Transformer Encoder. entity The hidden vector at time t can be fully mined by the second encoder to extract the textual features, image features, and temporal features of the entity, and finally concatenated into the multimodal features of the entity.
[0070] The multimodal features of the text to be processed and the multimodal features of the entity information are combined to obtain the first fusion feature h of the text to be processed. c :h c =[h struct ,h topic ,h entity ,h text ].
[0071] The above characteristics are explained below:
[0072] Text structure features of text (e.g., text to be processed, text sample): Scanning the text from left to right, identifying the parts of speech and order of verbs, nouns, etc. contained therein, can help the image generation module to more fully understand the semantics of the text.
[0073] Text topic features: By finely dividing the text topic, such as technical topics like product layout, technological breakthroughs, and financial report announcements, we can accurately locate the general semantics of the text and more accurately capture the topic of the generated image.
[0074] Textual semantic features: Mapping text to a high-dimensional vector can fully represent the semantic information of the text.
[0075] Textual features of entities (e.g., entities of the text to be processed, entity samples of text samples): features that describe the textual information of the entity.
[0076] Image features of an entity: Features that describe the image information of an entity.
[0077] Temporal characteristics of an entity: Features that describe the temporal information of an entity.
[0078] Furthermore, generating an image of the text to be processed based on the first fusion feature of the text to be processed includes: performing feature processing on the first fusion feature of the text to be processed to obtain a second fusion feature of the text to be processed; and obtaining an image of the text to be processed based on the second fusion feature of the text to be processed.
[0079] In order to generate a more refined and accurate image, this embodiment of the application performs feature processing on the first fusion feature of the text to be processed to obtain a more refined second fusion feature, so as to obtain a more refined and accurate image based on the second fusion feature. This embodiment of the application does not limit the specific method of feature processing, such as feature extraction, feature enhancement, etc.
[0080] Here, the Diffusion Transformer (DiT) model and image decoder can be used to generate images. For example, the first / second fusion features of the sample to be processed can be input into DiT for a reverse process of forward noise addition, i.e.:
[0081] h DiT =Linear Decoder(AdaLN(W DiT *(DiT Encoder(h′ c ))+b DiT ));
[0082] Among them, W DiT b DiT These are the weights of the Diffusion Transformer model, AdaLN is the adaptive layer normalization, Linear Decoder is the linear decoding layer, DiT Encoder is the encoder of the Diffusion Transformer model, and h′ is the weights of the Diffusion Transformer model. c This represents the hidden vector of the encoder in the Diffusion Transformer model representing the first / second fusion feature of the sample to be processed.
[0083] The latent space representation h is obtained after encoding using the Diffusion Transformer model. DiT Then, the image decoder VAE Decoder is used to process h. DiT The image can be obtained by decoding:
[0084] Res = VAE Decoder(h DiT );
[0085] Here, Res represents the image of the text to be processed.
[0086] In one optional embodiment, generating an image of the text to be processed based on a first fusion feature of the text to be processed includes:
[0087] The first fusion feature of the text to be processed is input into the trained feature processing model to obtain the second fusion feature of the text to be processed; the trained feature processing model is obtained by comparing and learning positive fusion feature samples and negative fusion feature samples of text samples.
[0088] An image of the text to be processed is obtained based on the second fusion feature of the text to be processed.
[0089] In this embodiment of the application, through comparative learning, the first fusion feature of the sample to be processed is made closer to its own positive fusion feature (i.e., closest) and farther away from its own negative fusion feature (i.e. farthest), which helps to generate a second fusion feature that is closer to the real professional field, thereby generating a more refined image that is more in line with human cognition and greatly improving the accuracy of the text-based image results.
[0090] In an optional embodiment, the positive and negative fusion feature samples of the text sample are obtained through the following steps:
[0091] By using a pre-constructed second knowledge graph, positive entity information samples and negative entity information samples are extracted from the text samples;
[0092] Feature extraction and fusion are performed on the text sample and the positive entity information sample to obtain the positive fused feature sample of the text sample;
[0093] Feature extraction and fusion are performed on the text sample and the negative entity information sample to obtain the negative fused feature sample of the text sample.
[0094] It is worth noting that the second knowledge graph is a realistic and accurate knowledge graph built based on expert knowledge, which can provide guidance for text-to-graph results. In the second knowledge graph, the text information, image information, time information, and other entity information of each entity are stored to construct multimodal entity information, thereby obtaining multimodal entity features.
[0095] Specifically, the step of extracting positive entity information samples and negative entity information samples from the text samples using a pre-constructed second knowledge graph includes:
[0096] Entity information samples of the text samples are extracted using a pre-constructed second knowledge graph.
[0097] according to The distance between the vector of the text sample and the vector of the entity information sample; where, dis T,E Let T represent the distance between the vector T of the text sample and the vector E of the entity information sample, where T represents the vector of the text sample and E represents the vector of the entity information sample. text E represents a vector of entity text information samples. img E represents a vector of entity image information samples. time A vector representing a sample of entity time information;
[0098] The closest entity information sample is taken as the positive entity information sample of the text sample;
[0099] The entity information sample that is furthest away is taken as the negative entity information sample of the text sample.
[0100] Specifically, the step of extracting and fusing features from the text sample and the positive entity information sample to obtain the positive fused feature sample of the text sample includes:
[0101] The text sample is input into a pre-trained first encoder to obtain multimodal feature samples of the text sample;
[0102] The positive entity information sample is input into a pre-trained second encoder to obtain the multimodal feature sample of the positive entity information sample;
[0103] The multimodal feature samples of the text sample and the multimodal feature samples of the positive entity information sample are combined to obtain the positive fusion feature sample of the text sample.
[0104] The positive fusion feature sample generation method of this application embodiment is similar to the first fusion feature generation method of the text to be processed, and will not be described in detail here.
[0105] Specifically, the step of extracting and fusing features from the text sample and the negative entity information sample to obtain the negative fused feature sample of the text sample includes:
[0106] The text sample is input into a pre-trained first encoder to obtain multimodal feature samples of the text sample;
[0107] The negative entity information sample is input into a pre-trained second encoder to obtain the multimodal feature sample of the negative entity information sample;
[0108] The multimodal feature samples of the text sample and the multimodal feature samples of the negative entity information sample are combined to obtain the negative fusion feature sample of the text sample.
[0109] The method for generating negative fusion feature samples in this application is similar to the method for generating the first fusion feature of the text to be processed, and will not be described in detail here.
[0110] In one alternative embodiment, during the contrastive learning process, the feature processing model performs contrastive learning based on the following loss function:
[0111]
[0112] Among them, L gen The loss function of the feature processing model. This is the first fusion feature sample of the text sample. These are positive fusion feature samples for text samples. Let be the i-th negative fusion feature sample of the text sample, k be the number of negative fusion feature samples of the text sample, τ be the temperature coefficient used to adjust the distribution of similarity scores, exp be the exponential function with the natural base e, and log be the logarithmic function.
[0113] This application embodiment constructs a loss function so that the parameter update direction of the feature processing model during contrastive learning training is such that the first fused feature sample of the text sample is oriented towards the first fused feature sample of the text sample. Positive fusion feature samples that are closer to text samples The auxiliary generation of second fusion features is closer to those in real professional fields. The method for generating the first fusion feature samples of the text samples is similar to that for the text to be processed, and will not be elaborated further here.
[0114] This application provides a text-to-image (TPE) method based on a knowledge graph. It extracts entity information from the text to be processed using a first knowledge graph. The entity information includes at least one of entity text information, entity image information, and entity time information. Features are extracted and fused from the text to be processed and the entity information to obtain a first fused feature of the text to be processed. Based on the first fused feature, an image of the text to be processed is generated. Therefore, this application not only combines the text features of the text to be processed but also at least one of the entity text information, entity image information, and entity time information, fully utilizing the text to be processed and the knowledge graph. This helps generate more refined images that are more consistent with human cognition, improving the accuracy of the TPE result, i.e., the generated image.
[0115] See Figure 2 , Figure 2 This is a structural block diagram of a text-to-graph method device 10 based on a knowledge graph, provided in an embodiment of this application. The text-to-graph method device 10 based on a knowledge graph includes:
[0116] Module 11 is used to acquire the text to be processed;
[0117] Extraction module 12 is used to extract entity information of the text to be processed through a first knowledge graph; the entity information includes at least one of entity text information, entity image information and entity time information;
[0118] The fusion module 13 is used to extract and fuse features of the text to be processed and the entity information to obtain the first fused feature of the text to be processed;
[0119] The generation module 14 is used to generate an image of the text to be processed based on the first fusion feature of the text to be processed.
[0120] Optionally, the generation module 14 is specifically used for:
[0121] The first fusion feature of the text to be processed is input into the trained feature processing model to obtain the second fusion feature of the text to be processed; the trained feature processing model is obtained by comparing and learning positive fusion feature samples and negative fusion feature samples of text samples.
[0122] An image of the text to be processed is obtained based on the second fusion feature of the text to be processed.
[0123] Optionally, the generation module 14 is further configured to:
[0124] By using a pre-constructed second knowledge graph, positive entity information samples and negative entity information samples are extracted from the text samples;
[0125] Feature extraction and fusion are performed on the text sample and the positive entity information sample to obtain the positive fused feature sample of the text sample;
[0126] Feature extraction and fusion are performed on the text sample and the negative entity information sample to obtain the negative fused feature sample of the text sample.
[0127] Optionally, the generation module 14 is further configured to:
[0128] During the contrastive learning process, the feature processing model performs contrastive learning based on the following loss function:
[0129]
[0130] Among them, L gen The loss function of the feature processing model. This is the first fusion feature sample of the text sample. These are positive fusion feature samples for text samples. Let be the i-th negative fusion feature sample of the text sample, k be the number of negative fusion feature samples of the text sample, τ be the temperature coefficient, exp be the exponential function with the natural base e, and log be the logarithmic function.
[0131] Optionally, the fusion module 13 is specifically used for:
[0132] The text to be processed is input into a pre-trained first encoder to obtain the multimodal features of the text to be processed; wherein, the multimodal features of the text to be processed include at least one of the text structure features, text topic features, and text semantic features of the text to be processed; the pre-trained first encoder is obtained by machine learning training through multimodal feature samples of text samples;
[0133] The entity information is input into a pre-trained second encoder to obtain the multimodal features of the entity information; wherein, the pre-trained second encoder is obtained by machine learning training through multimodal feature samples of entity information samples of text samples;
[0134] The multimodal features of the text to be processed and the multimodal features of the entity information are combined to obtain the first fusion feature of the text to be processed.
[0135] Optionally, the extraction module 12 is specifically used for:
[0136] Entity extraction is performed on the text to be processed to obtain the entities of the text to be processed;
[0137] The entity is queried in the first knowledge graph;
[0138] If the entity is found, the entity information is obtained;
[0139] If the entity is not found, the first knowledge graph is updated based on the webpage search results for the entity, and the entity is then queried in the updated first knowledge graph.
[0140] It is worth noting that the working process of each module in the knowledge graph-based text-to-graph method apparatus 10 described in this application embodiment can refer to the working process of the knowledge graph-based text-to-graph method described in the above embodiment, and will not be repeated here.
[0141] This application provides a knowledge graph-based text-to-image method and apparatus 10, which extracts entity information from the text to be processed using a first knowledge graph. The entity information includes at least one of entity text information, entity image information, and entity time information. Feature extraction and fusion are performed on the text to be processed and the entity information to obtain a first fusion feature of the text to be processed. Based on the first fusion feature, an image of the text to be processed is generated. Therefore, this application not only combines the text features of the text to be processed but also at least one of the entity text information, entity image information, and entity time information, fully utilizing the text to be processed and the knowledge graph. This helps to generate more refined images that are more consistent with human cognition, improving the accuracy of the text-to-image result, i.e., the generated image.
[0142] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the knowledge graph-based text-to-graph method as described in any of the above embodiments.
[0143] Furthermore, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the knowledge graph-based text-to-graph method as described in any of the above embodiments.
[0144] See Figure 3 , Figure 3 This is a structural block diagram of a text-to-graph method device 20 based on a knowledge graph, provided in an embodiment of this application. The device 20 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps described in the above-described knowledge graph-based text-to-graph method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.
[0145] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the knowledge graph-based text-to-image method device 20.
[0146] The knowledge graph-based text-to-graph method device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the knowledge graph-based text-to-graph method device 20 and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the knowledge graph-based text-to-graph method device 20 may also include input / output devices, network access devices, buses, etc.
[0147] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the knowledge graph-based text-to-text method device 20, connecting all parts of the device using various interfaces and lines.
[0148] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the knowledge graph-based text-to-image method device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0149] The modules / units integrated into the knowledge graph-based text graph method device 20, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0150] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0151] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. A text-to-graph method based on knowledge graphs, characterized in that, include: Get the text to be processed; Entity information of the text to be processed is extracted using the first knowledge graph; The entity information includes at least one of the following: entity text information, entity image information, and entity time information; Feature extraction and fusion are performed on the text to be processed and the entity information to obtain the first fused feature of the text to be processed. An image of the text to be processed is generated based on the first fusion feature of the text to be processed.
2. The text-to-graph method based on knowledge graphs as described in claim 1, characterized in that, The step of generating an image of the text to be processed based on the first fusion feature of the text to be processed includes: The first fusion feature of the text to be processed is input into the trained feature processing model to obtain the second fusion feature of the text to be processed; the trained feature processing model is obtained by comparing and learning positive fusion feature samples and negative fusion feature samples of text samples. An image of the text to be processed is obtained based on the second fusion feature of the text to be processed.
3. The text-to-graph method based on knowledge graphs as described in claim 2, characterized in that, The positive and negative fusion feature samples of the text sample are obtained through the following steps: By using a pre-constructed second knowledge graph, positive entity information samples and negative entity information samples are extracted from the text samples; Feature extraction and fusion are performed on the text sample and the positive entity information sample to obtain the positive fused feature sample of the text sample; Feature extraction and fusion are performed on the text sample and the negative entity information sample to obtain the negative fused feature sample of the text sample.
4. The knowledge graph-based text-to-graph method as described in any one of claims 2-3, characterized in that, During the contrastive learning process, the feature processing model performs contrastive learning based on the following loss function: Among them, L gen The loss function of the feature processing model. This is the first fusion feature sample of the text sample. These are positive fusion feature samples for text samples. Let be the i-th negative fusion feature sample of the text sample, k be the number of negative fusion feature samples of the text sample, τ be the temperature coefficient, exp be the exponential function with the natural base e, and log be the logarithmic function.
5. The text-to-graph method based on knowledge graphs as described in claim 1, characterized in that, The step of extracting and fusing features from the text to be processed and the entity information to obtain the first fused feature of the text to be processed includes: The text to be processed is input into a pre-trained first encoder to obtain the multimodal features of the text to be processed; wherein, the multimodal features of the text to be processed include at least one of the text structure features, text topic features, and text semantic features of the text to be processed; the pre-trained first encoder is obtained by machine learning training through multimodal feature samples of text samples; The entity information is input into a pre-trained second encoder to obtain the multimodal features of the entity information; wherein, the pre-trained second encoder is obtained by machine learning training through multimodal feature samples of entity information samples of text samples; The multimodal features of the text to be processed and the multimodal features of the entity information are combined to obtain the first fusion feature of the text to be processed.
6. The text-to-graph method based on knowledge graphs as described in claim 1, characterized in that, The step of extracting entity information from the text to be processed using the first knowledge graph includes: Entity extraction is performed on the text to be processed to obtain the entities of the text to be processed; The entity is queried in the first knowledge graph; If the entity is found, the entity information is obtained; If the entity is not found, the first knowledge graph is updated based on the webpage search results for the entity, and the entity is then queried in the updated first knowledge graph.
7. A text-to-graph method apparatus based on knowledge graphs, characterized in that, include: The acquisition module is used to acquire the text to be processed; An extraction module is used to extract entity information from the text to be processed using a first knowledge graph; The entity information includes at least one of the following: entity text information, entity image information, and entity time information; The fusion module is used to extract and fuse features from the text to be processed and the entity information to obtain the first fused feature of the text to be processed. The generation module is used to generate an image of the text to be processed based on the first fusion feature of the text to be processed.
8. A knowledge graph-based text-to-graph method device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the knowledge graph-based text-to-graph method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the text-to-text graph method based on any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the text-to-text graph method based on knowledge graphs as described in any one of claims 1 to 6.