Knowledge graph-combined brilliance large-scale model culture knowledge generation and retrieval method

Abstract

The invention relates to a brilliance large-scale model culture knowledge generation and retrieval method combined with a knowledge graph. The method comprises the steps that entity extraction, relation modeling and dynamic completion processing are carried out according to multi-modal culture data, a culture knowledge graph is constructed in combination with a fuzzy time reasoning algorithm and a cross-modal alignment mechanism, and the multi-modal culture data comprises an unstructured culture text, image data and space-time metadata; the space-time metadata comprises text space-time metadata and image moment metadata; performing semantic alignment processing by combining a large model through a user query statement and a culture knowledge graph to obtain joint representation of cross-modal semantic alignment; and performing multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge description and associated multi-modal content. According to the method, through multi-modal data fusion, dynamic knowledge completion and fuzzy time reasoning, culture knowledge description can be accurately generated, associated multi-modal content can be retrieved, and the accuracy and the intelligent level of culture knowledge service are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph. Background Art

[0002] With the rapid development of artificial intelligence technology, the combination of knowledge graphs and natural language processing technology has gradually become a research hotspot in the fields of cultural knowledge management and intelligent retrieval. However, traditional cultural knowledge processing methods often rely on structured databases or single-modal data extraction. For example, entity relationships are extracted from texts through rule matching, or visual feature annotations are performed based on image classification models. On the one hand, the semantic association between text and image data in this method is weak, and cross-modal alignment mainly relies on artificially defined rules, making it difficult to handle large-scale unstructured data. On the other hand, the temporal descriptions of cultural events are often ambiguous, such as "mid-Tang Dynasty", but this method uses precise time point annotations and cannot effectively handle temporal uncertainties. In addition, existing cultural retrieval methods usually rely on keyword matching or simple semantic retrieval, lacking cross-modal joint reasoning ability, resulting in semantic deviations between retrieval results and user needs. Summary of the Invention

[0003] Based on this, in view of the above technical problems, the object of the present invention is to provide a method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph to further improve the accuracy and efficiency of cultural knowledge generation and retrieval and enhance the user experience.

[0004] In a first aspect, the present application provides a method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph, the method comprising:

[0005] Performing entity extraction, relationship modeling, and dynamic completion processing on multi-modal cultural data, and constructing a cultural knowledge graph by combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism, where the multi-modal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata, and the spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata;

[0006] Performing semantic alignment processing through a user query statement and the cultural knowledge graph in combination with a large model to obtain a joint representation of cross-modal semantic alignment;

[0007] Performing multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content.

[0008] In one embodiment, performing entity extraction, relationship modeling, and dynamic completion processing on multi-modal cultural data, and constructing a cultural knowledge graph by combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism, includes:

[0009] Entity and relationship extraction processing is performed on unstructured cultural texts and image data through a cross-modal joint embedding model to obtain an entity vector set and a relationship set. The cross-modal joint embedding model consists of a text encoder and an image encoder;

[0010] A time probability density function is constructed using a Gaussian distribution model through the time fuzzy description of spatio-temporal metadata to obtain a fuzzy time label;

[0011] According to the entity vector set and the relationship set, entity association paths are generated through a graph traversal algorithm, and sparse entity paths are filtered out through a preset path quantity threshold;

[0012] According to the sparse entity paths, path completion processing is performed through a reinforcement learning strategy to generate missing triples. The reward function of the reinforcement learning strategy consists of cultural domain consistency and user feedback confidence. Among them, cultural domain consistency is calculated through the distance in a pre-trained cultural semantic space, and user feedback confidence is obtained by statistically analyzing the scoring data of users on historical generation results;

[0013] The entity vector set, the relationship set, the fuzzy time label, and the missing triples are integrated to obtain a cultural knowledge graph.

[0014] In one embodiment, semantic alignment processing is performed through a user query statement and a cultural knowledge graph in combination with a large model to obtain a joint representation of cross-modal semantic alignment, including:

[0015] According to the user query statement, text hidden layer representation generation processing is performed through a large model to obtain a text hidden layer representation. The large model is a pre-trained language model based on the Transformer architecture;

[0016] Based on the user query statement, subgraph retrieval processing is performed from the cultural knowledge graph to obtain an associated subgraph;

[0017] Based on a cross-modal attention mechanism, the text hidden layer representation and the associated subgraph are fused to obtain a preliminary fusion representation;

[0018] The entity vectors of the preliminary fusion representation and the entity vectors of the cultural knowledge graph are semantically aligned through a contrastive learning loss function to obtain a joint representation.

[0019] In one embodiment, multi-modal retrieval-generation joint optimization is performed based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content, including:

[0020] According to the fuzzy time words in the user query statement, event retrieval processing is performed through probability time interval mapping to obtain a candidate event set;

[0021] Input the candidate event set and the joint representation into the large model for context semantic understanding to obtain the initial text description;

[0022] According to the explicit feedback behavior of the user on the initial text description, perform adaptive correction processing through the reinforcement learning algorithm to obtain the updated user preference. The explicit feedback behavior includes user clicks, user ratings, or text correction operations;

[0023] According to the initial text description and the updated user preference, generate a cultural knowledge description through the multimodal matching algorithm, and retrieve it in the cultural knowledge graph to output the associated multimodal content.

[0024] In one embodiment, the method further includes:

[0025] Retrieve the associated multimodal content from the cultural knowledge graph according to the entity keywords in the cultural knowledge description. The associated multimodal content includes associated text data and associated image data, and calculate the cross-modal semantic similarity between the associated multimodal content and the cultural knowledge description;

[0026] If the cross-modal semantic similarity is lower than the preset threshold, perform context semantic correction processing on the cultural knowledge description through the large model to generate an optimized cultural knowledge description, and retrieve it again from the cultural knowledge graph to obtain the optimized associated multimodal content;

[0027] Based on the updated user preference, perform weighted fusion processing on the optimized cultural knowledge description and the optimized associated multimodal content to output the final cultural knowledge description and the final associated multimodal content.

[0028] In one embodiment, the calculation formula of the contrastive learning loss function is:

[0029]

[0030] where, L align is the result of the contrastive loss function, e i is the entity vector in the cultural knowledge graph, h LLM is the text hidden layer representation generated by the large model, τ is the temperature parameter used to adjust the strictness of semantic alignment, and N is the number of negative samples.

[0031] In a second aspect, the present application also provides a Guangming large model cultural knowledge generation and retrieval system combined with a knowledge graph. The system includes:

[0032] The knowledge graph construction module is used to perform entity extraction, relationship modeling, and dynamic completion processing based on multimodal cultural data, and construct a cultural knowledge graph by combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. The multimodal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata. The spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata;

[0033] The semantic alignment processing module is used to perform semantic alignment processing through user query statements and the cultural knowledge graph, combined with a large model, to obtain a joint representation of cross-modal semantic alignment;

[0034] The joint optimization processing module is used to perform multimodal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multimodal content.

[0035] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the first aspect are implemented.

[0036] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the first aspect are implemented.

[0037] The above-mentioned method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph performs entity extraction, relationship modeling, and dynamic completion processing through multimodal cultural data, effectively making up for the defect of insufficient utilization of single-modal data in traditional cultural knowledge processing methods. It can comprehensively integrate unstructured cultural texts, image data, and spatio-temporal metadata, thereby providing a rich data foundation for subsequent cultural knowledge generation and retrieval. And by using a fuzzy time reasoning algorithm and a cross-modal alignment mechanism to construct a cultural knowledge graph, it can not only accurately represent cultural entities and their relationships, but also dynamically update the cultural knowledge graph in real time, avoiding the limitation that a static knowledge graph cannot adapt to the dynamic changes of cultural knowledge. Secondly, this method performs semantic alignment processing through user query statements and the cultural knowledge graph combined with a large model, which can effectively improve the cross-modal semantic understanding and fusion ability and generate semantic representations that meet user needs. Finally, this method performs multimodal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multimodal content, which can provide users with comprehensive cultural knowledge services.

[0038] Compared with traditional methods, this method improves the efficiency and quality of cultural knowledge generation and retrieval through multimodal data fusion, dynamic knowledge graph construction, and joint optimization, providing effective technical support for the dissemination and utilization of cultural knowledge. Brief Description of the Drawings

[0039] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the accompanying drawings required for the description of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 Flowchart of the method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph provided by an exemplary embodiment of the present invention;

[0041] Figure 2 Schematic structural diagram of the system for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph provided by an exemplary embodiment of the present invention. Detailed implementation manners

[0042] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In one embodiment, as Figure 1 shown, a method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph is provided. In this embodiment, the method is exemplified by being applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] S101: Perform entity extraction, relationship modeling, and dynamic complement processing on multimodal cultural data, and construct a cultural knowledge graph in combination with a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. The multimodal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata. The spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata.

[0045] Specifically, the named entity recognition algorithm in natural language processing technology can be used to process unstructured cultural texts such as historical documents and news reports, and extract entities such as characters, locations, and cultural events from the texts. For example, for an article introducing ancient poems, entities such as the names of poets, the locations where the poems were created, and the historical events depicted in the poems can be recognized. And the relation extraction algorithm can be used to analyze the semantic relations between entities in the text, such as the creation relation between a poet and a poem, and the association relation between a poem and the described event. For image data, the object detection algorithm in computer vision technology can be used to identify cultural-related objects in the image, such as characters, buildings, and cultural relics in ancient paintings. Then, through image semantic segmentation technology, the positions and ranges of each entity in the image can be further clarified, and combined with the image feature extraction algorithm, the visual feature vectors of these entities can be obtained for subsequent cross-modal association with text entities. And the text spatio-temporal metadata is usually included in the unstructured cultural text. For example, the relevant descriptions of the time and location of historical events mentioned in the text can use the fuzzy time reasoning algorithm to process cultural events with fuzzy time descriptions, such as "the middle of the Tang Dynasty", and associate it with a specific time range. In addition, the image moment metadata can be obtained through the shooting time information of the image, the era characteristics of the objects in the image, etc. to determine the time information of the cultural content reflected in the image. Through the cross-modal alignment mechanism, the consistency and relevance of data in different modalities in the knowledge graph can be ensured. For example, the events described in the text can be aligned with the corresponding image content to construct the corresponding cultural knowledge graph.

[0046] S102: Through the user query statement and the cultural knowledge graph, combined with the large model for semantic alignment processing, to obtain the joint representation of cross-modal semantic alignment.

[0047] Specifically, by receiving the user's query statement, such as "What are the representative works of the Tang Dynasty poet Li Bai", its semantic analysis can be carried out to understand the user's needs and intentions. Secondly, the user query statement is combined with the cultural knowledge graph, and the large model such as the pre-trained language model is used for semantic alignment processing. The large model can understand the semantic information in the query statement and match it with the entities and relations in the knowledge graph to obtain the joint representation of cross-modal semantic alignment, and then ensure that the user query statement and the information in the cultural knowledge graph can be accurately matched semantically. Schematically, the "Tang Dynasty poet Li Bai" in the query statement can be aligned with the corresponding entity in the knowledge graph, and its relationship with "representative works" can be understood.

[0048] S103: Based on the joint representation, perform multi-modal retrieval-generation joint optimization to obtain cultural knowledge descriptions and associated multi-modal content.

[0049] Specifically, based on the obtained joint representation, multi-modal retrieval and generation optimization are performed. Among them, multi-modal retrieval can search for cultural knowledge descriptions and associated multi-modal content related to the user's query in the knowledge graph, such as retrieving the text information of Li Bai's representative works and related calligraphy work pictures. Generation optimization can generate high-quality cultural knowledge descriptions according to the retrieval results and user needs, such as a detailed introduction to Li Bai's representative works and their status in the history of literature. Finally, the retrieved multi-modal content and the generated cultural knowledge descriptions can be integrated and presented to the user in an intuitive and rich way. Schematically, the life, works and related cultural background of Li Bai can be displayed in a form with both pictures and texts.

[0050] In the above-mentioned method for generating and retrieving cultural knowledge of the Guangming large model combined with the knowledge graph, by performing entity extraction, relationship modeling and dynamic complement processing on multi-modal cultural data, and using the fuzzy time reasoning algorithm and cross-modal alignment mechanism to construct the cultural knowledge graph, it can break through the limitations of constructing the knowledge graph with traditional single-modal data, deeply excavate the complex associations between cultural knowledge, lay a solid foundation for subsequent knowledge retrieval and generation, and improve the comprehensiveness and accuracy of cultural knowledge representation. Secondly, through the semantic alignment processing of the user's query statement and the cultural knowledge graph combined with the large model, the problem that it is difficult to accurately match the user's query and cultural knowledge in traditional retrieval is effectively solved, and the accuracy and comprehensiveness of semantic understanding are further improved. Finally, based on the generated joint representation, multi-modal retrieval-generation joint optimization can obtain cultural knowledge descriptions and associated multi-modal content, that is, it can not only output text descriptions, but also associate with relevant images, audio and other multi-modal content, which can bring a richer and more intuitive knowledge experience to users.

[0051] Compared with the traditional methods for processing cultural knowledge, this method significantly improves the efficiency and accuracy of obtaining cultural knowledge through multi-modal fusion, semantic alignment and multi-modal retrieval-generation joint optimization, thus providing effective technical support for fields such as cultural research, education, and dissemination, and promoting the efficient utilization, inheritance and development of cultural knowledge.

[0052] In an exemplary embodiment, according to the multi-modal cultural data, entity extraction, relationship modeling and dynamic complement processing are performed, and combined with the fuzzy time reasoning algorithm and cross-modal alignment mechanism, a cultural knowledge graph is constructed, including:

[0053] According to the unstructured cultural text and image data, entity and relationship extraction processing is performed through a cross-modal joint embedding model, and an entity vector set and a relationship set are obtained. The cross-modal joint embedding model is composed of a text encoder and an image encoder;

[0054] Through the time fuzzy description of spatio-temporal metadata, a Gaussian distribution model is used to construct a time probability density function to obtain fuzzy time tags;

[0055] Generate entity association paths through a graph traversal algorithm based on the entity vector set and the relationship set, and filter out sparse entity paths through a preset path quantity threshold;

[0056] Perform path completion processing on the sparse entity paths through a reinforcement learning strategy to generate missing triples. The reward function of the reinforcement learning strategy is composed of cultural domain consistency and user feedback confidence. Among them, the cultural domain consistency is calculated through the distance in the pre-trained cultural semantic space, and the user feedback confidence is obtained by statistically analyzing the scoring data of users on historical generation results;

[0057] Integrate the entity vector set, the relationship set, the fuzzy time label, and the missing triples to obtain a cultural knowledge graph.

[0058] Specifically, the text encoder can, through natural language processing techniques, such as using a pre-trained model based on the Transformer architecture, perform operations such as lexical analysis, syntactic analysis, and semantic understanding on unstructured cultural texts, and then identify various entities such as people, places, events, etc., and convert them into corresponding vector representations to form part of the entity vector set. And the text encoder can also analyze and extract the relationships between entities in the text, such as the "creation" relationship between a poet and a work, the "occur in" relationship between an event and a place, etc., to form part of the relationship set. In addition, an image encoder can be used to process image data. The image encoder can, based on computer vision techniques, such as using a convolutional neural network architecture, perform feature extraction operations such as edge detection and object recognition on images, identify various culture-related entities such as cultural relics, buildings, artworks, etc., and convert them into corresponding vector representations and add them to the entity vector set. And the image encoder can also analyze and extract the spatial or semantic relationships between entities in the image, such as the "contain" relationship, the "adjacent" relationship, etc., and add them to the relationship set. Therefore, through the collaborative work of the cross-modal joint embedding model, entities and relationships can be extracted from unstructured cultural texts and image data to obtain the corresponding entity vector set and relationship set.

[0059] Specifically, spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata. However, the text spatio-temporal metadata may be a vague description of the time when a cultural event occurred, such as "the mid-Tang Dynasty", "the early 20th century", etc., and the image moment metadata may be the time when the image was taken or the background of the era it reflects. For the vague time description in the text spatio-temporal metadata, a Gaussian distribution model can be used for processing. Among them, the Gaussian distribution model is a commonly used probability and statistical model, which can describe the probability distribution of data. Furthermore, the vague time description can be transformed into the parameters of the Gaussian distribution, such as the mean and variance, and the time probability density function can be obtained through calculation. This function represents the probability of occurrence at different time points. According to this probability density function, a corresponding reasonable time interval can be determined, which is the fuzzy time label. Schematically, taking "the mid-Tang Dynasty" as an example, the approximate time range of this period, such as from 713 AD to 766 AD, can be determined, and then the Gaussian distribution model is used to transform this vague time description into a probability density function. This function has the highest probability density value at the central position of the time range, such as 739 AD, and the probability density gradually decreases as the deviation from the central position increases. That is, each vague time description can be transformed into a continuous time probability distribution, namely the fuzzy time label. This label can represent the time information of cultural events more flexibly, avoiding the limitations of traditional precise time annotation, and at the same time providing a basis for the processing and reasoning of time information in the subsequent knowledge graph.

[0060] After obtaining the entity vector set and the relationship set, the entities can be used as nodes and the relationships as edges to construct an initial cultural knowledge graph. For this graph, graph traversal algorithms such as depth-first search or breadth-first search can be used to generate entity association paths starting from each entity node. For example, starting from the entity "Li Bai", through the relationship "lived in", the entity "Tang Dynasty" can be reached, and then through the relationship "poets of the same period", the entity "Du Fu" can be reached, thus forming a path "Li Bai - lived in - Tang Dynasty - poets of the same period - Du Fu". This graph traversal algorithm can systematically explore the entity associations in the knowledge graph and generate a large number of entity paths. These entity paths reflect the complex relationships and semantic connections between different entities in cultural knowledge. However, since the number of generated entity association paths may be large, and some of these paths may contribute less to the understanding and application of cultural knowledge, a preset path number threshold can be set, for example, only the top K paths with higher occurrence frequencies or important cultural significance are retained, and the paths with fewer occurrences, weaker associations or redundancies are removed, so as to obtain a sparse entity path set. This screening process helps to reduce the data volume, improve the efficiency and accuracy of subsequent processing, and retain the entity association information that is most valuable for the construction of the cultural knowledge graph.

[0061] For incomplete paths or missing triples (i.e., subject, relation, object) in the sparse entity path set, a reinforcement learning strategy can be adopted for path completion. Reinforcement learning is a machine learning method that enables an agent to interact with the environment and learn the optimal strategy based on the reward signals feedback by the environment, that is, continuously optimize by designing a reward function. In this embodiment, the reward function consists of two parts: cultural domain consistency and user feedback confidence. The cultural domain consistency can be calculated through the distance in the pre-trained cultural semantic space. That is, in the cultural semantic space, the closer the semantic distance between the completed triple and the existing cultural knowledge, the more it conforms to the logic and rules of the cultural domain, and the higher the reward value. The user feedback confidence can be obtained by statistically analyzing the scoring data of the user on the historical generation results. If the completed triple can get a high score from the user, it means that it meets the user's expectations and cognitions, and the reward value will also increase accordingly. Under the reinforcement learning framework, the agent, that is, the path completion algorithm, can add new entities and relations to complete the path according to the current sparse entity path state, generate missing triples, calculate the reward value of the triple according to the reward function, and continuously feedback and adjust to maximize the cumulative reward, find the optimal path completion strategy, generate missing triples that conform to the logic of the cultural domain and the user's expectations, so as to enrich and improve the entity association information in the knowledge graph, and improve the integrity and accuracy of the knowledge graph.

[0062] Integrate the entity vector set, relation set, fuzzy time label, and the missing triples generated through path completion processing. Specifically, the entity vector set and relation set can be used as the basic structure of the knowledge graph, the fuzzy time label can be associated with the corresponding entity or relation to add information about the time dimension to the knowledge graph. And integrate the completed triples into the knowledge graph to fill the original gaps and deficiencies, then a more complete and rich cultural knowledge graph can be formed. In addition, during the integration process, the knowledge graph can be further optimized, such as removing duplicate or conflicting information, adjusting the representation of entities and relations to improve query efficiency, etc. Finally, the integrated cultural knowledge graph can be stored in formats such as RDF format or graph database format for subsequent calls and queries by the cultural knowledge generation and retrieval system. This cultural knowledge graph not only contains rich information in multi-modal cultural data, but also has stronger dynamics and integrity through fuzzy time reasoning and path completion processing, etc., and can provide a solid foundation for subsequent retrieval, generation, and analysis of cultural knowledge.

[0063] In an exemplary embodiment, through the user query statement and the cultural knowledge graph, combined with the large model for semantic alignment processing, a cross-modal semantic alignment joint representation is obtained, including:

[0064] According to the user's query statement, text hidden layer representation generation processing is performed through a large model to obtain the text hidden layer representation. The large model is a pre-trained language model based on the Transformer architecture;

[0065] Based on the user's query statement, subgraph retrieval processing is performed from the cultural knowledge graph to obtain an associated subgraph;

[0066] Based on the cross-modal attention mechanism, the text hidden layer representation and the associated subgraph are fused to obtain a preliminary fusion representation;

[0067] Through the contrastive learning loss function, the entity vectors of the preliminary fusion representation and the entity vectors of the cultural knowledge graph are semantically aligned to obtain a joint representation.

[0068] Specifically, the user can submit a query statement through a terminal or a server, such as "Please introduce the representative works and creative styles of the Tang Dynasty poet Li Bai". After receiving the query statement, preprocessing can be performed on it, such as removing stop words, stemming, lemmatization, etc., to further simplify the text and improve the efficiency of subsequent processing. Subsequently, a pre-trained language model based on the Transformer architecture can be used to encode the preprocessed query statement. Among them, the Transformer architecture can capture the long-distance dependencies between words in the text through the multi-head attention mechanism and generate a text hidden layer representation with rich semantic information. For example, the model can map words such as "Tang Dynasty", "poet", "Li Bai", "representative works", and "creative style" to vector representations in a high-dimensional semantic space. This vector contains the semantic features of the word and its context information in the sentence, providing a basis for subsequent semantic alignment processing. When constructing the cultural knowledge graph, an index structure can be constructed based on key information such as entity names, relationship types, and fuzzy time tags for quick retrieval. For example, an inverted index is established for each entity (such as "Li Bai", "Tang Dynasty", "poetry", etc.) in the knowledge graph to record its position and related relationships in the graph. According to the keywords and semantic information in the user's query statement, the associated subgraph is retrieved from this cultural knowledge graph. For example, for the query statement "Introduction to the representative works and creative styles of the Tang Dynasty poet Li Bai", the core entities "Li Bai", "Tang Dynasty", "representative works", and "creative style" can be identified, and then the subgraph directly related to the corresponding entity or connected through a certain relationship path is searched in the knowledge graph, so as to provide more comprehensive structured information for subsequent semantic fusion.

[0069] The cross-modal attention mechanism can automatically learn the correlation weights between different parts of the text and the subgraph, and perform weighted fusion on the text hidden layer representation and information such as entity vectors and relationship vectors in the associated subgraph to obtain a preliminary fusion representation. For example, for the semantic unit of "creative style" in the text hidden layer representation, this mechanism can calculate the correlation scores between it and each node in the associated subgraph, and assign different attention weights according to the scores, so that the subgraph nodes more relevant to "creative style" have a greater influence in the fusion process. Finally, operations such as concatenation and weighted summation are performed on the text hidden layer representation vector and the subgraph vector adjusted by the attention weights to obtain a preliminary fusion representation. This preliminary fusion representation contains both the text semantic information of the user query statement and the structured knowledge of the relevant subgraph in the cultural knowledge graph, providing a comprehensive semantic representation for further semantic alignment processing. In addition, contrastive learning is a machine learning method that can learn better feature representations by comparing the similarities and differences between different samples. Therefore, a contrastive learning loss function can be constructed to perform semantic alignment processing on the entity vectors of the preliminary fusion representation and the entity vectors of the cultural knowledge graph. This loss function aims to minimize the semantic difference between the entity vectors of the preliminary fusion representation and the corresponding entity vectors in the cultural knowledge graph, while maximizing the semantic distance between different entity vectors. For example, a contrastive loss function based on cosine similarity can be adopted, and the entity vectors in the preliminary fusion representation are continuously adjusted through the backpropagation algorithm, so that its entity vectors are more aligned with the entity vectors in the cultural knowledge graph in the semantic space, and finally a joint representation of cross-modal semantic alignment is obtained. This joint representation can accurately reflect the semantic relationship between the user query statement and the cultural knowledge graph, providing a high-quality semantic basis for subsequent multi-modal retrieval and generation.

[0070] In an exemplary embodiment, multi-modal retrieval-generation joint optimization is performed based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content, including:

[0071] According to the fuzzy time words in the user query statement, event retrieval processing is performed through probabilistic time interval mapping to obtain a candidate event set;

[0072] The candidate event set and the joint representation are input into a large model for context semantic understanding to obtain an initial text description;

[0073] According to the explicit feedback behavior of the user on the initial text description, adaptive correction processing is performed through a reinforcement learning algorithm to obtain updated user preferences. The explicit feedback behavior includes user clicks, user ratings, or text correction operations;

[0074] Generate a cultural knowledge description through a multimodal matching algorithm based on the initial text description and the updated user preferences, and retrieve it in the cultural knowledge graph to output associated multimodal content.

[0075] Specifically, analyze the user's query statement to identify fuzzy time words such as "ancient times", "modern times", "last century", etc., and use a predefined probability time interval mapping table to map the fuzzy time word to the corresponding time interval probability distribution. For example, "ancient China" may be mapped to multiple time intervals, such as "Pre-Qin period", "Han and Tang dynasties", etc., and each interval has a corresponding probability value indicating the likelihood of the fuzzy time word corresponding to that interval. Based on the mapped time interval probability distribution, combined with the cultural knowledge graph, retrieve cultural events related to these time intervals to generate a candidate event set. Input the generated candidate event set and the previously obtained joint representation into the large model. The large model, based on its powerful semantic understanding and generation capabilities, comprehensively processes the information to generate an initial text description related to the user's query. For example, if the user's query is about "the characteristics of ancient Chinese calligraphy art", the large model can generate an initial text describing the characteristics of ancient Chinese calligraphy art based on events such as "Wei and Jin calligraphy styles" and "Tang dynasty calligraphy schools" in the candidate event set and the semantic information in the joint representation. This text can include information such as the style characteristics of calligraphy in different dynasties, representative calligraphers and their works, providing basic text materials for subsequent text optimization and multimodal content retrieval. Subsequently, the explicit feedback behavior of the user on the initial text description can be monitored. Among them, the user's click can be to click on specific content in the text to obtain more information, the user's rating can be to rate the accuracy, integrity, etc. of the generated text, and the text correction operation can be that the user directly modifies the incorrect content in the text, etc. Through this explicit feedback behavior, the user preferences can be adaptively corrected using a reinforcement learning algorithm. Schematically, a reward function can be defined, taking the user's positive feedback such as high ratings and no modifications as positive reward signals, and negative feedback such as low ratings and frequent modifications as negative reward signals, continuously adjusting the strategy, and finally obtaining the updated user preferences, thereby realizing the personalized adaptation of user preferences.

[0076] Specifically, by combining the updated user preferences with the initial text description, the final cultural knowledge description can be generated through a multimodal matching algorithm. Among them, the multimodal matching algorithm can comprehensively consider various modal factors such as text semantics, image features, spatio-temporal information, etc., to ensure that the generated cultural knowledge description not only conforms to the user preferences in terms of content but also matches the multimodal information in the cultural knowledge graph. Guided by this cultural knowledge description, the cultural knowledge graph can be retrieved to obtain the associated multimodal content. This multimodal content can include relevant cultural images such as historical relic pictures, art work images, etc., video materials, audio resources, etc. Outputting the associated multimodal content and the cultural knowledge description to the user can provide the user with a rich and comprehensive cultural knowledge experience, meeting the user's diverse needs for cultural knowledge.

[0077] In an exemplary embodiment, the method further includes:

[0078] According to the entity keywords in the cultural knowledge description, retrieve the associated multimodal content from the cultural knowledge graph. The associated multimodal content includes associated text data and associated image data, and calculate the cross-modal semantic similarity between the associated multimodal content and the cultural knowledge description;

[0079] If the cross-modal semantic similarity is lower than the preset threshold, then perform context semantic correction processing on the cultural knowledge description through a large model to generate an optimized cultural knowledge description, and retrieve again from the cultural knowledge graph to obtain optimized associated multimodal content;

[0080] Based on the updated user preferences, perform weighted fusion processing on the optimized cultural knowledge description and the optimized associated multimodal content, and output the final cultural knowledge description and the final associated multimodal content.

[0081] Specifically, key entity keywords can be extracted from the generated cultural knowledge descriptions. Based on these keywords, retrieval operations can be carried out in the cultural knowledge graph to obtain corresponding associated multimodal content. Secondly, for the retrieved associated multimodal content and the original cultural knowledge descriptions, corresponding semantic features are extracted respectively. Schematically, in the associated multimodal content, for text data, natural language processing techniques such as word embedding and semantic analysis can be used to convert it into a semantic vector representation; for image data, visual feature vectors of the image can be extracted through image recognition and feature extraction algorithms. And corresponding semantic feature extraction is also carried out on the cultural knowledge descriptions, converting them into comparable semantic vector forms. Subsequently, similarity calculation methods such as cosine similarity and Euclidean distance can be used to measure the semantic similarity degree between the associated multimodal content and the cultural knowledge descriptions. By comparing the calculated cross-modal semantic similarity with a preset similarity threshold, it can be determined whether the associated multimodal content and the cultural knowledge descriptions reach a satisfactory semantic matching degree. If the similarity is lower than the preset threshold, it means that there is a large deviation in semantics between the associated multimodal content and the cultural knowledge descriptions, and subsequent optimization processing is still required.

[0082] Schematically, when it is found that the cross-modal semantic similarity is lower than the preset threshold, a large model can be used to perform context semantic correction on the cultural knowledge descriptions. The large model can deeply analyze the context of the cultural knowledge descriptions, identify parts where the semantic expression is unclear, inaccurate or does not match the associated multimodal content. For example, if the time expression of a certain historical event in the cultural knowledge description is vague, and the associated image data contains elements that can reflect the specific time of the event, the large model can accurately correct the time expression according to the context logic and associated information, generating an optimized cultural knowledge description. This description is more accurate and clear semantically, and the semantic consistency with the associated multimodal content is also significantly improved.

[0083] Based on the optimized cultural knowledge descriptions, retrieving associated multimodal content from the cultural knowledge graph again can more accurately locate the multimodal content that matches it. User preferences reflect the degree of importance that users attach to different types of information in the process of obtaining cultural knowledge. According to the updated user preferences, the weighting ratios of the cultural knowledge descriptions and the associated multimodal content can be determined, that is, different types of information can be organically integrated according to the weights to obtain the final cultural knowledge descriptions and the final associated multimodal content. This process can continuously optimize and accurately integrate the cultural knowledge descriptions and the associated multimodal content, ensuring that the output results not only meet user preferences but also have high-quality semantic consistency and content richness, greatly improving the level of cultural knowledge services and the user experience.

[0084] In an exemplary embodiment, the calculation formula of the contrastive learning loss function is:

[0085]

[0086] Among them, L align is the result of the contrast loss function, e i is the entity vector in the cultural knowledge graph, h LLM is the hidden layer representation of the text generated by the large model, τ is the temperature parameter used to adjust the strictness of semantic alignment, and N is the number of negative samples.

[0087] Specifically, the above formula evaluates the quality of semantic alignment by calculating the similarity between the target entity vector and the hidden layer representation of the text, and comparing it with the similarities of all candidate entities, namely positive samples and negative samples. Through this formula, the entity vector in the cultural knowledge graph can be more accurately aligned with the hidden layer representation of the text generated by the large model in the semantic space, thereby improving the effect of cross-modal semantic understanding and fusion, and providing a more reliable semantic basis for subsequent cultural knowledge generation and retrieval tasks.

[0088] Based on the same inventive concept, as Figure 2 shown, the embodiment of the present application further provides a Bright large model cultural knowledge generation and retrieval system 200 combined with a knowledge graph. The system includes:

[0089] A knowledge graph construction module 201, configured to perform entity extraction, relationship modeling, and dynamic completion processing according to multi-modal cultural data, and construct a cultural knowledge graph in combination with a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. The multi-modal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata. The spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata;

[0090] A semantic alignment processing module 202, configured to perform semantic alignment processing through a user query statement and a cultural knowledge graph in combination with a large model to obtain a joint representation of cross-modal semantic alignment;

[0091] A joint optimization processing module 203, configured to perform multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content.

[0092] In the above-mentioned Bright Model cultural knowledge generation and retrieval system 200 integrated with a knowledge graph, the knowledge graph construction module 201 constructs a cultural knowledge graph by performing entity extraction, relationship modeling, and dynamic completion processing on multi-modal cultural data, and combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. This module can comprehensively integrate unstructured cultural texts, image data, and spatio-temporal metadata, and can break through the limitations of constructing a knowledge graph with traditional single data sources, thereby providing a richer data foundation for subsequent cultural knowledge generation and retrieval. The semantic alignment processing module 202 can perform semantic alignment processing through the combination of user query statements and the cultural knowledge graph with the large model to obtain a joint representation of cross-modal semantic alignment. This process can effectively improve the cross-modal semantic understanding and fusion ability, generate a semantic representation that better meets the user's needs, and thus provide an accurate semantic foundation for subsequent retrieval and generation tasks. The joint optimization processing module 203 can perform multi-modal retrieval-generation joint optimization based on the joint representation, can simultaneously achieve accurate retrieval and high-quality generation, obtain corresponding cultural knowledge descriptions and associated multi-modal content, provide comprehensive cultural knowledge services for users, significantly improve the efficiency and quality of cultural knowledge generation and retrieval, and provide effective technical support for the dissemination and utilization of cultural knowledge.

[0093] Furthermore, the knowledge graph construction module 201 includes:

[0094] A fuzzy time reasoning sub-unit, used for:

[0095] According to unstructured cultural texts and image data, perform entity and relationship extraction processing through a cross-modal joint embedding model to obtain an entity vector set and a relationship set. The cross-modal joint embedding model consists of a text encoder and an image encoder;

[0096] Through the time fuzzy description of spatio-temporal metadata, construct a time probability density function using a Gaussian distribution model to obtain a fuzzy time label;

[0097] A cross-modal alignment sub-unit, used for:

[0098] According to the entity vector set and the relationship set, generate entity association paths through a graph traversal algorithm, and filter out sparse entity paths through a preset path quantity threshold;

[0099] According to the sparse entity paths, perform path completion processing through a reinforcement learning strategy to generate missing triples. The reward function of the reinforcement learning strategy consists of cultural domain consistency and user feedback confidence. Among them, cultural domain consistency is calculated through the distance in a pre-trained cultural semantic space, and user feedback confidence is statistically obtained based on the scoring data of the user's evaluation of historical generation results;

[0100] Integrate the entity vector set, relation set, fuzzy time tags, and missing triples to obtain a cultural knowledge graph.

[0101] Further, the semantic alignment processing module 202 includes:

[0102] A data processing sub-unit for:

[0103] According to the user's query statement, perform text hidden layer representation generation processing through a large model to obtain a text hidden layer representation. The large model is a pre-trained language model based on the Transformer architecture;

[0104] Based on the user's query statement, perform subgraph retrieval processing from the cultural knowledge graph to obtain an associated subgraph;

[0105] A data fusion sub-unit for:

[0106] Based on the cross-modal attention mechanism, fuse the text hidden layer representation and the associated subgraph to obtain a preliminary fusion representation;

[0107] Through a contrastive learning loss function, perform semantic alignment processing on the entity vectors of the preliminary fusion representation and the entity vectors of the cultural knowledge graph to obtain a joint representation.

[0108] Further, the joint optimization processing module 203 includes:

[0109] A text optimization sub-unit for:

[0110] According to the fuzzy time words in the user's query statement, perform event retrieval processing through probability time interval mapping to obtain a set of candidate events;

[0111] Input the set of candidate events and the joint representation into the large model for context semantic understanding to obtain an initial text description;

[0112] According to the user's explicit feedback behavior on the initial text description, perform adaptive correction processing through a reinforcement learning algorithm to obtain updated user preferences. The explicit feedback behavior includes user clicks, user ratings, or text correction operations;

[0113] A content output sub-unit for generating cultural knowledge descriptions through a multi-modal matching algorithm based on the initial text description and the updated user preferences, and retrieving in the cultural knowledge graph to output associated multi-modal content.

[0114] Further, the system also includes a content retrieval and generation module for:

[0115] Retrieve the associated multimodal content from the cultural knowledge graph according to the entity keywords in the cultural knowledge description. The associated multimodal content includes associated text data and associated image data, and calculate the cross-modal semantic similarity between the associated multimodal content and the cultural knowledge description;

[0116] If the cross-modal semantic similarity is lower than the preset threshold, perform context semantic correction processing on the cultural knowledge description through a large model to generate an optimized cultural knowledge description, and retrieve it again from the cultural knowledge graph to obtain optimized associated multimodal content;

[0117] Based on the updated user preferences, perform weighted fusion processing on the optimized cultural knowledge description and the optimized associated multimodal content, and output the final cultural knowledge description and the final associated multimodal content.

[0118] Schematically, the calculation formula of the contrastive learning loss function is:

[0119]

[0120] where, L align is the result of the contrastive loss function, e i is the entity vector in the cultural knowledge graph, h LLM is the text hidden layer representation generated by the large model, τ is the temperature parameter, used to adjust the strictness of semantic alignment, and N is the number of negative samples.

[0121] In an exemplary embodiment, the present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the Bright large model cultural knowledge generation and retrieval method combined with the knowledge graph of the present application. A multi-core processor is preferred to improve the parallel processing ability of the system. Memory: Provide sufficient temporary storage space to support the operation of the program and the processing of data. The memory capacity should be large enough to accommodate a large amount of supply information and computing tasks.

[0122] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating and retrieving cultural knowledge of the Guangming large model in combination with a knowledge graph according to the present application are implemented. The computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), solid state drives (SSD, Solid State Drives), or optical discs, etc. Among them, the random access memory may include resistive random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).

[0123] The above-described embodiments merely represent several implementation manners of the embodiments of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A method for generating and retrieving cultural knowledge of the Guangming large model combined with a knowledge graph, characterized in that, The method includes: Performing entity extraction, relationship modeling, and dynamic completion processing on multi-modal cultural data, and constructing a cultural knowledge graph by combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. The multi-modal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata, and the spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata; Performing semantic alignment processing through a user query statement and the cultural knowledge graph, in combination with a large model, to obtain a joint representation of cross-modal semantic alignment; Performing multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content.

2. The method according to claim 1, characterized in that The performing entity extraction, relationship modeling, and dynamic completion processing on multi-modal cultural data, and constructing a cultural knowledge graph by combining a fuzzy time reasoning algorithm and a cross-modal alignment mechanism, includes: Performing entity and relationship extraction processing on the unstructured cultural text and the image data through a cross-modal joint embedding model to obtain an entity vector set and a relationship set. The cross-modal joint embedding model consists of a text encoder and an image encoder; Constructing a time probability density function using a Gaussian distribution model through the time fuzzy description of the spatio-temporal metadata to obtain a fuzzy time label; Generating entity association paths according to the entity vector set and the relationship set through a graph traversal algorithm, and screening out sparse entity paths through a preset path quantity threshold; Performing path completion processing on the sparse entity paths through a reinforcement learning strategy to generate missing triples. The reward function of the reinforcement learning strategy consists of cultural domain consistency and user feedback confidence. Among them, the cultural domain consistency is calculated through the distance in a pre-trained cultural semantic space, and the user feedback confidence is statistically obtained based on the scoring data of the user's evaluation of the historical generation results; Integrating the entity vector set, the relationship set, the fuzzy time label, and the missing triples to obtain the cultural knowledge graph.

3. The method according to claim 1, characterized in that The performing semantic alignment processing through a user query statement and the cultural knowledge graph, in combination with a large model, to obtain a joint representation of cross-modal semantic alignment, includes: Performing text hidden layer representation generation processing on the user query statement through a large model to obtain a text hidden layer representation. The large model is a pre-trained language model based on the Transformer architecture; Performing subgraph retrieval processing from the cultural knowledge graph based on the user query statement to obtain an associated subgraph; Fusing the text hidden layer representation and the associated subgraph based on a cross-modal attention mechanism to obtain a preliminary fusion representation; Performing semantic alignment processing on the entity vectors of the preliminary fusion representation and the entity vectors of the cultural knowledge graph through a contrastive learning loss function to obtain the joint representation.

4. The method according to claim 1, characterized in that The performing multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge descriptions and associated multi-modal content, includes: Performing event retrieval processing on the fuzzy time words in the user query statement through probability time interval mapping to obtain a candidate event set; Input the set of candidate events and the joint representation into the large model for context semantic understanding to obtain an initial text description; According to the explicit feedback behavior of the user on the initial text description, perform adaptive correction processing through a reinforcement learning algorithm to obtain updated user preferences. The explicit feedback behavior includes user clicks, user ratings, or text correction operations; According to the initial text description and the updated user preferences, generate the cultural knowledge description through a multimodal matching algorithm, and retrieve it in the cultural knowledge graph to output the associated multimodal content.

5. The method according to claim 1, characterized in that, The method further includes: According to the entity keywords in the cultural knowledge description, retrieve the associated multimodal content from the cultural knowledge graph. The associated multimodal content includes associated text data and associated image data, and calculate the cross-modal semantic similarity between the associated multimodal content and the cultural knowledge description; If the cross-modal semantic similarity is lower than a preset threshold, perform context semantic correction processing on the cultural knowledge description through the large model to generate an optimized cultural knowledge description, and retrieve it again from the cultural knowledge graph to obtain optimized associated multimodal content; Based on the updated user preferences, perform weighted fusion processing on the optimized cultural knowledge description and the optimized associated multimodal content to output the final cultural knowledge description and the final associated multimodal content.

6. The method according to claim 1, wherein The calculation formula of the contrastive learning loss function is: Among them, L align is the result of the contrastive loss function, e i is the entity vector in the cultural knowledge graph, h LLM is the text hidden layer representation generated by the large model, τ is the temperature parameter, which is used to adjust the strictness of semantic alignment, and N is the number of negative samples.

7. The Bright Large Model Cultural Knowledge Generation and Retrieval System combined with a knowledge graph is characterized in that The system includes: A knowledge graph construction module for performing entity extraction, relationship modeling, and dynamic completion processing according to multimodal cultural data, and constructing a cultural knowledge graph in combination with a fuzzy time reasoning algorithm and a cross-modal alignment mechanism. The multimodal cultural data includes unstructured cultural texts, image data, and spatio-temporal metadata. The spatio-temporal metadata includes text spatio-temporal metadata and image moment metadata; A semantic alignment processing module for performing semantic alignment processing through user query statements and the cultural knowledge graph, in combination with a large model, to obtain a jointly represented cross-modal semantic alignment; A joint optimization processing module for performing multimodal retrieval-generation joint optimization based on the joint representation to obtain a cultural knowledge description and associated multimodal content.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images