Multi-source data fusion knowledge retrieval method and device, electronic equipment and medium

Through the knowledge retrieval method of multi-source data fusion, multimodal and multilingual data features are obtained and extracted, entity linking and knowledge fusion are performed, and a knowledge graph is constructed. This solves the problem of insufficient utilization of different languages ​​and multimodal data by traditional knowledge graphs and improves the accuracy of knowledge retrieval.

CN120670606AActive Publication Date: 2025-09-19PING AN TECH (SHENZHEN) CO LTD

Patent Information

Application Number
CN202510772092.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-10
Publication Date
2025-09-19
Estimated Expiration
2045-06-10

AI Technical Summary

Technical Problem

Traditional knowledge retrieval methods rely on knowledge graphs that only contain text data and lack the use of data in different language versions and text-related images, audio and other data, resulting in poor knowledge retrieval accuracy.

Method used

By acquiring target multi-source data, including target text, target image, target audio and at least two language data, feature extraction and knowledge extraction, entity linking, and fusing with predefined ontology knowledge data, a knowledge graph of multi-source data fusion is constructed for knowledge retrieval.

Benefits of technology

It improves the accuracy of knowledge retrieval, solves the problem of insufficient utilization of data in different language versions and multimodal data in traditional methods, realizes cross-modal and cross-language knowledge integration, and dynamically expands the knowledge content of the knowledge graph.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120670606A_ABST
    Figure CN120670606A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a knowledge retrieval method and device for multi-source data fusion, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing entity linking on a target image entity, a target audio entity and a language entity corresponding to multi-source data and a target text entity to obtain a target knowledge entity, and performing knowledge fusion on a relationship between the target knowledge entity and the target text entity and predefined ontology knowledge data to obtain a target knowledge entity; and performing knowledge retrieval on the retrieval indication information from a target knowledge graph constructed based on the fused knowledge data. According to the embodiment of the invention, the knowledge graph containing the multi-source data is constructed, and the knowledge retrieval is carried out from the knowledge graph, so that the knowledge retrieval can be carried out from the knowledge graph fusing the multi-modal data and the at least two kinds of language data, and the accuracy of the knowledge retrieval is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a knowledge retrieval method and device, electronic equipment, and medium for multi-source data fusion. Background Art

[0002] Traditional knowledge retrieval methods usually match the keywords of user queries with entity nodes in a pre-built knowledge graph and return the triple information associated with the node. In the Buddhist application scenario, when a user queries for Prajna, the Prajna and Buddhist concept entity nodes are retrieved by searching the pre-built knowledge graph, and the triple information that Prajna belongs to the Buddhist concept is returned. However, the knowledge graph that this method relies on only contains text data, which makes the knowledge graph lack the use of data in different language versions, as well as images, audio and other data related to the text. In addition, traditional knowledge graphs find it difficult to accurately understand the semantic differences between knowledge from different data sources, resulting in poor accuracy in knowledge retrieval. Therefore, how to improve the accuracy of knowledge retrieval has become an urgent problem to be solved. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a knowledge retrieval method and device, electronic device and medium for multi-source data fusion, aiming to improve the accuracy of knowledge retrieval.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a knowledge retrieval method for multi-source data fusion, the method comprising:

[0005] In response to a knowledge graph construction request, target multi-source data is obtained; wherein the target multi-source data includes target text, target image, target audio, and at least two language data;

[0006] Performing feature extraction on the target multi-source data to obtain multi-source data features;

[0007] Performing knowledge extraction on the multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities and target text entity relationships;

[0008] Perform entity linking on the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity;

[0009] Performing knowledge fusion on the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data;

[0010] Constructing a target knowledge graph based on the fused knowledge data;

[0011] Acquire retrieval indication information, and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

[0012] In some embodiments, entity linking the target image entity, the target audio entity, and the language entity with the target text entity to obtain a target knowledge entity includes:

[0013] Calculating similarity between the target image entity and the target text entity to obtain image-text entity similarity, and performing image-text entity linking between the target image entity and the target text entity based on the image-text entity similarity to obtain an image-text linked entity;

[0014] Calculating similarity between the target audio entity and the target text entity to obtain audio-text entity similarity, and performing audio-text entity linking between the target audio entity and the target text entity based on the audio-text entity similarity to obtain an audio-text linked entity;

[0015] Performing cross-language alignment processing on the language entity and the target text entity to obtain a language-text link entity;

[0016] The target knowledge entity is determined based on the image text link entity, the audio text link entity, and the language text link entity.

[0017] In some embodiments, the multi-source knowledge data further includes a multi-source data time entity and a multi-source data spatial location entity;

[0018] The determining the target knowledge entity based on the image text link entity, the audio text link entity, and the language text link entity includes:

[0019] Performing time entity embedding processing on the multi-source data time entity to obtain time entity features, performing space location entity embedding processing on the multi-source data space location entity to obtain space location entity features, and performing entity embedding processing on the target text entity to obtain text entity features;

[0020] Performing weighted fusion on the temporal entity feature, the spatial location entity feature, and the text entity feature to obtain a fused spatiotemporal entity feature;

[0021] Based on the fused spatiotemporal entity features, the multi-source data time entity, the multi-source data spatial position entity and the target text entity are entity fused to obtain a spatiotemporal text entity, and the target knowledge entity is determined based on the image text link entity, the audio text link entity, the language text link entity and the spatiotemporal text entity.

[0022] In some embodiments, the multi-source knowledge data further includes target text attributes; the ontology knowledge data includes ontology entities, ontology attributes, and ontology entity relationships;

[0023] The step of fusing the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data includes:

[0024] Perform entity alignment processing on the target knowledge entity and the ontology entity to obtain an aligned entity;

[0025] Performing attribute alignment processing on the target text attributes and the ontology attributes to obtain aligned attributes;

[0026] Performing relationship alignment processing on the target text entity relationship and the ontology entity relationship to obtain an aligned entity relationship;

[0027] The fused knowledge data is determined based on the aligned entities, the aligned attributes, and the aligned entity relationships.

[0028] In some embodiments, constructing a target knowledge graph based on the fused knowledge data includes:

[0029] Constructing an initial knowledge graph based on the fused knowledge data;

[0030] Performing data update detection on the target multi-source data to obtain updated multi-source data;

[0031] Determining updated multi-source knowledge data based on the updated multi-source data, and performing knowledge value scoring on the updated multi-source knowledge data to obtain an updated knowledge value score;

[0032] Performing entity updating on the target knowledge entity based on the updated knowledge value score to obtain an updated knowledge entity;

[0033] Performing entity relationship updates on the text entity relationship based on the updated knowledge entity to obtain an updated entity relationship;

[0034] updating the fused knowledge data based on the updated knowledge entities and the updated entity relationships to obtain updated fused knowledge data;

[0035] The initial knowledge graph is updated based on the updated fusion knowledge data to obtain the target knowledge graph.

[0036] In some embodiments, the multi-source data features include target text features, target image features, target audio features, and language features;

[0037] The extracting features of the target multi-source data to obtain multi-source data features includes:

[0038] Extracting text semantic features from the target text to obtain target text features;

[0039] Performing visual object detection on the target image to obtain an image detection object, and performing object semantic feature extraction on the image detection object to obtain the target image feature;

[0040] Converting the target audio into audio text, and performing audio text feature extraction on the audio text to obtain the target audio feature;

[0041] Language semantic features are extracted from the at least two language data to obtain the language features.

[0042] In some embodiments, after performing knowledge fusion on the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data, the method further includes:

[0043] Performing anomaly verification on the fused knowledge data to obtain anomaly verification data;

[0044] Performing integrity verification on the fused knowledge data to obtain integrity verification data;

[0045] Performing consistency check on the fused knowledge data to obtain consistency check data;

[0046] The fused knowledge data is optimized according to the abnormality check data, the integrity check data and the consistency check data to obtain optimized fused knowledge data, and the optimized fused knowledge data is used as the fused knowledge data.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a knowledge retrieval device for multi-source data fusion, the device comprising:

[0048] A multi-source data acquisition module is configured to acquire target multi-source data in response to a knowledge graph construction request; wherein the target multi-source data includes target text, target image, target audio, and data in at least two languages;

[0049] A feature extraction module is used to extract features from the target multi-source data to obtain multi-source data features;

[0050] A knowledge extraction module, configured to extract knowledge from the multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities, and target text entity relationships;

[0051] An entity linking module, configured to perform entity linking on the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity;

[0052] A knowledge fusion module is used to fuse the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data;

[0053] A knowledge graph construction module, configured to construct a target knowledge graph based on the fused knowledge data;

[0054] The knowledge retrieval module is used to obtain retrieval indication information and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

[0055] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0056] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of the above-mentioned first aspect.

[0057] The knowledge retrieval method and device, electronic device and medium for multi-source data fusion proposed in this application firstly acquire and extract the features of target multi-source data including target text, target image, target audio and at least two language data, so as to capture the semantic information of different modal data and different language data, and provide rich data support for the subsequent knowledge graph construction; secondly, the target image entity, target audio entity and language entity are entity-linked with the target text entity to obtain the target knowledge entity, which can associate multi-modal and multi-language entities, realize cross-modal and cross-language knowledge integration, and associate the target knowledge entity with the target text entity. The system fuses knowledge with predefined ontology knowledge data, which not only maintains the structured characteristics of knowledge but also can dynamically expand new knowledge. It helps to solve the problem that traditional knowledge graphs lack the use of data in different language versions, as well as images, audio and other data related to text, and traditional knowledge graphs are difficult to accurately understand the semantic differences of knowledge from different data sources, which helps to improve the accuracy of knowledge retrieval in the future; finally, knowledge retrieval of retrieval indication information from the target knowledge graph constructed based on the fused knowledge data can be performed through the knowledge graph that integrates multimodal and at least two language data, thereby improving the accuracy of knowledge retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the knowledge retrieval method for multi-source data fusion provided by an embodiment of the present application;

[0059] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0060] Figure 3 yes Figure 1 Flowchart of step S104 in FIG.

[0061] Figure 4 yes Figure 3 Flowchart of step S304 in FIG.

[0062] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0063] Figure 6 This is another flow chart of the knowledge retrieval method for multi-source data fusion provided by an embodiment of the present application;

[0064] Figure 7 yes Figure 1 Flowchart of step S106 in FIG.

[0065] Figure 8 This is a schematic diagram of the structure of a knowledge retrieval device for multi-source data fusion provided in an embodiment of the present application;

[0066] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0068] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0070] First, let’s analyze some of the terms used in this application:

[0071] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0072] The embodiments of the present application provide a knowledge retrieval method and device, electronic device, and medium for multi-source data fusion, aiming to improve the accuracy of knowledge retrieval.

[0073] The knowledge retrieval method and device, electronic device and medium for multi-source data fusion provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the knowledge retrieval method for multi-source data fusion in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0075] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The knowledge retrieval method for multi-source data fusion provided in the embodiment of the present application relates to the field of artificial intelligence technology. The knowledge retrieval method for multi-source data fusion provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of the knowledge retrieval method for multi-source data fusion, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] Figure 1 This is an optional flowchart of the knowledge retrieval method for multi-source data fusion provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0079] Step S101, in response to a knowledge graph construction request, obtain target multi-source data; wherein the target multi-source data includes target text, target image, target audio and at least two language data.

[0080] Step S102 : extracting features from the target multi-source data to obtain multi-source data features.

[0081] Step S103 , performing knowledge extraction on the multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities and target text entity relationships.

[0082] Step S104 , entity linking the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity.

[0083] Step S105 , performing knowledge fusion on the target knowledge entity and the target text entity relationship with the predefined ontology knowledge data to obtain fused knowledge data.

[0084] Step S106: construct a target knowledge graph based on the fused knowledge data.

[0085] Step S107: Obtain retrieval indication information, and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

[0086] In the steps S101 to S107 shown in the embodiment of the present application, first, by acquiring and extracting the features of the target multi-source data including the target text, target image, target audio and at least two language data, the semantic information of the different modal data and different language data can be captured, providing rich data support for the subsequent knowledge graph construction; secondly, the target image entity, target audio entity and language entity are entity-linked with the target text entity to obtain the target knowledge entity, which can associate the multimodal and multilingual entities, realize cross-modal and cross-lingual knowledge integration, and connect the target knowledge entity and the target text entity relationship with the predetermined The knowledge fusion of ontological knowledge data with semantics not only maintains the structured characteristics of knowledge, but also can dynamically expand new knowledge, which helps to solve the problem that traditional knowledge graphs lack the use of data in different language versions, as well as images, audio and other data related to text, and traditional knowledge graphs are difficult to accurately understand the semantic differences of knowledge from different data sources, which helps to improve the accuracy of knowledge retrieval in the future; finally, knowledge retrieval of retrieval indication information from the target knowledge graph constructed based on the fused knowledge data can be performed through the knowledge graph that integrates multimodal and at least two language data, thereby improving the accuracy of knowledge retrieval.

[0087] In step S101 of some embodiments, specifically, the target multi-source data may be a comprehensive data set including target text, target image, target audio, and at least two language data. The multi-source data may be a comprehensive data set in the field of Buddhism.

[0088] Specifically, the target text may be various types of Buddhist text materials, including but not limited to Buddhist classics, classic annotations, and papers of Buddhist scholars in the field of Buddhism.

[0089] Specifically, the target image may be visual materials in the field of Buddhism, including but not limited to images extracted from murals, sculptures, paintings, and paper Buddhist texts in the field of Buddhism.

[0090] Specifically, the target audio may be audio data in the field of Buddhism, including but not limited to audio of Buddhist lectures, recitation audio, and Buddhist music.

[0091] Specifically, language data refers to Buddhist data in various languages, including but not limited to Sanskrit, Pali, Tibetan, and Chinese. The language data can be different language data obtained from at least one modal data of target text, target image, and target audio.

[0092] Specifically, in the process of digital processing of paper-based Buddhist data, a high-resolution, high-speed document scanner is first used to quickly scan the paper-based Buddhist data to obtain complete text and image information. The image quality is improved through pre-processing techniques such as image enhancement, denoising and tilt correction. Optical character recognition technology, combined with the professional vocabulary library of Buddhist data and contextual semantic analysis, is used to accurately identify the text in the paper-based Buddhist data, especially special fonts and ancient characters, and convert them into editable text data. Then, the text after OCR recognition is structured according to the Buddhist knowledge ontology structure, and the chapters, paragraphs, sentences and key knowledge elements of the paper-based Buddhist data are extracted and imported into the Buddhist electronic database to realize the digital storage and management of paper-based Buddhist data.

[0093] Specifically, target multi-source data can be collected from major websites, digital libraries, social media and other platforms through web crawler technology and digital collection equipment (such as cameras, drones and microphones).

[0094] In this embodiment, target multi-source data is obtained by responding to a knowledge graph construction request; wherein, the target multi-source data includes target text, target image, target audio and at least two language data, thereby avoiding the situation where the knowledge system of the constructed knowledge graph is incomplete due to insufficient collection of non-text data such as data in different language versions, art paintings, sculptures, music, etc., and providing rich and multi-dimensional data support for subsequent knowledge graph construction.

[0095] See also Figure 2 In some embodiments, the multi-source data features include target text features, target image features, target audio features, and language features. Step S102 includes, but is not limited to, steps S201 to S204:

[0096] Step S201 : extracting text semantic features from the target text to obtain target text features.

[0097] Step S202 , performing visual object detection on the target image to obtain an image detection object, and performing object semantic feature extraction on the image detection object to obtain target image features.

[0098] Step S203: convert the target audio into audio text, and perform audio text feature extraction on the audio text to obtain target audio features.

[0099] Step S204: extracting language semantic features from at least two language data to obtain language features.

[0100] In step S201 of some embodiments, specifically, the multi-source data features may be a set of semantic features extracted from target text, target image, target audio, and at least two language data, and may specifically include target text features, target image features, target audio features, and language features.

[0101] Specifically, the target text feature refers to the text semantic feature vector extracted from the target text.

[0102] Specifically, the BERT model can be used to perform word embedding processing on the target text to convert the target text into a text vector and extract the semantic features of the text vector.

[0103] For example, in a Buddhist application scenario, word embedding processing can be performed on the Buddhist concept of "Prajnaparamita" to extract the semantic vector of the Buddhist concept.

[0104] In step S202 of some embodiments, specifically, the target image feature refers to a visual semantic feature vector extracted from the target image.

[0105] Specifically, a faster region-based convolutional neural network (such as Faster R-CNN) can be used to perform target detection calculations on the target image to extract key objects in the target image.

[0106] For example, in a mural depicting a Buddha scene, Faster R-CNN can be used to identify image detection objects such as portraits, thrones, and disciples.

[0107] Furthermore, the features of the image detection objects can be encoded through a deep residual network (such as ResNet) to extract the semantic feature vectors of the image detection objects.

[0108] For example, for a recognized portrait subject, the semantic features of the subject's hand gesture, as well as features such as the subject's clothing style and color, are extracted.

[0109] In step S203 of some embodiments, specifically, the target audio feature is an audio semantic feature vector extracted from the audio text.

[0110] Specifically, the target audio may be converted into audio text through automatic speech recognition (ASR) technology.

[0111] For example, for a Sanskrit audio recording of Buddhist recitations, the ASR technology can be used to convert the audio recording into the corresponding Sanskrit text.

[0112] Specifically, the BERT model can be used to perform word embedding processing on the audio text to extract the semantic feature vector of the audio text.

[0113] For example, extracting the semantic features of "Prajnaparamita" from Sanskrit text.

[0114] In step S204 of some embodiments, specifically, the semantic feature refers to a semantic feature vector extracted from different language data that can represent language content.

[0115] Specifically, by using cross-lingual models (such as XLM-RoBERTa, Cross-lingual LanguageModel-RoBERTa), we can compare and learn words in different languages ​​to narrow the cross-lingual distance of words with the same concept, map words in different languages ​​to the same vector space, and calculate the semantic similarity of words in different languages. Combined with attention alignment technology, we can align the same concept words in different languages ​​to generate an alignment matrix and its corresponding attention weights. Based on the alignment matrix and the attention weights, we can determine the expression form of the same concept words in different languages.

[0116] In this embodiment, by extracting language semantic features from data in at least two languages, the same semantic information in different languages ​​can be converted into unified semantic features, making it easier for the subsequently constructed knowledge graph to understand and process multilingual data.

[0117] Through steps S201 to S204, by extracting semantic features from multi-source data, key information in different modalities and different languages ​​can be comprehensively captured, which helps to comprehensively consider text semantic information, visual information, audio information and multilingual information when constructing the subsequent knowledge graph, thereby improving the understanding and processing capabilities of the subsequently constructed knowledge graph for different data sources.

[0118] In step S103 of some embodiments, specifically, multi-source knowledge data refers to structured knowledge data extracted from multi-source data features, and the multi-source knowledge data may include but is not limited to target text entities, target image entities, target audio entities, language entities, multi-source data time entities, multi-source data spatial position entities, target text entity relationships, target image entity relationships, target audio entity relationships, language entity relationships, and target text attributes, etc.

[0119] Specifically, named entity recognition technology can be used to extract target text entities, target image entities, target audio entities, language entities, etc. from multi-source data features.

[0120] For example, in the field of Buddhism, for the target text feature of "Prajnaparamita", text entities such as "Prajna" and "Paramita" can be identified; for the target image feature, target image entities such as "cross-legged sitting" and "dhyana seal" can be identified; for the target audio feature, rhythm entities such as "tranquility" and "solemnity" can be identified; for the language feature, Sanskrit entities such as "Prajna" and "dhyana" can be identified.

[0121] Furthermore, dependency syntactic analysis can be performed on multimodal entities and multilingual entities to extract entity relationships between entities. The entity relationships can be synonymous relationships, hierarchical relationships, causal relationships, guiding relationships, and inclusion relationships between entities.

[0122] For example, the entity relationship between text entities such as "Prajna" and "Paramita" is a guiding relationship in which "Prajna" guides "Paramita"; the entity relationship between the image entities of "cross-legged sitting" and "dhyana seal" is an accompanying relationship; the entity relationship between rhythm entities such as "tranquility" and "solemnity" is a collaborative relationship; the entity relationship between Sanskrit entities such as "Prajna" and "dhyana" is an inclusion relationship.

[0123] In this embodiment, by performing knowledge extraction on multi-source data features, entities and entity relationships in the multi-source data features can be extracted, further providing comprehensive and rich data support for the subsequent construction of structured knowledge data containing multiple modalities and multiple languages.

[0124] See also Figure 3 In some embodiments, step S104 includes but is not limited to steps S301 to S304:

[0125] Step S301 : Calculate the similarity between the target image entity and the target text entity to obtain the image-text entity similarity, and perform image-text entity linking between the target image entity and the target text entity based on the image-text entity similarity to obtain the image-text linked entity.

[0126] Step S302 : Calculate the similarity between the target audio entity and the target text entity to obtain the audio-text entity similarity, and perform audio-text entity linking between the target audio entity and the target text entity based on the audio-text entity similarity to obtain an audio-text linked entity.

[0127] Step S303 : performing cross-language alignment processing on the language entity and the target text entity to obtain a language-text link entity.

[0128] Step S304: determining a target knowledge entity based on the image text link entity, the audio text link entity, and the language text link entity.

[0129] In step S301 of some embodiments, specifically, the cosine similarity calculation is performed on the target image entity and the target text entity to determine the image-text entity similarity.

[0130] For example, the image entity of the "Awakening of Faith" gesture extracted from the Buddha mural is subjected to cosine similarity calculation with the text entity of "Giving Awakening", and the image-text entity similarity is 0.9.

[0131] Specifically, the image-text link entity refers to associating the target image entity with the target text entity to form a unified entity representation.

[0132] For example, if the image-text entity similarity is 0.9, which is higher than the set threshold (such as 0.85), the "Avrajnaparamita" gesture is linked to "Avrajnaparamita" to form an image-text linked entity "Avrajnaparamita-Avrajnaparamita".

[0133] In step S302 of some embodiments, specifically, cosine similarity calculation is performed on the target audio entity and the target text entity to determine the audio-text entity similarity.

[0134] For example, the cosine similarity calculation between the rhythmic entity “peaceful” extracted from the audio of Buddhist recitation and the text entity “compassion” was performed, and the audio-text entity similarity was 0.86.

[0135] Specifically, the audio-text linked entity refers to associating the target audio entity with the target text entity to form a unified entity representation.

[0136] For example, if the audio text entity similarity is 0.86, which is higher than the set threshold (such as 0.85), "peace" and "compassion" are linked to form the audio text link entity "peace-compassion".

[0137] In step S303 of some embodiments, specifically, linking the language text entity refers to associating the language entity with the target text entity to form a unified entity representation.

[0138] Specifically, different language entities and target text entities can be mapped to the same vector space by using a cross-lingual model (such as XLM-RoBERTa) to calculate the cosine similarity between language entities and text entities, and then perform entity linking on the language entities and target text entities based on the cosine similarity.

[0139] For example, the cosine similarity calculation is performed on the Sanskrit entity of "everything is empty" and the Chinese entity of "all dharmas are empty", and a language text entity similarity of 0.95 is obtained. The language text entity similarity of 0.95 is higher than the set threshold (such as 0.85), so the Sanskrit entity of "everything is empty" and the Chinese entity of "all dharmas are empty" are linked to form a language text linked entity "everything is empty Sanskrit-all dharmas are empty Chinese".

[0140] See also Figure 4 In some embodiments, the multi-source knowledge data further includes a multi-source data time entity and a multi-source data space location entity. Step S304 includes but is not limited to steps S401 to S403:

[0141] Step S401, perform time entity embedding processing on the time entity of multi-source data to obtain time entity features, perform spatial location entity embedding processing on the spatial location entity of multi-source data to obtain spatial location entity features, and perform entity embedding processing on the target text entity to obtain text entity features.

[0142] Step S402 : performing weighted fusion on the temporal entity features, spatial location entity features, and text entity features to obtain fused spatiotemporal entity features.

[0143] Step S403, based on the fusion of spatiotemporal entity features, the multi-source data time entity, the multi-source data spatial location entity and the target text entity are entity-fused to obtain the spatiotemporal text entity, and the target knowledge entity is determined based on the image text link entity, the audio text link entity, the language text link entity and the spatiotemporal text entity.

[0144] In step S401 of some embodiments, specifically, the multi-source data time entity may be the creation time of a Buddhist work, the construction time of a Buddhist portrait, etc.

[0145] Specifically, the multi-source data spatial location entity may be the unearthed site of Buddhist statues, the performance venue of Buddhist music, etc.

[0146] Specifically, the BERT model can be used to perform entity embedding processing on multi-source data time entities and target text entities to convert time information and text information into feature vectors.

[0147] For example, for historical events in Buddhist literature, such as "A Buddhist figure attained enlightenment in Bodh Gaya in 528 BC", the time entity "528 BC" is converted into a time entity feature vector through word embedding processing.

[0148] For example, "Buddha" in the target text entity description is converted into a text entity feature vector through word embedding processing.

[0149] Furthermore, the spatial location entities of multi-source data can be encoded through a geographic information system (GIS) to convert geographic location information into a feature vector.

[0150] For example, for the geographical location of Buddhist murals, obtain geographic coordinate entities (such as longitude and latitude entities), and use GIS to convert the longitude and latitude coordinate entities into low-dimensional vectors to convert geographic location information into spatial location entity features.

[0151] In step S402 of some embodiments, specifically, fusing spatiotemporal entity features refers to an entity feature vector that integrates temporal entity features, spatial location entity features, and text entity features.

[0152] Specifically, the fused spatiotemporal entity feature can be obtained by obtaining the time weight of the time entity feature, the spatial weight of the spatial location entity feature, and the text weight of the text entity feature, and adding the product of the time entity feature and the time weight, the product of the spatial location entity feature and the spatial weight, and the product of the entity feature and the text weight.

[0153] For example, the features of “528 BC”, “geographical coordinates of Buddhist murals” and “Buddha” are weightedly fused to obtain a comprehensive fused spatiotemporal entity feature.

[0154] In step S403 of some embodiments, specifically, the multi-source data time entity, the multi-source data spatial location entity, and the target text entity can be fused through graph convolutional networks (GCN).

[0155] For example, by integrating the entities “528 BC”, “geographical coordinates of Buddhist murals” and “Buddha”, we can obtain a unified spatiotemporal text entity “Buddha attained enlightenment in Bodh Gaya in 528 BC”.

[0156] Furthermore, by combining image text link entities (such as "Avrahamuddin - the one who gives fearlessness"), audio text link entities (such as "peace - compassion") and language text link entities (such as "all is empty in Sanskrit - all dharmas are empty in Chinese"), the target knowledge entity containing the above link entities can be determined.

[0157] Provide comprehensive multimodal and multilingual knowledge for the construction of knowledge graphs.

[0158] Through steps S401 to S403, by embedding multi-source data time entities and multi-source data spatial location entities, and integrating them with text entities, a comprehensive knowledge representation containing time, space and semantic information can be constructed, which can enrich the content of the knowledge graph and help the subsequently constructed knowledge graph to more accurately represent the time and place of historical events, so as to support more complex historical and cultural research.

[0159] Through steps S301 to S304, through entity linking of multimodal and multilingual data, it is possible to solve the problem that the knowledge graph constructed subsequently lacks the use of data in different language versions, as well as images, audio and other data related to text. It also improves the ability of the knowledge graph constructed subsequently to understand and process different data sources, so as to better support knowledge retrieval in a multilingual environment, thereby improving the accuracy of subsequent knowledge retrieval.

[0160] See also Figure 5 In some embodiments, the multi-source knowledge data further includes target text attributes; the ontology knowledge data includes ontology entities, ontology attributes, and ontology entity relationships. Step S105 includes but is not limited to steps S501 to S504:

[0161] Step S501: align the target knowledge entity with the ontology entity to obtain an aligned entity.

[0162] Step S502 : aligning the target text attributes with the ontology attributes to obtain aligned attributes.

[0163] Step S503 , performing relationship alignment processing on the target text entity relationship and the ontology entity relationship to obtain aligned entity relationship.

[0164] Step S504: determining fused knowledge data based on the aligned entities, aligned attributes, and aligned entity relationships.

[0165] In step S501 of some embodiments, specifically, ontology knowledge data refers to structured knowledge data in the target domain, and may include ontology entities, which refer to basic concepts or objects in the target domain knowledge system.

[0166] For example, in the field of Buddhism, ontological entities can be "meditation" and "prajna", etc.

[0167] Specifically, the knowledge fusion of target knowledge entities and ontology entities can be achieved through a pre-built Buddhist ontology model. The Buddhist ontology model consists of a core concept layer, an attribute relationship layer, and an instance data layer. The core concept layer defines ontology entities (such as Buddhist figures, Buddhist concepts, and Buddhist texts); the attribute relationship layer defines the ontological attributes of ontology entities (such as the Buddhist sects to which Buddhist figures belong) and entity relationships (such as the spatiotemporal relationship between Buddhist sculptures and Buddhist texts); and the instance data layer is used to instantiate and concretize specific multi-source data with the ontology knowledge in the ontology model. That is, the instance data layer maps the entities, attributes, and relationships of multi-source data to the ontology knowledge to form specific knowledge fusion instances.

[0168] Specifically, ontology attributes can be used to describe semantic annotations of ontology entities using the Web Ontology Language (OWL) to define entity relationships or attributes between ontology entities.

[0169] Specifically, the semantic similarity between the target knowledge entity and the ontology entity can be calculated through the instance data layer, and the target knowledge entity and the ontology entity can be aligned based on the semantic similarity.

[0170] For example, the target knowledge entity "wisdom" is a Buddhist concept entity after multi-source data fusion. There is also an entity "Prajna" in the ontology knowledge data. Although their names are different, through semantic analysis, it is found that "Prajna" is usually interpreted as "wisdom" in Buddhism. It can be determined that the two entities are different manifestations of the same concept. Through entity alignment processing, "Prajna" and "wisdom" are aligned, that is, mapped to the same concept, thereby obtaining the aligned entity "Prajna-wisdom".

[0171] In this embodiment, by aligning the target knowledge entity with the ontology entity, entities from different data sources can be integrated into a unified knowledge system through ontology mapping, which can effectively eliminate semantic differences and concept conflicts between different data sources.

[0172] In step S502 of some embodiments, specifically, the target text attribute refers to a structural feature used to describe the specific meaning of the target text entity.

[0173] Specifically, ontology knowledge data also includes ontology attributes, which refer to structured features used to describe the specific meaning of ontology entities in a predefined ontology knowledge system.

[0174] Specifically, alignment attributes refer to structural features that integrate the target text attributes and ontology attributes.

[0175] Specifically, the cosine similarity between the target text attribute and the ontology attribute may be calculated through the instance data layer, and the target text attribute and the ontology attribute may be aligned based on the cosine similarity.

[0176] For example, the target text attribute of the target text entity "Alaya consciousness" is "having the three meanings of being able to store, being stored, and holding the stored", and the ontological attributes are "being able to store", "being stored", and "holding the stored" respectively. Through similarity calculation, it is ensured that being able to store, being stored, and holding the stored are accurately mapped to the ontological model to obtain the aligned attributes "being able to store", "being stored", and "holding the stored".

[0177] In step S503 of some embodiments, specifically, ontology entity relationships refer to structured relationships that define and describe logical or semantic connections between ontology entities in an ontology knowledge system.

[0178] Specifically, the relational semantic similarity between the target text entity relationship and the ontology entity relationship can be calculated through the instance data layer, and the target text entity relationship and the ontology entity relationship can be aligned based on the relational semantic similarity.

[0179] For example, the target text entity relationship is the entity relationship "Alaya consciousness is the root of ignorance" extracted from Buddhist literature and "ignorance". The ontological entity relationship has a causal relationship of "ignorance leads to the attachment of Alaya consciousness". Through entity relationship alignment, it can be confirmed that the semantics of the two are related and logically consistent. Then the target text entity relationship "Alaya consciousness is the root of ignorance" is aligned with the ontological entity relationship "ignorance leads to the attachment of Alaya consciousness" to obtain the aligned entity relationship "causal relationship".

[0180] In step S504 of some embodiments, specifically, the fused knowledge data refers to knowledge system data including aligned entities, aligned attributes, and aligned entity relationships.

[0181] For example, the fused knowledge data can be represented as: entity "Alaya consciousness", attributes "capable of storing, stored, and holding stored", and the inclusion relationship "Alaya consciousness has the three meanings of capable of storing, stored, and holding stored".

[0182] In this embodiment, the fused knowledge data is determined based on aligned entities, aligned attributes, and aligned entity relationships, which can effectively integrate knowledge elements in multi-source data with elements in a predefined ontology knowledge system to generate a structured and systematic knowledge system.

[0183] Through steps S501 to S504, by performing entity alignment, attribute alignment, and relationship alignment on multi-source data and ontology knowledge data respectively, the knowledge elements in the multi-source data can be effectively associated with the elements in the predefined ontology knowledge system, making the data after knowledge fusion more accurate and complete, and facilitating the subsequent integration of the aligned elements into a unified knowledge system, forming a structured knowledge graph that eliminates semantic differences and conceptual conflicts between data from different sources, and helping to improve the accuracy of subsequent knowledge retrieval.

[0184] In an optional embodiment of the present application, when defining ontology knowledge data, the quality of the ontology knowledge data is ensured through annotation guidelines, and the annotated data is audited for format compliance, entity integrity and basic logic verification in combination with cross-group review to ensure the quality of the ontology knowledge data.

[0185] For example, in the field of Buddhism, the data that need to be annotated include Buddhist ontological entities (such as Buddhist concepts, figures, and scriptures), Buddhist ontological relationships (such as master-disciple, interpretation), and the annotation must follow the standard rules, including the label system and boundary rules. For example, the "Four Noble Truths" must be fully labeled with suffering, origin, extinction, and path; combined with the cross-audit within the group, the format compliance, entity integrity and basic logic of the annotated data are verified, such as checking whether the "Twelve Links of Dependent Origination" are fully annotated, and whether the time when Buddhist figures discovered Buddhist scriptures is correct. In addition, the language level review focuses on the accuracy of the annotated terms, grammatical standardization and disambiguation processing, such as Sanskrit. It is uniformly translated as "emptiness" and so on.

[0186] See also Figure 6 In some embodiments, the knowledge retrieval method for multi-source data fusion further includes but is not limited to steps S601 to S604:

[0187] Step S601: perform anomaly check on the fused knowledge data to obtain anomaly check data.

[0188] Step S602: perform integrity check on the fused knowledge data to obtain integrity check data.

[0189] Step S603: Perform consistency check on the fused knowledge data to obtain consistency check data.

[0190] Step S604 , optimizing the fused knowledge data according to the abnormality check data, the integrity check data and the consistency check data to obtain optimized fused knowledge data, and using the optimized fused knowledge data as the fused knowledge data.

[0191] In step S601 of some embodiments, specifically, the preset knowledge verification rules can be learned through the LSTM model to perform anomaly verification on the fused knowledge data.

[0192] For example, in the field of Buddhism, the LSTM model can be used to perform anomaly verification on the "Four Noble Truths" in the fused knowledge data to determine whether the "Four Noble Truths" are incorrectly written as "Four Holy Emperors", or to perform anomaly verification on the birth and death years of Buddhist figures in 702 AD to determine that the birth and death years of the Buddhist figures are between 602 AD and 664 AD. In addition, cross-validation can be used to verify the source of the cited Buddhist data, if there is a difference between the content recorded in the cited classics shown in the Buddhist data and the Buddhist data.

[0193] In this embodiment, by performing anomaly verification on the fused knowledge data, errors in the fused knowledge data can be discovered and corrected in a timely manner, thereby improving the accuracy and reliability of the data.

[0194] In step S602 of some embodiments, specifically, the integrity of the fused knowledge data may be verified by methods such as field fill rate monitoring, association integrity checking, and cross-source completion.

[0195] For example, in the fusion of Buddhist knowledge, the field fill rate can be used to monitor the fill rate of core Buddhist fields (such as concepts, figures, and classics) to ensure that the missing rate does not exceed 2%. The knowledge system path query can also be used to verify whether the association between entities is complete, such as whether the classic school is correctly associated with the Buddhist figure. Further, when it is found that the fused knowledge data is missing, the missing data can be supplemented through cross-source completion methods, such as calling the Buddhist classics API to supplement Pali literature.

[0196] In step S603 of some embodiments, specifically, consistency verification can be performed on the fused knowledge data through methods such as spatiotemporal logic verification, cross-version comparison, and relationship conflict detection.

[0197] Specifically, we can identify the consistency of time logic in the fused knowledge data by constructing an event timeline, and analyze whether the fused knowledge data of different language versions are consistent through cross-version comparison, and further combine relationship conflicts to detect whether there are relationship conflicts in the fused knowledge data.

[0198] For example, we can construct a timeline of events to ensure that the year of writing of Buddhist scriptures is no earlier than the year of death of the corresponding Buddhist figures. By comparing the Sanskrit, Chinese and Tibetan versions of the same Buddhist scriptures, we can ensure that the similarity of the textual expressions exceeds 0.75, and further test whether the core concepts of Buddhism of different sects are mistakenly marked as completely consistent.

[0199] In this embodiment, by performing consistency check on the fused knowledge data, the logical consistency of the fused knowledge data can be ensured, and confusion in the knowledge system caused by logical errors or relationship conflicts can be avoided.

[0200] In step S604 of some embodiments, specifically, optimizing the fused knowledge data refers to fused knowledge data after correcting abnormal data, supplementing missing data, and adjusting inconsistent data.

[0201] For example, incorrectly marked terms or year contradictions in the integrated knowledge data are corrected, missing Pali documents are supplemented based on the integrity verification data, and incorrect relationship annotations or logical conflicts are adjusted based on the consistency verification data.

[0202] Through steps S601 to S604, through anomaly verification, integrity verification and consistency verification, errors, omissions and inconsistencies in the fused knowledge data can be fully identified and corrected, ensuring the accuracy and integrity of the data, which will help to subsequently build a more accurate knowledge graph to improve the accuracy of knowledge retrieval.

[0203] See also Figure 7 In some embodiments, step S106 includes but is not limited to steps S701 to S707:

[0204] Step S701: construct an initial knowledge graph based on the fused knowledge data.

[0205] Step S702: Perform data update detection on the target multi-source data to obtain updated multi-source data.

[0206] Step S703 : determining updated multi-source knowledge data based on the updated multi-source data, and performing knowledge value scoring on the updated multi-source knowledge data to obtain an updated knowledge value score.

[0207] Step S704: update the target knowledge entity based on the updated knowledge value score to obtain an updated knowledge entity.

[0208] Step S705 : updating the text entity relationship based on the updated knowledge entity to obtain an updated entity relationship.

[0209] Step S706: updating the fused knowledge data based on the updated knowledge entities and updated entity relationships to obtain updated fused knowledge data.

[0210] Step S707: Update the initial knowledge graph based on the updated fusion knowledge data to obtain the target knowledge graph.

[0211] In step S701 of some embodiments, specifically, entities in the fused knowledge data are used as nodes, attributes are used as labels of nodes, and entity relationships are used as edges connecting two nodes, so as to organize all fused knowledge data into a graph structure to form an initial knowledge graph.

[0212] In this embodiment, an initial knowledge graph is constructed based on the fused knowledge data, which can present complex knowledge data in an intuitive graphical form, providing a data basis for subsequent knowledge updating and knowledge retrieval.

[0213] In step S702 of some embodiments, specifically, whether the multi-source data has been updated can be determined by detecting whether the timestamp of the target text has changed, or detecting whether a new image has been added to the image database, or detecting whether the timestamp of the audio file has changed. If a change is detected in the text, image or audio data, the changed data is marked as updated multi-source data.

[0214] In this embodiment, by performing data update detection on the target multi-source data, changes in the multi-source data can be discovered in a timely manner, ensuring that the constructed knowledge graph can reflect the latest knowledge information.

[0215] In step S703 of some embodiments, specifically, the knowledge value score is used to evaluate the importance and contribution of updating multi-source data to the knowledge graph.

[0216] Specifically, the importance and contribution of updated multi-source data to the knowledge graph can be assessed through a knowledge value scoring model. This model includes multiple evaluation dimensions, such as innovation, authority, and completeness. Innovation is used to assess whether the knowledge data provides new perspectives or discoveries; authority is used to assess the credibility of the knowledge source; and completeness is used to assess whether the knowledge data comprehensively covers the relevant topic.

[0217] For example, for research papers on Buddhist scriptures, the updated multi-source knowledge data is scored by analyzing the citation rate and professional terminology ratio of the research paper. If the citation rate is 60% and the professional terminology density exceeds the set threshold (such as 70%), the knowledge value score obtained by the research paper can be 65 points.

[0218] In this embodiment, updating multi-source knowledge data is determined based on updating multi-source data, and the updated multi-source knowledge data is scored for knowledge value, which can help identify and prioritize updated data that makes important contributions to the knowledge graph, ensure that the knowledge graph prioritizes high-value updated data, and improve the quality and practicality of the knowledge graph.

[0219] In step S704 of some embodiments, specifically, high-value update data can be screened based on the updated knowledge value score to update the target knowledge entity in the knowledge graph. For example, update data with a knowledge value score exceeding a certain threshold (such as 0.8) is selected. For example, if a newly discovered image of a Buddha statue shows new gesture features, it may be necessary to update the attributes of the "Buddha statue" entity and add a new gesture description.

[0220] In this embodiment, by performing data update detection on the target multi-source data, it is possible to ensure that the entities in the knowledge graph always remain in the latest and most accurate state, which helps to improve the accuracy and reliability of the knowledge graph.

[0221] In step S705 of some embodiments, specifically, updating entity relationships refers to the relationships between updated entities.

[0222] For example, after detecting the update of the "Buddhist Figure" entity through the knowledge graph reasoning engine, the entity relationship related to "gesture" can be identified, and the entity relationship can be adjusted according to the updated entity information. If the updated gesture is related to the "Namaste Seal", the relationship between "Buddhist Figure" and "Namaste Seal" will be used as the updated entity relationship.

[0223] In step S706 of some embodiments, specifically, the updated knowledge entities (such as Buddhist figures) and updated entity relationships (such as the relationship between "Buddhist figures" and "Namaste Seal") can be integrated into the fused knowledge data to replace the original fused knowledge data.

[0224] In step S707 of some embodiments, specifically, graph embedding technology can be used to convert the updated knowledge nodes and the nodes in the initial knowledge graph into low-dimensional vector representations, and the cosine similarity between the vectors can be calculated to identify the semantic relationship between the updated knowledge nodes and the existing entity nodes. The relationship between the updated knowledge nodes and the existing nodes is expanded through a path sorting algorithm to generate updated entity relationship edges, and the updated entity relationship edges are added to the initial knowledge graph to form a target knowledge graph.

[0225] For example, for updating the knowledge node "Theravada Buddhist Thought", the updated entity relationship edge between "Theravada Buddhist Thought" and "Consciousness-Only School" is determined and marked as "related relationship", and the "difference relationship" edge between "Theravada Buddhist Thought" and "Northern Buddhist Thought" is determined through the path sorting algorithm, and the "difference relationship" edge connects the "Theravada Buddhist Thought" node and the "Consciousness-Only School" node.

[0226] In an optional embodiment of the present application, when updating the graph, OWL technology is also used to perform logical verification on the updated knowledge relations in the updated fusion knowledge data.

[0227] For example, if the updated knowledge relationship has a logical conflict with the existing axioms, an updated knowledge relationship branch is created through OWL technology to store the updated knowledge relationship in the branch, while keeping the existing axioms unchanged to ensure the logical consistency of the constructed knowledge graph; if the updated knowledge relationship does not conflict with the existing axioms, an RDF triple update transaction is generated to add the updated entity relationship to the initial knowledge graph.

[0228] In this embodiment, the initial knowledge graph is updated based on the updated fusion knowledge data, which can dynamically expand the knowledge graph according to the updated knowledge entities to ensure the integrity and timeliness of the final generated knowledge graph. It can also be combined with the logical verification of the updated entity relationship to resolve the conflict between the updated knowledge entities and the existing knowledge, thereby ensuring the logical consistency and integrity of the final constructed knowledge graph.

[0229] Through steps S701 to S707, by constructing the initial knowledge graph and continuously monitoring the update of the target multi-source data, the entities and their relationships in the initial knowledge graph are accurately updated based on the value scores of the updated multi-source knowledge data to ensure that the knowledge graph always remains in the latest and most accurate state, and the knowledge graph is dynamically expanded according to the updated knowledge entities to ensure the integrity and timeliness of the knowledge graph finally generated.

[0230] In step S107 of some embodiments, specifically, the search instruction information refers to a query request input by a user, which usually appears in the form of keywords, phrases, or natural language questions and is used to specify the target content to be retrieved.

[0231] For example, in the Buddhist knowledge graph, a user may enter "the definition of Alaya consciousness" as search indication information.

[0232] Specifically, the target retrieval knowledge information refers to the knowledge content related to the retrieval instruction information.

[0233] Specifically, word embedding processing is performed on the retrieval indication information to extract the key semantic feature vector of the retrieval indication information, and knowledge retrieval is performed in the target knowledge graph to quickly locate the target retrieval node corresponding to the key semantics, and all other nodes associated with the target retrieval node and the relationship between the nodes are used as target retrieval knowledge information.

[0234] For example, if a user inputs the definition of "Alaya consciousness", by searching in the target knowledge graph, the node related to "Alaya consciousness" can be located and the attribute information of the node can be extracted, such as the three meanings of "the ability to store, what is stored, and the holder of the storage".

[0235] In this embodiment, retrieval indication information is obtained, and knowledge retrieval is performed on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information, which can quickly and accurately respond to user query requests and provide knowledge content that is highly relevant to the retrieval indication information from the constructed target knowledge graph. Moreover, since the target knowledge graph is a knowledge system constructed by integrating data of different language versions and text-related images, audio and other data, the accuracy of knowledge retrieval is significantly improved.

[0236] The embodiment of the present application firstly obtains and extracts features of target multi-source data including target text, target image, target audio and at least two language data, thereby capturing semantic information of different modal data and different language data, and providing rich data support for subsequent knowledge graph construction; secondly, the target image entity, target audio entity and language entity are entity-linked with the target text entity to obtain the target knowledge entity, which can associate multimodal and multilingual entities, realize cross-modal and cross-lingual knowledge integration, and fuse the target knowledge entity and target text entity relationship with predefined ontology knowledge data, which not only maintains the structured characteristics of knowledge but also dynamically expands new knowledge, helps to solve the problem that traditional knowledge graphs lack the use of different language versions of data and text-related images, audio and other data, and traditional knowledge graphs have difficulty in accurately understanding the semantic differences of knowledge from different data sources, which helps to improve the accuracy of knowledge retrieval in the future; finally, knowledge retrieval is performed on the retrieval indication information from the target knowledge graph constructed based on the fused knowledge data, and knowledge retrieval can be performed in the knowledge graph that fuses multimodal and at least two language data, thereby improving the accuracy of knowledge retrieval.

[0237] See also Figure 8 The present application also provides a multi-source data fusion knowledge retrieval device, which can implement the multi-source data fusion knowledge retrieval method described above. The device includes:

[0238] A multi-source data acquisition module is used to obtain target multi-source data in response to a knowledge graph construction request; wherein the target multi-source data includes target text, target image, target audio, and at least two language data;

[0239] Feature extraction module, used to extract features from target multi-source data to obtain multi-source data features;

[0240] A knowledge extraction module is used to extract knowledge from multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities and target text entity relationships;

[0241] An entity linking module is used to perform entity linking on the target image entity, target audio entity, and language entity with the target text entity to obtain the target knowledge entity;

[0242] The knowledge fusion module is used to fuse the target knowledge entity and target text entity relationship with the predefined ontology knowledge data to obtain fused knowledge data;

[0243] The knowledge graph construction module is used to construct the target knowledge graph based on the fused knowledge data;

[0244] The knowledge retrieval module is used to obtain retrieval indication information and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

[0245] The specific implementation of the multi-source data fusion knowledge retrieval device is basically the same as the specific embodiment of the multi-source data fusion knowledge retrieval method described above, and will not be repeated here.

[0246] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described knowledge retrieval method for multi-source data fusion. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0247] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0248] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0249] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the processing system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the knowledge retrieval method for multi-source data fusion in the embodiments of this application.

[0250] Input / output interface 903, used to implement information input and output;

[0251] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0252] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0253] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0254] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned knowledge retrieval method for multi-source data fusion.

[0255] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0256] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0257] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0258] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0259] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0260] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0261] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0262] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0263] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0264] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0266] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A knowledge retrieval method for multi-source data fusion, characterized in that: The method comprises: In response to a knowledge graph construction request, target multi-source data is obtained; wherein the target multi-source data includes target text, target image, target audio, and at least two language data; Performing feature extraction on the target multi-source data to obtain multi-source data features; Performing knowledge extraction on the multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities and target text entity relationships; Perform entity linking on the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity; Performing knowledge fusion on the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data; Constructing a target knowledge graph based on the fused knowledge data; Acquire retrieval indication information, and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

2. The method according to claim 1, characterized in that The step of entity linking the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity includes: Calculating similarity between the target image entity and the target text entity to obtain image-text entity similarity, and performing image-text entity linking between the target image entity and the target text entity based on the image-text entity similarity to obtain an image-text linked entity; Calculating similarity between the target audio entity and the target text entity to obtain audio-text entity similarity, and performing audio-text entity linking between the target audio entity and the target text entity based on the audio-text entity similarity to obtain an audio-text linked entity; Performing cross-language alignment processing on the language entity and the target text entity to obtain a language-text link entity; The target knowledge entity is determined based on the image text link entity, the audio text link entity, and the language text link entity.

3. The method according to claim 2, characterized in that The multi-source knowledge data also includes a multi-source data time entity and a multi-source data space location entity; The determining the target knowledge entity based on the image text link entity, the audio text link entity, and the language text link entity includes: Performing time entity embedding processing on the multi-source data time entity to obtain time entity features, performing space location entity embedding processing on the multi-source data space location entity to obtain space location entity features, and performing entity embedding processing on the target text entity to obtain text entity features; Performing weighted fusion on the temporal entity feature, the spatial location entity feature, and the text entity feature to obtain a fused spatiotemporal entity feature; Based on the fused spatiotemporal entity features, the multi-source data time entity, the multi-source data spatial position entity and the target text entity are entity fused to obtain a spatiotemporal text entity, and the target knowledge entity is determined based on the image text link entity, the audio text link entity, the language text link entity and the spatiotemporal text entity.

4. The method according to claim 1, wherein The multi-source knowledge data also includes target text attributes; the ontology knowledge data includes ontology entities, ontology attributes and ontology entity relationships; The step of fusing the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data includes: Perform entity alignment processing on the target knowledge entity and the ontology entity to obtain an aligned entity; Performing attribute alignment processing on the target text attributes and the ontology attributes to obtain aligned attributes; Performing relationship alignment processing on the target text entity relationship and the ontology entity relationship to obtain an aligned entity relationship; The fused knowledge data is determined based on the aligned entities, the aligned attributes, and the aligned entity relationships.

5. The method according to claim 1, wherein The constructing of a target knowledge graph based on the fused knowledge data includes: Constructing an initial knowledge graph based on the fused knowledge data; Performing data update detection on the target multi-source data to obtain updated multi-source data; Determining updated multi-source knowledge data based on the updated multi-source data, and performing knowledge value scoring on the updated multi-source knowledge data to obtain an updated knowledge value score; Performing entity updating on the target knowledge entity based on the updated knowledge value score to obtain an updated knowledge entity; Performing entity relationship updates on the text entity relationship based on the updated knowledge entity to obtain an updated entity relationship; updating the fused knowledge data based on the updated knowledge entities and the updated entity relationships to obtain updated fused knowledge data; The initial knowledge graph is updated based on the updated fusion knowledge data to obtain the target knowledge graph.

6. The method according to claim 1, characterized in that The multi-source data features include target text features, target image features, target audio features and language features; The extracting features of the target multi-source data to obtain multi-source data features includes: Extracting text semantic features from the target text to obtain target text features; Performing visual object detection on the target image to obtain an image detection object, and performing object semantic feature extraction on the image detection object to obtain the target image feature; Converting the target audio into audio text, and performing audio text feature extraction on the audio text to obtain the target audio feature; Language semantic features are extracted from the at least two language data to obtain the language features.

7. The method according to any one of claims 1 to 6, characterized in that After fusing the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data, the method further includes: Performing anomaly verification on the fused knowledge data to obtain anomaly verification data; Performing integrity verification on the fused knowledge data to obtain integrity verification data; Performing consistency check on the fused knowledge data to obtain consistency check data; The fused knowledge data is optimized according to the abnormality check data, the integrity check data and the consistency check data to obtain optimized fused knowledge data, and the optimized fused knowledge data is used as the fused knowledge data.

8. A knowledge retrieval device for multi-source data fusion, characterized in that: The device comprises: A multi-source data acquisition module is configured to acquire target multi-source data in response to a knowledge graph construction request; wherein the target multi-source data includes target text, target image, target audio, and data in at least two languages; A feature extraction module is used to extract features from the target multi-source data to obtain multi-source data features; A knowledge extraction module, configured to extract knowledge from the multi-source data features to obtain multi-source knowledge data; wherein the multi-source knowledge data includes target text entities, target image entities, target audio entities, language entities, and target text entity relationships; An entity linking module, configured to perform entity linking on the target image entity, the target audio entity, the language entity, and the target text entity to obtain a target knowledge entity; A knowledge fusion module is used to fuse the target knowledge entity and the target text entity relationship with predefined ontology knowledge data to obtain fused knowledge data; A knowledge graph construction module, configured to construct a target knowledge graph based on the fused knowledge data; The knowledge retrieval module is used to obtain retrieval indication information and perform knowledge retrieval on the retrieval indication information from the target knowledge graph to obtain target retrieval knowledge information.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the knowledge retrieval method for multi-source data fusion according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the knowledge retrieval method for multi-source data fusion according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Information search method and device based on large model, electronic equipment and storage medium

    CN119917672A

  • Knowledge-derived search suggestion

    US20220253477A1

Cited By

  • Ancient book text labeling method and system based on graph neural network

    CN121457439A