Media field-based multi-modal knowledge graph construction method and information retrieval method

Abstract

The application relates to the technical field of news media, and discloses a multi-modal knowledge graph construction method and an information retrieval method based on a media field, aiming to solve the problem that the retrieval effect and user experience of a retrieval engine based on a knowledge graph construction are poor, and the scheme mainly comprises the following steps: acquiring and filtering news data; performing sentence processing on news texts corresponding to the news data, and extracting entity words and domain event triples in each sentence; training a CLIP fine-tuning model, performing matching prediction on corresponding entities and events according to the CLIP fine-tuning model, and expanding a multi-modal attribute set of the entities and events according to the matching prediction result; constructing a knowledge base according to the extracted data, performing anaphora resolution and entity disambiguation processing on the data in the knowledge base, constructing an ontology of domain entities and domain events, and writing the ontology into a graph database to obtain a multi-modal knowledge graph of the media field. The application improves the retrieval effect and user experience of news information.

Description

Technical Field

[0001] This invention relates to the field of news media technology, and more specifically to a method for constructing a multimodal knowledge graph and an information retrieval method based on the media field. Background Technology

[0002] With the rapid development of the internet, the media industry is also expanding its influence, accompanied by an increasing amount of online news data. As a crucial way for users to obtain information, search engines often fail to effectively present the information users want through simple text searches, directly impacting the user experience. Therefore, improving the accuracy and diversity of search results has become an urgent problem to solve.

[0003] A knowledge graph (KG) is essentially a large-scale semantic network with entities (concepts) as nodes and various semantic relationships between them as edges. However, most existing knowledge graphs are represented in the form of text (symbols), which weakens the machine's ability to describe and understand the real world. Although existing search engines are gradually introducing knowledge graphs as knowledge reserves to improve search performance, there are few cases of applying multimodal data. In the media field, a large amount of news data is equipped with rich multimodal attributes such as images and videos to enhance user understanding. However, these multimodal attributes are often not fully utilized in downstream applications such as retrieval and recommendation, resulting in poor retrieval performance and user experience. Summary of the Invention

[0004] This invention aims to address the problem of poor search performance and user experience in existing knowledge graph-based search engines, and proposes a multimodal knowledge graph construction method and information retrieval method based on the media domain.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] Firstly, a method for constructing a multimodal knowledge graph based on the media domain is provided, including the following steps:

[0007] Step 1: Obtain news data. After preprocessing the news data, obtain and save the first image data and the first audio / video data from the news data.

[0008] Step 2: Segment the news text corresponding to the news data and extract entity words and domain event triples from each sentence;

[0009] Step 3: Search for the corresponding second image data and second audio / video data in the search engine based on the entity words and domain event triples. Use the entity words and domain events and their corresponding filtered second image data and second audio / video data as the training set to fine-tune the CLIP multimodal pre-trained language model to obtain the CLIP fine-tuned model.

[0010] Step 4: Match and predict the corresponding entities and events according to the CLIP fine-tuning model, and expand the multimodal attribute set of entities and events according to the matching prediction results and the first image data and the first audio and video data;

[0011] Step 5: Construct a knowledge base with entities and events as the main body, and with the relationships between entities, the attributes of entities, the relationships between entities and events, the relationships between events, and the multimodal data in the multimodal attribute set as attribute fields of entities and events. After performing referential resolution and entity disambiguation on the data in the knowledge base, construct the ontology of domain entities and domain events, and write the ontology into the graph database to obtain the media domain multimodal knowledge graph.

[0012] Furthermore, to ensure the quality, quantity, diversity, and timeliness of news data, step 1, namely, acquiring news data, specifically includes:

[0013] Obtain raw news data from multiple news data sources;

[0014] Each news data source is weighted and scored based on the quality, quantity, diversity, and timeliness of the original news data, and the top news data sources with the highest weight scores are selected.

[0015] Based on the web crawling algorithm, news data is crawled from each selected news data source according to the corresponding weight score; the higher the weight score, the more data is crawled from the corresponding news data source, and the lower the weight score, the less data is crawled from the corresponding news data source.

[0016] Furthermore, to further improve the quality of news data, step 1, the preprocessing of news data, specifically includes:

[0017] Categorize news data and filter out news data that cannot be categorized or has incomplete information;

[0018] Filter sensitive and inappropriate information from news data;

[0019] Perform dirty data cleaning and deduplication on news data.

[0020] Furthermore, in order to achieve entity extraction, step 2 includes the following entity word extraction method:

[0021] For each news data, based on RoBERTa, the news title and body sentence are embedded into semantic vectors, and the news title is used as the center vector. TextRank is used to model and score the body sentences to obtain the importance score of each body sentence.

[0022] Entity recognition and extraction are performed using a combination of domain dictionaries and deep learning. Sentence-level joint entity relation extraction is performed based on the CASREL framework to obtain triples of entity and attribute, and entity and relation. The extracted entity scores in the news data are calculated based on the importance score of each sentence in the text and TF-IDF. After normalizing the entity scores, entities whose sum of scores is greater than a first preset score are taken as entity words in the text.

[0023] Furthermore, in order to achieve event extraction, step 2 includes the following method for extracting the domain event triples:

[0024] Based on the DEGREE model, trigger word recognition and argument recognition are performed on each sentence in the text to obtain the corresponding event triples. The network structure for argument recognition includes at least an embedding layer and an attention layer. The initial value of the embedding layer is a fine-tuned BERT model.

[0025] The weighted score of each event triple is obtained by weighted summing of the scores of the entities in the event triple. Event triples with a weighted score greater than a second preset score are designated as domain event triples.

[0026] Furthermore, to further improve the diversity of data, step 4 also includes:

[0027] Based on the entity term and domain event triple, supplementary data for entities and events is searched in the search engine, and the supplementary data is filtered and added to the extracted data. The supplementary data is structured data containing the entity term and domain event.

[0028] Secondly, a media-based information retrieval method is provided, including the multimodal knowledge graph construction method in the media domain as described in the first aspect, and further including the following steps:

[0029] Step 6: Construct an ES retrieval library based on the multimodal knowledge graph;

[0030] Step 7: Obtain the information to be retrieved and determine the type of the information to be retrieved. If the information to be retrieved is text, extract entities and events from the text, and retrieve relevant data based on the extracted entities and events and the ES retrieval library. If the information to be retrieved is an image, use the perceptual PHash algorithm to obtain the 64-bit hash code of the image, and retrieve relevant data based on the hash code and the bit value of the ES retrieval library.

[0031] Furthermore, in order to construct the ES search library, step 6 specifically includes:

[0032] The triplet data in the multimodal knowledge graph is structured and stored in the ES database to build an ES retrieval library.

[0033] Based on the pre-trained language model BERT, a low-dimensional vector is learned for each entity and event, and the low-dimensional vector is stored in the vector field of the ES retrieval library;

[0034] The perceptual PHash algorithm is used to perform 64-bit hash encoding on images related to entities or events, and the resulting hash value is stored in the image encoding field of the ES retrieval database.

[0035] Furthermore, to further improve the accuracy and effectiveness of information retrieval, step 7 involves recalling relevant data based on the extracted entities and events and the ES retrieval database, specifically including:

[0036] The text is vectorized, and the similarity between the vector of the recalled entity or event and the vector of the text is calculated. Entities or events with similarity greater than a threshold are used as recalled data, and the multimodal attributes related to the entity or event are displayed. The multimodal attributes include: image data and / or audio and video data.

[0037] Furthermore, to avoid the impact of data changes in the knowledge graph on retrieval accuracy, the method also includes:

[0038] Periodically scan to determine data changes in the multimodal knowledge graph, and update the ES retrieval database based on the changed data.

[0039] The beneficial effects of this invention are as follows: The multimodal knowledge graph construction method and information retrieval method based on the media domain described in this invention construct a multimodal knowledge graph in the media domain based on full-network multimodal data crawling technology, information extraction technology (entities, relationships, attributes, events, etc.), entity localization technology, knowledge graph technology, NLP-related technologies, and information retrieval technology. A retrieval engine is then built based on this multimodal knowledge graph. Users can retrieve news information through the retrieval engine using text or images. Furthermore, the retrieval engine can display multimodal news data such as images, audio, and video based on the retrieved recall data, thereby improving the effectiveness of information retrieval services and user experience. Quantitative analysis of the entire network data source ensures the reliability and diversity of the data. Cleaning of dirty data and filtering of sensitive data after crawling greatly ensures high data quality. Entity extraction is performed using a combination of domain dictionaries and deep learning. Sentence-level joint extraction of entity relationships is conducted based on the CASREL framework, and then the top K entity words are selected through weighted scoring, greatly ensuring the quality of extracted entities. Joint training of the event extraction task based on a deep learning model reduces error propagation caused by the multi-level tasks of traditional methods. The ability to model long-distance dependencies is improved by adding pre-trained models and attention layers. Weighted scores based on sentence triples and entity weights are used to obtain descriptions of news events. The obtained entities and events are used to extract corresponding multimodal attributes (images, audio, video, etc.) from search engines and encyclopedia data. After manual screening, text-image (audio-video) matching data is used as the training set to fine-tune the CLIP multimodal pre-trained language model. The fine-tuned model is then used to perform corresponding entity and event matching predictions on previously crawled image and audio-video data, enriching the multimodal attributes of entities and events. Domain knowledge is stored in the form of a domain entity and event knowledge graph, making the domain knowledge more structured and hierarchical, improving its subsequent maintainability and scalability. A retrieval library for recall is built using ES middleware, BERT is used to vectorize text entities, and the perceptual PHash algorithm is used to obtain image hash codes for post-recall similarity ranking, improving the performance and effectiveness of the retrieval engine. Two retrieval processes based on text and images are constructed to enhance the diversity of retrieval functions. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method for constructing a multimodal knowledge graph based on the media domain as described in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the entity word extraction process according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the process for extracting domain event triples according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the entity positioning process described in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the process for constructing a multimodal knowledge graph according to an embodiment of the present invention;

[0045] Figure 6 This is a flowchart illustrating the information retrieval method based on the media domain as described in an embodiment of the present invention. Detailed Implementation

[0046] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] This invention aims to provide a method for constructing a multimodal knowledge graph and an information retrieval method based on the media domain, in order to improve the retrieval effect of news information and user experience. The main technical solution includes: acquiring news data; preprocessing the news data; acquiring and saving first image data and first audio / video data from the news data; segmenting the news text corresponding to the news data into sentences and extracting entity words and domain event triples from each sentence; searching for corresponding second image data and second audio / video data in a search engine based on the entity words and domain event triples; and using the entity words and domain events and their corresponding filtered second image data and second audio / video data as a training set to fine-tune C. The CLIP multimodal pre-trained language model is used to obtain the CLIP fine-tuned model. Matching and prediction of corresponding entities and events are performed based on the CLIP fine-tuned model. The multimodal attribute sets of entities and events are expanded based on the matching and prediction results, the first image data, and the first audio / video data. A knowledge base is constructed using entities and events as the main body, with relationships between entities, entity attributes, relationships between entities and events, relationships between events, and multimodal data in the multimodal attribute set as attribute fields for entities and events. After performing referential resolution and entity disambiguation on the data in the knowledge base, an ontology of domain entities and domain events is constructed, and the ontology is written into a graph database to obtain a media domain multimodal knowledge graph. An ES retrieval library is constructed based on the multimodal knowledge graph; the information to be retrieved is obtained, and the type of the information to be retrieved is determined. If the information to be retrieved is text, entities and events are extracted from the text, and relevant data is retrieved based on the extracted entities and events and the ES retrieval library. If the information to be retrieved is an image, the 64-bit hash code of the image is obtained using the perceptual PHash algorithm, and relevant data is retrieved based on the hash code and the bit value of the ES retrieval library.

[0048] Specifically, this invention constructs a multimodal knowledge graph in the media field based on multimodal data crawling technology, information extraction technology (entities, relationships, attributes, events, etc.), entity localization technology, knowledge graph technology, NLP (Natural Language Processing) related technologies, and information retrieval technology, and builds a retrieval engine based on the multimodal knowledge graph. The process involves several key components: First, a multimodal web crawling technique is used to crawl and filter news data across the entire internet, extracting and storing corresponding image and audio / video data. Second, information extraction techniques are used to extract event triples (entities, attributes, and relationships), and entity words and domain event triples are determined from the extracted data. Third, entity localization techniques are used to construct multimodal attribute sets for entities and events, including image and audio / video data. Fourth, knowledge graph techniques are used to construct a multimodal knowledge graph based on the extracted event triples and multimodal attribute sets. Finally, NLP and information retrieval techniques are used to build an Elasticsearch (ES) retrieval library based on the constructed multimodal knowledge graph, and corresponding API services are developed. Users can retrieve news information by inputting the information they wish to search. Specifically, after a user inputs the information to be searched, the interface service determines the type of the information and selects the appropriate method to retrieve recall data from the ES retrieval library, then outputs the recalled data in a structured format. Example

[0049] Please see Figure 1 The method for constructing a multimodal knowledge graph based on the media domain, as described in this embodiment of the invention, includes the following steps:

[0050] Step 1: Obtain news data. After preprocessing the news data, obtain and save the first image data and the first audio / video data from the news data.

[0051] In this embodiment, the method of obtaining news data specifically includes: obtaining raw news data from multiple news data sources; assigning weight scores to each news data source based on the quality, quantity, diversity, and timeliness of the raw news data, and selecting multiple news data sources with higher weight scores; crawling news data from each selected news data source based on a crawling algorithm and according to the corresponding weight scores; wherein, the higher the weight score, the more data is crawled from the corresponding news data source, and the lower the weight score, the less data is crawled from the corresponding news data source.

[0052] This step is used for news data crawling and filtering. This embodiment includes the following process:

[0053] First, a comprehensive analysis is conducted on major news websites across the internet, such as People's Daily Online, Xinhua Net, and CCTV. Data from these websites over a specific period is analyzed, focusing on attributes such as news quality, quantity, diversity, and timeliness. Each attribute dimension is manually sampled, evaluated, and scored, and then the average score is calculated to determine the weight of each source. Next, for the identified news sources, the top-scoring sources are selected, and crawling resources are allocated according to their weight scores. More data is crawled from higher-scoring sources, and less from lower-scoring sources. This embodiment utilizes the open-source Python Scrapy package to build a crawler algorithm and periodically updates the data.

[0054] The method for preprocessing news data in this embodiment includes: classifying the crawled news data, mainly into categories such as politics, sports, entertainment, and finance, and filtering out news data that cannot be classified or has incomplete information; constructing a sensitivity detection algorithm based on dictionary, pinyin, variant characters, and deep learning to filter sensitive and harmful information (pornography, violence, advertisements, etc.) from the classified news data; and then performing dirty data cleaning and deduplication on the news data.

[0055] Step 2: Segment the news text corresponding to the news data and extract entity words and domain event triples from each sentence;

[0056] Please see Figure 2 The method for extracting entity words in this embodiment specifically includes:

[0057] For each news data, based on RoBERTa, the news title and body sentence are embedded into semantic vectors, and the news title is used as the center vector. TextRank is used to model and score the body sentences to obtain the importance score of each body sentence.

[0058] Entity recognition and extraction are performed using a combination of domain dictionaries and deep learning. Sentence-level joint entity relation extraction is performed based on the CASREL framework to obtain triples of entity and attribute, and entity and relation. The extracted entity scores in the news data are calculated based on the importance score of each sentence in the text and TF-IDF. After normalizing the entity scores, entities whose sum of scores is greater than a first preset score are taken as entity words in the text.

[0059] The first preset score can be set according to the actual situation. This embodiment does not limit this, for example, 90%.

[0060] Please see Figure 3 The method for extracting domain event triples in this embodiment specifically includes:

[0061] Trigger word and argument recognition are performed on each sentence in the text based on the DEGREE model to obtain the corresponding event triples. The network structure for argument recognition includes at least an embedding layer and an attention layer. The initial value of the embedding layer is a fine-tuned BERT model. By adding the attention layer, long-distance dependencies can be modeled.

[0062] The weighted score of each event triple is obtained by weighted summing of the scores of the entities in the event triple. Event triples with a weighted score greater than a second preset score are designated as domain event triples.

[0063] The first preset score can be set according to the actual situation, and this embodiment does not impose any restrictions on it.

[0064] Step 3: Search for the corresponding second image data and second audio / video data in the search engine based on the entity words and domain event triples. Use the entity words and domain events and their corresponding filtered second image data and second audio / video data as the training set to fine-tune the CLIP multimodal pre-trained language model to obtain the CLIP fine-tuned model.

[0065] Step 4: Match and predict the corresponding entities and events according to the CLIP fine-tuning model, and expand the multimodal attribute set of entities and events according to the matching prediction results and the first image data and the first audio and video data;

[0066] Please see Figure 4 In this embodiment, steps 3-4 are used to achieve entity localization, and specifically include the following process:

[0067] Step A: Using the entities and events obtained above, search for the corresponding second image data and second audio / video data from search engines (Baidu, Sogou, etc.) and encyclopedias (Baidu Encyclopedia, Wikipedia);

[0068] Step B: Manually filter the second image data and second audio / video data obtained above, filtering out some data that does not match the entity or event, and selecting the second image data and second audio / video data with a high degree of matching as additional attributes of the entity and event.

[0069] Step C: Use the text-image (audio-video) matching data mentioned above as a training set to fine-tune the CLIP (Contrastive Language-Image Pre-Training) multimodal pre-trained language model. Use the fine-tuned model to perform corresponding entity and event matching predictions on the previously crawled first image data and first audio-video data, expanding the multimodal attribute set corresponding to entities and events.

[0070] Step 5: Construct a knowledge base with entities and events as the main body, and with the relationships between entities, the attributes of entities, the relationships between entities and events, the relationships between events, and the multimodal data in the multimodal attribute set as attribute fields of entities and events. After performing referential resolution and entity disambiguation on the data in the knowledge base, construct the ontology of domain entities and domain events, and write the ontology into the graph database to obtain the media domain multimodal knowledge graph.

[0071] Please see Figure 5 This step is used to construct a knowledge graph, and this embodiment specifically includes the following process:

[0072] Based on the entity term and domain event triple, supplementary data for entities and events is searched in the search engine, and the supplementary data is filtered and added to the extracted data. The supplementary data is structured data containing the entity term and domain event.

[0073] A knowledge base is constructed based on the extracted entity relationship attributes and event data: the knowledge base is constructed with entities and events as the main body, and the relationships between entities, the various attributes of entities, the relationships between entities and events, the relationships between events, and supplementary multimodal data as attribute fields of entities and events.

[0074] Based on the entity word data collected in the early stage, external data obtained from the encyclopedia, and the corresponding image data and audio and video data of the entities, the referential resolution is carried out, and entity disambiguation related operations are performed through word2vec vectorization technology of related words;

[0075] A knowledge graph is built using domain entities and domain events as ontology, including text, image, audio and video attributes. If an entity can be matched with a corresponding event, a relationship between the entity and the event is established.

[0076] Please see Figure 6 Based on the above-mentioned method for constructing a multimodal knowledge graph in the media domain, this embodiment also proposes an information retrieval method based on the media domain, including the aforementioned method for constructing a multimodal knowledge graph in the media domain, and further including the following steps:

[0077] Step 6: Construct an ES retrieval library based on the multimodal knowledge graph;

[0078] In this embodiment, the process of building the ES search library specifically includes:

[0079] Step 61: After structuring the triplet data in the multimodal knowledge graph, store it in the ES database to build an ES retrieval library, including fields such as entity, relation, attribute (text, image, etc.), vector, image encoding, and time;

[0080] Step 62: Based on the pre-trained language model BERT, a low-dimensional vector is learned for each entity and event, and the low-dimensional vector is stored in the vector field of the ES retrieval library; at the same time, the crawled full network data is used to fine-tune the training of BERT to improve the accuracy of the model.

[0081] Step 63: Use the perceptual PHash algorithm to perform 64-bit hash encoding on the images related to entities or events, and save the resulting hash value in the image encoding field of the ES retrieval library.

[0082] After constructing the ES retrieval library, this embodiment also includes: periodically scanning to determine data changes in the multimodal knowledge graph, and updating the ES retrieval library based on the changed data.

[0083] Step 7: Obtain the information to be retrieved and determine the type of the information to be retrieved. If the information to be retrieved is text, extract entities and events from the text, and retrieve relevant data based on the extracted entities and events and the ES retrieval library. If the information to be retrieved is an image, use the perceptual PHash algorithm to obtain the 64-bit hash code of the image, and retrieve relevant data based on the hash code and the bit value of the ES retrieval library.

[0084] Please see Figure 6 This embodiment develops and encapsulates an interface service to determine the type of the input query and selects the appropriate method for news information retrieval based on the determination result. Specifically, when the query is text, entities and events are extracted from the text, and then the extracted entities and events are retrieved using Elasticsearch (ES). Simultaneously, BERT is used to vectorize the query. By calculating the similarity between the vectors of the retrieved entities (events) and the vectors of the input entities (events), a threshold is given to filter the top k similarity data as search results, and multimodal attributes such as images and videos related to the entities (events) are displayed.

[0085] When the query is an image, the perceptual PHash algorithm is used to obtain the 64-bit hash code of the query. The relevant data is retrieved in Elasticsearch based on the bit value. The higher the number of bit matches, the more similar the two images are. A threshold is given to filter the top k similarity data as search results and display multimodal attributes such as entities (events) and videos related to the image.

[0086] In summary, the multimodal knowledge graph construction and information retrieval method based on the media domain described in this embodiment performs quantitative analysis on data sources from the entire network, ensuring the reliability and diversity of the data. The crawled data undergoes cleaning of dirty data and filtering of sensitive data, followed by manual review, significantly guaranteeing high data quality. Entity extraction is performed using a combination of domain dictionaries and deep learning. Sentence-level entity relation joint extraction is conducted based on the CASREL framework, and then the top K entity words are selected through weighted scoring, greatly ensuring the quality of extracted entities. Joint training of the event extraction task based on a deep learning model reduces error propagation caused by multi-level tasks in traditional methods. The ability to model long-distance dependencies is improved by adding a pre-trained model and attention layers. A weighted score based on sentence triples and entity weights is used to obtain descriptions of news events. The obtained entities and events are used to acquire corresponding multimodal attributes such as images, audio, and video from search engines and encyclopedia data. These attributes are manually screened, and then text-image (audio-video) matching data is used as the training set to fine-tune the CLIP multimodal pre-trained language model. The finely tuned model is used to perform entity and event matching and prediction on previously crawled image, audio, and video data, enriching the multimodal attributes of entities and events. Domain knowledge is stored in the form of a domain entity and event knowledge graph, making the domain knowledge more structured and hierarchical, improving its maintainability and scalability. NLP technologies such as word2vec are used to perform operations such as graph referencing resolution and entity disambiguation. An ES middleware is used to build a retrieval library for image recall, using BERT to vectorize text entities and a perceptual PHash algorithm to obtain image hash codes for post-recall similarity ranking, improving the performance and effectiveness of the retrieval engine. Two retrieval processes are constructed, one based on text and the other on images, enhancing the diversity of retrieval functions.

Claims

1. A method for constructing a multimodal knowledge graph based on the media domain, characterized in that: Includes the following steps: Step 1: Obtain news data. After preprocessing the news data, obtain and save the first image data and the first audio / video data from the news data. Step 2: Segment the news text corresponding to the news data and extract entity words and domain event triples from each sentence; Step 3: Search for the corresponding second image data and second audio / video data in the search engine based on the entity words and domain event triples. Use the entity words and domain events and their corresponding filtered second image data and second audio / video data as the training set to fine-tune the CLIP multimodal pre-trained language model to obtain the CLIP fine-tuned model. Step 4: Match and predict the corresponding entities and events according to the CLIP fine-tuning model, and expand the multimodal attribute set of entities and events according to the matching prediction results and the first image data and the first audio and video data; Step 5: Construct a knowledge base with entities and events as the main body, and with the relationships between entities, the attributes of entities, the relationships between entities and events, the relationships between events, and the multimodal data in the multimodal attribute set as attribute fields of entities and events. After performing referential resolution and entity disambiguation on the data in the knowledge base, construct the ontology of domain entities and domain events, and write the ontology into the graph database to obtain the media domain multimodal knowledge graph.

2. The method for constructing a multimodal knowledge graph based on the media domain as described in claim 1, characterized in that, Step 1, specifically acquiring news data, includes: Obtain raw news data from multiple news data sources; Each news data source is weighted and scored based on the quality, quantity, diversity, and timeliness of the original news data, and the top news data sources with the highest weight scores are selected. Based on the web crawling algorithm, news data is crawled from each selected news data source according to the corresponding weight score; the higher the weight score, the more data is crawled from the corresponding news data source, and the lower the weight score, the less data is crawled from the corresponding news data source.

3. The method for constructing a multimodal knowledge graph based on the media domain as described in claim 1, characterized in that, Step 1 involves preprocessing the news data, specifically including: Categorize news data and filter out news data that cannot be categorized or has incomplete information; Filter sensitive and inappropriate information from news data; Perform dirty data cleaning and deduplication on news data.

4. The method for constructing a multimodal knowledge graph based on the media domain as described in claim 1, characterized in that, In step 2, the entity word extraction method includes: For each news data, based on RoBERTa, the news title and body sentence are embedded into semantic vectors, and the news title is used as the center vector. TextRank is used to model and score the body sentences to obtain the importance score of each body sentence. Entity recognition and extraction are performed using a combination of domain dictionaries and deep learning. Sentence-level joint entity relation extraction is performed based on the CASREL framework to obtain triples of entity and attribute, and entity and relation. The extracted entity scores in the news data are calculated based on the importance score of each sentence in the text and TF-IDF. After normalizing the entity scores, entities whose sum of scores is greater than a first preset score are taken as entity words in the text.

5. The method for constructing a multimodal knowledge graph based on the media domain as described in claim 4, characterized in that, In step 2, the method for extracting the domain event triples includes: Based on the DEGREE model, trigger word recognition and argument recognition are performed on each sentence in the text to obtain the corresponding event triples. The network structure for argument recognition includes at least an embedding layer and an attention layer. The initial value of the embedding layer is a fine-tuned BERT model. The weighted score of each event triple is obtained by weighted summing of the scores of the entities in the event triple. Event triples with a weighted score greater than a second preset score are designated as domain event triples.

6. The method for constructing a multimodal knowledge graph based on the media domain as described in claim 1, characterized in that, Step 4 also includes: Based on the entity term and domain event triple, supplementary data for entities and events is searched in the search engine, and the supplementary data is filtered and added to the extracted data. The supplementary data is structured data containing the entity term and domain event.

7. An information retrieval method based on the media field, characterized in that, The method for constructing a multimodal knowledge graph in the media domain as described in any one of claims 1 to 6 further includes the following steps: Step 6: Construct an ES retrieval library based on the multimodal knowledge graph; Step 7: Obtain the information to be retrieved and determine the type of the information to be retrieved. If the information to be retrieved is text, extract entities and events from the text, and retrieve relevant data based on the extracted entities and events and the ES retrieval library. If the information to be retrieved is an image, use the perceptual PHash algorithm to obtain the 64-bit hash code of the image, and retrieve relevant data based on the hash code and the ES retrieval library by bit value.

8. The information retrieval method based on the media domain as described in claim 7, characterized in that, Step 6 specifically includes: The triplet data in the multimodal knowledge graph is structured and stored in the ES database to build an ES retrieval library. Based on the pre-trained language model BERT, a low-dimensional vector is learned for each entity and event, and the low-dimensional vector is stored in the vector field of the ES retrieval library; The perceptual PHash algorithm is used to perform 64-bit hash encoding on images related to entities or events, and the resulting hash value is stored in the image encoding field of the ES retrieval database.

9. The information retrieval method based on the media domain as described in claim 7, characterized in that, Step 7 involves retrieving relevant data based on the extracted entities and events and the ES retrieval database, specifically including: The text is vectorized, and the similarity between the vector of the recalled entity or event and the vector of the text is calculated. Entities or events with similarity greater than a threshold are used as recalled data, and the multimodal attributes related to the entity or event are displayed. The multimodal attributes include: image data and / or audio and video data.

10. The information retrieval method based on the media domain as described in claim 7, characterized in that, The method further includes: Periodically scan to determine data changes in the multimodal knowledge graph, and update the ES retrieval database based on the changed data.

Images