Multi-modal power knowledge base construction method and retrieval method based on AI large language model

Abstract

The invention belongs to the technical field of power system informatization, and provides a multi-modal power knowledge base construction method and retrieval method based on an AI large language model.The method comprises the following steps that S1, classification and named entity extraction are conducted on an original database, and a data set and an entity set are obtained; s2, aligning the data set and the entity set based on an AI large language model; s3, extracting and complementing entity relationships, and constructing an electric power diagram database; s4, performing joint embedding on the entities and the associated data, mapping the entities and the associated data to the same vector space, constructing a multi-modal power knowledge base, and generating a data index; and performing retrieval by using the constructed multi-modal power knowledge base, and performing source file backtracking through a data index. According to the method, multi-modal data are fused, the constructed multi-modal electric power knowledge base is rich and accurate, and source file backtracking can be carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of power system informatization, and specifically relates to a multimodal power knowledge base construction method and a retrieval method based on an AI large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the advancement of large language models, its capabilities in text processing, image processing, audio processing, and video processing have been significantly improved. These advances have greatly promoted the intelligentization of multiple industries. However, in the key field of the power industry, despite the accumulation of rich research results and data resources, the application of AI technology is still in its early stages and has not been able to fully realize its potential.

[0003] The power industry is undergoing a major transformation from traditional thermal power generation and hydropower generation to new energy power generation methods such as wind power and solar power. A large amount of valuable data is generated in this process, including but not limited to production data, industry experience and technological achievements. These data provide key support for the sustainable development of the power system, the efficient use of energy and the optimized operation of the power grid. However, the current knowledge base construction technology in the power industry is still not mature enough and mainly relies on text data, while the use of unstructured data such as images, videos and audio is relatively limited and the utilization rate is not high. Therefore, it has become a difficult problem to better integrate these multi-source heterogeneous data and transform them into knowledge to support decision-making in the power industry and build a comprehensive multimodal power knowledge base.

[0004] Patent document CN111831792B proposes a method for constructing an electric power knowledge base. For existing unstructured electric power data, a named entity recognition model is used to extract entities; then a named entity relationship extraction model including a first extraction model and a second extraction module is proposed. The first extraction model is used to obtain the relationship words between the named entities, and the relationship words are then used to reversely search the corresponding named entity pairs in the text through the second extraction model to obtain the named entity relationship data. Finally, the named entity relationship data is stored in the process to construct an electric power knowledge base. For structured data, the named entity relationship extraction model is directly used to obtain the named entity relationship data, and then the electric power knowledge base is constructed; finally, the TransE model is used to complete the established knowledge base. However, this patent mainly constructs the power knowledge base with structured data and unstructured data. In the data collection and preprocessing process, it does not consider adding AI big language models, and cannot fully extract the data information. In addition, the data used is mainly text data, and images, videos and audio data are not considered. In the process of building the knowledge base, traditional named entity extraction and named entity relationship extraction models are used, and the use of AI big language models is not considered. This patent also does not discuss the retrieval method of the power knowledge base.

[0005] Patent document CN118093768A proposes a method for constructing a knowledge base of electric power scientific research achievements based on a large language model. The method uses the Scrapy crawler framework to crawl the power data in the network, and then converts the ChatGLM2 6B large language model into an electric power keyword expander to expand the keyword information; the keyword vocabulary obtained by the large model is combined with the word embedding model to obtain alias / synonym information and construct a keyword vocabulary of electric power scientific research achievements; the expanded keyword vocabulary is vectorized using the word embedding model, and the text similarity is performed considering the keyword information itself, the keyword context semantic information and the key text length, and the alias / synonym information of the keyword is obtained through a high threshold, and it is classified into one category, and the keywords are efficiently and comprehensively semantically represented in the vector space based on the BGE Large zh model, so as to construct a knowledge base. However, this patent mainly constructs the power knowledge base based on text data, without considering image, video and audio data. In the data collection and preprocessing process, the main function of the AI ​​large language model is to expand the keyword vocabulary, which is not used in the extraction process of power data in this patent, and the patent also does not describe the retrieval method of the power knowledge base.

[0006] Therefore, the current construction methods of the power knowledge base are mainly based on common text data, and data is extracted by text entity recognition and entity relationship extraction. However, how to improve the utilization of video, audio and image data has not been given major consideration, and the AI ​​big language model has not been fully used in the construction of the knowledge base, so that the extracted data cannot be fully integrated and cannot support the intelligent development of the power industry. In addition, the existing technology does not involve the discussion of searching the power knowledge base based on the AI ​​big language model and improving the retrieval effect. Therefore, it is necessary to propose a construction method and retrieval method for a multimodal power knowledge base to solve the current problems of low efficiency in data resource utilization and weak knowledge base retrieval capabilities. Summary of the invention

[0007] The present invention aims to address the deficiencies in the prior art and provide a method for constructing a multimodal electric power knowledge base and a retrieval method based on an AI large language model, which integrates text, image, audio and video data of the electric power industry, uses an AI large language model to align the data in the data set with the entities in the entity set, and completes the entity relationship between the entities and the image screen data, audio data and video screen data, and can perform source file backtracking.

[0008] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0009] A method for constructing a multimodal power knowledge base based on an AI large language model comprises the following steps:

[0010] S1. Construct a multimodal power original database, classify the data in the multimodal power original database to obtain text data, image data, audio data and video data, and perform named entity extraction respectively to obtain a data set and an entity set;

[0011] S2. Based on the AI ​​large language model, align the data in the data set with the entities in the entity set, and associate each entity with the text data corresponding to the entity in the data set, the image screen data contained in the image data, the image-to-text data contained in the image data, the audio data, the audio-to-text data, the video screen data contained in the video data, and the video-to-text data contained in the video data;

[0012] S3, for the entity set in step S2, extract entity relationships, build a preliminary database, and complete the entity relationships between the entities and the image screen data, audio data and video screen data to build a power map database;

[0013] S4. Jointly embed the entities in the power graph database and the image data, audio data and video data associated with the entities, so that data in different formats are mapped to the same vector space, entity vectors are obtained, a multimodal power knowledge base is constructed, and data indexes are generated from the entity vectors through scripts.

[0014] Preferably, in step S1, the Scrapy crawler framework is used to crawl the power industry related data in the network, sort out a multimodal power original database, and divide the data in the multimodal power original database into text data, image data, audio data and video data through a script.

[0015] Preferably, in step S1,

[0016] Preprocess the text data:

[0017] The BERT-Bi-LSTM-CRF model is used to extract named entities, and the distance between entities is calculated by cosine similarity. When the cosine similarity is greater than or equal to the preset value a, they are marked as the same entity, otherwise they are marked as different entities, and an entity set consisting of different entities is obtained.

[0018] Preferably, in step S1,

[0019] Preprocess the image data:

[0020] For image data containing image frames, the ViT model is used for classification to obtain an image frame dataset;

[0021] For image data containing text content, use PaddleOCR to perform direction correction and classification detection, extract the text content in the image data, generate text data, and obtain the image-to-text dataset;

[0022] For the text data in the image-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0023] Preferably, in step S1,

[0024] Preprocess the audio data:

[0025] The speech in the audio data is classified and labeled through the BERT model, and the starting frame and frame length are recorded to obtain the audio data set;

[0026] Use the Whisper model to convert audio into text, generate text data, and obtain an audio-to-text dataset;

[0027] For the text data in the audio-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0028] Preferably, in step S1,

[0029] Preprocess the video data:

[0030] Use the YOLOv9 model to annotate the video screen data in the video data, and record the starting frame and frame length to obtain the video screen data set;

[0031] Use the Whisper model to convert the audio in the video into text, generate text data, and obtain a video-to-text dataset;

[0032] For the text data in the video-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0033] Preferably, in step S2, the audio data and the video picture data are both obtained by recording the starting frame and frame length to obtain entity parts, thereby achieving entity alignment.

[0034] Preferably, in step S3, an entity relationship extraction model is used to extract entity relationships, and a multimodal knowledge graph completion model is used to complete entity relationships between entities and image data, voice data, and video data;

[0035] Preferably, in step S4, a joint embedding model is used to jointly embed the entities in the power graph database and the image data, voice data and video data associated with the entities.

[0036] The present invention also provides a method for searching a multimodal power knowledge base constructed by the construction method as described above, using an AI large language model for retrieval. When searching, entities in the query question are converted into entity vectors through embedding, input into the multimodal power knowledge base, and matching results are screened out by calculating cosine similarity. Text, image, audio and video content matching the entity are output, and image indexes, audio indexes and video indexes of the matching results are found. Source files are traced back through data indexes to locate the specific position of the matching results in the data source.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] (1) Data sets and entity sets are obtained by preprocessing text, image, audio, and video data, and the data in the data sets are aligned with the entities in the entity set based on the AI ​​big language model, so that data utilization is higher and the multimodal power knowledge base finally constructed is richer and more accurate;

[0039] (2) By using a multimodal knowledge graph completion model that adds video data input to complete the entity relationships between entities and image data, audio data, and video data, the accuracy of extracting relationships between entities is effectively improved compared to extracting only entity relationships containing text data.

[0040] (3) By using a joint embedding model to jointly embed entities in the power map database and the image data, audio data, and video data associated with the entities, data of different modalities can be mapped to the same vector space, making it easier to generate corresponding data indexes through scripts, thereby improving the subsequent AI large language model's ability to understand the multimodal power knowledge base;

[0041] (4) When using the AI ​​large language model for retrieval, matching results can be screened out for entities in the query, and the source files of the matching results can be traced back to improve the credibility and traceability of the source of the answer, thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1Flowchart of a multimodal power knowledge base construction method and retrieval method based on an AI large language model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Example 1

[0045] Combination Figure 1 As shown, an embodiment of the present invention provides a method for constructing a multimodal power knowledge base based on an AI large language model, comprising the following steps:

[0046] S1. Construct a multimodal power original database, classify the data in the multimodal power original database, obtain text data, image data, audio data and video data, and perform named entity extraction respectively to obtain a data set and an entity set.

[0047] Specifically, by using the Scrapy crawler framework, crawling the power industry related data on the network, sorting out a multimodal power original database, and dividing the data in the multimodal power original database into text data, image data, audio data and video data through a script;

[0048] Preprocess the text data:

[0049] The named entities are extracted using the BERT-Bi-LSTM-CRF model, and the distance between entities is calculated by cosine similarity. When the cosine similarity is greater than or equal to the preset value a, they are marked as the same entity, otherwise they are marked as different entities, and an entity set consisting of different entities is obtained. In this embodiment, the preset value of cosine similarity is 0.7, and the following description of the preset value a of cosine similarity is all 0.7;

[0050] Preprocess the image data:

[0051] For image data containing image content, the ViT model is used for classification to obtain an image picture dataset;

[0052] For image data containing text content, use PaddleOCR to perform direction correction and classification detection, extract the text content in the image data, generate text data, and obtain the image-to-text dataset;

[0053] For the text data in the image-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0054] Preprocess the audio data:

[0055] The speech in the audio data is classified and labeled through the BERT model, and the starting frame and frame length are recorded to obtain the audio data set;

[0056] Use the Whisper model to convert audio into text, generate text data, and obtain an audio-to-text dataset;

[0057] For the text data in the audio-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0058] Preprocess the video data:

[0059] Use the YOLOv9 model to annotate the video screen data in the video data, and record the starting frame and frame length to obtain the video screen data set;

[0060] Use the Whisper model to convert the audio in the video into text, generate text data, and obtain a video-to-text dataset;

[0061] For the text data in the video-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

[0062] S2. Based on the AI ​​large language model, the data in the data set are aligned with the entities in the entity set, and each entity is associated with the text data corresponding to the entity in the data set, the image screen data contained in the image data, the image-to-text data contained in the image data, the audio data, the audio-to-text data, the video screen data contained in the video data, and the video-to-text data contained in the video data.

[0063] Among them, for image data containing image content, the features extracted by the ViT model are aligned with the entities in the entity set, thereby achieving an alignment effect;

[0064] The audio data and the video picture data both obtain the entity part by recording the starting frame and the frame length, so as to realize the entity alignment and further realize the entity alignment effect.

[0065] By preprocessing text, image, audio and video data, we obtain data sets and entity sets, and align the data in the data sets with the entities in the entity sets based on the AI ​​big language model, so that data utilization is higher and the multimodal power knowledge base finally constructed is richer and more accurate.

[0066] S3. For the entity set in step S2, extract entity relationships, build a preliminary database, and complete the entity relationships between the entities and the image screen data, audio data and video screen data to build a power map database.

[0067] Furthermore, an entity relationship extraction model is used to extract the entity relationship of the entity set in step S2. Specifically, entity relationship extraction models such as BERT, SpERT, CasRel, PFN, PL-Marker, etc. can be used to build a preliminary database. For image data, audio data, and video data, a multimodal knowledge graph completion model with an added video data input part is used to complete the entity relationship between the entity and the image data, audio data, and video data. Specifically, multimodal knowledge graph completion models such as AdaMF-MAT, MKGFormer, NativE, etc. can be used to build a complete power graph database.

[0068] By using a multimodal knowledge graph completion model that adds video data input to complete the entity relationships between entities and image data, audio data, and video data, the accuracy of extracting relationships between entities is effectively improved compared to only extracting entity relationships containing text data.

[0069] S4. Use a joint embedding model to jointly embed the entities in the power map database and the image data, audio data and video data associated with the entities. Specifically, joint embedding models such as CLIP, BLIP-2, Chinese-CLIP, Long-CLIP, ImageBind, etc. can be used to map data in different formats to the same vector space to obtain entity vectors, build a multimodal power knowledge base, and generate data indexes from entity vectors through scripts.

[0070] By using a joint embedding model to jointly embed entities in the power graph database and the image data, audio data, and video data associated with the entities, data of different modalities can be mapped to the same vector space, making it easier to generate corresponding data indexes through scripts and improving the subsequent AI large language model's ability to understand the multimodal power knowledge base.

[0071] Example 2

[0072] Different from Example 1, the present invention provides a method for searching a multimodal power knowledge base constructed by the construction method as described above, and uses an AI large language model for retrieval. When searching, the entities in the query question are converted into entity vectors through embedding, and input into the multimodal power knowledge base. The matching results are screened out by calculating the cosine similarity, and the text, image, audio and video content matching the entity are output. The image index, audio index and video index of the matching result are found, and the source file is traced back through the data index to locate the specific position of the matching result in the data source.

[0073] When using the AI ​​large language model for retrieval, matching results can be screened out for entities in the queried question, and the source files of the matched results can be traced back. Specifically, the source files of image data, audio data, and video data can be traced back, and the corresponding positions of the matched results in the data source can be given. For example, image data can be located at the coordinates of the entity, and audio and video data can be located at the starting frame. This function is relatively rare in the construction method and retrieval method of the electric power knowledge base, which can improve the credibility and traceability of the source of the answer and enhance the user experience.

[0074] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the application scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a multimodal power knowledge base based on an AI large language model, characterized in that: The following steps are involved: S1. Construct a multimodal power original database, classify the data in the multimodal power original database to obtain text data, image data, audio data and video data, and perform named entity extraction respectively to obtain a data set and an entity set; S2. Based on the AI ​​large language model, align the data in the data set with the entities in the entity set, and associate each entity with the text data corresponding to the entity in the data set, the image screen data contained in the image data, the image-to-text data contained in the image data, the audio data, the audio-to-text data, the video screen data contained in the video data, and the video-to-text data contained in the video data; S3, for the entity set in step S2, extract entity relationships, build a preliminary database, and complete the entity relationships between the entities and the image screen data, audio data and video screen data to build a power map database; S4. Jointly embed the entities in the power graph database and the image data, audio data and video data associated with the entities, so that data in different formats are mapped to the same vector space, entity vectors are obtained, a multimodal power knowledge base is constructed, and data indexes are generated from the entity vectors through scripts.

2. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 1 is characterized in that: In step S1, the Scrapy crawler framework is used to crawl the power industry related data in the network, sort out a multimodal power original database, and divide the data in the multimodal power original database into text data, image data, audio data and video data through a script.

3. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 2 is characterized in that: In step S1, Preprocess the text data: The BERT-Bi-LSTM-CRF model is used to extract named entities, and the distance between entities is calculated by cosine similarity. When the cosine similarity is greater than or equal to the preset value a, they are marked as the same entity, otherwise they are marked as different entities, and an entity set consisting of different entities is obtained.

4. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 2 is characterized in that: In step S1, Preprocess the image data: For image data containing image frames, the ViT model is used for classification to obtain an image frame dataset; For image data containing text content, use PaddleOCR to perform direction correction and classification detection, extract the text content in the image data, generate text data, and obtain the image-to-text dataset; For the text data in the image-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

5. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 2 is characterized in that: In step S1, Preprocess the audio data: The speech in the audio data is classified and labeled through the BERT model, and the starting frame and frame length are recorded to obtain the audio data set; Use the Whisper model to convert audio into text, generate text data, and obtain an audio-to-text dataset; For the text data in the audio-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

6. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 2 is characterized in that: In step S1, Preprocess the video data: Use the YOLOv9 model to annotate the video screen data in the video data, and record the starting frame and frame length to obtain the video screen data set; Use the Whisper model to convert the audio in the video into text, generate text data, and obtain a video-to-text dataset; For the text data in the video-to-text dataset, the BERT-Bi-LSTM-CRF model is used to extract named entities and calculate cosine similarity with the entities in the entity set. When the cosine similarity is greater than or equal to the preset value a, the entity set is not added. When the cosine similarity is less than the preset value a, the entity set is added.

7. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 6 is characterized in that: In step S2, the audio data and the video picture data are both obtained by recording the starting frame and frame length to obtain the entity part, thereby achieving entity alignment.

8. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 7 is characterized in that: In step S3, an entity relationship extraction model is used to extract entity relationships, and a multimodal knowledge graph completion model is used to complete the entity relationships between entities and image data, voice data, and video data.

9. The method for constructing a multimodal power knowledge base based on an AI large language model according to claim 8 is characterized in that: In step S4, a joint embedding model is used to jointly embed the entities in the power graph database and the image data, voice data and video data associated with the entities.

10. A method for searching a multimodal power knowledge base constructed using the construction method according to any one of claims 1 to 9, characterized in that: The AI ​​language model is used for retrieval. During retrieval, the entities in the query are converted into entity vectors through embedding, input into the multimodal power knowledge base, and the matching results are screened out by calculating the cosine similarity. The text, image, audio and video content of the body is matched, and the image index, audio index and video index of the matching results are found. The source file is traced back through data indexing to locate the specific location of the matching result in the data source.

Images