Document retrieval method, system and electronic device based on power semantic enhancement

Abstract

The application relates to a document retrieval method and system based on power semantic enhancement and electronic equipment, and belongs to the technical field of natural language processing. The application solves the problem of low utilization of complex knowledge and insufficient power professional semantic understanding, which leads to low retrieval accuracy. The application comprises the following steps: receiving user query content, obtaining a query embedding vector by using a modal joint embedding model; obtaining a structured query vector of the user query content by using a large language model and a text embedding model based on a power knowledge graph; obtaining a plurality of candidate documents, a document-level similarity score and a page-level similarity score of the candidate documents according to the query embedding vector and the structured query vector, and then obtaining a comprehensive similarity score of each candidate document by using a double-path prediction model; obtaining a total score according to the document-level similarity score, the page-level similarity score and the comprehensive similarity score of each candidate document, and selecting a plurality of candidate documents with the highest total score as retrieval results. The application improves the retrieval precision.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a document retrieval method, system, and electronic device based on power semantic enhancement. Background Technology

[0002] With the increasing complexity and scale of power engineering projects, and the continuous updating of relevant technical standards, the amount of engineering documents involved—including design specifications, construction drawings, equipment manuals, technical standards, regulations, and various reports—is exploding. Therefore, how to quickly and accurately retrieve key materials closely related to current review needs from this massive and highly specialized power engineering documentation, and use this information to assist reviewers or automated artificial intelligence algorithms in efficient and accurate extraction, reasoning, and decision-making, has become a major challenge that the power industry urgently needs to address.

[0003] Current practices for retrieving and reviewing power engineering documents still primarily rely on traditional information retrieval techniques, supplemented by extensive human experience and judgment. While these traditional techniques have made some progress in general text retrieval, their inherent limitations become increasingly apparent when processing highly specialized documents like those related to power engineering, which are heavily reliant on domain knowledge. These limitations lead to problems such as low retrieval efficiency, insufficient accuracy of results, and high labor costs.

[0004] Early document retrieval solutions primarily relied on keyword matching. These methods filter information by comparing keywords in the user's query with the document text content, typically implemented using Boolean search and term frequency-inverse document frequency models. However, these solutions suffer from significant semantic gaps, leading to low retrieval accuracy, including difficulty in identifying numerous synonyms, near-synonyms, and technical abbreviations in the field of electrical engineering. Furthermore, these solutions fail to understand the deeper intent behind the user's query and the complex logical connections between words, lacking an understanding of domain-specific knowledge and struggling to aggregate the overall semantics of long texts. While manually building keyword databases can mitigate this, it is costly and time-consuming, and the retrieval level remains at the word and sentence level, failing to effectively summarize the document content.

[0005] With the development of deep learning and distributed representation technologies, retrieval schemes based on word embedding have gradually emerged. These schemes learn to map words to a low-dimensional vector space, bringing semantically similar words closer together and matching them based on vector similarity. While these schemes offer significant improvements, they still have considerable limitations when processing power engineering documents. The core deficiency lies in the fact that word vector models trained on general corpora struggle to deeply understand the specialized vocabulary, subtle differences between concepts, and complex relationships specific to the power industry. Their "generalization" ability is insufficient to support the "vertical" depth of understanding required in the power field. This results in insufficient word disambiguation and deep semantic reasoning capabilities within the power industry's professional context. Furthermore, word vector models cannot explicitly represent the complex structured relationships between entities in the power industry, making it difficult for the models to perform complex semantic matching and reasoning based on domain logic or rules. Faced with multi-step reasoning or highly abstract expert queries, single word vector matching is insufficient to effectively aggregate and parse query semantics, often failing to retrieve the most accurate documents. Finally, purely text-based word vector models cannot effectively utilize the non-textual structured multimodal information (such as charts and standard codes) contained in power engineering documents, thus limiting their application in comprehensive document retrieval. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a document retrieval method, system, and electronic device based on power semantic enhancement, in order to solve the problems of low retrieval accuracy caused by low utilization of complex knowledge and insufficient understanding of power professional semantics.

[0007] On one hand, embodiments of the present invention provide a document retrieval method based on power semantic enhancement, comprising the following steps:

[0008] Receive user query content and obtain query embedding vector using modal joint embedding model; based on power knowledge graph, obtain structured query vector of user query content using large language model and text embedding model;

[0009] Based on the query embedding vector and structured query vector of the user query content, multiple candidate documents and their document-level similarity scores and page-level similarity scores are obtained. Then, a dual-path prediction model is used to obtain the comprehensive similarity score of each candidate document.

[0010] The total score is calculated based on the document-level similarity score, page-level similarity score, and overall similarity score of each candidate document. The candidate documents with the highest total scores are then selected as the search results.

[0011] Based on further improvements to the above method, multiple candidate documents and their document-level similarity scores and page-level similarity scores are obtained according to the query embedding vector and structured query vector of the user query content, including:

[0012] Based on the similarity between the structured semantic vector of each power document and the structured query vector of the user query content, obtain multiple first similar documents and their document-level similarity scores;

[0013] Based on the similarity between the page embedding vector of each power document and the query embedding vector of the user query content, multiple second similar documents and their page-based semantic vectors and page-level similarity scores are obtained.

[0014] Multiple first-similar documents and multiple second-similar documents are merged to obtain candidate documents; based on the query embedding vector and structured query vector of the user query content, the document-level similarity score, page-based semantic vector and page-level similarity score of each candidate document are obtained.

[0015] Based on further improvements to the above method, multiple second-similar documents, their paged semantic vectors, and page-level similarity scores are obtained according to the similarity between the query embedding vector of the user's query content and the embedding vector of each page of each power document, including:

[0016] Obtain the page embedding vectors with similarity greater than the page similarity threshold to get the page to be processed; based on the power document where the page to be processed is located, obtain the second similar document by deduplication;

[0017] For each second-similar document, obtain multiple neighboring pages of the page to be processed with the highest similarity. Take the average of the page embedding vectors and similarities of the multiple neighboring pages to obtain the page-based semantic vector and page-level similarity score.

[0018] Further improvements to the above method, based on the power knowledge graph, utilize a large language model and a text embedding model to obtain structured query vectors for user queries, including:

[0019] Based on elements with hierarchical nesting relationships in the power knowledge graph, we construct and extract the content of the elements and organize the elements and content into JSON format prompt words according to the hierarchical nesting relationship. We then use a large language model to obtain the JSON format query results corresponding to the user's query content.

[0020] The text embedding model is used to convert JSON-formatted query results into structured query vectors.

[0021] Further improvements to the above method utilize a text embedding model to convert JSON-formatted query results into structured query vectors, including:

[0022] Each key-value pair in the JSON-formatted query results is encoded into a path string, which is then input into the text embedding model to obtain the embedding vector for each path string. The average of the embedding vectors of all path strings is then used to obtain the structured query vector.

[0023] Based on the further improvement of the above method, the embedding vector of each page of each power document is obtained by preprocessing each page of each power document and then inputting each page into the modal joint embedding model.

[0024] Based on further improvements to the above method, the modal joint embedding model includes: a text encoder, an image encoder, and a cross-modal fusion module; wherein, the text encoder and the image encoder extract feature vectors from the input text and image respectively, and then pass them to the cross-modal fusion module to align and fuse the two feature vectors.

[0025] Based on further improvements to the above method, the loss function of the modal joint embedding model is a weighted sum of the text contrast learning loss of the text encoder, the image contrast learning loss of the image encoder, and the cross-modal alignment loss of the cross-modal fusion module; the cross-modal contrast loss includes: text-to-image alignment loss and image-to-text alignment loss.

[0026] Based on the above method, in the modal joint embedding model, the text encoder adopts the BGE model, and the text contrastive learning loss adopts the multi-negative sample ranking loss function.

[0027] The image encoder uses the Vit model, and the image contrast learning loss uses the SimCLR loss function.

[0028] The cross-modal fusion module uses the Transformer model, and the cross-modal contrast loss uses the InfoNCE loss function.

[0029] Based on the further improvement of the above method, the dual-path prediction model includes two parallel encoder branches and a fully connected layer; wherein, both the first encoder branch and the second encoder branch adopt the Transformer model, and the fully connected layer is used to fuse the aggregated representation of the outputs of the two encoder branches and map it to a score.

[0030] Based on further improvements to the above method, a dual-path prediction model is used to obtain the comprehensive similarity score for each candidate document, including:

[0031] The query embedding vector of the user's query content and the structured semantic vector of each candidate document are concatenated and fed into the first encoder branch;

[0032] The query embedding vector of the user's query content and the paged semantic vector of the corresponding candidate document are concatenated and passed into the second encoder branch;

[0033] The two encoder branches output two aggregate representations through their respective self-attention mechanisms and pass them to the fully connected layer. After fusion by the fully connected layer, the composite similarity score is mapped and output.

[0034] Based on the further improvements to the above method, the formula for the loss function of the dual-path prediction model is as follows:

[0035] ,

[0036] Where L represents the loss function of the dual-path prediction model, Indicates the first The set of candidate samples corresponding to each query sample and They represent The Middle The candidate sample and the first The candidate samples respectively for the first The relevance level of each query sample and They represent The Middle The candidate sample and the first The predicted score of each candidate sample. Represents the weighting factor, calculated... The Middle The candidate sample and the first The result is obtained by taking the absolute value of the change in the cumulative normalized loss before and after the position swap of each candidate sample.

[0037] Based on the further improvement of the above method, the modal joint embedding model is obtained by training on text data and image data obtained by data preprocessing and modal separation of various power documents in the power corpus;

[0038] Text data includes: text content extracted from Word, PDF and scanned documents, text content in JSON or Markdown format obtained from Excel spreadsheets, and descriptive or summary text content generated using large language models;

[0039] Image data includes: equipment photos and CAD drawings.

[0040] Based on the above method, the structured semantic vector of each power document is also obtained by using a large language model and a text embedding model, based on the power knowledge graph.

[0041] Based on further improvements to the above methods, the hierarchy in the power knowledge graph includes, in order: project phase, equipment type, technical specifications, specification / standard category, risk / failure type, document type, and professional field; each level includes multiple elements.

[0042] Based on the further improvement of the above method, the similarity between the structured query vector and the structured semantic vector of each power document, as well as the similarity between the query embedding vector and the embedding vector of each page of each power document, are both cosine similarities.

[0043] Based on the further improvement of the above method, the total score is obtained by normalizing the document-level similarity score, page-level similarity score and comprehensive similarity score of each candidate document, and then summing them by weight.

[0044] Based on further improvements to the above methods, the power corpus includes: project materials, element rule materials, and domain materials. Project materials are documents in various heterogeneous formats generated during the technical and economic review process. Element rule materials include: technical standards, design guidelines, construction specifications, and equipment selection recommendations issued by the power grid company. Domain materials include: project regulations, technical standards, and academic papers.

[0045] On the other hand, embodiments of the present invention provide a document retrieval system based on power semantic enhancement, comprising:

[0046] The query feature extraction module receives user query content and obtains query embedding vectors using a modal joint embedding model; based on the power knowledge graph, it uses a large language model and a text embedding model to obtain structured query vectors of user query content.

[0047] The similarity retrieval module is used to obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores based on the query embedding vector and structured query vector of the user's query content, and then use a dual-path prediction model to obtain the comprehensive similarity score of each candidate document.

[0048] The search results display module is used to calculate the total score based on the document-level similarity score, page-level similarity score, and comprehensive similarity score of each candidate document, and select the candidate documents with the highest total scores as the search results.

[0049] This invention also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it executes any of the above-mentioned document retrieval methods based on power semantic enhancement.

[0050] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0051] 1. By integrating large language models, multimodal large models, power knowledge graphs, and multi-level document similarity mechanisms, this approach overcomes the limitations of traditional keyword or general word vector matching, achieving deep semantic understanding and matching of power-related documents. Specifically, it obtains candidate documents and similarity scores at different levels through page-by-page comparison and structured matching driven by the power knowledge graph. Then, it predicts a comprehensive similarity score by fusing features from the candidate documents. Finally, it weights the similarity scores at each level and the comprehensive similarity score to obtain a total score. The order of the candidate documents is then rearranged to obtain the final retrieval results, improving retrieval accuracy and comprehensively enhancing the efficiency of power grid companies in acquiring information from massive amounts of power documents.

[0052] 2. By constructing a modal joint embedding model and extracting page-level embedding vectors, the applicability and recall effect of document retrieval are increased, improving the applicability of the original document retrieval strategy from 20% to 90%.

[0053] 3. By constructing a power knowledge graph and combining it with the powerful semantic understanding capabilities of a pre-trained large language model, we can achieve deep semantic understanding and matching of power professional documents. By focusing on the document's logical structure and overall information, we can further increase matching efficiency and recall effect, improving the Top-5 recall accuracy from 10% to 70%.

[0054] 4. By retrieving documents through embedded vector retrieval on the page, combined with knowledge graph-driven enhanced document matching and the predictive capabilities of neural network models, a powerful and highly intelligent power semantic-enhanced document similarity retrieval system has been achieved. This significantly enhances the intelligence level of the retrieval, reduces manual costs, and enables more efficient and accurate auxiliary review, better serving the daily operation, fault diagnosis, planning and design and other business needs of power grid companies.

[0055] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0056] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0057] Figure 1 This is a flowchart of a document retrieval method based on power semantic enhancement in Embodiment 1 of the present invention;

[0058] Figure 2This is a schematic diagram of the modal joint embedding model in Embodiment 1 of the present invention;

[0059] Figure 3 This is a flowchart of the method for obtaining candidate documents and their similarity scores at different levels in Embodiment 1 of the present invention;

[0060] Figure 4 This is a schematic diagram illustrating the process of obtaining document-level similarity scores in Embodiment 1 of the present invention;

[0061] Figure 5 This is a schematic diagram of a document retrieval system based on power semantic enhancement in Embodiment 2 of the present invention. Detailed Implementation

[0062] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the invention, but are not intended to limit the scope of the invention. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0063] Those skilled in the art should understand that the following specific embodiments or implementation methods are a series of optimized configurations listed to further explain the specific content of the invention. These configurations can be combined or used in conjunction with each other, unless the invention explicitly states that some or a specific embodiment or implementation method cannot be associated with or used in conjunction with other embodiments or implementation methods. Furthermore, the following specific embodiments or implementation methods are merely optimized configurations and are not intended to limit the scope of protection of the invention.

[0064] Example 1

[0065] A specific embodiment of the present invention discloses a document retrieval method based on power semantic enhancement, such as... Figure 1 As shown, it includes the following steps:

[0066] S1. Receive user query content and obtain query embedding vector using modal joint embedding model; based on power knowledge graph, obtain structured query vector of user query content using large language model and text embedding model;

[0067] S2. Based on the query embedding vector and structured query vector of the user query content, obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores, and then use the dual-path prediction model to obtain the comprehensive similarity score of each candidate document.

[0068] S3. Calculate the total score based on the document-level similarity score, page-level similarity score, and overall similarity score of each candidate document, and select the candidate documents with the highest total scores as the search results.

[0069] During implementation, by integrating large language models, multimodal large models, power knowledge graphs, and multi-level document similarity mechanisms, the limitations of traditional keyword or general word vector matching are overcome, enabling deep semantic understanding and matching of power-related documents. Specifically, page-by-page comparison and structured matching driven by the power knowledge graph are used to obtain candidate documents and similarity scores at different levels. Then, by fusing the features of the candidate documents, a comprehensive similarity score is predicted. Finally, the weighted similarity scores at each level and the comprehensive similarity score are summed to obtain the total score. The order of the candidate documents is then rearranged to obtain the final retrieval results, improving retrieval accuracy and comprehensively enhancing the efficiency of power grid companies in acquiring information from massive amounts of power documents.

[0070] It should be noted that document retrieval involves finding the most similar power documents among various power documents collected from the power corpus. The power documents in the power corpus include: project materials, element rule materials, and domain materials.

[0071] The project materials consist of documents in various heterogeneous formats generated during the technical and economic review process. A complete set of project materials includes documents in various heterogeneous formats such as Word text, Excel spreadsheets, PDF reports, CAD drawings, scanned copies, and equipment photos.

[0072] The essential rules and materials include: technical standards, design guidelines, construction specifications, and recommended parameters for equipment selection issued by the power grid company.

[0073] The materials in this field include: project procedures, technical standards, and academic papers.

[0074] This embodiment constructs a modal joint embedding model to map data from different modalities (text, tables, images, CAD, etc.) into a unified semantic vector space, forming unique semantics in the power field, thereby achieving cross-modal similarity matching.

[0075] First, the text and image data obtained by preprocessing and modal separation of power documents in the power corpus are used to facilitate the training of the modal joint embedding model.

[0076] The text data includes: text content extracted from Word, PDF and scanned documents, text content in JSON or Markdown format obtained from Excel spreadsheets, and descriptive or summarizing text content generated using a large language model; the image data includes: equipment photos and CAD drawings.

[0077] Specifically, the text data is obtained in the following ways:

[0078] The LayoutLM deep learning-based document layout analysis model is used to perform fine layout analysis on various power documents to accurately distinguish between text, table and image areas.

[0079] The python-docx and PyPDF2 tools were used to extract text content from text areas in Word and PDF documents, respectively.

[0080] PaddleOCR was used to extract text content from scanned documents, equipment photos, CAD drawings, and image areas in Word and PDF documents.

[0081] It should be noted that manual annotation can be used to classify equipment photos (such as transformers, circuit breakers, etc.) and add detailed annotations so that key information such as equipment model and installation location can be extracted from the equipment photos when extracting text data; for CAD drawings, equipment nameplates, legends, text annotations or structural information can be extracted to obtain semantically enhanced content.

[0082] Use tools such as Pandas to convert Excel tables into JSON format text content;

[0083] Tools such as MarkItDown are used to extract table content from table areas in Word and PDF, including row, column, cell and header information, and convert it into Markdown format text content;

[0084] The table area is analyzed using a large language model to generate descriptive text content containing table content and header attributes; for example: "Parameter: Voltage level, Value: 110kV";

[0085] Large language models or multimodal large models are used to process documents containing multiple modalities into summative text content.

[0086] By writing regular expressions and verifying with a large language model, the extracted text content is automatically cleaned to remove non-semantic information such as special symbols and garbled characters, resulting in the final text data.

[0087] For example, the multimodal large model adopts the Qwen-VL model.

[0088] It should be noted that the equipment photos and CAD drawings are processed by denoising, binarization, and resizing before being used as image data.

[0089] Secondly, a strategy of using different modal encoders for processing and then performing cross-modal feature fusion and alignment is adopted to construct an integrated modal joint embedding model.

[0090] Specifically, such as Figure 2 As shown, the modal joint embedding model includes a text encoder, an image encoder, and a cross-modal fusion module. The text encoder and the image encoder extract feature vectors from the input text and image, respectively, and then pass them to the cross-modal fusion module to align and fuse the two feature vectors to obtain a multimodal embedding vector.

[0091] (a) Text encoder

[0092] The text encoder uses a BGE model, such as the bge-base-zh-v1.5 model, and fine-tunes the text encoder based on Sentence Transformers (a Python library based on PyTorch and Transformers) using text data. It employs a contrastive learning training strategy to train the text encoder.

[0093] Before training, text segments with semantic similarity or relevance in the text data are identified through a combination of regular expressions, expert annotations, and large language models, and these are constructed as positive sample pairs. That is, the content of positive sample pairs includes: statements in different chapters of the same project material that describe the same equipment or process, and different expressions in the element rule material that have the same technical specifications or describe the same concept.

[0094] For example, regular expressions are used to identify similarity or relevance of document metadata (titles, chapter names, etc.); experts manually mark the data as examples, prompts are used to describe the extraction principles, and the authors are required to quickly analyze and expand the data according to the examples. A large language model is then used to identify related text, and finally, unreasonable parts are manually filtered out.

[0095] During training, the multi-negative sample ranking loss function is used as the text contrastive learning loss to fine-tune the text encoder. The multi-negative sample ranking loss function combines all samples in a batch into pairs, treating all pairs except positive ones as negative pairs, and calculates the cross-entropy loss. It is particularly suitable for semantic retrieval and information recall tasks.

[0096] Specifically, the text contrastive learning loss uses the multi-negative-sample ranking loss function. The formula is shown below:

[0097] ,

[0098] in, and These represent the first in the batch. text and its positive samples ;

[0099] Indicates batch size;

[0100] This represents the text embedding vector output by the text encoder;

[0101] This indicates the calculation of cosine similarity;

[0102] Represents an exponential function;

[0103] This represents the temperature coefficient, which is set to 0.07 here.

[0104] The denominator represents the sum of the cosine similarities between the embedding vectors of all sample pairs in the batch, after exponential and temperature scaling.

[0105] (b) Image encoder

[0106] The image encoder uses the Vit (Vision Transformer) model to segment the image data in the power document into multiple image blocks of fixed size. Then, each image block is unfolded into a one-dimensional vector. The image block embedding is generated by linear projection and positional encoding is added to preserve the spatial information of the image. Finally, the image encoder efficiently transforms the image data into an image embedding vector aligned with the text modality.

[0107] The image encoder also employs a contrastive learning training strategy, using the SimCLR loss function as the image contrastive learning loss for fine-tuning training.

[0108] When constructing positive and negative sample pairs, a self-supervised contrastive learning method is adopted, which uses the enhanced version of the image itself to construct positive sample pairs and constructs negative samples through in-batch negative sampling.

[0109] Specifically, for each image in the batch, an enhanced image is generated using random data augmentation methods, such as random cropping, color jittering, and flipping. The original image and its enhanced image are considered positive sample pairs. For each original image, all other images in the batch, as well as the enhanced images of those other images, are considered negative samples.

[0110] The image contrast learning loss uses the SimCLR loss function. The formula is shown below:

[0111] ,

[0112] in, and These represent the first in the batch. One image and its enhanced image ;

[0113] This represents the first image in a batch, along with the enhanced image of each image. One image;

[0114] This represents the image embedding vector output by the image encoder;

[0115] Indicates an indicator function, when It returns 1 if the condition is met, otherwise it returns 0, which is used to exclude the case where each image is compared with its own image;

[0116] Indicates the batch The total number obtained after enhancing each image.

[0117] (c) Cross-modal fusion module

[0118] The cross-modal fusion module uses the Transformer model, which leverages its multi-head self-attention mechanism to interact with features from text and image modalities, bringing semantically related samples from different modalities closer to each other in a shared embedding space.

[0119] The cross-modal fusion module also adopts a contrastive learning training strategy, using the InfoNCE loss function as the cross-modal contrastive loss for training. This effectively promotes the mutual attraction of semantically related cross-modal embeddings while repelling irrelevant embeddings.

[0120] Positive sample pairs during training Data originates from different modalities within the same project materials. For example, in a power project document, the text descriptions, equipment photos, and parameter tables contain inherent semantic relationships. For cases with only single-modal data, a zero-vector imputation strategy is employed for the other modality of the sample pair, enabling the data to pass smoothly through the cross-modal fusion module and effectively encode even single-modal data.

[0121] Negative sample pairs are derived from different modal data in materials from different projects, avoiding zero-vector padding. For example, images from project A are randomly combined with text from project B to form negative sample pairs. ,in .

[0122] In this embodiment, the cross-modal contrast loss includes: text-to-image alignment loss and image-to-text alignment loss, that is, considering the matching relationship in both the text-to-image and image-to-text directions.

[0123] Specifically, in a batch Image-text positive sample pairs Image-to-text alignment loss This is used to measure how close each image embedding is to its matching text embedding, while keeping it away from all other text embeddings in the batch. The formula is as follows:

[0124] .

[0125] Text-to-image alignment loss This is used to measure how close each text embedding is to its matching image embedding, while keeping it away from all other image embeddings in the batch. The formula is as follows:

[0126] .

[0127] The cross-modal contrast loss is obtained by adding the image-to-text alignment loss to the text-to-image alignment loss. The formula is shown below:

[0128] .

[0129] After the text encoder and image encoder are fine-tuned and trained separately, they are trained end-to-end again together with the cross-modal fusion module. The total loss function is the weighted sum of the text contrast learning loss of the text encoder, the image contrast learning loss of the image encoder, and the cross-modal alignment loss of the cross-modal fusion module, as shown in the following formula:

[0130] ,

[0131] in, The loss function represents the modal joint embedding model;

[0132] , and These represent the weights of text contrastive learning loss, image contrastive learning loss, and cross-modal alignment loss, respectively. By default, they are set to 1, but can be adjusted according to the actual situation.

[0133] After training, the resulting modal joint embedding model can map any input plain text, plain image, or combination of text and image to a unified, low-dimensional semantic vector space to represent multimodal embedding vectors.

[0134] It should be noted that in existing technologies, the various modalities in a document are typically extracted, and the embedding representation of the entire document is directly obtained using a modal joint embedding model. However, this embodiment takes into account the complexity of power documents, preprocessing each power document page by page. Single-modal or multi-modal data is extracted based on the content of each page and then input into the modal joint embedding model to obtain the embedding vector for each page of each power document. Subsequently, the embedding vector of the document is determined based on the retrieved page embedding vectors.

[0135] Furthermore, to improve the accuracy of retrieval, this embodiment constructs a power knowledge graph and combines it with the powerful semantic understanding capabilities of a pre-trained Large Language Model (LLM) to achieve deep semantic understanding and matching of power-related documents, focusing on the document's logical structure and overall information.

[0136] It should be noted that the power knowledge graph is a high-concept knowledge graph (HCKG) customized for the power field, which embodies a lightweight, high-level knowledge system for the power field.

[0137] The schema of the power knowledge graph is mainly based on the element rule materials in the power corpus. It defines the high-level conceptual elements with core significance in power documents and represents the organizational relationship between these high-level conceptual elements through hierarchical nesting relationships.

[0138] In this embodiment, the hierarchy of the power knowledge graph includes, in order: project stage, equipment type, technical specifications, specification / standard category, risk / failure type, document type, and professional field; each level includes multiple elements, as shown below:

[0139] Project phases: design, construction, acceptance, operation, maintenance, and repair;

[0140] Equipment types: transformers, switches, lines, protection devices, control systems;

[0141] Technical specifications: voltage level, current parameters, insulation level, reliability indicators, and test data;

[0142] Categories of specifications / standards: design specifications, construction specifications, operation procedures, acceptance standards, safety procedures;

[0143] Risk / failure types: short circuit, overload, grounding, abnormal operation, natural disaster;

[0144] File types: design documents, construction drawings, test reports, acceptance reports, operation logs;

[0145] Areas of expertise: power systems, relay protection, high-voltage electrical systems, and communication automation.

[0146] It should be noted that the different levels do not necessarily have to be nested one by one. For example, the project phase (construction) can be directly followed by the specification category (Safety Construction Specification DL / T XXXXX).

[0147] Furthermore, the power knowledge graph is used as a guiding framework for information extraction and structured representation by the large language model, and the knowledge framework is learned and improved by the large language model.

[0148] Specifically, through carefully designed few-shot learning and cue word engineering, a small number of power document examples and their corresponding expected JSON structures are provided to the large language model. Based on these examples, the model learns how to identify the hierarchy (key) of elements in the text and fill in the corresponding values ​​(value) for the elements. Expert manual proofreading and feedback are then provided based on the extraction results from the large language model to continuously optimize the cue words and examples until the large language model can accurately identify the keys and generate values ​​for elements that conform to the power knowledge graph.

[0149] For example, the constructed prompt is: "Please extract key information from the following text according to the structure of power industry technical documents and organize it into JSON format. Focus on core elements and their specific content such as 'Project Phase', 'Equipment Type', 'Technical Specifications', and 'Specification / Standard Category'. The output format should be similar to: {'Equipment Type':{'Transformer':{'Technical Specifications':['220kV','50MVA']}}}."

[0150] It should be noted that traditional knowledge graphs maintain relationships between entities based on various relationships. However, in this embodiment, the inherent knowledge and logical reasoning capabilities of the large language model are fully utilized, avoiding the explicit definition of trivial and common-sense relationships found in traditional knowledge graphs, thereby reducing construction and maintenance costs.

[0151] After constructing the power knowledge graph and pre-training the large language model based on the power knowledge graph, the key information of each power document in the power corpus is extracted in advance using the large language model. That is, the elements and content in the power knowledge graph are organized into JSON format according to the hierarchical nesting relationship.

[0152] It should be noted that the key information extracted includes:

[0153] The design includes clear prompts that instruct the large language model to identify the level of each element in the given text based on the hierarchical nesting relationships of the elements in the power knowledge graph, and to extract the associated values ​​of each element.

[0154] With its powerful semantic understanding and reasoning capabilities, the large language model can infer and fill in values ​​based on context and inherent knowledge, even if some values ​​are not explicitly labeled in the text. For complex or long texts, a multi-round extraction strategy is adopted to gradually refine and improve the values ​​in the JSON structure.

[0155] For example, the first round of extraction identifies the "equipment type" and "technical specifications", and the second round extracts the specific parameters of the "technical specifications".

[0156] Ultimately, each power document in the power corpus is converted into one or more key information in JSON format that conforms to the power knowledge graph, resulting in one or more structured data corresponding to each power document.

[0157] Furthermore, based on the structured data corresponding to each power document, a structured semantic vector for each power document is obtained, providing deeper semantic association information for the power documents to improve retrieval accuracy.

[0158] Specifically, the structured semantic vector for each electricity document is obtained through the following steps:

[0159] ① Path encoding:

[0160] Each key-value pair in the JSON-formatted structured data is encoded into a path string. Specifically, for each value in the structured data, its corresponding nested key in the JSON format is expanded into a single dimension, concatenated with the value, and then a unique path string is generated. This string preserves the hierarchical relationship while facilitating pairwise comparisons.

[0161] For example, the value in “{'Equipment Type':{'Transformer':{'Technical Specifications':['220kV','50MVA']}}}” is: ['220kV','50MVA']. The nested hierarchical relationship is represented by dots, and the final encoding is: Equipment Type.Transformer.Technical Specifications:['220kV','50MVA'].

[0162] ②Embedded generation:

[0163] Each path string is input into the text embedding model in turn to obtain the embedding vector of each path string.

[0164] It should be noted that since the string input to the text embedding model is extracted from a large language model based on the power knowledge graph, its semantic space is compact and clear, and the resulting path encoding has a high semantic level. Therefore, there is no need to fine-tune the text embedding model. A general text embedding model, such as the BGE-Base-zh-v1.5 model, can be used directly.

[0165] ③Embedded aggregation:

[0166] The average of the embedding vectors of all path strings corresponding to each power document is taken to obtain the structured semantic vector of each power document, which is expressed by the following formula:

[0167] ,

[0168] in, Indicates the first The structured semantic vector of each power document;

[0169] Indicates the first The number of string paths generated by the structured data encoding corresponding to each power document;

[0170] express The Middle An embedding vector of a string path.

[0171] By using the trained modal joint embedding model to obtain the embedding vector of each page of each power document, and by using the constructed power knowledge graph, the pre-trained large language model and the general text embedding model to obtain the structured semantic vector of each power document, the basic offline data preparation is completed.

[0172] After receiving the user's query content in step S1, the system identifies the data of different modalities in the user's query content, and inputs the data of the text modality and the image modality (if it is a single modality, the other modality is filled with zero vector) into the trained modality joint embedding model to obtain the query embedding vector.

[0173] Furthermore, based on the power knowledge graph, a large language model and a text embedding model are used to obtain the structured query vector of the user's query content. The method of obtaining the structured semantic vector of each electronic document is similar to that described above, and is specifically carried out through the following steps:

[0174] Based on elements with hierarchical nesting relationships in the power knowledge graph, we construct and extract the content of the elements and organize the elements and content into JSON format prompt words according to the hierarchical nesting relationship. We then use a large language model to obtain the JSON format query results corresponding to the user's query content.

[0175] The text embedding model is used to convert JSON-formatted query results into structured query vectors.

[0176] It's important to note that the large language model doesn't directly extract elements from the user query. Instead, it first infers the query results based on the user query, then extracts the elements and content from the results and organizes them into JSON format. The main purpose of this is to unify the input user query and the retrieved power documents under the constraints of the power knowledge graph, while also adhering to the semantic space of path encoding, facilitating semantic matching that enhances conceptual hierarchy and logical structure.

[0177] For example, if a user wants to query whether the construction details mentioned in the project materials meet the specifications, the large language model infers the query results from the power corpus based on the power knowledge graph, and organizes the query results into a unified JSON format, similar to answering the user's query in JSON format based on the power knowledge graph. The obtained query results are not required to be completely accurate in content; the actual values ​​in the power documents may differ from the inferred query results. However, converting the query results into vectors based on this format makes it easier to match precise content.

[0178] The text embedding model is used to convert JSON-formatted query results into structured query vectors, specifically following the path encoding, embedding generation, and embedding aggregation methods described above.

[0179] Each key-value pair in the JSON-formatted query results is encoded into a path string, which is then input into the text embedding model to obtain the embedding vector for each path string. The average of the embedding vectors of all path strings is then used to obtain the structured query vector.

[0180] In step S2, based on the query embedding vector and structured query vector of the user's query content, similar power documents are retrieved from the power corpus as candidate documents through steps S21-S23, and document-level and page-level similarity scores are obtained. In step S24, the dual-path prediction model efficiently integrates the high-order structured knowledge (structured semantic vector) and fine-grained page semantics (page-based semantic vector) of the candidate documents to predict the comprehensive similarity score of the candidate documents, comprehensively characterizing the relevance between the candidate documents and the user's query content, thereby significantly improving the accuracy of the retrieval structure.

[0181] like Figure 3 As shown, step S2 specifically includes:

[0182] S21. Based on the similarity between the structured semantic vector of each power document and the structured query vector of the user query content, obtain multiple first similar documents and their document-level similarity scores.

[0183] It should be noted that the structured semantic vector of each power document was obtained by using a pre-trained large language model and text embedding model after the power knowledge graph was constructed.

[0184] like Figure 4 As shown, the similarity between the structured semantic vector of each power document and the structured query vector of the user query content is obtained by calculating the cosine similarity, and the calculation formula is as follows:

[0185] ,

[0186] in, Indicates the first One power document Structured semantic vectors Structured query vectors of user query content Similarity;

[0187] This indicates the calculation of the dot product of two vectors;

[0188] This represents the Euclidean norm of a vector.

[0189] Furthermore, power documents with a similarity greater than a preset document similarity threshold are taken as the first similar documents, and the similarity calculated at this time is the document-level similarity score.

[0190] Document-level similarity scores reflect higher-order similarity at the conceptual, relational, and logical levels of documents, integrating dimensions such as key element weighted matching, hierarchical path precise matching, and value semantic similarity.

[0191] S22. Based on the similarity between the page embedding vector of each power document and the query embedding vector of the user query content, obtain multiple second similar documents and their page-based semantic vectors and page-level similarity scores.

[0192] It should be noted that the similarity between the embedding vector of each page of each power document and the query embedding vector of the user query content is also obtained by calculating the cosine similarity.

[0193] Furthermore, based on the calculated similarity of each page, aggregated features are extracted to reflect the overall page relevance of the document, which serve as the page-level similarity of the document, including: taking the maximum similarity, average similarity, and average similarity of neighboring pages.

[0194] In practical use, the required page embedding vectors are selected based on the page similarity threshold. Multiple page embedding vectors may correspond to pages belonging to the same document. Therefore, the page embedding vector aggregation method should be flexibly selected according to the specific task.

[0195] This embodiment prioritizes using the strategy of average similarity of neighboring pages to obtain multiple second-similar documents and their paged semantic vectors and page-level similarity scores.

[0196] Specifically, it can be obtained through the following steps:

[0197] Obtain the page embedding vectors with similarity greater than the page similarity threshold to get the page to be processed; based on the power document where the page to be processed is located, obtain the second similar document by deduplication;

[0198] For each second-similar document, obtain multiple neighboring pages of the page to be processed with the highest similarity. Take the average of the page embedding vectors and similarities of the multiple neighboring pages to obtain the page-based semantic vector and page-level similarity score.

[0199] Page-level similarity scores reflect higher-order similarity at the document multimodal and contextual levels, integrating dimensions such as key page similarity and its contextual page similarity.

[0200] S23. Merge multiple first-similar documents and multiple second-similar documents to obtain candidate documents; based on the query embedding vector and structured query vector of the user query content, obtain the document-level similarity score, page-based semantic vector and page-level similarity score of each candidate document.

[0201] It should be noted that multiple first-similar documents and multiple second-similar documents may contain partially identical power documents, or they may be completely different. Therefore, all candidate documents are obtained by taking the union of the two.

[0202] For each candidate document, the following features need to be extracted: document-level similarity score, page-based semantic vector, and page-level similarity score.

[0203] If a candidate document is both the first and second most similar document, no further processing is required.

[0204] If the candidate document is only the first similar document obtained through step S21, then the paged semantic vector and page-level similarity score are obtained according to the method in step S22, that is:

[0205] Calculate the similarity between the page embedding vector of each candidate document and the query embedding vector of the user query content, and obtain the page-based semantic vector and page-level similarity score by the average similarity of neighboring pages;

[0206] If the candidate document is only the second similar document obtained through step S22, then the document-level similarity score is obtained according to the method of step S21, that is:

[0207] Calculate the similarity between the candidate structured semantic vector and the structured query vector of the user query content.

[0208] S24. Based on the query embedding vector of the user's query content, and the structured semantic vector and paged semantic vector of each candidate document, a dual-path prediction model is used to obtain the comprehensive similarity score of each candidate document.

[0209] It should be noted that the dual-path prediction model is a deep learning model based on the Transformer architecture, which adopts a dual-input cross-coding structure, including two parallel encoder branches and a fully connected layer. The first encoder branch and the second encoder branch both adopt the Transformer model, and the fully connected layer is used to fuse the aggregated representation of the outputs of the two encoder branches and map it to a score.

[0210] Before training the dual-path prediction model, a dataset for training the model is constructed, including:

[0211] Multiple query samples are set up. For each query sample, the query embedding vector and structured query vector are obtained according to the methods described above. Multiple candidate samples are obtained based on similarity, along with the structured query vector and paged semantic vector for each candidate sample. Power industry experts then label each candidate sample with a relevance level based on its correlation with the corresponding query sample, including: 0: irrelevant, 1: weakly relevant, 2: moderately relevant, 3: strongly relevant, and 4: perfect match. The query samples, candidate samples, and labels are then placed into the dataset.

[0212] Furthermore, each query sample is paired with its corresponding candidate sample. For each sample pair, the query embedding vector of the query sample and the structured semantic vector of the candidate sample are concatenated and fed into the first encoder branch to focus on the macro-themes and structural relationships of the document, outputting a first aggregate representation. The query embedding vector of the query sample and the page-based semantic vector of the candidate sample are concatenated and fed into the second encoder branch to focus on the fusion of semantic and multimodal information within the page, outputting a second aggregate representation. The first and second aggregate representations are fused through a fully connected layer to learn how to balance page-level details and document-level concepts, and finally output a comprehensive relevance score.

[0213] It's important to note that, to improve the model's discriminative ability, the training data includes a large number of hard negative examples. These hard negative examples refer to documents with high similarity scores but low actual relevance, or documents with high page-level similarity scores at the text level but low document-level similarity scores due to structural and semantic mismatches. By selecting these types of samples, the model is forced to learn more refined discrimination criteria.

[0214] For a query sample and its related retrieved candidate samples, the goal of the dual-path prediction model is to capture the relative ranking relationship between the candidate samples of each query sample, so that the candidate samples with higher relevance are ranked higher in the ranking list.

[0215] The loss function of the dual-path prediction model in this embodiment is obtained by weighting the score errors between candidate samples using the absolute value of the change in the normalized loss cumulative gain as the weight, as shown in the following formula:

[0216] ,

[0217] Where L represents the loss function of the dual-path prediction model, Indicates the first The set of candidate samples corresponding to each query sample;

[0218] and They represent The Middle The candidate sample and the first The candidate samples respectively for the first The relevance level of each query sample;

[0219] and They represent The Middle The candidate sample and the first The predicted score of each candidate sample;

[0220] Represents the weighting factor, calculated... The Middle The candidate sample and the first The result is obtained by taking the absolute value of the change in the cumulative normalized loss before and after the position swap of each candidate sample.

[0221] Furthermore, It is calculated using the following formula:

[0222] ,

[0223] in, and They represent The Middle The candidate sample and the first The location of each candidate sample; Indicates the first The maximum cumulative gain of a single query sample.

[0224] It should be noted that the first The cumulative gain of a query sample will vary depending on the order in which multiple candidate samples are sorted. This can be calculated using the following formula:

[0225] ,

[0226] in, This represents the cumulative gain resulting from the current ranking of candidate samples.

[0227] Indicates the first The candidate sample set of the query sample. The candidate sample pairs The relevance level of each query sample;

[0228] Indicates the th candidate sample in the current ranking. The location of each candidate sample.

[0229] As can be seen from the above formula, the magnitude of the weight factor in the loss function depends on two factors: first, the difference in the true relevance levels of the two candidate samples that swap positions; the greater the difference, the greater the weight; and second, the ranking position of the two candidate samples. This allows the model to focus its training efforts on correcting the ranking errors that have the greatest impact on the cumulative gain of the normalization loss.

[0230] After the dual-path prediction model is trained, the comprehensive similarity score of each candidate document is obtained using the dual-path prediction model, including:

[0231] The query embedding vector of the user query content and the structured semantic vector of each candidate document are concatenated and fed into the first encoder branch; the query embedding vector of the user query content and the paged semantic vector of the corresponding candidate document are concatenated and fed into the second encoder branch; the two encoder branches output two aggregated representations through their respective self-attention mechanisms and are fed into the fully connected layer. After being fused by the fully connected layer, the comprehensive similarity score is mapped and output.

[0232] In step S3, the total score is obtained based on the document-level similarity score, page-level similarity score, and comprehensive similarity score of each candidate document. First, these three similarity scores are normalized. Preferably, the minimum-maximum normalization process is used to normalize the similarity scores to the [0,1] interval to eliminate the influence of dimensions.

[0233] Furthermore, the total score is obtained by weighted summation using the following formula:

[0234] ,

[0235] in, Indicates the first One power document Total score;

[0236] , and They represent the first One power document Page-level similarity score, document-level similarity score, and overall similarity score;

[0237] , and These represent the weights of the page-level similarity score, document-level similarity score, and overall similarity score, respectively; for example, they are set to 0.3, 0.3, and 0.4, respectively.

[0238] Finally, based on the calculated total score, the candidate documents are sorted, and the top-scoring candidate documents are selected as the search results.

[0239] Compared with existing technologies, this embodiment provides a document retrieval method based on power semantic enhancement. By integrating large language models, multimodal large models, power knowledge graphs, and multi-level document similarity mechanisms, it breaks through the limitations of traditional keyword or general word vector matching, achieving deep semantic understanding and matching of power professional documents. Specifically, it obtains candidate documents and similarity scores at different levels through page-by-page comparison and structured matching driven by the power knowledge graph. Then, it predicts a comprehensive similarity score by fusing the features of the candidate documents. Finally, it weights the similarity scores at each level and the comprehensive similarity score to obtain the total score. The order of the candidate documents is then rearranged to obtain the final retrieval results, improving retrieval accuracy and comprehensively enhancing the information acquisition efficiency of power grid companies from massive amounts of power documents.

[0240] By constructing a modal joint embedding model and extracting page-level embedding vectors, the applicability and recall effect of document retrieval are increased, improving the applicability of the original document retrieval strategy from 20% to 90%.

[0241] By constructing a power knowledge graph and combining it with the powerful semantic understanding capabilities of a pre-trained large language model, we can achieve deep semantic understanding and matching of power-related documents. By focusing on the document's logical structure and overall information, we can further increase matching efficiency and recall effect, improving Top-5 recall accuracy from 10% to 70%.

[0242] By retrieving documents through page-embedded vector retrieval, combined with knowledge graph-driven enhanced document matching, and the predictive capabilities of neural network models, a powerful and highly intelligent power semantic-enhanced document similarity retrieval system has been achieved. This significantly enhances the intelligence level of retrieval, reduces manual costs, and enables more efficient and accurate auxiliary review, better serving the daily operation, fault diagnosis, planning and design, and other business needs of power grid companies.

[0243] Example 2

[0244] Another embodiment of the present invention discloses a document retrieval system based on power semantic enhancement, thereby implementing the document retrieval method based on power semantic enhancement in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. Figure 5 As shown, the system includes:

[0245] The query feature extraction module 101 is used to receive user query content and obtain query embedding vector using a modal joint embedding model; based on the power knowledge graph, it uses a large language model and a text embedding model to obtain the structured query vector of the user query content.

[0246] The similarity retrieval module 102 is used to obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores based on the query embedding vector and structured query vector of the user query content, and then use a dual-path prediction model to obtain the comprehensive similarity score of each candidate document.

[0247] The search results display module 103 is used to obtain a total score based on the document-level similarity score, page-level similarity score, and comprehensive similarity score of each candidate document, and select the candidate documents with the highest total scores as the search results.

[0248] Furthermore, the similarity retrieval module is used to obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores based on the query embedding vector and structured query vector of the user's query content, including:

[0249] Based on the similarity between the structured semantic vector of each power document and the structured query vector of the user query content, obtain multiple first similar documents and their document-level similarity scores;

[0250] Based on the similarity between the page embedding vector of each power document and the query embedding vector of the user query content, multiple second similar documents and their page-based semantic vectors and page-level similarity scores are obtained.

[0251] Multiple first-similar documents and multiple second-similar documents are merged to obtain candidate documents; based on the query embedding vector and structured query vector of the user query content, the document-level similarity score, page-based semantic vector and page-level similarity score of each candidate document are obtained.

[0252] It should be noted that the system in this embodiment also includes:

[0253] The modality joint embedding model training module is used to preprocess the text data and image data obtained from modality separation in the power corpus, and then train the constructed modality joint embedding model.

[0254] The power knowledge graph construction module is used to extract elements from power documents based on the power corpus and represent the organizational relationships between elements through hierarchical nesting.

[0255] Since the document retrieval system based on power semantic enhancement in this embodiment and the aforementioned document retrieval method based on power semantic enhancement can be mutually referenced, this description is redundant and will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.

[0256] Example 3

[0257] Another embodiment of the present invention discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0258] It should be noted that when the processor executes the computer program, it implements the steps of the document retrieval method based on power semantic enhancement in the method embodiment.

[0259] The electronic device in this embodiment can be a server, a terminal, or any other electronic device with the necessary computing and / or processing capabilities.

[0260] In one specific embodiment, the electronic device includes a processor, a memory, and a communication interface connected via a system bus. The processor provides the necessary computing, processing, and / or control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs, and serves as a runtime working area, temporarily storing intermediate and final results. The internal memory provides the operating environment for the operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device is used for network connection and communication with external devices. When the computer program is executed by the processor, it performs the steps of the document retrieval method based on power semantic enhancement in the method embodiment.

[0261] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0262] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A document retrieval method based on electricity semantic enhancement, characterized in that, Includes the following steps: Receive user query content and obtain query embedding vector using modal joint embedding model; based on power knowledge graph, obtain structured query vector of the user query content using large language model and text embedding model; Based on the query embedding vector and structured query vector of the user query content, obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores, including: based on the similarity between the structured semantic vector of each power document and the structured query vector of the user query content, obtain multiple first similar documents and their document-level similarity scores; Based on the similarity between the page embedding vector of each power document and the query embedding vector of the user query content, multiple second similar documents, their page-based semantic vectors, and page-level similarity scores are obtained; multiple first similar documents and multiple second similar documents are merged to obtain candidate documents; based on the query embedding vector and structured query vector of the user query content, the document-level similarity score, page-based semantic vector, and page-level similarity score of each candidate document are obtained; and then a dual-path prediction model is used to obtain the comprehensive similarity score of each candidate document. The total score is calculated based on the document-level similarity score, page-level similarity score, and overall similarity score of each candidate document. The candidate documents with the highest total scores are then selected as the search results.

2. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The step of obtaining multiple second-similar documents and their page-based semantic vectors and page-level similarity scores based on the similarity between the page embedding vector of each power document and the query embedding vector of the user query content includes: Obtain the page embedding vectors with similarity greater than the page similarity threshold to get the page to be processed; based on the power document where the page to be processed is located, obtain the second similar document by deduplication; For each second-similar document, obtain multiple neighboring pages of the page to be processed with the highest similarity. Take the average of the page embedding vectors and similarities of the multiple neighboring pages to obtain the page-based semantic vector and page-level similarity score.

3. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The process of obtaining a structured query vector for the user's query content based on a power knowledge graph, using a large language model and a text embedding model, includes: Based on elements with hierarchical nesting relationships in the power knowledge graph, the content of the extracted elements is constructed and the elements and content are organized into JSON format prompt words according to the hierarchical nesting relationship. The JSON format query results corresponding to the user query content are obtained by using a large language model. The text embedding model is used to convert JSON-formatted query results into structured query vectors.

4. The document retrieval method based on power semantic enhancement according to claim 3, characterized in that, The process of converting JSON-formatted query results into structured query vectors using a text embedding model includes: Each key-value pair in the JSON-formatted query results is encoded into a path string, which is then input into the text embedding model to obtain the embedding vector for each path string. The average of the embedding vectors of all path strings is then used to obtain the structured query vector.

5. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The embedding vector for each page of each power document is obtained by preprocessing each page of each power document and then inputting each page into the modal joint embedding model.

6. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The modal joint embedding model includes a text encoder, an image encoder, and a cross-modal fusion module. The text encoder and the image encoder extract feature vectors from the input text and image, respectively, and then pass them to the cross-modal fusion module to align and fuse the two feature vectors.

7. The document retrieval method based on power semantic enhancement according to claim 6, characterized in that, The loss function of the modal joint embedding model is a weighted sum of the text contrast learning loss of the text encoder, the image contrast learning loss of the image encoder, and the cross-modal alignment loss of the cross-modal fusion module. The cross-modal alignment loss includes: text-to-image alignment loss and image-to-text alignment loss.

8. The document retrieval method based on power semantic enhancement according to claim 6, characterized in that, In the modal joint embedding model, the text encoder adopts the BGE model, and the text contrastive learning loss adopts the multi-negative sample ranking loss function. The image encoder uses the Vit model, and the image contrast learning loss uses the SimCLR loss function. The cross-modal fusion module uses the Transformer model, and the cross-modal alignment loss uses the InfoNCE loss function.

9. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The dual-path prediction model includes two parallel encoder branches and a fully connected layer; wherein, both the first encoder branch and the second encoder branch adopt the Transformer model, and the fully connected layer is used to fuse the aggregated representation of the outputs of the two encoder branches and map it to a score.

10. The document retrieval method based on power semantic enhancement according to claim 9, characterized in that, The method of obtaining a comprehensive similarity score for each candidate document using a dual-path prediction model includes: The query embedding vector of the user's query content and the structured semantic vector of each candidate document are concatenated and passed into the first encoder branch; The query embedding vector of the user's query content and the paged semantic vector of the corresponding candidate document are concatenated and passed into the second encoder branch; The two encoder branches output two aggregate representations through their respective self-attention mechanisms and pass them to the fully connected layer. After fusion by the fully connected layer, the composite similarity score is mapped and output.

11. The document retrieval method based on power semantic enhancement according to claim 9, characterized in that, The formula for the loss function of the dual-path prediction model is as follows: ， in, This represents the loss function of the dual-path prediction model. Indicates the first The set of candidate samples corresponding to each query sample and They represent The Middle The candidate sample and the first The candidate samples respectively for the first The relevance level of each query sample and They represent The Middle The candidate sample and the first The predicted score of each candidate sample. Represents the weighting factor, calculated... The Middle The candidate sample and the first The result is obtained by taking the absolute value of the change in the cumulative normalized loss before and after the position swap of each candidate sample.

12. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The modality joint embedding model is obtained by training on text data and image data obtained from data preprocessing and modality separation of various power documents in the power corpus. The text data includes: text content extracted from Word, PDF and scanned documents, text content in JSON or Markdown format obtained from Excel spreadsheets, and descriptive or summary text content generated using a large language model. The image data includes: device photos and CAD drawings.

13. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The structured semantic vector for each power document is also obtained based on the power knowledge graph, using a large language model and a text embedding model.

14. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The hierarchy in the power knowledge graph includes, in order: project stage, equipment type, technical specifications, specification / standard category, risk / failure type, document type, and professional field; each level includes multiple elements.

15. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The similarity between the structured query vector and the structured semantic vector of each power document, and the similarity between the query embedding vector and the embedding vector of each page of each power document, are both cosine similarities.

16. The document retrieval method based on power semantic enhancement according to claim 1, characterized in that, The total score is obtained by weighted summation of the document-level similarity score, page-level similarity score, and comprehensive similarity score of each candidate document after normalization.

17. The document retrieval method based on power semantic enhancement according to claim 12, characterized in that, The power corpus includes: project materials, element rule materials, and domain materials; the project materials are documents in various heterogeneous formats generated during the technical and economic review process; the element rule materials include: technical standards, design guidelines, construction specifications, and equipment selection recommendation parameters issued by the power grid company; and the domain materials include: project procedures, technical standards, and academic papers.

18. A document retrieval system based on electricity semantic enhancement, characterized in that, include: The query feature extraction module is used to receive user query content and obtain query embedding vectors using a modal joint embedding model; Based on the power knowledge graph, a structured query vector for the user's query content is obtained using a large language model and a text embedding model. The similarity retrieval module is used to obtain multiple candidate documents and their document-level similarity scores and page-level similarity scores based on the query embedding vector and structured query vector of the user query content. This includes: obtaining multiple first-similar documents and their document-level similarity scores based on the similarity between the structured semantic vector of each power document and the structured query vector of the user query content; obtaining multiple second-similar documents and their page-based semantic vectors and page-level similarity scores based on the similarity between each page embedding vector of each power document and the query embedding vector of the user query content; merging multiple first-similar documents and multiple second-similar documents to obtain candidate documents; obtaining the document-level similarity score, page-based semantic vector, and page-level similarity score of each candidate document based on the query embedding vector and structured query vector of the user query content; and then using a dual-path prediction model to obtain the comprehensive similarity score of each candidate document. The search results display module is used to calculate the total score based on the document-level similarity score, page-level similarity score, and comprehensive similarity score of each candidate document, and select the candidate documents with the highest total scores as the search results.

19. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the document retrieval method based on power semantic enhancement as described in any one of claims 1-17.

Images