RAG-based transformer substation patrol intelligent processing method and system

Abstract

The invention belongs to the field of electric power system operation and maintenance, and particularly relates to a substation patrol intelligent processing method and system based on RAG. The method comprises the following steps: collecting context information of a current patrol task, and converting the context information into vector representation; the context information comprises a device type, a working condition parameter, a historical defect record and a timestamp corresponding to the inspection task; searching a set number of knowledge entry vectors with the highest correlation degree with the context information vectors in the constructed multi-modal vector database; each knowledge entry vector in the multi-modal vector database comprises a text vector, an image feature vector and a timestamp; the correlation degree is determined through the similarity between the text vector and the context information vector, the similarity between the image feature vector and the context information vector, and the timestamp difference between the knowledge item vector and the context information vector; and splicing the context information with the retrieval result, generating an enhanced cue word, and inputting the enhanced cue word into the large language model to obtain a processing result.

Description

Technical Field

[0001] This invention belongs to the field of power system operation and maintenance, specifically relating to a substation inspection intelligent processing method and system based on RAG. Background Technology

[0002] Currently, substation inspection systems have accumulated massive amounts of data, including images, videos, defect records, historical work orders, and maintenance procedures. Existing intelligent analysis solutions mainly rely on single image recognition or rule engines, which have the following main shortcomings:

[0003] 1) Knowledge silos: Maintenance manuals and defect experience from different equipment manufacturers are scattered and difficult to retrieve;

[0004] 2) Model illusion: Large language models often generate suggestions that are inconsistent with the facts when reasoning alone;

[0005] 3) Lack of closed-loop: It is impossible to carry out knowledge iteration and optimization, and it is difficult to continuously improve the quality of diagnosis.

[0006] Chinese invention patent application CN120563072A discloses an intelligent power work order processing method integrating RPA and a multimodal large model. This method parses multimodal work order data (text, voice, images, etc.) using a domain-adaptive large language model, and combines this with a dynamic knowledge graph to extract key fault information and resolve conflicts. It utilizes spatiotemporal constraint reinforcement learning for work order priority scoring and resource allocation. The RPA engine converts the analysis results into automated execution scripts, achieving a closed-loop process for dispatching, processing, and feedback. Simultaneously, it constructs a collaborative mechanism for feedback optimization and conflict resolution, continuously iterating the knowledge graph and model accuracy. This method improves work order processing efficiency and parsing accuracy, enhances the scientific nature of decision-making, and is suitable for intelligent operation and maintenance scenarios in power systems. Summary of the Invention

[0007] The purpose of this invention is to provide a substation inspection intelligent processing method and system based on RAG, which solves the problem that the intelligent analysis scheme of the existing substation inspection system requires scattered information such as operation and maintenance standards and defect experience, which is difficult to retrieve.

[0008] To achieve the above objectives, this invention provides a substation inspection intelligent processing method based on RAG, comprising:

[0009] Collect the context information of the current inspection task and convert it into a vector representation; the context information includes the equipment type, operating parameters, historical defect records and timestamps corresponding to the inspection task;

[0010] In the constructed multimodal vector database, a set number of knowledge entry vectors with the highest relevance to the context information vector are retrieved. Each knowledge entry vector in the multimodal vector database contains a text vector, an image feature vector, and a timestamp. The text vector is obtained by converting text information containing substation inspection procedures or historical work orders into vector representations, and the image feature vector is obtained by converting image information containing defect images into vector representations. The relevance is determined by the similarity between the text vector, the image feature vector, and the context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector.

[0011] The context information is concatenated with the search results to generate enhanced prompt words, which are then input into a large language model to obtain the processing results.

[0012] Furthermore, it also includes: calculating the confidence level of the processing result based on the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints; when the confidence level is less than the confidence level threshold, the processing result is pushed to the manual review stage.

[0013] Furthermore, it also includes: if the processing result is pushed to the manual review stage, then the corresponding knowledge entry vector in the multimodal vector database is updated according to the result obtained after the manual review stage; if the processing result does not need to be pushed to the manual review stage, then the corresponding knowledge entry vector in the multimodal vector database is updated directly according to the processing result.

[0014] Furthermore, the methods for determining the relevance by the similarity between the text vector, image feature vector, and context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector, include:

[0015] The correlation between the knowledge entry vector and the context information vector is obtained by weighted summing the similarity between the text vector and the image feature vector and the context information vector, respectively, and the reciprocal of the timestamp difference between the knowledge entry vector and the context information vector.

[0016] Furthermore, the confidence level of the processing result is calculated by means of the similarity between the processing result and the retrieval result, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints.

[0017] The confidence level of the processing result is obtained by weighted summing the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; the weighted summation method includes:

[0018]

[0019] in, The confidence level of the processing result. The semantic similarity between the processing result and the retrieval result. The similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; , These are the semantic similarity between the processing result and the retrieval result, and the weight coefficients corresponding to the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints, respectively.

[0020] Furthermore, based on the results obtained after the manual review process or the processing results, the methods for updating the corresponding knowledge entry vectors in the multimodal vector database include:

[0021]

[0022] in For the learning rate, and ; This is the vector of the i-th knowledge entry before the update. This is the updated vector of the i-th knowledge entry; For use in sending relevant information Transform it into a vector encoding model that uses vector representation. The result obtained after manual review or the processing result.

[0023] Furthermore, the processing results include defect type, recommended measures, and confidence score.

[0024] Furthermore, the similarity between the text vector, the image feature vector, and the context information vector is calculated using cosine similarity.

[0025] The weighted summation methods, which utilize the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the reciprocal of the timestamp difference between knowledge entry vectors and context information vectors, include:

[0026]

[0027] in, Indicates the degree of relevance. Text vectors With context information vector cosine similarity Image feature vectors With context information vector cosine similarity Knowledge Item Vector With context information vector The weighting coefficients corresponding to the timestamp differences, and , This represents the time difference obtained based on the timestamp difference between the knowledge entry vector and the context information vector.

[0028] Furthermore, methods for concatenating contextual information with search results include: concatenating contextual information with search results using a dynamic slot filling function.

[0029] The above-described technical solution of this invention provides a novel intelligent processing method for substation inspection based on RAG, the beneficial effects of which include:

[0030] This study fully considers the problem in traditional inspection systems where knowledge is stored in a scattered manner in the form of unstructured documents, images, and work orders, leading to high retrieval difficulty and low recall. By constructing a unified multimodal vector space, the features of text, images, and video frames are mapped to the same vector space, and retrieval is performed based on vector similarity, which significantly improves the efficiency of inspection and diagnosis, reduces the cost of manual screening, and avoids the impact of scattered knowledge storage. Furthermore, through multimodal vector retrieval, efficient fusion of multi-source knowledge such as text and images is achieved, while also considering the influence of knowledge novelty, which can comprehensively improve the accuracy and consistency of diagnostic results and reduce the false negative rate. On this basis, context-enhanced prompts ensure that the prompts contain sufficient factual constraints, thereby limiting the model's reasoning space to the range of credible knowledge, significantly reducing the large model illusion rate, and improving the reliability of reasoning.

[0031] This invention also provides a RAG-based intelligent processing system for substation inspection, including a processor storing executable program instructions. These instructions are executed to implement the following RAG-based intelligent processing method for substation inspection, specifically including:

[0032] Collect the context information of the current inspection task and convert it into a vector representation; the context information includes the equipment type, operating parameters, historical defect records and timestamps corresponding to the inspection task;

[0033] In the constructed multimodal vector database, a set number of knowledge entry vectors with the highest relevance to the context information vector are retrieved. Each knowledge entry vector in the multimodal vector database contains a text vector, an image feature vector, and a timestamp. The text vector is obtained by converting text information containing substation inspection procedures or historical work orders into vector representations, and the image feature vector is obtained by converting image information containing defect images into vector representations. The relevance is determined by the similarity between the text vector, the image feature vector, and the context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector.

[0034] The context information is concatenated with the search results to generate enhanced prompt words, which are then input into a large language model to obtain the processing results.

[0035] Furthermore, it also includes: calculating the confidence level of the processing result based on the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints; when the confidence level is less than the confidence level threshold, the processing result is pushed to the manual review stage.

[0036] Furthermore, it also includes: if the processing result is pushed to the manual review stage, then the corresponding knowledge entry vector in the multimodal vector database is updated according to the result obtained after the manual review stage; if the processing result does not need to be pushed to the manual review stage, then the corresponding knowledge entry vector in the multimodal vector database is updated directly according to the processing result.

[0037] Furthermore, the methods for determining the relevance by the similarity between the text vector, image feature vector, and context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector, include:

[0038] The correlation between the knowledge entry vector and the context information vector is obtained by weighted summing the similarity between the text vector and the image feature vector and the context information vector, respectively, and the reciprocal of the timestamp difference between the knowledge entry vector and the context information vector.

[0039] Furthermore, the confidence level of the processing result is calculated by means of the similarity between the processing result and the retrieval result, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints.

[0040] The confidence level of the processing result is obtained by weighted summing the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; the weighted summation method includes:

[0041]

[0042] in, The confidence level of the processing result. The semantic similarity between the processing result and the retrieval result. The similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; , These are the semantic similarity between the processing result and the retrieval result, and the weight coefficients corresponding to the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints, respectively.

[0043] Furthermore, based on the results obtained after the manual review process or the processing results, the methods for updating the corresponding knowledge entry vectors in the multimodal vector database include:

[0044]

[0045] in For the learning rate, and ; This is the vector of the i-th knowledge entry before the update. This is the updated vector of the i-th knowledge entry; For use in sending relevant information Transform it into a vector encoding model that uses vector representation. The result obtained after manual review or the processing result.

[0046] Furthermore, the processing results include defect type, recommended measures, and confidence score.

[0047] Furthermore, the similarity between the text vector, the image feature vector, and the context information vector is calculated using cosine similarity.

[0048] The weighted summation methods, which utilize the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the reciprocal of the timestamp difference between knowledge entry vectors and context information vectors, include:

[0049]

[0050] in, Indicates the degree of relevance. Text vectors With context information vector cosine similarity Image feature vectors With context information vector cosine similarity Knowledge Item Vector With context information vector The weighting coefficients corresponding to the timestamp differences, and , This represents the time difference obtained based on the timestamp difference between the knowledge entry vector and the context information vector.

[0051] Furthermore, methods for concatenating contextual information with search results include: concatenating contextual information with search results using a dynamic slot filling function.

[0052] The technical solution of the RAG-based intelligent substation inspection processing system described above can achieve the same beneficial effects as the RAG-based intelligent substation inspection processing method described above. Attached Figure Description

[0053] Figure 1 This is a flowchart of the intelligent processing method for substation inspection based on RAG in the implementation of the present invention.

[0054] Figure 2 This is an example diagram illustrating the formula representation of the intelligent processing method for substation inspection based on RAG in the implementation of the present invention.

[0055] Figure 3 This is a flowchart illustrating the process of determining whether manual review and knowledge base updates are needed based on the processing results after obtaining the processing results in the intelligent processing method for substation inspection based on RAG of the present invention.

[0056] Figure 4 This is an example diagram illustrating the architectural principle of the intelligent processing system for substation inspection based on RAG in the implementation of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] Implementation of a RAG-based Intelligent Processing Method for Substation Inspection

[0059] This implementation presents a technical solution for an intelligent substation inspection processing method based on RAG (Retrieval-Augmented Generation). By constructing a unified multimodal vector space, the features of text, images, and video frames are mapped to the same vector space. A time factor is also added to facilitate the retrieval of the latest and most relevant knowledge based on the similarity between inspection information and knowledge information pre-existing in the database. This knowledge is then used as part of the input to a large language model, improving the reliability of the large language model's output and overcoming the problem of knowledge dispersion.

[0060] The method includes:

[0061] Collect the context information of the current inspection task and convert it into a vector representation; the context information includes the equipment type, operating parameters, historical defect records and timestamps corresponding to the inspection task;

[0062] In the constructed multimodal vector database, a set number of knowledge entry vectors with the highest relevance to the context information vectors are retrieved. Each knowledge entry vector in the multimodal vector database contains a text vector, an image feature vector, and a timestamp. The text vector is obtained by converting text information containing substation inspection procedures or historical work orders into vector representations, and the image feature vector is obtained by converting image information containing defect images into vector representations. The relevance is determined by the similarity between the text vector, the image feature vector, and the context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector.

[0063] The context information is concatenated with the search results to generate enhanced prompt words, which are then input into a large language model to obtain the processing results.

[0064] Therefore, this method addresses the problem in traditional inspection systems where knowledge is scattered across multiple sources such as unstructured documents, images, and work orders, leading to high retrieval difficulty and low recall. It constructs a unified multimodal vector space, mapping text features, image features, and video frame features to the same vector representation domain. Relevance retrieval is then performed based on vector similarity, thereby improving cross-modal knowledge recall, reducing the burden of manual screening, and avoiding the inefficiency caused by knowledge dispersion. Furthermore, this method introduces time decay weights during multimodal vector retrieval, ensuring that candidate knowledge relies not only on semantic similarity but also on the age of the knowledge entries. This achieves efficient fusion of multi-source knowledge such as text and images, improving the accuracy and consistency of diagnostic results and reducing the probability of missed and false positives. Building upon this, a context-enhanced prompt mechanism embeds retrieved factual evidence into generated prompts, providing structured constraints for large-scale model reasoning, limiting the model's generation space, and significantly reducing the probability of large models generating illusory information, thus improving the reliability and controllability of the reasoning process.

[0065] Reference Figure 2Specifically, in this embodiment, the method of concatenating contextual information with search results includes: fully considering the problem in traditional inspection systems where knowledge is stored in a scattered manner in the form of unstructured documents, images, work orders, etc., leading to high retrieval difficulty and low recall, a unified multimodal vector space is constructed to map the features of text, images, and video frames into the same vector space. Retrieval is then performed based on vector similarity, significantly improving inspection and diagnostic efficiency, reducing manual screening costs, and avoiding the impact of scattered knowledge storage. Furthermore, multimodal vector retrieval achieves efficient fusion of multi-source knowledge such as text and images, while also considering the influence of knowledge novelty, comprehensively improving the accuracy and consistency of diagnostic results and reducing the false negative rate. Based on this, context-enhanced prompts ensure that the prompts contain sufficient factual constraints, thereby limiting the model's reasoning space to the range of credible knowledge, significantly reducing the large model illusion rate, and improving reasoning reliability. A dynamic slot-filling function concatenates contextual information with search results to ensure that the generated context is complete, coherent, and traceable. In other embodiments, other feasible concatenation methods may also be used.

[0066] Furthermore, in this embodiment, the methods for determining the relevance by the similarity between the text vector, image feature vector, and context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector, include:

[0067] The correlation between the knowledge entry vector and the context information vector is obtained by weighted summing the similarities between the text vector, image feature vector, and context information vector, respectively, and the reciprocal of the timestamp difference between the knowledge entry vector and the context information vector. Specifically, in this embodiment, the similarities between the text vector, image feature vector, and context information vector are calculated using cosine similarity.

[0068] The weighted summation methods, which utilize the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the reciprocal of the timestamp difference between knowledge entry vectors and context information vectors, include:

[0069]

[0070] in, Indicates the degree of relevance. Text vectors With context information vector cosine similarity Image feature vectors With context information vector cosine similarity Knowledge Item Vector With context information vector The weighting coefficients corresponding to the timestamp differences, and , This represents the time difference calculated based on the timestamp difference between the knowledge entry vector and the context information vector. This formula considers not only semantic similarity but also incorporates a time decay factor. It prioritizes recalling the newest and most relevant knowledge entries, thus overcoming the inefficiency of retrieval caused by knowledge dispersion. Specifically, the cosine similarity is calculated as follows:

[0071]

[0072] In one specific embodiment, contextual information such as the equipment type, inspection target, operating parameters, and historical defect records of the current inspection task are first collected, and a vector representation is constructed:

[0073]

[0074] in, For context information vectors, For the context encoding function, a sentence vector encoding model based on deep neural networks can be used, such as Sentence-BERT, but it is not limited to this. After vectorizing the context information to obtain the context vector c, the most relevant knowledge entry vector is retrieved from a multimodal vector database. The knowledge entries in the multimodal knowledge base include text-based knowledge and image-based knowledge. Text-based knowledge is obtained by encoding text vectors from operating procedures, maintenance specifications, historical work order texts, and tags using a text encoding model. Image-related knowledge is obtained by extracting image feature vectors from defective images using a pre-trained visual model. The system calculates the context vector c and the text vector respectively. Image feature vectors The similarity was calculated, and all knowledge items were weighted and ranked according to a weighted similarity formula, from which a weighted similarity value was selected. The top N knowledge items are selected as the candidate knowledge set. Weights α, β, and γ are dynamically determined based on fields such as device type, defect category, and task time within the task context, ensuring that textual knowledge, image knowledge, and time relevance have varying degrees of importance under different task conditions. Through this weighted recall strategy, a comprehensive retrieval and ranking of multi-source knowledge, including procedural documents, historical work orders, and defect images, is achieved, resulting in a precise candidate knowledge set.

[0075] Next, the context information is combined with the search results to generate enhanced suggestion words. :

[0076]

[0077] in It is a dynamic splicing function that ensures the generated context is complete, coherent, and traceable.

[0078] In one embodiment, an example template for generating enhanced prompt words is as follows:

[0079] Task context: Device=XXX, Defect history=[...];

[0080] Relevant regulations: Chapter XXX;

[0081] Historical work orders: most recent processing solution = [...];

[0082] Based on the above information, please determine whether the defect has recurred and provide repair recommendations.

[0083] The prompt templates mentioned above support dynamic slots to ensure content completeness and relevance.

[0084] After obtaining the enhanced prompt words, input the prompt words into the large language model. (e.g., LLaMA, GPT), to obtain the processing results. The processing results are output in JSON format, including fields such as defect type, recommended measures, and confidence score, to facilitate subsequent program parsing.

[0085]

[0086] Reference Figure 1 and Figure 2 Similarly, considering the lack of constraints in the reasoning process of traditional large language models, which can easily lead to "illusory" results, the intelligent processing method for substation inspection in this embodiment also includes:

[0087] The confidence level of the processing result is calculated by comparing the similarity between the processing result obtained after inputting into the large language model and the search result, as well as the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints. When the confidence level is less than the confidence level threshold, the processing result is pushed to the manual review stage.

[0088] By designing such a dual-channel confidence assessment mechanism (similarity between the processed result and the retrieved result + similarity between the parameters involved in the processed result and the parameters in the set procedural constraints), a quantitative measure is provided for the results generated by the large language model, ensuring that only confidence is considered. Only the results of the confidence threshold are automatically output. In other cases, the results generated by the large language model will be manually reviewed, which significantly improves the controllability of reasoning. That is, through the confidence assessment and manual review mechanism, the generated results are guaranteed to be traceable and controllable.

[0089] like Figure 3 As shown, in a preferred embodiment, when In some cases, the processing results can be used to automatically generate work orders; otherwise, manual review is required. For example, when an inspection task detects an abnormal oil temperature in a transformer, the system retrieves historical defect work orders similar to those for that equipment (e.g., equipment B's oil temperature was too high on [date in 2024]), and combines this with the 85℃ temperature limit constraint in the operating procedures. If the calculated... This allows for the generation of a draft work order for "immediate shutdown for maintenance." This provides structured output, enabling direct work order generation and improving the automation level of the operation and maintenance process.

[0090] In this embodiment, the confidence level of the above-mentioned calculation results specifically includes the following methods:

[0091] The confidence score of the processing result is obtained by weighted summing the similarity between the processed result and the retrieval result after inputting into the large language model, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints; wherein the weighted summation method includes:

[0092]

[0093] in, The confidence score of the processing result (i.e., the vector similarity between the output and the retrieved evidence, which is the retrieval result mentioned above). The semantic similarity between the processing results and the search results is used as a semantic consistency score. The similarity between the parameters involved in the processing results and the parameters in the set procedure constraints (i.e., whether the key fields in the output, such as voltage level and equipment model, meet the corresponding matching degree of the procedure requirements) is used as the factual consistency score; , These are the weight coefficients corresponding to the semantic similarity between the processing results and the retrieval results, and the similarity between the parameters involved in the processing results and the parameters in the set procedural constraints.

[0094] refer to Figure 1 Considering the existing technology, the results of manual confirmation (i.e., the results obtained after a manual review process, also known as manual review results) are subject to manual verification. (This indicates that) the results of manual review are often difficult to accumulate, preventing intelligent processing methods from adaptively optimizing. Therefore, this implementation also introduces an incremental learning mechanism to incorporate the results of manual review. Write back to the knowledge base and update the index, specifically including updating the corresponding knowledge entry vectors in the multimodal vector database based on the results obtained after the manual review process.

[0095] This approach, which involves manually verifying results and gradually updating pre-stored knowledge vectors, ensures that subsequent searches can more accurately and efficiently recall similar defect cases. This forms an adaptive architecture that becomes more accurate the more it is used, thus solving the problem of long-term performance degradation in database-based processing architectures combined with large language models. In other words, it utilizes human feedback for dynamic updates, forming a dual iterative optimization loop of knowledge and large language models. Through continuous learning, the adaptive architecture can be continuously optimized with the accumulation of operational experience, improving long-term performance and possessing long-term economic and social value.

[0096] Specifically, such as Figure 3 As shown, based on the results obtained after manual review, the methods for updating the corresponding knowledge entry vectors in the multimodal vector database (which can be called incremental updates) include:

[0097]

[0098] in For the learning rate, and ; This is the vector of the i-th knowledge entry before the update. This is the updated vector of the i-th knowledge entry; For use in This is transformed into a vector encoding model that uses vector representations (high-dimensional feature vectors consistent with the knowledge base vector space), and its output dimension is the same as the knowledge entry vector. The same applies; both are used to support incremental updates of the knowledge base. This refers to the results obtained after manual review (a structured result reflecting accurate diagnostic information or factual content obtained after manual review of the i-th knowledge entry) or the results that do not require manual review. This update method supports incremental learning and online learning mechanisms, ensuring that the knowledge base is dynamically updated over time without compromising the stability of existing knowledge, thus improving future retrieval performance. Through this incremental update method, the knowledge entry vector gradually approaches the actual information confirmed by humans, thereby improving the accuracy and stability of subsequent retrieval and diagnosis.

[0099] In one specific embodiment, after manual review, the final result (i.e., the result obtained after the manual review process or the processing result that does not require manual review) is stored in the knowledge base, the vector index is updated, and the ranking for the next search is improved. At the same time, reinforcement learning is performed on the prompt word template to improve the generation quality.

[0100] In summary, the RAG-based intelligent substation inspection processing method in this embodiment theoretically guarantees knowledge utilization, diagnostic accuracy, and continuous learning capability of the system. Moreover, it has been verified that in practical applications, it can improve the defect identification accuracy by ≥15% and reduce the workload of manual review by ≥30%, which is significantly better than traditional schemes based on rule engines or static knowledge bases.

[0101] Implementation of a RAG-based Intelligent Processing System for Substation Inspection

[0102] This embodiment provides a technical solution for a RAG-based intelligent processing system for substation inspection, including a processor containing executable program instructions. These instructions are executed to implement the following RAG-based intelligent processing method for substation inspection:

[0103] Collect the context information of the current inspection task and convert it into a vector representation; the context information includes the equipment type, operating parameters, historical defect records and timestamps corresponding to the inspection task;

[0104] In the constructed multimodal vector database, a set number of knowledge entry vectors with the highest relevance to the context information vectors are retrieved. Each knowledge entry vector in the multimodal vector database contains a text vector, an image feature vector, and a timestamp. The text vector is obtained by converting text information containing substation inspection procedures or historical work orders into vector representations, and the image feature vector is obtained by converting image information containing defect images into vector representations. The relevance is determined by the similarity between the text vector, the image feature vector, and the context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector.

[0105] The context information is concatenated with the search results to generate enhanced prompt words, which are then input into a large language model to obtain the processing results.

[0106] Therefore, addressing the issues of traditional inspection systems where knowledge is scattered across multiple sources such as unstructured documents, images, and work orders, leading to high retrieval difficulty and low recall, this method constructs a unified multimodal vector space. This space maps textual, image, and video frame features to the same vector representation domain, and performs relevance retrieval based on vector similarity. This improves cross-modal knowledge recall, reduces the burden of manual screening, and avoids the inefficiency caused by knowledge dispersion. Furthermore, this method introduces time decay weights during multimodal vector retrieval, ensuring that candidate knowledge relies not only on semantic similarity but also on the age of the knowledge entries. This achieves efficient fusion of multi-source knowledge such as text and images, improving the accuracy and consistency of diagnostic results and reducing the probability of missed or false positives. Building upon this, a context-enhanced prompt mechanism embeds retrieved factual evidence into generated prompts, providing structured constraints for large-scale model reasoning, limiting the model's generation space, and significantly reducing the probability of large models generating illusory information, thereby improving the reliability and controllability of the reasoning process.

[0107] Specifically, in this embodiment, the method of concatenating context information with search results includes: concatenating context information with search results using a dynamic slot filling function to ensure that the generated context is complete, coherent, and traceable. In other embodiments, other feasible concatenation methods may also be used.

[0108] Furthermore, in this embodiment, the methods for determining the relevance by the similarity between the text vector, image feature vector, and context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector, include:

[0109] The correlation between the knowledge entry vector and the context information vector is obtained by weighted summing the similarities between the text vector, image feature vector, and context information vector, respectively, and the reciprocal of the timestamp difference between the knowledge entry vector and the context information vector. Specifically, in this embodiment, the similarities between the text vector, image feature vector, and context information vector are calculated using cosine similarity.

[0110] The weighted summation methods, which utilize the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the reciprocal of the timestamp difference between knowledge entry vectors and context information vectors, include:

[0111]

[0112] in, Indicates the degree of relevance. Text vectors With context information vector cosine similarity Image feature vectors With context information vector cosine similarity Knowledge Item Vector With context information vector The weighting coefficients corresponding to the timestamp differences, and , This represents the time difference calculated based on the timestamp difference between the knowledge entry vector and the context information vector. This formula considers not only semantic similarity but also incorporates a time decay factor. It prioritizes recalling the newest and most relevant knowledge entries, thus overcoming the inefficiency of retrieval caused by knowledge dispersion. Specifically, the cosine similarity is calculated as follows:

[0113]

[0114] In one specific embodiment, contextual information such as the equipment type, inspection target, operating parameters, and historical defect records of the current inspection task are first collected, and a vector representation is constructed:

[0115]

[0116] in, For context information vectors, For the context encoding function, a sentence vector encoding model based on deep neural networks can be used, such as Sentence-BERT, but it is not limited to this. (See also...) Figure 4 The collection of context information is mainly achieved through the context management module.

[0117] After vectorizing the context information to obtain the context vector c, the most relevant knowledge entry vector is retrieved from the multimodal vector database. The knowledge entries in the multimodal knowledge base include text-based knowledge and image-based knowledge. Text-based knowledge is obtained by encoding text vectors from operating procedures, maintenance specifications, historical work order texts, and tags using a text encoding model. Image-related knowledge is obtained by extracting image feature vectors from defective images using a pre-trained visual model. The system calculates the context vector c and the text vector respectively. Image feature vectors The similarity was calculated, and all knowledge items were weighted and ranked according to a weighted similarity formula, from which a weighted similarity value was selected. The top N knowledge items are selected as the candidate knowledge set. Weights α, β, and γ are dynamically determined based on fields such as device type, defect category, and task time within the task context, ensuring that textual knowledge, image knowledge, and time relevance have varying degrees of importance under different task conditions. Through this weighted recall strategy, a comprehensive retrieval and ranking of multi-source knowledge, including procedural documents, historical work orders, and defect images, is achieved, resulting in a precise candidate knowledge set. (Refer to...) Figure 4 Retrieval based on multimodal vector database and The most relevant knowledge item vectors are primarily implemented through the multimodal retrieval module.

[0118] Next, the context information is combined with the search results to generate enhanced suggestion words. :

[0119]

[0120] in It is a dynamic splicing function that ensures the generated context is complete, coherent, and traceable.

[0121] In one embodiment, an example template for generating enhanced prompt words is as follows:

[0122] Task context: Device=XXX, Defect history=[...];

[0123] Relevant regulations: Chapter XXX;

[0124] Historical work orders: most recent processing solution = [...];

[0125] Based on the above information, please determine whether the defect has recurred and provide repair recommendations.

[0126] The above prompt templates support dynamic slots to ensure content completeness and relevance. (See also...) Figure 4 The generation of enhanced prompts is mainly achieved through the prompt generation module.

[0127] After obtaining the enhanced prompt words, input the prompt words into the large language model. (e.g., LLaMA, GPT), to obtain the processing results. ;reference Figure 4 The generation of processing results from the large model is primarily achieved through the inference generation module. The processing results are output in JSON format, containing fields such as defect type, recommended measures, and confidence score, facilitating subsequent program parsing.

[0128]

[0129] Similarly, considering the lack of constraints in traditional large language model reasoning, which can easily lead to "illusory" results, the substation inspection intelligent processing system in this embodiment also includes:

[0130] The confidence level of the processing result is calculated by comparing its similarity to the search results obtained after inputting into the large language model, and by comparing the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints. When the confidence level is less than the confidence threshold, the processing result is sent to the manual review stage. (Refer to...) Figure 4 The confidence level of the calculation results is mainly achieved through the confidence assessment module.

[0131] By designing such a dual-channel confidence assessment mechanism (similarity between the processed result and the retrieved result + similarity between the parameters involved in the processed result and the parameters in the set procedural constraints), a quantitative measure is provided for the results generated by the large language model, ensuring that only confidence is considered. Only the results of the confidence threshold are automatically output. In other cases, the results generated by the large language model will be manually reviewed, which significantly improves the controllability of reasoning. That is, through the confidence assessment and manual review mechanism, the generated results are guaranteed to be traceable and controllable.

[0132] In a preferred embodiment, when In some cases, the processing results can be used to automatically generate work orders; otherwise, manual review is required. For example, when an inspection task detects an abnormal oil temperature in a transformer, the system retrieves historical defect work orders similar to those for that equipment (e.g., equipment B's oil temperature was too high on [date in 2024]), and combines this with the 85℃ temperature limit constraint in the operating procedures. If the calculated... This allows for the generation of a draft work order for "immediate shutdown for maintenance." This provides structured output, enabling direct work order generation and improving the automation level of the operation and maintenance process.

[0133] In this embodiment, the confidence level of the above-mentioned calculation results specifically includes the following methods:

[0134] The confidence score of the processing result is obtained by weighted summing the similarity between the processed result and the retrieval result after inputting into the large language model, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints; wherein the weighted summation method includes:

[0135]

[0136] in, The confidence score of the processing result (i.e., the vector similarity between the output and the retrieved evidence, which is the retrieval result mentioned above). The semantic similarity between the processing results and the search results is used as a semantic consistency score. The similarity between the parameters involved in the processing results and the parameters in the set procedure constraints (i.e., whether the key fields in the output, such as voltage level and equipment model, meet the corresponding matching degree of the procedure requirements) is used as the factual consistency score; , These are the weight coefficients corresponding to the semantic similarity between the processing results and the retrieval results, and the similarity between the parameters involved in the processing results and the parameters in the set procedural constraints.

[0137] Considering the existing technology, the results of manual verification (i.e., the results obtained after a manual review process, also known as manual review results) are subject to manual verification. (This indicates that) results are often difficult to accumulate, preventing the intelligent processing system from adaptively optimizing. Therefore, this implementation also introduces an incremental learning mechanism to incorporate manually reviewed results or processing results that do not require manual review. Write back to the knowledge base and update the index. Specifically, if the processing result is pushed to the manual review stage, the corresponding knowledge entry vector in the multimodal vector database is updated based on the result obtained after the manual review stage; if the processing result does not need to be pushed to the manual review stage, the corresponding knowledge entry vector in the multimodal vector database is updated directly based on the processing result.

[0138] This approach, which involves manually verifying results and gradually updating pre-stored knowledge vectors, ensures that subsequent searches can more accurately and efficiently recall similar defect cases. This forms an adaptive architecture that becomes more accurate the more it is used, thus solving the problem of long-term performance degradation in database-based processing architectures combined with large language models. In other words, it utilizes human feedback for dynamic updates, forming a dual iterative optimization loop of knowledge and large language models. Through continuous learning, the adaptive architecture can be continuously optimized with the accumulation of operational experience, improving long-term performance and possessing long-term economic and social value.

[0139] Specifically, based on the results obtained after the manual review process, the methods for updating the corresponding knowledge entry vectors in the multimodal vector database include:

[0140]

[0141] in For the learning rate, and ; This is the vector of the i-th knowledge entry before the update. This is the updated vector of the i-th knowledge entry; For use in This is transformed into a vector encoding model that uses vector representations (high-dimensional feature vectors consistent with the knowledge base vector space), and its output dimension is the same as the knowledge entry vector. The same applies; both are used to support incremental updates of the knowledge base. This refers to the result obtained after manual review (a structured result reflecting accurate diagnostic information or factual content obtained after manual review of the i-th knowledge entry) or processing result. This update method supports incremental learning and online learning mechanisms, ensuring that the knowledge base is dynamically updated over time without compromising the stability of existing knowledge, thus improving future retrieval performance. (Refer to...) Figure 4 The process of receiving manual confirmation results and updating the database is mainly achieved through the feedback learning module.

[0142] In one specific embodiment, after manual review, the final result (i.e., the result obtained after the manual review process) is stored in the knowledge base along with the corresponding prompt words and retrieval evidence, updating the vector index and improving the ranking for the next retrieval. Simultaneously, reinforcement learning is applied to the prompt word template to improve the generation quality. The context management module, multimodal retrieval module, prompt word generation module, generation reasoning module, credibility assessment module, and feedback learning module communicate asynchronously via message queues to ensure system real-time performance and scalability.

[0143] In summary, the RAG-based intelligent substation inspection system in this embodiment theoretically guarantees knowledge utilization, diagnostic accuracy, and continuous learning capability. Moreover, it has been verified that in practical applications, it can improve defect identification accuracy by ≥15% and reduce manual review workload by ≥30%, which is significantly better than traditional solutions based on rule engines or static knowledge bases.

[0144] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. A substation inspection intelligent processing method based on RAG, characterized in that, include: Collect the context information of the current inspection task and convert it into a vector representation; the context information includes the equipment type, operating parameters, historical defect records and timestamps corresponding to the inspection task; In the constructed multimodal vector database, a set number of knowledge entry vectors with the highest relevance to the context information vector are retrieved. Each knowledge entry vector in the multimodal vector database contains a text vector, an image feature vector, and a timestamp. The text vector is obtained by converting text information containing substation inspection procedures or historical work orders into vector representations, and the image feature vector is obtained by converting image information containing defect images into vector representations. The relevance is determined by the similarity between the text vector, the image feature vector, and the context information vector, respectively, and the timestamp difference between the knowledge entry vector and the context information vector. The context information is concatenated with the search results to generate enhanced prompt words, which are then input into a large language model to obtain the processing results.

2. The intelligent processing method for substation inspection based on RAG according to claim 1, characterized in that, Also includes: The confidence level of the processing result is calculated by comparing the similarity between the processing result and the search result, and by comparing the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints. When the confidence level is less than the confidence level threshold, the processing result is pushed to the manual review stage.

3. The intelligent processing method for substation inspection based on RAG according to claim 2, characterized in that, Also includes: If the processing result is pushed to the manual review stage, the corresponding knowledge entry vector in the multimodal vector database will be updated based on the result obtained after the manual review stage. If the processing result does not need to be pushed to the manual review stage, the corresponding knowledge entry vector in the multimodal vector database is updated directly based on the processing result.

4. The intelligent processing method for substation inspection based on RAG according to any one of claims 1-3, characterized in that, The methods for determining the relevance by the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the timestamp difference between knowledge entry vectors and context information vectors, include: The correlation between the knowledge entry vector and the context information vector is obtained by weighted summing the similarity between the text vector and the image feature vector and the context information vector, respectively, and the reciprocal of the timestamp difference between the knowledge entry vector and the context information vector.

5. The intelligent processing method for substation inspection based on RAG according to any one of claims 1-3, characterized in that, The confidence level of the processing result is calculated by using the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints. The methods include: The confidence level of the processing result is obtained by weighted summing the similarity between the processing result and the search result, and the similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; the weighted summation method includes: ； in, The confidence level of the processing result. The semantic similarity between the processing result and the retrieval result. The similarity between the parameters involved in the processing result and the parameters in the set procedure constraints; , These are the semantic similarity between the processing result and the retrieval result, and the weight coefficients corresponding to the similarity between the parameters involved in the processing result and the parameters in the set procedural constraints, respectively.

6. The intelligent processing method for substation inspection based on RAG according to claim 3, characterized in that, The methods for updating the corresponding knowledge entry vectors in the multimodal vector database based on the results obtained after manual review or the processing results mentioned above include: ； in For the learning rate, and ; This is the vector of the i-th knowledge entry before the update. This is the updated vector of the i-th knowledge entry; For use in sending relevant information Transform it into a vector encoding model that uses vector representation. The result obtained after manual review or the processing result.

7. The intelligent processing method for substation inspection based on RAG according to any one of claims 1-3, characterized in that, The processing results include the defect type, recommended measures, and confidence score.

8. The intelligent processing method for substation inspection based on RAG according to claim 4, characterized in that, The similarity between text vectors, image feature vectors, and context information vectors is calculated using cosine similarity. The weighted summation methods, which utilize the similarity between text vectors, image feature vectors, and context information vectors, respectively, and the reciprocal of the timestamp difference between knowledge entry vectors and context information vectors, include: ； in, Indicates the degree of relevance. Text vectors With context information vector cosine similarity Image feature vectors With context information vector cosine similarity Knowledge Entry Vector With context information vector The weighting coefficients corresponding to the timestamp differences, and , This represents the time difference obtained based on the timestamp difference between the knowledge entry vector and the context information vector.

9. The intelligent processing method for substation inspection based on RAG according to any one of claims 1-3, characterized in that, Methods for concatenating context information with search results include using dynamic slot filling functions to concatenate context information with search results.

10. A substation inspection intelligent processing system based on RAG, comprising a processor, wherein the processor stores executable program instructions, characterized in that, The executable program instructions are executed to implement the RAG-based intelligent processing method for substation inspection as described in any one of claims 1-9.

Images