Automatic intelligent operation and maintenance method and system for power grid dispatching based on multi-modal large model

Abstract

The invention discloses an automatic intelligent operation and maintenance method and system for power grid dispatching based on a multi-modal large model, the multi-modal large model composed of a language processing model, an image recognition model and an AIGC operation and maintenance model is constructed, input information of different information sources is processed, different operation and maintenance scheme texts are obtained, and the operation and maintenance scheme texts are stored in the multi-modal large model. And searching a historical operation and maintenance scheme based on a keyword of the input information, carrying out three-party confirmation, and when the matching degree is lower than a threshold value, considering that an output result is unreliable. Through innovation points of multi-modal information input, dual operation and maintenance scheme output, matching degree judgment and the like, the problem that the generative AI is easy to express mistakenly is solved, so that the operation and maintenance system can judge and prompt the reliability of the result when outputting the result, frequent recheck of operation and maintenance personnel is avoided, the operation and maintenance reliability and efficiency are greatly improved, and the operation and maintenance cost is reduced. Comprehensive understanding and efficient processing of operation and maintenance scenes are realized, and powerful technical guarantee is provided for safe and stable operation of a power system.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and system for intelligent operation and maintenance of power grid dispatching automation based on a multi-modal large model. Background Art

[0002] With the rapid development of technologies such as 5G, cloud computing, and the Internet of Things, the scale and complexity of networks are increasing rapidly. The traditional manual operation and maintenance model is gradually unable to cope with massive equipment and complex network problems, and the operation and maintenance efficiency is low, fault handling is delayed, and the operation and maintenance costs continue to rise. Based on this demand, AI (artificial intelligence) technology has begun to be gradually introduced into the field of operation and maintenance, forming the concept of intelligent operation and maintenance (AIOps). Existing technologies have proposed a variety of architectures. For example, an intelligent operation and maintenance solution that combines AI, big data, RPA (robotic process automation) and other technologies creates "operation and maintenance digital employees" through the combination of multimodal input, intent understanding, knowledge graphs, and AIGC (generative AI) models to improve the intelligence level of network operation and maintenance.

[0003] Thanks to the emergence of the AIGC model, the output results of various intelligent operation and maintenance solutions have become more and more accurate, providing a new path to improve operation and maintenance efficiency. However, due to the large number of knowledge points involved in operation and maintenance, and the feature conversion process after multimodal input may lead to inaccurate information transmission and other problems, the intelligent operation and maintenance solutions based on the AIGC model will still have occasional unrealistic output results or other errors. The most serious problem is that the model usually only outputs content step by step, and it cannot judge whether the result is reliable. Therefore, for operation and maintenance personnel, in order to ensure the reliability of operation and maintenance, each output result must be manually reviewed, which narrows the efficiency gap between intelligent operation and maintenance and manual operation and maintenance, and also makes intelligent operation and maintenance meaningless.

[0004] Therefore, how to make the operation and maintenance system aware of its own limitations and promptly alert risks when accurate judgments cannot be made is the key to improving operation and maintenance efficiency, and is also a technical problem that is currently difficult to solve. Summary of the invention

[0005] In view of the problem that the operation and maintenance system in the prior art cannot judge whether the output result is reliable, resulting in the difficulty in ensuring the operation and maintenance effect and efficiency, the present invention provides an intelligent operation and maintenance method and system for power grid dispatch automation based on a multimodal large model. By processing the input information from different information sources, different operation and maintenance plan texts are obtained, and then the historical operation and maintenance plans are retrieved based on the keywords of the input information, and a three-party confirmation is performed. When the matching degree is lower than the threshold, the output result is considered unreliable. The present invention solves the problem that generative AI is prone to misexpression through a three-party confirmation method, so that the operation and maintenance system can judge and prompt the reliability of the results when outputting the results, eliminating the frequent review of the operation and maintenance personnel, and greatly improving the reliability and efficiency of the operation and maintenance.

[0006] The following is the technical solution of the present invention.

[0007] The method for intelligent operation and maintenance of power grid dispatching automation based on multi-modal large model includes the following steps: S1: Pre-build a multimodal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model; S2: input text input information and image input information, and superimpose mark information on the image input information; S3: The image recognition model in the multimodal large model recognizes the image input information and the label information, and outputs text description information; S4: The language processing model in the multimodal large model compares and synthesizes the text input information and the text description information to generate a collection of text clues; S5: The AIGC operation and maintenance model in the multimodal large model independently reads the text input information and the text clue collection, outputs the corresponding different operation and maintenance plan texts, and searches for historical operation and maintenance plans based on the keywords of the text clue collection to obtain search results; S6: Determine the matching degree between different operation and maintenance solution texts and search results. If the matching degrees are all lower than the threshold, execute S7. Otherwise, output the operation and maintenance solution texts and search results whose matching degrees are not lower than the threshold. S7: Output the operation and maintenance plan text and search results, and prompt risk information. If a change instruction is received, obtain the changed text input information and / or text description information based on the change instruction, and return to S4. If a reset instruction is received, return to S2.

[0008] In the present invention, a multimodal large model composed of different sub-models is used to support the input and processing of text and images, wherein the text input is directly converted into text input information, and the image input information and marking information are converted into text description information. The text input information and the text description information are used to generate a text clue collection to express the complete input content. Subsequently, the present invention independently reads the text input information and the different operation and maintenance plan texts obtained by the text clue collection, and retrieves the historical operation and maintenance plans according to the text clue collection. Since one of the operation and maintenance plan texts only considers the text input information, and the other operation and maintenance plan text considers both the text input information and the text description information (indirectly considering the image content), when it is uncertain whether the expression of the text or image is clear, the reliability of the two operation and maintenance plan texts is questionable. Therefore, the present invention continues to verify by comparing the historical operation and maintenance plans, and can determine the reliability of the operation and maintenance plan text, and clearly prompt the risks while displaying them. On the one hand, it can be used as a reference for the operation and maintenance personnel, and on the other hand, it does not require the operation and maintenance personnel to verify the reliability of each result, thereby improving the operation and maintenance efficiency.

[0009] Preferably, the S1: pre-builds a multimodal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model, including: Building a BERT model based on a number of Transformer encoders, and training the BERT model to obtain a language processing model; Building a YOLO model based on a real-time target detection algorithm, and training the YOLO model to obtain an image recognition model; Based on the Transformer architecture, a GPT model is constructed using a multi-layer self-attention mechanism and position encoding, and the GPT model is trained to obtain the AIGC operation and maintenance model.

[0010] Preferably, the step S2: inputting text input information and image input information, and superimposing marking information on the image input information, includes: Enter text input information by voice or text input; Input image input information by shooting or importing; The image input information is displayed, an operator draws a closed figure in the image input information to form marking information, the coordinates of the marking information are recorded, and the marking information is superimposed on the image input information based on the coordinate position.

[0011] Preferably, the step S3: recognizing the image input information and the marking information by the image recognition model in the multimodal large model and outputting text description information includes: The image recognition model in the multimodal large model performs target detection on the image input information, identifies the scene in the image, outputs scene description information, and performs separate target detection on the image area involved in the marking information, identifies and outputs device description information about the target device in the image area; Based on the preset description template, the scene description information and the device description information are integrated to output text description information.

[0012] Preferably, the step S4: comparing and synthesizing the text input information and the text description information by the language processing model in the multimodal large model to generate a text clue collection includes: The language processing model in the multimodal large model recognizes the text input information and text description information, obtains keywords related to the operation and maintenance task as clue information, and represents the extracted clue information in a structured manner; The clue information is aligned and fused to obtain a collection of text clues.

[0013] Preferably, in S5: the AIGC operation and maintenance model in the multimodal large model independently reads the text input information and the text clue collection, outputs the corresponding different operation and maintenance solution texts, and searches for historical operation and maintenance solutions based on keywords in the text clue collection to obtain search results, including: The AIGC operation and maintenance model in the multimodal large model reads the text input information and outputs the first operation and maintenance solution text based on the text input information; The AIGC operation and maintenance model in the multimodal large model reads the text clue collection, and outputs the second operation and maintenance solution text based on the text clue collection; At the same time, historical operation and maintenance plans are retrieved, and the keywords in the text clue collection are used to search among the historical operation and maintenance plans, and the operation and maintenance plan corresponding to the training set with the highest relevance is used as the search result.

[0014] In the present invention, the text input information and the text description information respectively correspond to the information content carried by the text input and the picture input in the operation and maintenance scenario, and the text clue collection carries all the input content in the current operation and maintenance scenario. Since it is impossible to ensure whether the content is complete or whether each model understands the meaning, two different operation and maintenance plan texts are output respectively, and the historical operation and maintenance plans are retrieved by keywords as a reference. The output result is considered credible only when the description of the operation and maintenance plan text has a high degree of match with the historical operation and maintenance plan. This solves the problem that the existing generative AI model cannot evaluate whether its own output content is credible, and also avoids the problems of missing information and inadequate scene understanding that may be caused by keyword retrieval alone.

[0015] Preferably, the step S6: determining the matching degree between different operation and maintenance solution texts and the search results, if the matching degrees are all lower than a threshold, then executing S7, otherwise outputting the operation and maintenance solution texts and the search results whose matching degrees are not lower than the threshold, includes: Determine the judgment dimension of the matching degree, compare different operation and maintenance solution texts with the search results based on the judgment dimension, and obtain the matching degree between each operation and maintenance solution text and the search results; If the matching degrees are all lower than the threshold, execute S7; Otherwise, the operation and maintenance solution text and search results whose matching degree is not less than the threshold are output, and the operation and maintenance solution text and search results will be recognized as reliable solutions.

[0016] Preferably, the S7: outputting the operation and maintenance plan text and the search results, and prompting the risk information, if a change instruction is received, obtaining the changed text input information and / or text description information based on the change instruction, and returning to S4, if a reset instruction is received, returning to S2, including: Use different colors to mark the same and different parts of the operation and maintenance plan text and search results, and use risk information prompts to cover the display area of ​​the operation and maintenance plan text and search results; After a confirmation button in the risk information is clicked, the risk information is cleared and the operation and maintenance plan text and search results are redisplayed; If the text input information and / or text description information is selected to be changed, the text input information and / or text description information is fully displayed and set to be editable, and after editing based on the change instruction, the changed text input information and / or text description information is generated; If a reset command is received, return to S2.

[0017] The present invention also provides an automated intelligent operation and maintenance system for power grid dispatching based on a multimodal large model, comprising an operation and maintenance server and an operation and maintenance terminal, wherein the operation and maintenance server and the operation and maintenance terminal are configured to execute the above-mentioned automated intelligent operation and maintenance method for power grid dispatching based on a multimodal large model.

[0018] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the above-mentioned power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model are implemented.

[0019] The present invention also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned power grid dispatching automation intelligent operation and maintenance method based on a multi-modal large model are implemented.

[0020] The substantial effects of the present invention include: The present invention realizes the comprehensive processing capability of text and image information by constructing a multimodal large model composed of a language processing model, an image recognition model, and an AIGC operation and maintenance model. This multimodal input method allows operation and maintenance personnel to input information in a variety of ways, improving the flexibility and accuracy of information input. At the same time, the image input information is subjected to target detection through the image recognition model, and the key information in the image is converted into text description information, which further enriches the description of the operation and maintenance scenario and provides a more comprehensive information basis for the subsequent generation of operation and maintenance solutions.

[0021] The present invention outputs two different operation and maintenance plan texts by independently reading text input information and a collection of text clues synthesized from the text input information and text description information. At the same time, the consistency between the operation and maintenance plan text and the search results is evaluated through a matching judgment mechanism in combination with the search results of historical operation and maintenance plans based on keywords in the text clue collection, which can further verify the reliability and feasibility of the operation and maintenance plan. When the matching degree is lower than the threshold, the system will prompt risk information and allow the operation and maintenance personnel to make changes or reset operations as needed. This solves the problem that generative AI may "talk nonsense" in the prior art, thereby improving the accuracy of operation and maintenance decisions.

[0022] Finally, the present invention enhances the readability and comprehensibility of the operation and maintenance plan by marking the same and different parts of the operation and maintenance plan text and search results with different colors, and covering the display area with risk information prompts. These humanized designs enable operation and maintenance personnel to grasp key information more quickly and improve operation and maintenance efficiency.

[0023] In summary, the present invention realizes a comprehensive understanding and efficient processing of operation and maintenance scenarios through innovations such as multimodal information input, dual operation and maintenance solution output, matching judgment, and humanized display methods, improves operation and maintenance efficiency and accuracy, and provides a strong technical guarantee for the safe and stable operation of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution will be clearly and completely described below in combination with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0027] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0028] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0029] The technical solution of the present invention is described in detail with specific embodiments below. The embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0030] Embodiment 1: A method for intelligent operation and maintenance of power grid dispatching automation based on a multi-modal large model, such as Figure 1 As shown, the following steps are included: S1: Pre-build a multimodal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model.

[0031] The method comprises: building a BERT model based on a plurality of Transformer encoders, and training the BERT model to obtain a language processing model; Building a YOLO model based on a real-time target detection algorithm, and training the YOLO model to obtain an image recognition model; Based on the Transformer architecture, a GPT model is constructed using a multi-layer self-attention mechanism and position encoding, and the GPT model is trained to obtain the AIGC operation and maintenance model.

[0032] In this embodiment, language processing is performed by constructing a BERT model. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model based on the Transformer encoder. Its core is to capture contextual information in the text through a multi-layer bidirectional Transformer structure. During the construction process, the parameters of the Transformer encoder are first designed and initialized, including the dimensions of the input layer, hidden layer, and output layer. Then a large amount of text data is used to train the BERT model so that it can deeply understand the context and meaning of the language. After training, this model can be used for various language processing tasks, such as various semantic analyses related to operation and maintenance.

[0033] Secondly, in order to perform image recognition, this embodiment adopts the YOLO (You Only Look Once) model. YOLO is a real-time target detection algorithm that can quickly and accurately identify various objects in pictures. The network structure of YOLO includes convolutional layers, pooling layers, and fully connected layers to extract image features and predict target locations and categories. Then, the labeled image data set is used for training, and the model parameters are optimized through the back propagation algorithm and loss function so that the model can accurately identify targets in the image. By training the YOLO model, it can have powerful image recognition capabilities, so as to identify scenes, target devices, etc. during operation and maintenance.

[0034] Finally, in order to support AIGC operation and maintenance, this embodiment constructs a GPT (Generative Pre-trained Transformer) model. GPT is a generative pre-trained model based on the Transformer architecture, which uses a multi-layer self-attention mechanism and position encoding to generate coherent text content. During the construction process, the parameters of the Transformer decoder are first designed and initialized, including the dimensions of the input layer, hidden layer, and output layer. Next, a large-scale text corpus is used for pre-training. The training samples of this embodiment are mainly historical operation and maintenance solutions, etc. The model is trained through autoregressive language modeling tasks to enable it to learn to generate text that conforms to language rules. During the training process, an adaptive learning rate and optimization algorithm are used to update the model parameters until the model achieves optimal performance on the validation set. By training the GPT model, it can be enabled to generate high-quality text content, thereby supporting AIGC operation and maintenance tasks.

[0035] S2: Input text input information and image input information, and superimpose mark information on the image input information.

[0036] Includes: inputting text input information by voice or text input; Input image input information by shooting or importing; The image input information is displayed, an operator draws a closed figure in the image input information to form marking information, the coordinates of the marking information are recorded, and the marking information is superimposed on the image input information based on the coordinate position.

[0037] In this embodiment, the system receives text input, which can be achieved through voice-to-text technology or direct text input. For example, voice input "the oil level gauge reading of XX transformer is low", "XX equipment has abnormal operation sound and excessive vibration", etc.

[0038] At the same time, image input is completed by taking new photos or importing pictures from existing resources, which ensures the breadth of data sources. In the image display interface of this embodiment, the operator can intuitively see the image content and accurately mark the key areas or objects in the image by drawing closed graphics (such as rectangles, circles, etc.). These marks not only help to clarify the focus, but also provide accurate positioning information for subsequent analysis or processing. Generally speaking, the marked circled parts are mainly target devices. This method can facilitate the system to grasp the key content in subsequent processing.

[0039] To achieve this, the system records the coordinate data of each marker, which indicates the precise location of the marker information on the image. Based on these coordinates, the system can accurately overlay the marker information on the original image to form an information-rich image dataset containing an additional annotation layer.

[0040] S3: The image recognition model in the multimodal large model recognizes the image input information and label information, and outputs text description information.

[0041] Including: the image recognition model in the multimodal large model performs target detection on the image input information, identifies the scene in the image, outputs the scene description information, and performs separate target detection on the image area involved in the marking information, identifies and outputs the device description information about the target device in the image area; Based on the preset description template, the scene description information and the device description information are integrated to output text description information.

[0042] In this embodiment, the image recognition model in the multimodal large model performs comprehensive target detection on the image input information. This process involves the recognition of various elements in the image, including people, objects, background, etc., so as to generate description information of the overall scene. This description is not limited to the conspicuous or main elements in the image, but also covers the overall environment and atmosphere presented by the image. For example, the image recognition model in the multimodal large model performs comprehensive target detection on the picture, identifies the main elements in the picture, including transformers, fences, telephone poles, etc., and generates description information of the overall scene, such as "The picture shows a power grid operation and maintenance scene, with a large transformer in the center, surrounded by fences, and several telephone poles nearby."

[0043] The model then performs more detailed target detection on the image area specified by the tag information. This usually involves identifying specific equipment, objects, or specific parts of the image, and outputting detailed descriptions of these target equipment. This targeted detection and analysis makes the output description information more accurate and specific. For example, the tag information marks the area of ​​the transformer, and the model performs more detailed target detection on this specific equipment and outputs detailed descriptions of the transformer, such as "The transformer model is XX, the current oil level is abnormal, there may be signs of oil leakage, and the heat sink is in good condition."

[0044] Finally, based on the preset description template, the model integrates the scene description information with the device description information. This step ensures that the output text description information is both comprehensive and structured, which is convenient for users to quickly understand and use. The design of the description template can be adjusted according to the actual application scenario to meet the needs of different fields and uses. For example, based on the preset description template, the text description information is output: "Overall environment: a transformer is surrounded by a fence, there are several electric poles nearby, and the overall environment is safe; target device: transformer model XX; abnormal judgment: oil level is abnormal, heat sink is normal, and the current status is general." S4: The language processing model in the multimodal large model compares and synthesizes the text input information and the text description information to generate a collection of text clues.

[0045] The process includes: using a language processing model in a multimodal large model to identify text input information and text description information, obtaining keywords related to the operation and maintenance task as clue information, and representing the extracted clue information in a structured manner; The clue information is aligned and fused to obtain a collection of text clues.

[0046] For example, if the text input information mentions "equipment heating", and the text description information recognizes "equipment damage" from the image, the fusion will result in "equipment damage, heating". Alignment takes into account that when the information is complex, the description objects of the equipment need to be aligned to avoid the situation where the descriptions of different equipment are integrated into the description of one equipment.

[0047] S5: The AIGC operation and maintenance model in the multimodal large model reads the text input information and the text clue collection independently, outputs the corresponding different operation and maintenance plan texts, and searches for historical operation and maintenance plans based on the keywords of the text clue collection to obtain search results.

[0048] The method includes: reading text input information by an AIGC operation and maintenance model in the multimodal large model, and outputting a first operation and maintenance solution text based on the text input information; The AIGC operation and maintenance model in the multimodal large model reads the text clue collection, and outputs the second operation and maintenance solution text based on the text clue collection; At the same time, historical operation and maintenance plans are retrieved, and the keywords in the text clue collection are used to search among the historical operation and maintenance plans, and the operation and maintenance plan corresponding to the training set with the highest relevance is used as the search result.

[0049] In this embodiment, the text input information and the text description information respectively correspond to the information content carried by the text input and image input in the operation and maintenance scenario, and the text clue collection carries all the input content in the current operation and maintenance scenario. Since it is impossible to ensure the completeness of the content and whether each model understands the meaning, two different operation and maintenance plan texts are output respectively, and the historical operation and maintenance plans are retrieved by keywords as a reference. The output result is considered credible only when the description of the operation and maintenance plan text has a high degree of match with the historical operation and maintenance plan. This solves the problem that the existing generative AI model cannot evaluate whether its own output content is credible. At the same time, it also avoids the problems of missing information and inadequate scene understanding that may be caused by keyword retrieval alone.

[0050] S6: Determine the matching degree between different operation and maintenance solution texts and the search results. If the matching degrees are all lower than the threshold, execute S7. Otherwise, output the operation and maintenance solution text and the search results whose matching degrees are not lower than the threshold.

[0051] Including: determining the judgment dimension of the matching degree, comparing different operation and maintenance solution texts with the search results based on the judgment dimension, and obtaining the matching degree of each operation and maintenance solution text with the search results; If the matching degrees are all lower than the threshold, execute S7; Otherwise, the operation and maintenance solution text and search results whose matching degree is not less than the threshold are output, and the operation and maintenance solution text and search results will be recognized as reliable solutions.

[0052] In this embodiment, semantic similarity is selected as the judgment dimension of the matching degree. When judging, the text or vocabulary is represented as a vector in a high-dimensional space, and then the similarity between these vectors (such as cosine similarity, Euclidean distance, etc.) is calculated. By calculating the similarity between two vocabulary vectors, the semantic similarity between them can be evaluated, and then the matching degree can be obtained.

[0053] This embodiment can avoid situations where multiple output results are unreliable. For example, keywords may not accurately reflect the actual situation, the retrieved historical operation and maintenance solutions are not suitable, and the two different operation and maintenance solution texts do not match the search results because the subjective expressions of the operation and maintenance personnel are taken into account. In this case, this embodiment will prompt risks. For another example, the subjective expressions of the operation and maintenance personnel are misjudged or ambiguous, resulting in misunderstandings between the two different operation and maintenance solution texts, and the historical operation and maintenance solutions retrieved by keywords are more reasonable. In this case, this embodiment will also prompt risks. Only when the two output methods match each other will this embodiment recognize the reliability of the output results.

[0054] S7: Output the operation and maintenance plan text and search results, and prompt risk information. If a change instruction is received, obtain the changed text input information and / or text description information based on the change instruction, and return to S4. If a reset instruction is received, return to S2.

[0055] Including: using different colors to mark the same parts and different parts in the operation and maintenance plan text and search results, and using risk information prompts to cover the display area of ​​the operation and maintenance plan text and search results; After a confirmation button in the risk information is clicked, the risk information is cleared and the operation and maintenance plan text and search results are redisplayed; If the text input information and / or text description information is selected to be changed, the text input information and / or text description information is fully displayed and set to be editable, and after editing based on the change instruction, the changed text input information and / or text description information is generated; If a reset command is received, return to S2.

[0056] In the system interface of this embodiment, the operation and maintenance plan text and search results are clearly displayed in two adjacent areas. The operation and maintenance plan text is displayed in blue font, and the search results are displayed in green font, so that users can quickly distinguish them. Among them, the same parts are highlighted with a yellow background, and the different parts are marked with a red background. The risk information prompt is covered in the relevant area with a red border and a flashing effect. When the user clicks the confirmation button in the risk information, the system clears the risk prompt and displays the operation and maintenance plan text and search results in normal colors again.

[0057] If the user chooses to change the operation and maintenance plan or the text information in the search results, the system will display this information in full and set it to editable state. The user can directly modify the text content in the edit box, and the system will save the changes in real time and generate the modified text input information and / or text description information.

[0058] At any time, if the user wants to start over, he can click the "Reset" button on the interface. After receiving the reset instruction, the system clears all current comparison results and editing contents and returns to step S2.

[0059] In this embodiment, a multimodal large model composed of different sub-models is used to support the input and processing of text and images, wherein the text input is directly converted into text input information, and the image input information and the marking information are converted into text description information, and a text clue collection is generated using the text input information and the text description information to express the complete input content. Subsequently, the present invention independently reads the text input information and the different operation and maintenance solution texts obtained by the text clue collection, and retrieves the historical operation and maintenance solutions according to the text clue collection. Since one of the operation and maintenance solution texts only considers the text input information, and the other operation and maintenance solution text considers both the text input information and the text description information (indirectly considering the image content), in the case of uncertainty as to whether the expression of the text or image is clear, the reliability of the two operation and maintenance solution texts is questionable. Therefore, the present invention continues to verify by comparing the historical operation and maintenance solutions, and can determine the reliability of the operation and maintenance solution text, and clearly prompt the risks while displaying them. On the one hand, it can be used as a reference for the operation and maintenance personnel, and on the other hand, it does not require the operation and maintenance personnel to verify the reliability of each result, thereby improving the operation and maintenance efficiency.

[0060] Embodiment 2: A power grid dispatching automation intelligent operation and maintenance system based on a multimodal large model, comprising an operation and maintenance server and an operation and maintenance terminal, wherein the operation and maintenance server and the operation and maintenance terminal are configured to execute the above-mentioned power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model.

[0061] The operation and maintenance server mainly executes the deployment and operation of multimodal large models and generates operation and maintenance plans. It is also responsible for data storage and management, safely stores all operation data, and provides data query and backup functions.

[0062] The operation and maintenance terminal mainly provides a user interface to display the system status and operation and maintenance plan, and provides input modules such as touch screen, keyboard, microphone, etc., as well as sending requests to the operation and maintenance server and receiving and processing responses.

[0063] The operation and maintenance server works with the operation and maintenance terminal to realize the automated and intelligent operation and maintenance of the system, and improve the system stability and operating efficiency.

[0064] Embodiment 3: This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of the above-mentioned power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model are implemented.

[0065] Embodiment 4: This embodiment also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the above-mentioned power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model are implemented.

[0066] Through the above embodiments, the substantial effects that can be achieved include: By building a multimodal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model, the comprehensive processing capability of text and image information is achieved. This multimodal input method allows operation and maintenance personnel to input information in a variety of ways, improving the flexibility and accuracy of information input. At the same time, the image recognition model performs target detection on the image input information, converts the key information in the image into text description information, further enriches the description of the operation and maintenance scenario, and provides a more comprehensive information basis for the subsequent generation of operation and maintenance solutions.

[0067] By independently reading the text input information and the text clue collection synthesized from the text input information and the text description information, two different operation and maintenance plan texts are output. At the same time, the search results of historical operation and maintenance plans based on the keywords of the text clue collection are combined, and the consistency between the operation and maintenance plan text and the search results is evaluated through the matching judgment mechanism, which can further verify the reliability and feasibility of the operation and maintenance plan. When the matching degree is lower than the threshold, the system will prompt risk information and allow the operation and maintenance personnel to make changes or reset operations as needed. It solves the problem that generative AI may "talk nonsense" in the existing technology, thereby improving the accuracy of operation and maintenance decisions.

[0068] The readability and comprehensibility of the operation and maintenance plan are enhanced by marking the same and different parts of the operation and maintenance plan text and search results with different colors, and covering the display area with risk information prompts. These humanized designs enable operation and maintenance personnel to grasp key information more quickly and improve operation and maintenance efficiency.

[0069] In summary, this embodiment achieves a comprehensive understanding and efficient processing of operation and maintenance scenarios through innovations such as multimodal information input, dual operation and maintenance solution output, matching judgment, and humanized display methods, improves operation and maintenance efficiency and accuracy, and provides strong technical guarantees for the safe and stable operation of the power system.

[0070] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0071] In the embodiments provided in the present application, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the embodiments of the structure described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another structure, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0073] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0075] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for intelligent operation and maintenance of power grid dispatching automation based on a multi-modal large model, characterized in that: The following steps are involved: S1: Pre-build a multimodal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model; S2: input text input information and image input information, and superimpose mark information on the image input information; S3: The image recognition model in the multimodal large model recognizes the image input information and the label information, and outputs text description information; S4: The language processing model in the multimodal large model compares and synthesizes the text input information and the text description information to generate a collection of text clues; S5: The AIGC operation and maintenance model in the multimodal large model independently reads the text input information and the text clue collection, outputs the corresponding different operation and maintenance plan texts, and searches for historical operation and maintenance plans based on the keywords of the text clue collection to obtain search results; S6: Determine the matching degree between different operation and maintenance solution texts and search results. If the matching degrees are all lower than the threshold, execute S7. Otherwise, output the operation and maintenance solution texts and search results whose matching degrees are not lower than the threshold. S7: Output the operation and maintenance plan text and search results, and prompt risk information. If a change instruction is received, obtain the changed text input information and / or text description information based on the change instruction, and return to S4. If a reset instruction is received, return to S2.

2. The method for automatic intelligent operation and maintenance of power grid dispatching based on multimodal large model according to claim 1 is characterized in that: S1: pre-build a multi-modal large model consisting of a language processing model, an image recognition model, and an AIGC operation and maintenance model, including: Building a BERT model based on a number of Transformer encoders, and training the BERT model to obtain a language processing model; Building a YOLO model based on a real-time target detection algorithm, and training the YOLO model to obtain an image recognition model; Based on the Transformer architecture, a GPT model is constructed using a multi-layer self-attention mechanism and position encoding, and the GPT model is trained to obtain the AIGC operation and maintenance model.

3. The method for automatic intelligent operation and maintenance of power grid dispatching based on multimodal large model according to claim 1 is characterized in that: The step S2: inputting text input information and image input information, and superimposing marking information on the image input information, includes: Enter text input information by voice or text input; Input image input information by shooting or importing; The image input information is displayed, an operator draws a closed figure in the image input information to form marking information, the coordinates of the marking information are recorded, and the marking information is superimposed on the image input information based on the coordinate position.

4. The method for automatic intelligent operation and maintenance of power grid dispatching based on multi-modal large model according to claim 1 is characterized in that: S3: the image recognition model in the multimodal large model recognizes the image input information and the marking information, and outputs text description information, including: The image recognition model in the multimodal large model performs target detection on the image input information, identifies the scene in the image, outputs scene description information, and performs separate target detection on the image area involved in the marking information, identifies and outputs device description information about the target device in the image area; Based on the preset description template, the scene description information and the device description information are integrated to output text description information.

5. The method for automatic intelligent operation and maintenance of power grid dispatching based on multi-modal large model according to claim 1 is characterized in that: S4: the language processing model in the multimodal large model compares and synthesizes the text input information and the text description information to generate a text clue collection, including: The language processing model in the multimodal large model recognizes the text input information and text description information, obtains keywords related to the operation and maintenance task as clue information, and represents the extracted clue information in a structured manner; The clue information is aligned and fused to obtain a collection of text clues.

6. The method for automatic intelligent operation and maintenance of power grid dispatching based on multi-modal large model according to claim 1 is characterized in that: S5: The AIGC operation and maintenance model in the multimodal large model independently reads the text input information and the text clue collection, outputs the corresponding different operation and maintenance solution texts, and searches for historical operation and maintenance solutions based on the keywords of the text clue collection to obtain search results, including: The AIGC operation and maintenance model in the multimodal large model reads the text input information and outputs the first operation and maintenance solution text based on the text input information; The AIGC operation and maintenance model in the multimodal large model reads the text clue collection, and outputs the second operation and maintenance solution text based on the text clue collection; At the same time, historical operation and maintenance plans are retrieved, and the keywords in the text clue collection are used to search among the historical operation and maintenance plans, and the operation and maintenance plan corresponding to the training set with the highest relevance is used as the search result.

7. The method for automatic intelligent operation and maintenance of power grid dispatching based on multi-modal large model according to claim 6 is characterized in that: S6: Determine the matching degree between different operation and maintenance solution texts and search results. If the matching degree is lower than the threshold, execute S7. Otherwise, output the operation and maintenance solution text and search results whose matching degree is not lower than the threshold, including: Determine the judgment dimension of the matching degree, compare different operation and maintenance solution texts with the search results based on the judgment dimension, and obtain the matching degree between each operation and maintenance solution text and the search results; If the matching degrees are all lower than the threshold, execute S7; Otherwise, the operation and maintenance solution text and search results whose matching degree is not less than the threshold are output, and the operation and maintenance solution text and search results will be recognized as reliable solutions.

8. The method for automatic intelligent operation and maintenance of power grid dispatching based on multi-modal large model according to claim 1 is characterized in that: S7: outputting the operation and maintenance plan text and search results, and prompting risk information. If a change instruction is received, obtaining the changed text input information and / or text description information based on the change instruction, and returning to S4. If a reset instruction is received, returning to S2, including: Use different colors to mark the same and different parts of the operation and maintenance plan text and search results, and use risk information prompts to cover the display area of ​​the operation and maintenance plan text and search results; After a confirmation button in the risk information is clicked, the risk information is cleared and the operation and maintenance plan text and search results are redisplayed; If the text input information and / or text description information is selected to be changed, the text input information and / or text description information is fully displayed and set to be editable, and after editing based on the change instruction, the changed text input information and / or text description information is generated; If a reset command is received, return to S2.

9. A power grid dispatching automation intelligent operation and maintenance system based on a multi-modal large model, including an operation and maintenance server and an operation and maintenance terminal, characterized in that: The operation and maintenance server and the operation and maintenance terminal are configured to execute the power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model as described in any one of claims 1 to 8.

10. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the power grid dispatching automation intelligent operation and maintenance method based on a multimodal large model as described in any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium stores computer executable instructions, which, when loaded and executed by a processor, implement the steps of the method for automated intelligent operation and maintenance of power grid dispatching based on a multimodal large model as described in any one of claims 1 to 8.

Images