Fine adjustment method and device for multi-modal large model of electric power inspection

Abstract

The invention discloses an electric power inspection multi-mode large model fine tuning method and device, and the method comprises the steps: carrying out the time series data spectrum preprocessing of a one-dimensional current signal through a Grubrum angle field algorithm, and obtaining a two-dimensional current spectrum; combining the visible light / infrared image and the diagnosis text to construct an instruction fine tuning data set; on the basis of the universal multi-modal backbone model, designing a 1 + N LoRA fine tuning framework comprising a current map encoder and an electrical semantic optimization LoRA module, and obtaining an adjusted model; respectively outputting preliminary diagnosis probability vectors through the visual branch and the current expert branch, and calculating the cosine similarity of the two preliminary diagnosis probability vectors; if the cosine similarity is lower than a preset threshold value or the physical paradox exists, triggering reflection logic; and adjusting the weight of the electrical semantic optimization LoRA module, recalculating a feature fusion result, and outputting a final diagnosis conclusion. The method solves the problems that in the prior art, single-mode sensing is limited, and diagnosis precision is insufficient due to environmental interference.

Description

Technical Field

[0001] This invention relates to the fields of power equipment testing and artificial intelligence technology, specifically to a method and apparatus for fine-tuning a multimodal large model for power inspection. Background Technology

[0002] Power grid inspection is a core component of ensuring the safe and stable operation of the power grid. Its core objective is to identify power equipment defects in a timely and accurate manner and prevent the escalation of faults. Traditional power grid inspection mainly relies on manual on-site inspections, which is inefficient and limited by personnel experience and environmental conditions. With the development of artificial intelligence technology, single-modal automated models are gradually being applied to inspection scenarios. For example, visual models based on CNN or Transformer analyze visible light and infrared images to identify surface defects on equipment, playing an important role in detecting apparent anomalies such as foreign object adhesion, casing damage, and localized overheating. In recent years, multimodal large language models (MLLM) have shown broad application prospects in the field of power grid inspection due to their powerful multi-source data fusion capabilities, providing the possibility of integrating visual, electrical parameter, and other types of data for comprehensive diagnosis.

[0003] However, existing power inspection technologies still have many problems that urgently need to be solved: On the one hand, single-modal models have obvious perception limitations. Visual modalities can only capture the external appearance features of equipment and cannot perceive internal electrical problems without significant visual features, such as transformer winding deformation and cable hidden faults. Moreover, the one-dimensional current data sequences collected at high frequencies in substations are too long, far exceeding the processing window of multimodal large models, and the pure numerical sequences lack physical semantics, making them difficult for the model to understand effectively. On the other hand, single visual models are easily affected by environmental factors such as sunlight refraction, producing false hot spots and other "visual illusions," leading to incorrect diagnostic conclusions due to the lack of electrical physical constraints. In addition, there is a natural gap between current data and visual images in terms of data structure and semantic expression, and the lack of an effective cross-modal alignment mechanism makes multi-source data fusion fail, failing to achieve a diagnostic effect of 1+1>2, and making it difficult to meet the high standards of inspection accuracy and robustness required by power grid operation and maintenance.

[0004] Therefore, there is an urgent need for a multimodal large-scale model fine-tuning method for power line inspection to solve the problems of single-modal perception limitations and insufficient diagnostic accuracy due to susceptibility to environmental interference in existing technologies. Summary of the Invention

[0005] To address these issues, the present invention provides a method and apparatus for fine-tuning a multimodal large model in power line inspection, which solves the problems of limitations in single-modal perception, difficulty in understanding current time-series data by the model, lack of effective alignment mechanism for multimodal data, and susceptibility to environmental interference leading to insufficient diagnostic accuracy in existing power line inspection systems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for fine-tuning a multimodal large model of power line inspection, characterized in that it includes: The Gram angle field algorithm is used to perform time-series data mapping preprocessing on the one-dimensional current signal collected by the substation SCADA system to obtain a two-dimensional current spectrum. Based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text, a command fine-tuning dataset is constructed; Based on the instruction fine-tuning dataset, a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module is designed on the basis of a general multimodal backbone model to obtain the adjusted model. Based on the adjusted model, preliminary diagnostic probability vectors are output through the visual branch and the current expert branch, respectively, and the cosine similarity between the two preliminary diagnostic probability vectors is calculated. If the cosine similarity is lower than a preset threshold or a physical paradox exists, the reflection logic is triggered. The weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion results are recalculated, and the final diagnostic conclusion is output.

[0007] As a preferred scheme for fine-tuning a multimodal large model of power inspection, the preprocessing process for time-series data mapping of one-dimensional current signals using the Gram angle field algorithm includes: adaptive normalization and polar coordinate transformation, Gram angle and difference field generation, and multi-scale GAF fusion.

[0008] As a preferred scheme for fine-tuning a multimodal large model of power line inspection, the weight update formula of the 1+N LoRA fine-tuning framework is:

[0009] In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

[0010] As a preferred scheme for fine-tuning a multimodal large model of power line inspection, the formula for calculating the cosine similarity is:

[0011] In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

[0012] As a preferred scheme for a multimodal large-scale model fine-tuning method for power line inspection, the expression for the final diagnostic conclusion is:

[0013] In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

[0014] This invention also provides a power inspection multimodal large model fine-tuning device, employing the above-mentioned power inspection multimodal large model fine-tuning method, including: The one-dimensional current signal processing unit is used to perform time-series data mapping preprocessing on the one-dimensional current signal acquired by the substation SCADA system using the Gram angle field algorithm to obtain a two-dimensional current spectrum. The instruction fine-tuning dataset construction unit is used to construct an instruction fine-tuning dataset based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text; The 1+N LoRA fine-tuning framework design unit is used to fine-tune the dataset based on the instructions, and design a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module on the basis of a general multimodal backbone model, so as to obtain the adjusted model. The self-verifying reasoning framework construction unit is used to output preliminary diagnostic probability vectors through the visual branch and the current expert branch respectively based on the adjusted model, and calculate the cosine similarity between the two preliminary diagnostic probability vectors; if the cosine similarity is lower than a preset threshold or there is a physical paradox, the reflection logic is triggered; the weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion results are recalculated, and the final diagnostic conclusion is output.

[0015] As a preferred embodiment of a multimodal large-scale model fine-tuning device for power inspection, the one-dimensional current signal processing unit, in the process of performing time-series data mapping preprocessing on the one-dimensional current signal through the Gram angle field algorithm, includes: adaptive normalization and polar coordinate transformation, Gram angle and difference field generation, and multi-scale GAF fusion.

[0016] As a preferred embodiment of a multimodal large-scale model fine-tuning device for power line inspection, the weight update formula of the 1+N LoRA fine-tuning framework design unit is as follows:

[0017] In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

[0018] As a preferred embodiment of a multimodal large-scale model fine-tuning device for power line inspection, the formula for calculating the cosine similarity in the self-verifying inference framework construction unit is as follows:

[0019] In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

[0020] As a preferred embodiment of a multimodal large-scale model fine-tuning device for power line inspection, the expression for the final diagnostic conclusion in the self-verifying inference framework construction unit is:

[0021] In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

[0022] The present invention has the following advantages: First, it breaks through the barriers of modal fusion: by using the Gram corner field algorithm to transform one-dimensional current time series data into two-dimensional maps, it can achieve deep semantic alignment between current data and visual data without large-scale reconstruction of multimodal models, which greatly reduces the computational cost and training difficulty of cross-modal fusion.

[0023] Second, address the shortcomings of single-mode testing: integrate the internal electrical state of the equipment reflected by current data with the external appearance anomalies captured by visual data to form a cross-verification of physical mechanisms and appearance phenomena, solving the problem that traditional technologies cannot detect internal faults.

[0024] Third, improve diagnostic robustness: By injecting a 1+N LoRA fine-tuning framework with a diagonal constraint matrix of electrical terms, the electrical and physical logic constraints of the model are strengthened, effectively suppressing "visual illusions" caused by environmental interference such as sunlight refraction, and reducing erroneous diagnostic conclusions.

[0025] Fourth, optimize data adaptability: construct a three-in-one instruction set using a mixture of simulation and engineering data to make up for the lack of fault data; the multi-scale GAF fusion design takes into account both the local details and global trends of long-sequence current data, improving the model's adaptability to complex data.

[0026] Fifth, it possesses self-correcting capabilities: The innovative self-verifying reasoning framework dynamically adjusts model weights through conflict detection and reflective logic, highlighting weak signal characteristics and further improving the accuracy and reliability of fault diagnosis. Attached Figure Description

[0027] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0028] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0029] Figure 1 This is a flowchart illustrating a multimodal large-scale model fine-tuning method for power inspection provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the adjusted large model architecture in a multimodal large model fine-tuning method for power inspection provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the architecture of a multimodal large-scale model fine-tuning device for power inspection provided in Embodiment 2 of the present invention. Detailed Implementation

[0030] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0031] See Figure 1Embodiment 1 of the present invention provides a method for fine-tuning a multimodal large model of power inspection, comprising the following steps: S1. The one-dimensional current signal acquired by the substation SCADA system is preprocessed into a time-series data graph by using the Gram angle field algorithm to obtain a two-dimensional current graph. S2. Based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text, construct an instruction fine-tuning dataset; S3. Based on the instruction fine-tuning dataset, on the basis of the general multimodal backbone model, design a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module to obtain the adjusted model. S4. Based on the adjusted model, output preliminary diagnostic probability vectors through the visual branch and the current expert branch respectively, and calculate the cosine similarity between the two preliminary diagnostic probability vectors; if the cosine similarity is lower than the preset threshold or there is a physical paradox, trigger the reflection logic; adjust the weights of the electrical semantic optimization LoRA module, recalculate the feature fusion result, and output the final diagnostic conclusion.

[0032] In this embodiment, in step S1, the one-dimensional current signal acquired by the substation SCADA system (Supervisory Control And Data Acquisition) is preprocessed into a time-series data graph by using the Gram angle field algorithm to obtain a two-dimensional current graph.

[0033] Specifically, for one-dimensional current signals sampling frequency The Gram-angle field (GAF) algorithm is used for temporal data mapping preprocessing, mapping it into a two-dimensional image, laying the foundation for subsequent multimodal alignment.

[0034] The preprocessing process includes: adaptive normalization and polar coordinate transformation, Gram angle and / or difference field generation, and multi-scale GAF fusion.

[0035] Specifically, the adaptive normalization and polar coordinate transformation process is as follows: Adaptive normalization of current amplitude is performed. For current data containing surges, segmented normalization is adopted (normal segment [-1,1], transient segment retains relative amplitude) to avoid extreme value distortion angle encoding.

[0036]

[0037] In the formula, It is a current signal; This is the moving average of the current data within the current time window; This represents the standard deviation of current data within the same window.

[0038] Mapped to angles using the inverse cosine function:

[0039] In the formula, The angle is obtained by mapping one-dimensional current data after preprocessing. This is the one-dimensional current signal data after adaptive normalization.

[0040] In this embodiment, the formula for generating the Gram angle and / or difference field is:

[0041] In the formula, For the Gram angle and the field matrix, the first... i line, number j Column elements; The first angular difference field matrix in Gram i line, number j Column elements; , The first i , j The angle is obtained by mapping the normalized current data.

[0042] In this embodiment, the multi-scale GAF fusion process is as follows: Long sequences are windowed to generate an overlay image of local texture (high-frequency details) and global texture (trend features), thus solving the problem of excessive computational complexity of the original GAF for long sequences. First, the sequence is segmented into... Multiple overlapping windows:

[0043] In the formula, For windows; Step size; This represents the number of sampling points per cycle. It is the power frequency cycle, usually 50Hz.

[0044]

[0045] In the formula, The sampling frequency.

[0046] For each window Calculate the GASF matrix:

[0047] Calculate the GASF matrix for the downsampled sequence:

[0048] After unifying the local / global GAF to the target size, merge them:

[0049] In the formula, For the adjusted size of the first k A local current spectrum; The global current spectrum after size adjustment; The final fused current spectrum; K This represents the number of local windows; N The image size is defined.

[0050] In this embodiment, in step S2, a fine-tuning instruction dataset is constructed based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text.

[0051] Specifically, based on the aforementioned two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text, a three-in-one instruction fine-tuning dataset is constructed, comprising "current spectrum - visible light / infrared image - professional diagnostic text". This process is primarily implemented using a hybrid simulation / engineering data approach.

[0052] Among them, simulation data mainly uses simulation software to model and generate various types of fault current waveforms to supplement the problem of insufficient fault data.

[0053] Engineering data: Data is acquired from the substation SCADA system, the sampled current is extracted and timestamps are aligned to ensure tolerance. The inspection robot collects data on-site and completes the annotation of visible light / infrared images.

[0054] In this embodiment, in step S3, based on the instruction fine-tuning dataset, a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module is designed on the basis of the general multimodal backbone model to obtain the adjusted model.

[0055] Specifically, such as Figure 2 As shown, based on the frozen general multimodal backbone model (such as Qwen-VL), a dedicated current expert Adapter is designed. Its implementation mainly includes a lightweight current spectrum encoder (based on the projection layer of VisionTransformer) and a set of LoRA modules (Low-Rank Adaptation) optimized for electrical semantics.

[0056] In this embodiment, the rank decomposition matrix of the electrical semantic constraints is:

[0057] In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

[0058] Key domain knowledge injected:

[0059] In the formula, It is a diagonal matrix; diagonal elements Indicates the semantic weight of electrical terms.

[0060] In the feature fusion layer:

[0061] In the formula, The result of feature fusion; Image features extracted by the visual encoder; Sampled current feature map extracted for the current expert adapter; , The weight matrix adjusted for the electrical semantic LoRA module.

[0062] In this embodiment, in step S4, based on the adjusted model, preliminary diagnostic probability vectors are output through the visual branch and the current expert branch respectively, and the cosine similarity between the two preliminary diagnostic probability vectors is calculated; if the cosine similarity is lower than a preset threshold or there is a physical paradox, the reflection logic is triggered; the weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion result is recalculated, and the final diagnostic conclusion is output.

[0063] Specifically, based on the preliminary diagnostic probability vectors output by the visual branch and the current expert branch, the evidence consistency score between the two is calculated using cosine similarity:

[0064] In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

[0065] like ( (For example, if a preset conflict threshold is set, or a specific physical paradox is met, such as visually identifying a thermal defect but an electrical expert deeming it normal, then the reflection logic is triggered.)

[0066] During the reflection phase, the electrical semantic LoRA module defined in step 3 is adjusted to weight conflict-related electrical terms. A conflict adjustment vector is introduced. :

[0067] In the formula, This is an adjustment factor dynamically generated based on the type of conflict. For example, if visual detection indicates "sleeve heating," the adjustment factor corresponding to the terms "insulation aging" and "poor contact" is increased. The weight.

[0068] In reflective mode, the feature fusion layer It will be recalculated to highlight weak signals that were ignored in the initial inference:

[0069] In the formula, For the recipient Guided semantic attention weights ensure that the model is biased towards interpretations that conform to the logic of electrical physics.

[0070] The model will combine instructions with conflict context and modified features. The data is fed into a multimodal LLM decoder, which outputs the final conclusion:

[0071] In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

[0072] The application scenarios of this invention are as follows: In the daily inspection of substations, this invention can convert the current data collected by the SCADA system into a spectrum, which is then fused with equipment images and diagnostic text captured by the inspection robot. Through a finely tuned multimodal model, internal faults such as winding deformation and latent discharge can be accurately identified, replacing manual inspection and improving efficiency.

[0073] In emergency inspection scenarios of power grid faults, this invention can quickly align current data during the fault period with images of on-site equipment, and avoid misjudgments caused by environmental interference by using a self-verifying reasoning mechanism, providing reliable diagnostic conclusions for emergency repair decisions.

[0074] In the scenario of equipment inspection in new energy power plants, this invention can adapt to the high-frequency fluctuation of power generation equipment current data, integrate infrared thermal imaging images, realize multi-modal comprehensive diagnosis of equipment such as inverters and combiner boxes, and ensure the stable operation of new energy power generation systems.

[0075] In the intelligent operation and maintenance scenario of power equipment, this invention can continuously fine-tune a multimodal large model based on the instruction set constructed from historical current spectrum and equipment image data, thereby achieving early warning and location of equipment defects and reducing operation and maintenance costs.

[0076] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0077] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Example 2

[0078] See Figure 3 Embodiment 2 of the present invention also provides a multimodal large-scale power inspection fine-tuning device, comprising: The one-dimensional current signal processing unit 001 is used to perform time-series data mapping preprocessing on the one-dimensional current signal acquired by the substation SCADA system using the Gram angle field algorithm to obtain a two-dimensional current spectrum. The instruction fine-tuning dataset construction unit 002 is used to construct an instruction fine-tuning dataset based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text; The 1+N LoRA fine-tuning framework design unit 003 is used to fine-tune the dataset based on the instructions, and design a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module on the basis of a general multimodal backbone model, so as to obtain the adjusted model. The self-verifying reasoning framework construction unit 004 is used to output preliminary diagnostic probability vectors through the visual branch and the current expert branch respectively based on the adjusted model, and calculate the cosine similarity between the two preliminary diagnostic probability vectors; if the cosine similarity is lower than a preset threshold or there is a physical paradox, the reflection logic is triggered; the weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion result is recalculated, and the final diagnostic conclusion is output.

[0079] In this embodiment, in the one-dimensional current signal processing unit 001, during the time-series data mapping preprocessing of the one-dimensional current signal using the Gram angle field algorithm, the preprocessing process includes: adaptive normalization and polar coordinate transformation, Gram angle and difference field generation, and multi-scale GAF fusion.

[0080] In this embodiment, the weight update formula of the 1+N LoRA fine-tuning framework in the 1+N LoRA fine-tuning framework design unit 003 is as follows:

[0081] In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

[0082] In this embodiment, the formula for calculating the cosine similarity in the self-verifying reasoning framework construction unit 004 is as follows:

[0083] In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

[0084] In this embodiment, in the self-verifying reasoning framework construction unit 004, the expression for the final diagnostic conclusion is:

[0085] In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

[0086] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here. Example 3

[0087] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a power inspection multimodal large model fine-tuning method. The program code includes instructions for executing the power inspection multimodal large model fine-tuning method of Embodiment 1 or any possible implementation thereof.

[0088] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)). Example 4

[0089] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can call the program instructions to execute a power inspection multimodal large model fine-tuning method according to Embodiment 1 or any possible implementation thereof.

[0090] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0091] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0092] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0093] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for fine-tuning a multimodal large-scale power line inspection model, characterized in that, include: The Gram angle field algorithm is used to perform time-series data mapping preprocessing on the one-dimensional current signal collected by the substation SCADA system to obtain a two-dimensional current spectrum. Based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text, a command fine-tuning dataset is constructed; Based on the instruction fine-tuning dataset, a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module is designed on the basis of a general multimodal backbone model to obtain the adjusted model. Based on the adjusted model, preliminary diagnostic probability vectors are output through the visual branch and the current expert branch, respectively, and the cosine similarity between the two preliminary diagnostic probability vectors is calculated. If the cosine similarity is lower than a preset threshold or a physical paradox exists, the reflection logic is triggered. The weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion results are recalculated, and the final diagnostic conclusion is output.

2. The method for fine-tuning a multimodal large model of power line inspection according to claim 1, characterized in that, In the process of performing time-series data mapping preprocessing on one-dimensional current signals using the Gram angle field algorithm, the preprocessing process includes: adaptive normalization and polar coordinate transformation, Gram angle and difference field generation, and multi-scale GAF fusion.

3. The method for fine-tuning a multimodal large model of power line inspection according to claim 2, characterized in that, The weight update formula for the 1+NLoRA fine-tuning framework is as follows: ； In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

4. The method for fine-tuning a multimodal large model of power line inspection according to claim 3, characterized in that, The formula for calculating the cosine similarity is: ； In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

5. The method for fine-tuning a multimodal large model of power line inspection according to claim 4, characterized in that, The expression for the final diagnostic conclusion is: ； In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

6. A power line inspection multimodal large-scale model fine-tuning device, employing the power line inspection multimodal large-scale model fine-tuning method according to any one of claims 1-5, characterized in that, include: The one-dimensional current signal processing unit is used to perform time-series data mapping preprocessing on the one-dimensional current signal acquired by the substation SCADA system using the Gram angle field algorithm to obtain a two-dimensional current spectrum. The instruction fine-tuning dataset construction unit is used to construct an instruction fine-tuning dataset based on the two-dimensional current spectrum, combined with visible light / infrared images and diagnostic text; The 1+N LoRA fine-tuning framework design unit is used to fine-tune the dataset based on the instructions, and design a 1+N LoRA fine-tuning framework including a current spectrum encoder and an electrical semantic optimization LoRA module on the basis of a general multimodal backbone model, so as to obtain the adjusted model. The self-verifying reasoning framework construction unit is used to output preliminary diagnostic probability vectors through the visual branch and the current expert branch respectively based on the adjusted model, and calculate the cosine similarity between the two preliminary diagnostic probability vectors; if the cosine similarity is lower than a preset threshold or there is a physical paradox, the reflection logic is triggered; the weights of the electrical semantic optimization LoRA module are adjusted, the feature fusion results are recalculated, and the final diagnostic conclusion is output.

7. The multi-modal large-scale model fine-tuning device for power line inspection according to claim 6, characterized in that, In the one-dimensional current signal processing unit, during the time-series data mapping preprocessing of the one-dimensional current signal using the Gram angle field algorithm, the preprocessing process includes: adaptive normalization and polar coordinate transformation, Gram angle and difference field generation, and multi-scale GAF fusion.

8. A multi-modal large-scale model fine-tuning device for power line inspection according to claim 7, characterized in that, In the 1+NLoRA fine-tuning framework design unit, the weight update formula of the 1+N LoRA fine-tuning framework is: ； In the formula, The updated weights; These are the original weights of the pre-trained model; 、 This is a LoRA low-rank matrix transformation; This is a diagonal constraint matrix for electrical terminology.

9. A multi-modal large-scale model fine-tuning device for power line inspection according to claim 8, characterized in that, In the self-verifying reasoning framework construction unit, the formula for calculating the cosine similarity is: ； In the formula, Cosine similarity; This is the probability vector for visual branch diagnosis; This is the probability vector for the current expert branch diagnosis.

10. A multi-modal large-scale model fine-tuning device for power line inspection according to claim 9, characterized in that, In the self-verifying reasoning framework construction unit, the expression for the final diagnostic conclusion is: ； In the formula, This is the final diagnostic conclusion; This is the corrected feature fusion result; For activation functions; This is a decoder module for multimodal large language models; The conflict context instruction input when the reflexive logic is triggered.

Images