A method and system for identifying defects in power transmission channels based on multi-modal large models

Abstract

The present invention discloses a method and system for identifying defects in a power transmission channel based on a multimodal large model, and relates to the technical field of power channel defect identification. The method comprises: constructing a multimodal data set based on multi-source monitoring data of the power transmission channel; stratifying the defect categories of the power transmission channel, and using the stratified defect categories to add classification labels to the multimodal data set; using the multimodal data set to perform a first fine-tuning on the base model so that it has the ability to identify multimodal data; performing a second fine-tuning on the base model based on the hierarchical structure of the power transmission channel defect categories to improve its classification ability, and obtaining the final multimodal large model after compression; regularly using the multimodal large model to identify defects in the power transmission channel, and forwarding the identification results to relevant operation and maintenance personnel. The present invention can more comprehensively capture defect information in the power transmission channel, more accurately identify defects, and reduce missed detection and false detection.

Description

Technical Field

[0001] The present invention relates to the technical field of electric channel defect recognition, and in particular to a method and system for identifying electric channel defects based on a multi-modal large model. Background Art

[0002] The current status of the research on the transmission channel defect recognition method is mainly reflected in the application of deep learning technology, especially the target detection method based on deep learning. Researchers have proposed a variety of improved algorithms for different scenarios and defect types. For example, by adjusting the convolution kernel size of the CNN model and the rotation transformation of the image to expand the data set, the Faster R-CNN network is used to detect the images collected by the drone; the random forest classification method is used to segment the image to achieve the target recognition of the insulator, and then CNN is used to classify whether the insulator is defective; the region-based full convolution network R-FCN is used as a deep learning algorithm to detect insulator crack defects. Although the research on the transmission channel defect recognition method has made significant progress, there are still defects such as complex environmental interference, hardware limitations, and insufficient algorithm robustness. Therefore, the present invention proposes a transmission channel defect recognition method based on a multimodal large model, which avoids the limitations of single-source data, improves the robustness of the algorithm, and reduces the resource consumption of the algorithm, thereby reducing its requirements for hardware and saving costs. Summary of the invention

[0003] The present invention provides a method for identifying defects in a power transmission channel based on a multi-modal large model, comprising:

[0004] Step 1: Construct a multimodal dataset based on multi-source monitoring data of the transmission channel;

[0005] Step 2: stratify the defect categories of the transmission channel and use the stratified defect categories to add classification labels to the multimodal dataset;

[0006] Step 3: Use the multimodal dataset to fine-tune the base model for the first time, so that it has the ability to recognize multimodal data;

[0007] Step 4: Based on the hierarchical structure of the transmission channel defect categories, the base model is fine-tuned for the second time to improve its classification ability, and the final multi-modal large model is obtained after compression;

[0008] Step 5: Regularly use the multimodal large model to identify defects in the transmission channel and forward the identification results to relevant operation and maintenance personnel.

[0009] As described above, a method for identifying defects in a power transmission channel based on a multimodal large model is described, wherein a multimodal data set is constructed based on multi-source monitoring data of the power transmission channel, which is specifically divided into the following sub-steps:

[0010] Collect multi-source monitoring historical data of transmission channels;

[0011] Use feature engineering to extract features from the collected historical monitoring data;

[0012] The extracted features are aligned and stored in a multimodal dataset.

[0013] In the above-mentioned method for identifying defects in a power transmission channel based on a multimodal large model, a multimodal data set is used to perform a first fine-tuning on the base model, which is specifically divided into the following sub-steps:

[0014] Create a base model for implementing the classification task;

[0015] Input multimodal data into the base model in batches to fine-tune the base model;

[0016] During the fine-tuning process, the parameters of the base model are continuously iterated, and the optimal solution of the model parameters is obtained based on the minimum loss principle.

[0017] In the above-mentioned method for identifying power transmission channel defects based on a multimodal large model, the base model is fine-tuned for the second time based on the hierarchical structure of the power transmission channel defect categories, which is specifically divided into the following sub-steps:

[0018] Adjust the output structure of the base model according to the hierarchical structure of the transmission channel defect categories;

[0019] Verify layer by layer whether the output results of the base model meet expectations. If yes, ignore them; otherwise, proceed to the next processing step.

[0020] Perform targeted tuning on the model layers that fail verification.

[0021] As described above, a method for identifying defects in a power transmission channel based on a multimodal large model is described, in which the multimodal large model is used regularly to identify defects in the power transmission channel, and the identification results are forwarded to relevant operation and maintenance personnel. The method is specifically divided into the following sub-steps:

[0022] Preset an identification cycle for the power transmission channel and create a timer according to the identification cycle;

[0023] The timer periodically obtains various monitoring data of the transmission channel, extracts features and inputs them into the multi-modal large model to obtain defect identification results;

[0024] The defect identification results are organized into preset file types and forwarded to the operation and maintenance personnel.

[0025] The present invention also provides a transmission channel defect identification system based on a multimodal large model, comprising: a multimodal data set construction module, a defect category stratification module, a model fine-tuning module, and a defect identification module;

[0026] A multimodal dataset construction module, used to construct a multimodal dataset based on multi-source monitoring data of a transmission channel;

[0027] A defect category stratification module is used to stratify the defect categories of the transmission channel and use the stratified defect categories to add classification labels to the multimodal dataset;

[0028] The model fine-tuning module is used to fine-tune the base model using the multimodal dataset and obtain the final multimodal large model after compression;

[0029] The defect identification module is used to regularly use a large multi-modal model to identify defects in the transmission channel and forward the identification results to relevant operation and maintenance personnel.

[0030] The beneficial effects achieved by the present invention are as follows: integrating data from different sources and forms, such as text, images, videos, etc., so as to more comprehensively capture the defect information in the transmission channel; this comprehensive analysis capability enables the model to identify defects more accurately, reducing missed detections and false detections; at the same time, because the large model has powerful computing and learning capabilities, it can process and analyze data more quickly, improving the efficiency of defect identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a flow chart of a method for identifying power transmission channel defects based on a multi-modal large model provided in Example 1 of the present application;

[0033] Figure 2 This is a schematic diagram of a power transmission channel defect identification system based on a multimodal large model provided in Example 2 of the present application. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] like Figure 1As shown, the first embodiment of the present application provides a method for identifying defects in a power transmission channel based on a multi-modal large model, including:

[0037] Step S10: constructing a multimodal data set based on multi-source monitoring data of the transmission channel;

[0038] Multi-source monitoring data includes but is not limited to imaging data, electromagnetic wave data and vibration data of the transmission channel. Imaging data includes infrared imaging and visible light imaging. Infrared imaging is used to detect whether the temperature of the channel heating device is normal, and visible light imaging can detect the appearance defects of the channel. Electromagnetic wave data is used to detect whether the insulator is damaged, and vibration data is used to judge the stability of the channel. These data have certain defect detection effects, but they all have certain limitations. In order to avoid the limitations of a single data source and improve the accuracy of defect identification, this application proposes to use a multimodal large model to fuse and analyze the above data. In this step, it is necessary to build a multimodal data set to provide a data basis for the subsequent creation of a multimodal large model. The specific construction process includes:

[0039] Step S11: Collecting multi-source monitoring historical data of the transmission channel;

[0040] The vibration data and electromagnetic wave data are converted into waveform images by an oscilloscope in order to obtain characteristic values.

[0041] Step S12: extracting features from the collected historical monitoring data using feature engineering;

[0042] The characteristic value of infrared imaging is represented by the grayscale mean of the pixel part of the transmission channel. The visible light imaging feature must at least include the color feature, texture feature and shape feature of the pixel part of the transmission channel. There are many existing methods for image feature extraction, which will not be described here. The feature extraction formula of vibration data or electromagnetic wave data is expressed as:

[0043] Where f is the vibration frequency of the waveform image in the current monitoring period, p tx represents the value of the tth extreme point (peak or trough) on the X-axis, p t+1x represents the value of the t+1th extreme point on the X-axis, p ty represents the value of the tth extreme point on the Y axis, p t+1y Represents the value of the t+1th extreme point on the Y axis, are the resolution coefficients of the feature engineering for the changes in the X-axis and Y-axis respectively, t takes values ​​from 1 to T, T is the total number of extreme points contained in the waveform in the current monitoring period, and a is the characteristic value of the waveform in the current monitoring period.

[0044] Step S13: aligning the extracted features and storing them in a multimodal dataset;

[0045] The extracted imaging features, electromagnetic wave features, and vibration features need to be aligned according to the acquisition time and location. The aligned data are stored in a multimodal dataset and used as training data for subsequent multimodal large models.

[0046] Step S20: stratifying the defect categories of the transmission channel, and using the stratified defect categories to add classification labels to the multimodal data set;

[0047] Hierarchical defect categories of transmission channels can improve the accuracy of defect recognition and speed up recognition. For example, the general concept of physical defects can be used as the first layer, and the second layer can be refined into specific defect locations such as insulator defects, conductor defects, and tower defects. The insulator defects in the second layer can be further refined into specific defect conditions such as rupture / cracks, pollution / dust accumulation, and missing / detached in the third layer. Using hierarchical defect categories to add corresponding classification labels to multimodal data sets can automatically match classification labels based on existing association tables. For example, each diagnostic result in the defect diagnosis table has the corresponding monitoring data, so the defect diagnosis table and the monitoring data table must have an association relationship. Based on this association relationship, a script can be designed to complete the automatic matching of classification labels and multimodal data.

[0048] Step S30: using the multimodal data set to perform a first fine-tuning on the base model so that it has the ability to recognize multimodal data;

[0049] Using the large classification model as the base model, the multimodal dataset is used to fine-tune the base model for the first time, so that it can learn the correlation between multimodal data and classification labels. Specifically:

[0050] Step S31: creating a base model for implementing the classification task;

[0051] For the defect identification of the transmission channel, there are multiple classification results, so multiple classifiers are needed to output the probability of an input sample X belonging to its own classification result. Finally, the model summarizes the classification result with the highest probability (i.e., classification label) for final output. The mathematical expression of the created base model is:

[0052] Where Y is the output of the model, y k is the k-th classification label, Fk(X) is the classifier of the k-th classification label, X is the input multimodal data, is the classifier F k The weight matrix of (X), ω k is the classifier F k (X), k ranges from 1 to K, and K is the total number of classification labels.

[0053] Step S32: input the multimodal data into the base model in batches to fine-tune the base model;

[0054] The batching of multimodal data requires that the proportion of data of the same category in each batch is about 80%.

[0055] Step S33: During the fine-tuning process, the parameters of the base model are continuously iterated, and the optimal solution of the model parameters is obtained based on the minimum loss principle;

[0056] As the data is input, the loss function curve changes are observed, and the parameters in the base model are continuously iterated until the function curve stops decreasing or there is no obvious downward trend, then the fine-tuning is stopped. The loss function of the base model is expressed as:

[0057] Where Yu represents the output result of the classifier to which the true classification label of the u-th input sample belongs, represents the expected result of the classifier to which the true classification label of the u-th input sample belongs, Y uv is the output result of other j-th classifiers for the u-th input sample, v takes values ​​1 to V, V is the number of classifiers except the classifier to which the true classification label of the u-th input sample belongs, u takes values ​​1 to U, U is the number of input samples in the current training batch, and L is the loss value calculation result.

[0058] Step S40: fine-tune the base model for the second time based on the hierarchical structure of the transmission channel defect categories to improve its classification ability, and obtain the final multi-modal large model after compression;

[0059] The second fine-tuning needs to fine-tune the hierarchical structure of the base model in combination with the stratification results of the defect categories, so that the output of each layer of the model corresponds to the defect category labels at different levels, and the defect category can be identified layer by layer. Specifically:

[0060] Step S41: adjusting the output structure of the base model according to the hierarchical structure of the transmission channel defect categories;

[0061] The original base model is a random parallel execution of multiple classifiers. Now we need to adjust the execution order, that is, change the output structure of the base model, so as to reduce the number of classifier executions and increase the output rate of the model. The workflow of the adjusted base model is as follows:

[0062] First execute the classifier of the root classification label, and obtain the classification label with the highest prediction probability and assign it to δ. where F k (X) represents the classifier with subscript k, y k Represents the classification label with subscript k, k ranges from 1 to f, and f is the number of classifiers of the root classification label;

[0063] Then update the value of f = G(δ, ++g), where the G() function is used to return the number of classifiers with parent label δ in the g-th layer, and the initial value of g is 1, and then continue to execute The δ returned at this time represents the label in the second layer classification label that best matches the input data;

[0064] Repeat the update operation of f and δ until g = g total When , the calculated δ is output as the final classification label of the input sample, and gtotal is the total number of classification labels.

[0065] Step S42: Verify layer by layer whether the output result of the base model meets the expectation, if yes, ignore it, otherwise proceed to the next processing step;

[0066] After the adjustment, the next layer of the base model will output according to the output of the previous layer. In this step, it is necessary to verify whether the output result of each layer is the same as the expected classification label. For example, for an input sample X, the first layer should output the root label to which the sample X belongs, and the second layer should output the second-layer label to which the sample belongs, and so on, until the output reaches the last layer to obtain the final classification label. If the output result of a layer is inconsistent with the expected result, it is considered that the layer has failed the verification and needs to be processed in the next step.

[0067] Step S43: Targeted tuning of the model layers that failed the verification;

[0068] 200 to 500 data are extracted from the multimodal dataset, and the classification labels of the corresponding layers are used as output. The multimodal data is used as input for targeted training of the model layer. During the training process, the parameters in the layer are tuned based on the previously designed loss function until the output results meet expectations.

[0069] Step S50: regularly use the multi-modal large model to identify defects in the power transmission channel, and forward the identification results to relevant operation and maintenance personnel;

[0070] The recognition results can be organized into reports and then forwarded to the operation and maintenance personnel. This can be achieved by setting up templates and filling in the corresponding data. The specific process of regular recognition is divided into the following sub-steps:

[0071] Step S51: presetting an identification period for the power transmission channel and creating a timer according to the identification period;

[0072] The identification period set in this embodiment is one day, so the timer event will be triggered once every 24 hours.

[0073] Step S52: The timer periodically acquires various monitoring data of the power transmission channel, extracts features and inputs them into the multi-modal large model to obtain defect identification results;

[0074] The specific operation behavior of setting the timer event is to pull the monitoring data from each monitoring device. After the feature engineering when creating the multimodal data set, the characteristics of each monitoring data are obtained, and then the obtained features are input into the multimodal large model to obtain the defect identification results of the transmission channel.

[0075] Step S53: organize the defect identification results into a preset file type and forward them to the operation and maintenance personnel;

[0076] The preset file type in this embodiment is a report format, in which the identified defects are displayed one by one in the report, and the corresponding monitoring data is provided to enhance the readability of the report.

[0077] Embodiment 2

[0078] like Figure 2 As shown, the second embodiment of the present application provides a transmission channel defect identification system based on a multimodal large model, including: a multimodal data set construction module 21, a defect category stratification module 22, a model fine-tuning module 23, and a defect identification module 24;

[0079] A multimodal data set construction module 21 is used to construct a multimodal data set based on multi-source monitoring data of the transmission channel; specifically comprising: a historical monitoring data collection submodule, a feature extraction submodule, and a feature alignment submodule;

[0080] 1. Historical monitoring data collection submodule, used to collect multi-source monitoring historical data of the transmission channel;

[0081] The multi-source monitoring data includes but is not limited to imaging data, electromagnetic wave data and vibration data of the transmission channel. The imaging data includes infrared imaging and visible light imaging. The vibration data and electromagnetic wave data are converted into waveform images by an oscilloscope to obtain characteristic values.

[0082] 2. Feature extraction submodule, used to extract features from the collected historical monitoring data using feature engineering;

[0083] The characteristic value of infrared imaging is represented by the grayscale mean of the pixel part of the transmission channel. The visible light imaging feature must at least include the color feature, texture feature and shape feature of the pixel part of the transmission channel. There are many existing methods for image feature extraction, which will not be described here. The feature extraction formula of vibration data or electromagnetic wave data is expressed as:

[0084] Where f is the vibration frequency of the waveform image in the current monitoring period, p txrepresents the value of the tth extreme point (peak or trough) on the X-axis, p t+1x represents the value of the t+1th extreme point on the X-axis, p ty represents the value of the tth extreme point on the Y axis, p t+1y Represents the value of the t+1th extreme point on the Y axis, are the resolution coefficients of the feature engineering for the changes in the X-axis and Y-axis respectively, t takes values ​​from 1 to T, T is the total number of extreme points contained in the waveform in the current monitoring period, and a is the characteristic value of the waveform in the current monitoring period.

[0085] 3. Feature alignment submodule, used to align the extracted features and store them in a multimodal dataset;

[0086] The extracted imaging features, electromagnetic wave features, and vibration features need to be aligned according to the acquisition time and location. The aligned data are stored in a multimodal dataset and used as training data for subsequent multimodal large models.

[0087] A defect category stratification module 22, used to stratify the defect categories of the transmission channel and use the stratified defect categories to add classification labels to the multimodal data set;

[0088] Hierarchical defect categories of transmission channels can improve the accuracy of defect recognition and speed up recognition. For example, the general concept of physical defects can be used as the first layer, and the second layer can be refined into specific defect locations such as insulator defects, conductor defects, and tower defects. The insulator defects in the second layer can be further refined into specific defect conditions such as rupture / cracks, pollution / dust accumulation, and missing / detached in the third layer. Using hierarchical defect categories to add corresponding classification labels to multimodal data sets can automatically match classification labels based on existing association tables. For example, each diagnostic result in the defect diagnosis table has the corresponding monitoring data, so the defect diagnosis table and the monitoring data table must have an association relationship. Based on this association relationship, a script can be designed to complete the automatic matching of classification labels and multimodal data.

[0089] The model fine-tuning module 23 is used to fine-tune the base model using the multimodal data set and obtain the final multimodal large model after compression; specifically, it includes: a multimodal data recognition capability learning submodule and a multimodal data recognition capability improvement submodule;

[0090] 1. Multimodal data recognition capability learning submodule, which is used to use the multimodal dataset to perform the first fine-tuning on the base model so that it has the ability to recognize multimodal data;

[0091] Using the large classification model as the base model, the multimodal dataset is used to fine-tune the base model for the first time, so that it can learn the correlation between multimodal data and classification labels. Specifically:

[0092] ① Create a base model for implementing the classification task;

[0093] For the defect identification of the transmission channel, there are multiple classification results, so multiple classifiers are needed to output the probability of an input sample X belonging to its own classification result. Finally, the model summarizes the classification result with the highest probability (i.e., classification label) for final output. The mathematical expression of the created base model is:

[0094] Where Y is the output of the model, yk is the kth classification label, and F k (X) is the classifier of the k-th classification label, X is the input multimodal data, is the classifier F k The weight matrix of (X), ω k is the classifier F k (X), k ranges from 1 to K, and K is the total number of classification labels.

[0095] ② Input the multimodal data into the base model in batches to fine-tune the base model;

[0096] The batching of multimodal data requires that the proportion of data of the same category in each batch is about 80%.

[0097] ③ During the fine-tuning process, the parameters of the base model are continuously iterated, and the optimal solution of the model parameters is obtained based on the minimum loss principle;

[0098] As the data is input, the loss function curve changes are observed, and the parameters in the base model are continuously iterated until the function curve stops decreasing or there is no obvious downward trend, then the fine-tuning is stopped. The loss function of the base model is expressed as:

[0099] where Y u represents the output result of the classifier to which the true classification label of the u-th input sample belongs, represents the expected result of the classifier to which the true classification label of the u-th input sample belongs, Y uv is the output result of other j-th classifiers for the u-th input sample, v takes values ​​1 to V, V is the number of classifiers except the classifier to which the true classification label of the u-th input sample belongs, u takes values ​​1 to U, U is the number of input samples in the current training batch, and L is the loss value calculation result.

[0100] 2. The submodule for improving the multimodal data recognition capability is used to fine-tune the base model for the second time based on the hierarchical structure of the transmission channel defect categories to improve its classification capability;

[0101] The second fine-tuning needs to fine-tune the hierarchical structure of the base model in combination with the stratification results of the defect categories, so that the output of each layer of the model corresponds to the defect category labels at different levels, and the defect category can be identified layer by layer. Specifically:

[0102] ① Adjust the output structure of the base model according to the hierarchical structure of the transmission channel defect categories;

[0103] The original base model is a random parallel execution of multiple classifiers. Now we need to adjust the execution order, that is, change the output structure of the base model, so as to reduce the number of classifier executions and increase the output rate of the model. The workflow of the adjusted base model is as follows:

[0104] First execute the classifier of the root classification label, and obtain the classification label with the highest prediction probability and assign it to δ. where F k (X) represents the classifier with subscript k, y k Represents the classification label with subscript k, k ranges from 1 to f, and f is the number of classifiers of the root classification label;

[0105] Then update the value of f = G(δ, ++g), where the G() function is used to return the number of classifiers with parent label δ in the g-th layer, and the initial value of g is 1, and then continue to execute The δ returned at this time represents the label in the second layer classification label that best matches the input data;

[0106] Repeat the update operation of f and δ until g = g total When , the calculated δ is output as the final classification label of the input sample, and gtotal is the total number of classification labels.

[0107] ② Verify layer by layer whether the output results of the base model meet expectations. If yes, ignore them; otherwise, proceed to the next processing step.

[0108] After the adjustment, the next layer of the base model will output according to the output of the previous layer. In this step, it is necessary to verify whether the output result of each layer is the same as the expected classification label. For example, for an input sample X, the first layer should output the root label to which the sample X belongs, and the second layer should output the second-layer label to which the sample belongs, and so on, until the output reaches the last layer to obtain the final classification label. If the output result of a layer is inconsistent with the expected result, it is considered that the layer has failed the verification and needs to be processed in the next step.

[0109] ③Perform targeted optimization on the model layers that failed the verification;

[0110] 200 to 500 data are extracted from the multimodal dataset, and the classification labels of the corresponding layers are used as output. The multimodal data is used as input for targeted training of the model layer. During the training process, the parameters in the layer are tuned based on the previously designed loss function until the output results meet expectations.

[0111] The defect identification module 24 is used to regularly use the multi-modal large model to identify defects in the transmission channel and forward the identification results to relevant operation and maintenance personnel; specifically includes: an identification period preset submodule, a defect identification result acquisition submodule, and a defect identification result forwarding submodule;

[0112] 1. An identification cycle preset submodule is used to preset an identification cycle for a power transmission channel and create a timer according to the identification cycle;

[0113] The identification period set in this embodiment is one day, so the timer event will be triggered once every 24 hours.

[0114] 2. The defect identification result acquisition submodule is used to use a timer to regularly obtain various monitoring data of the transmission channel, extract features and input them into the multi-modal large model to obtain defect identification results;

[0115] The specific operation behavior of setting the timer event is to pull the monitoring data from each monitoring device. After the feature engineering when creating the multimodal data set, the characteristics of each monitoring data are obtained, and then the obtained features are input into the multimodal large model to obtain the defect identification results of the transmission channel.

[0116] 3. Defect identification result forwarding submodule, which is used to organize the defect identification results into preset file types and forward them to the operation and maintenance personnel;

[0117] The preset file type in this embodiment is a report format, in which the identified defects are displayed one by one in the report, and the corresponding monitoring data is provided to enhance the readability of the report.

[0118] Corresponding to the above embodiment, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0119] The memory is used to store one or more program instructions;

[0120] The processor is used to run one or more program instructions to execute a transmission channel defect identification method based on a multi-modal large model.

[0121] Corresponding to the above-mentioned embodiment, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used by a processor to execute a method for identifying defects in a power transmission channel based on a multi-modal large model.

[0122] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned method for identifying power transmission channel defects based on a multi-modal large model.

[0123] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0124] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0125] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0126] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0127] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0128] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0129] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0130] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying transmission channel defects based on a multimodal large model, characterized in that: include: Step 1: Construct a multimodal dataset based on multi-source monitoring data of the transmission channel; Step 2: stratify the defect categories of the transmission channel and use the stratified defect categories to add classification labels to the multimodal dataset; Step 3: Use the multimodal dataset to fine-tune the base model for the first time, so that it has the ability to recognize multimodal data; Step 4: Based on the hierarchical structure of the transmission channel defect categories, the base model is fine-tuned for the second time to improve its classification ability, and the final multi-modal large model is obtained after compression; Step 5: Regularly use the multi-modal large model to identify defects in the transmission channel and forward the identification results to relevant operation and maintenance personnel; The base model is fine-tuned for the second time based on the hierarchical structure of the transmission channel defect categories, which is divided into the following sub-steps: The output structure of the base model is adjusted according to the hierarchical structure of the transmission channel defect categories. The workflow of the adjusted base model is as follows: first execute the classifier of the root classification label, and obtain the classification label with the highest prediction probability and assign it to δ, where F k (X) represents the classifier with subscript k, y k Indicates the classification label with subscript k, k ranges from 1 to f, and f is the number of classifiers of the root classification label; then update the value of f = G(δ,++g), where the G() function is used to return the number of classifiers with parent label δ in the g-th layer, and the initial value of g is 1, and then continue to execute The returned δ at this time represents the label in the second-layer classification label that best matches the input data; repeat the update operation of f and δ until g = g total When , the calculated δ is output as the final classification label of the input sample, and gtotal is the total number of classification labels; Verify layer by layer whether the output results of the base model meet expectations. If yes, ignore them; otherwise, proceed to the next processing step. Perform targeted tuning on the model layers that fail verification.

2. According to claim 1, a method for identifying defects in a power transmission channel based on a multi-modal large model is characterized in that: The multimodal dataset is constructed based on the multi-source monitoring data of the transmission channel, which is divided into the following sub-steps: Collect multi-source monitoring historical data of transmission channels; Use feature engineering to extract features from the collected historical monitoring data; The extracted features are aligned and stored in a multimodal dataset.

3. The method for identifying power transmission channel defects based on a multimodal large model according to claim 1 is characterized in that: The first fine-tuning of the base model using the multimodal dataset is divided into the following sub-steps: Create a base model for implementing the classification task; Input multimodal data into the base model in batches to fine-tune the base model; During the fine-tuning process, the parameters of the base model are continuously iterated, and the optimal solution of the model parameters is obtained based on the minimum loss principle.

4. The method for identifying power transmission channel defects based on a multimodal large model according to claim 1 is characterized in that: Regularly use the multimodal large model to identify defects in the transmission channel and forward the identification results to relevant operation and maintenance personnel. The specific steps are as follows: Preset an identification cycle for the power transmission channel and create a timer according to the identification cycle; The timer periodically obtains various monitoring data of the transmission channel, extracts features and inputs them into the multi-modal large model to obtain defect identification results; The defect identification results are organized into preset file types and forwarded to the operation and maintenance personnel.

5. A transmission channel defect identification system based on a multi-modal large model, characterized in that: include: Multimodal dataset construction module, defect category stratification module, model fine-tuning module, defect identification module; A multimodal dataset construction module, used to construct a multimodal dataset based on multi-source monitoring data of a transmission channel; A defect category stratification module is used to stratify the defect categories of the transmission channel and use the stratified defect categories to add classification labels to the multimodal dataset; The model fine-tuning module is used to fine-tune the base model using the multimodal dataset and obtain the final multimodal large model after compression; Defect identification module, which is used to regularly identify defects in the transmission channel using a multi-modal large model and forward the identification results to relevant operation and maintenance personnel; Model fine-tuning module, specifically including: multimodal data recognition capability learning submodule, multimodal data recognition capability improvement submodule; The multimodal data recognition capability learning submodule is used to use the multimodal dataset to perform the first fine-tuning on the base model so that it has the ability to recognize multimodal data; The submodule for improving the multimodal data recognition capability is used to fine-tune the base model for the second time based on the hierarchical structure of the transmission channel defect categories to improve its classification capability; The base model is fine-tuned for the second time based on the hierarchical structure of the transmission channel defect categories, which is divided into the following sub-steps: The output structure of the base model is adjusted according to the hierarchical structure of the transmission channel defect categories. The workflow of the adjusted base model is as follows: first execute the classifier of the root classification label, and obtain the classification label with the highest prediction probability and assign it to δ, where F k (X) represents the classifier with subscript k, y k Indicates the classification label with subscript k, k ranges from 1 to f, and f is the number of classifiers of the root classification label; then update the value of f = G(δ,++g), where the G() function is used to return the number of classifiers with parent label δ in the g-th layer, and the initial value of g is 1, and then continue to execute The returned δ at this time represents the label in the second-layer classification label that best matches the input data; repeat the update operation of f and δ until g = g total When , the calculated δ is output as the final classification label of the input sample, and gtotal is the total number of classification labels; Verify layer by layer whether the output results of the base model meet expectations. If yes, ignore them; otherwise, proceed to the next processing step. Perform targeted tuning on the model layers that fail verification.

6. A transmission channel defect identification system based on a multi-modal large model according to claim 5, characterized in that: The multimodal dataset construction module includes: historical monitoring data collection submodule, feature extraction submodule, and feature alignment submodule; A historical monitoring data collection submodule is used to collect multi-source monitoring historical data of the transmission channel; The feature extraction submodule is used to extract features from the collected historical monitoring data using feature engineering; The feature alignment submodule is used to align the extracted features and store them in the multimodal dataset.

7. The power transmission channel defect identification system based on multi-modal large model according to claim 5 is characterized in that: The defect recognition module specifically includes: a recognition period preset submodule, a defect recognition result acquisition submodule, and a defect recognition result forwarding submodule; An identification cycle presetting submodule is used to presetting an identification cycle for a power transmission channel and to create a timer according to the identification cycle; The defect identification result acquisition submodule is used to use a timer to periodically acquire various monitoring data of the transmission channel, extract features and input them into the multi-modal large model to obtain defect identification results; The defect identification result forwarding submodule is used to organize the defect identification results into preset file types and forward them to the operation and maintenance personnel.

8. A computer storage medium, characterized in that: include: at least one memory and at least one processor; A memory for storing one or more program instructions; A processor, used to run one or more program instructions to execute a method for identifying power transmission channel defects based on a multi-modal large model as described in any one of claims 1 to 4.

Images