Power transmission line foreign matter identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a power transmission line foreign matter recognition method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-recognized first power transmission line picture and a first cue word used for representing information content needing to be recognized from the first power transmission line picture; wherein the information content comprises the type and position information of the foreign matter; inputting the first power transmission line picture and the corresponding first cue word into a multi-modal model, so that the multi-modal model generates the type and position information of the foreign matter in the first power transmission line picture; the accuracy of foreign matter identification of the power transmission line can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method, device, electronic equipment and storage medium for identifying foreign objects in power transmission lines. Background Art

[0002] As an important part of power transmission, the operation status of transmission lines is directly related to the stability and safety of the power system. The inspection of transmission lines has always been a key link in ensuring the reliability of power supply. Foreign object identification is one of the important tasks in the inspection of transmission lines. Foreign objects, such as branches, bird nests, plastic bags, etc., may pose a serious threat to transmission lines and even cause safety accidents such as short circuits and fires. Therefore, timely identification and removal of foreign objects on transmission lines is of great significance to ensure the normal operation of the power system and prevent accidents.

[0003] Existing foreign object recognition technology usually requires separate detection of two targets, transmission lines and foreign objects. The location of the transmission line is identified through image analysis or other detection methods and used as basic data. The foreign object target in the image is then detected, and the coordinate information of the foreign object is obtained to determine whether it is located on the transmission line. Since the detection of these two targets is carried out separately, the whole process is cumbersome and inefficient, and multiple steps are required to obtain the location information of the foreign object. This method not only increases the complexity of processing, but also makes it impossible to directly and quickly identify foreign objects on the transmission line. It lacks real-time and high efficiency, which limits its application in actual inspections. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, electronic device and storage medium for identifying foreign objects in power transmission lines. By implementing the present invention, the type and location information of foreign objects in power transmission lines can be directly generated using a multimodal model, thereby simplifying the foreign object identification process in power transmission lines and improving the efficiency of foreign object identification in power transmission lines, thereby improving inspection efficiency and ensuring the safe operation of power transmission lines.

[0005] An embodiment of the present invention provides a method for identifying foreign objects in a transmission line, comprising:

[0006] Acquire a first transmission line image to be identified and a first prompt word for characterizing information content to be identified from the first transmission line image; wherein the information content includes: type and location information of the foreign object;

[0007] Inputting the first transmission line image and the corresponding first prompt word into the multimodal model, so that the multimodal model generates type and location information of the foreign object in the first transmission line image;

[0008] Wherein, the training of the multimodal model includes:

[0009] Acquire a plurality of second transmission line images, second prompt words, and image labels corresponding to the second transmission line images for training; wherein the image labels of the second transmission line images include the type of foreign matter and the location information of the foreign matter;

[0010] constructing a plurality of training samples according to the second transmission line picture, the second prompt word, and the picture label corresponding to the second transmission line picture;

[0011] Input each training sample into the multimodal model in turn, and train the multimodal model until a preset number of training times is reached; wherein, when the multimodal model receives each training sample, it outputs the type and location information of the foreign body corresponding to the training sample; calculates the loss function value through the loss function according to the foreign body information and the corresponding image label; and updates the multimodal model according to the loss function value.

[0012] Furthermore, the multimodal model includes a large language model, a visual encoder, and a language adapter;

[0013] Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including:

[0014] The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

[0015] Furthermore, when training the multimodal model, the weights of the Qwen-7B model are used to initialize the network parameters of the large language model; the weights of the ViT-bigG model are used to initialize the network parameters of the visual encoder.

[0016] Furthermore, after inputting the first transmission line picture and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign matter in the first transmission line picture, the method further includes:

[0017] A foreign object information report is generated according to the type and location information of the foreign object in the first transmission line image, and the foreign object report is sent to the user end according to a preset push time.

[0018] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0019] An embodiment of the present invention provides a transmission line foreign object recognition device, comprising: a sample acquisition module to be recognized, a foreign object recognition module and a model training module.

[0020] The to-be-identified sample acquisition module is used to acquire a first transmission line image to be identified and a first prompt word used to characterize the information content to be identified from the first transmission line image; wherein the information content includes: the type and location information of the foreign object;

[0021] The foreign object recognition module is used to input the first transmission line picture and the corresponding first prompt word into the multimodal model, so that the multimodal model generates the type and location information of the foreign object in the first transmission line picture;

[0022] The model training module is used to obtain a number of second transmission line pictures, second prompt words and picture labels corresponding to the second transmission line pictures for training; wherein the picture labels of the second transmission line pictures include the type of foreign matter and the location information of the foreign matter; construct a number of training samples based on the second transmission line pictures, the second prompt words and the picture labels corresponding to the second transmission line pictures; input each training sample into the multimodal model in turn, and train the multimodal model until a preset number of training times is reached; wherein, when the multimodal model receives a training sample each time, it outputs the type and location information of the foreign matter corresponding to the training sample; calculates a loss function value through a loss function based on the foreign matter information and the corresponding picture label; and updates the multimodal model based on the loss function value.

[0023] Furthermore, in the transmission line foreign body identification device, the multimodal model includes a large language model, a visual encoder and a language adapter;

[0024] Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including:

[0025] The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

[0026] Furthermore, the transmission line foreign object recognition device, when training a multimodal model, uses the weights of the Qwen-7B model to initialize the network parameters of the large language model; and uses the weights of the ViT-bigG model to initialize the network parameters of the visual encoder.

[0027] Furthermore, the transmission line foreign body identification device further includes a foreign body information report generation module;

[0028] The foreign object information report generation module is used to generate a foreign object information report according to the type and location information of the foreign object in the first transmission line image, and send the foreign object report to the user end according to a preset push time.

[0029] Based on the above method item embodiments, the present invention provides corresponding electronic device item embodiments.

[0030] An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for identifying foreign objects in a transmission line as described in any one of the above method embodiments can be implemented.

[0031] Based on the above method item embodiments, the present invention provides a corresponding storage medium item embodiment.

[0032] An embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying foreign objects in a transmission line as described in any one of the above method embodiments can be implemented.

[0033] An embodiment of the present invention provides a method for identifying foreign objects in a transmission line, comprising:

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The embodiment of the present invention provides a method, device, electronic device and storage medium for identifying foreign objects on a transmission line. The method obtains a first transmission line image to be identified and its corresponding first prompt word, and inputs them into a multimodal model so that the multimodal model can directly generate the type and location information of the foreign object, thereby simplifying the process of separately identifying the transmission line and the foreign object in the traditional method. In the training stage, a training set is constructed by collecting and annotating a number of second transmission line images, second prompt words and corresponding image labels, and the recognition ability of the multimodal model is improved through batch training. Through the introduction of the multimodal model, the entire foreign object recognition process is simplified, the cumbersome steps of separately detecting the transmission line and the foreign object in the traditional method are avoided, and the foreign objects on the transmission line and their location information can be directly output, which improves the operation efficiency and accuracy. At the same time, since the multimodal model can be trained in combination with image and text information, the recognition process no longer relies on separate target detection, which reduces the occurrence of errors and missed judgments, making the recognition of foreign objects faster and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1The present invention is a flowchart of a method for identifying foreign objects in a transmission line provided by an embodiment of the present invention.

[0037] Figure 2 It is a flowchart of training a multimodal model provided by an embodiment of the present invention.

[0038] Figure 3 It is a structural schematic diagram of a transmission line foreign object identification device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, an embodiment of the present invention provides a method for identifying foreign objects in a transmission line, which at least includes the following steps:

[0041] Step S1: Acquire a first power transmission line image to be identified and a first prompt word used to characterize information content to be identified from the first power transmission line image.

[0042] Specifically, the information content includes: the type and location information of the foreign object; obtaining the first transmission line image to be identified, and the first prompt word used to characterize the information content that needs to be identified from the first transmission line image. The design of the first prompt word is to efficiently guide the model to focus on the key features in the image to accurately identify and extract relevant information content. Specifically, the information content includes but is not limited to the type and location information of foreign objects around or on the surface of the transmission line, such as specific categories of branches, bird nests, hanging objects, etc. that may affect the normal operation of the transmission line, and their precise coordinates or areas in the picture. In addition, the first prompt word can be further customized according to task requirements, such as adding relevant keywords describing the status of the transmission line (such as wear and tear, fracture), thereby expanding the scope of information extraction and providing a more comprehensive basis for subsequent processing. In this way, comprehensive monitoring and intelligent management of the operating status of the transmission line can be achieved.

[0043] Preferably, the selection range of the prompt words includes the following content, so as to clearly and efficiently guide the model to extract information from the transmission line picture: for example, "Please describe the type and location of foreign objects on the transmission line in this picture", "What foreign objects are there on the transmission line in this picture, and where are they?" and "At which positions on the transmission line in this picture are there foreign objects, and what are they?" etc.

[0044] Step S2: input the first transmission line image and the corresponding first prompt word into the multimodal model, so that the multimodal model generates the type and location information of the foreign matter in the first transmission line image.

[0045] The first transmission line image and the corresponding first prompt word are input into the multimodal model to generate the type and location information of the foreign object in the first transmission line image by using the model's combined image and text understanding ability. In the specific operation process, the first transmission line image is used as a visual input, which contains the image features of the transmission line and its surrounding environment, while the first prompt word is used as a text input to provide contextual semantic guidance for the multimodal model, thereby helping the model focus on specific task objectives. Through this combined image and text input method, the multimodal model can analyze the visual information of different areas in the image, and generate clear and structured output results in combination with the specific requirements in the prompt word.

[0046] The output results include the specific type of foreign objects, such as bird nests, branches, hanging objects, etc., as well as the location information of the foreign objects in the picture, such as the precise coordinates relative to the transmission line, coverage or area annotation. At the same time, the model can automatically filter out irrelevant information through its built-in feature matching capabilities to ensure that the output results are consistent with the prompt word requirements. For example, if the prompt word requires "marking dangerous foreign objects that may exist near the transmission line", the model will prioritize the analysis of the potential impact of foreign objects on the transmission line and generate a higher priority output. Through this input and processing method, intelligent monitoring and efficient management of the status of the transmission line can be achieved, providing data support for subsequent maintenance and risk warnings.

[0047] In a preferred embodiment, the multimodal model includes a large language model, a visual encoder, and a language adapter;

[0048] Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including:

[0049] The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

[0050] In actual operation, the multimodal model can use the Qwen-VL model as the basic model for further optimization.

[0051] In an optional embodiment, when training a multimodal model, the network parameters of the large language model are initialized using the weights of the Qwen-7B model; and the network parameters of the visual encoder are initialized using the weights of the ViT-bigG model.

[0052] Specifically, the visual encoder uses the VIT architecture. VIT is an algorithm that extracts features by dividing images into patches. Due to its excellent performance, it is often used in the visual feature extraction part of large models. In actual training, the visual encoder divides the image into patches with a step size of 14 to generate a set of image features; the language adapter contains a randomly initialized single-layer cross-attention module. This module uses a set of trainable vectors (embeddings) as query vectors and uses the image features from the visual encoder as keys for cross-attention operations.

[0053] Optionally, after inputting the first transmission line picture and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign matter in the first transmission line picture, the method further includes:

[0054] A foreign object information report is generated according to the type and location information of the foreign object in the first transmission line image, and the foreign object report is sent to the user end according to a preset push time.

[0055] Specifically, a structured foreign object information report is generated based on the type and location information of the foreign objects extracted from the first transmission line image. The report includes the specific category of the foreign object (such as branches, bird nests, hanging objects, etc.), its location information (such as precise coordinates or areas marked in the image), and the degree of impact that may be caused to the operation of the transmission line (such as potential short circuit risks or mechanical damage). In addition, the report can also attach key area annotations in the image so that users can understand the distribution of foreign objects more intuitively.

[0056] The format of the foreign body information report can be customized according to user needs. For example, a comprehensive form or document containing the following contents can be generated:

[0057] Basic information: includes background information such as the transmission line number, location, and image acquisition time.

[0058] Foreign object details: List the type, quantity, and specific location (such as distance from the transmission line or area name) of each foreign object, and provide a risk rating for its impact.

[0059] Additional data: Such as historical record analysis to determine whether the area is prone to foreign matter or whether emergency maintenance measures are required.

[0060] Visualization support: with original pictures and annotated analysis results, highlighting the foreign body area, making it easier for users to locate and make decisions quickly.

[0061] After the report is generated, the foreign body report is sent to the user through the system according to the preset push time. The push method can be selected in the form of SMS, email, mobile application notification, etc. to ensure the timely delivery of information. If the push time is set periodically, such as a fixed time every day or immediate push after the detection is completed, the system will automatically complete the task according to the configuration. In addition, users can also set real-time push under specific conditions, such as when the foreign body risk rating exceeds a certain threshold, a high priority notification is immediately triggered for quick response.

[0062] In this way, foreign object information reporting not only improves the intelligence and automation of transmission line monitoring, but also helps users understand the operating status of transmission lines in real time, effectively plan subsequent maintenance and management work, and improve the safety and stability of the system.

[0063] like Figure 2 As shown, in a preferred embodiment, the training of the multimodal model includes:

[0064] Step S2.1, obtaining a plurality of second power transmission line images, second prompt words, and image labels corresponding to the second power transmission line images for training.

[0065] Specifically, the image label of the second transmission line image includes the type of foreign matter and the location information of the foreign matter. Obtain a number of second transmission line images, second prompt words, and image labels corresponding to the second transmission line images for training. These data together constitute the core data set for training multimodal models to support efficient learning and optimization of the model in the task of foreign matter detection on transmission lines. Specifically, the second transmission line images come from a wide range of sources and can be generated through drone inspection photography, fixed monitoring equipment capture, or saved image data in historical maintenance records to ensure sample data diversity and coverage.

[0066] The second prompt word is a text guide designed for the training task, which can clearly indicate the goal that the model needs to learn. For example, "Identify the type and location of foreign objects on the power transmission line in the picture", "Label the category of each foreign object in the picture and the specific coordinates in the picture", or "What foreign objects are there on the power transmission line in this picture, and in which areas are they located?" The design of prompt words should cover a variety of scenarios and language expressions to help the model adapt to the diverse inputs that may be encountered in actual applications.

[0067] The image labels of the second transmission line images serve as supervisory information, including precisely labeled foreign object types and location information of foreign objects. These labels are usually presented in the form of structured data, such as JSON or XML format, which clearly records the category of each foreign object (such as branches, bird nests, foreign object hanging objects, etc.), as well as its location information (such as rectangular bounding boxes, pixel-level segmentation masks or relative coordinates). The generation of labels can be completed by manual annotation or in combination with existing mature image segmentation and detection algorithms to ensure the accuracy and consistency of annotations. In order to further improve the generalization ability and robustness of the model, the acquired second transmission line images and labels should cover a variety of complex scenes, including different weather conditions (sunny, rainy, foggy, etc.), lighting conditions (strong light, backlight, weak light, etc.), and the diversity of foreign object types and distribution. In addition, the sample data should cover different transmission line environments, such as mountainous areas, forest areas, urban areas and open plains, to simulate the diverse scenarios that may occur in actual operation. By integrating the second transmission line image, the second prompt word and the image label, the constructed training dataset can provide sufficient supervision signals for the multimodal model, helping the model to efficiently learn the correlation features between vision and text, thereby achieving accurate and reliable recognition and analysis in the actual transmission line foreign object detection task.

[0068] In a preferred embodiment, in order to enable the multimodal model to acquire target recognition capabilities other than common foreign objects, in addition to the second transmission line image, the training samples can also be expanded using other target types annotated in public datasets, such as: COCO dataset, Pascal VOC dataset, and Cityscapes dataset.

[0069] Step S2.2: construct a plurality of training samples according to the second transmission line picture, the second prompt word, and the picture label corresponding to the second transmission line picture.

[0070] Specifically, based on the second transmission line image, the second prompt word, and the corresponding image label, constructing training samples requires integrating these data into input-output pairs that can effectively guide the learning of the multimodal model. The second transmission line image is used as a visual input. When constructing the sample, it needs to undergo preprocessing operations, such as adjusting the resolution, denoising, or normalization, to ensure that it adapts to the input requirements of the model while retaining key information. These images should cover different scene conditions, including weather changes (such as sunny, rainy, and foggy days), lighting environments (such as daytime, nighttime, and backlight), and the diverse geographical locations of transmission lines (such as mountainous areas, forest areas, and urban areas), thereby increasing the diversity of training data and the generalization ability of the model.

[0071] The second prompt word, as a text input, is a key part used to guide the model to focus on specific task objectives. The design of these prompt words should be diverse and meet the needs of actual scenarios. They can be clear and direct questions, such as "Please mark the type and specific location of foreign objects on the power line in the picture", or more open or natural language expressions, such as "What foreign objects are there on the power line in this picture, and where are they located?" Through such a design, prompt words can help the model learn the relationship between pictures and language during training and improve its ability to understand tasks.

[0072] The label of the second transmission line image contains a pixel-level segmentation mask that clearly marks the category and location information of each foreign object. These label data not only provide supervision signals for the model, but also ensure the accuracy of the training results. The label can describe the location of the foreign object through a rectangular box, segmentation mask or pixel-level area, and record the category of the foreign object (such as bird nests, branches, hanging objects, etc.). In the process of constructing the training samples, the picture and the corresponding prompt word and label are integrated into a set of input-output pairs, where the picture and prompt word are input, and the type and location of the foreign object are output.

[0073] The final training sample set should cover a variety of complex scenarios and practical problems, ensuring that the model can learn to extract key features from diverse inputs and output accurate recognition results. The diversity and high quality of these samples are the basis for ensuring the reliability and robustness of multimodal models in practical applications.

[0074] Step S2.3: input each training sample into the multimodal model in sequence, and train the multimodal model until a preset number of training times is reached.

[0075] It should be noted here that when the multimodal model receives a training sample each time, it outputs the type and location information of the foreign object corresponding to the training sample; based on the foreign object information and the corresponding image label, the loss function value is calculated through the loss function; and the multimodal model is updated according to the loss function value.

[0076] Each training sample is input into the multimodal model in turn, and the multimodal model is iteratively trained until the preset number of training times is reached or the set training convergence condition is met. During the training process, each training sample consists of the input second transmission line picture and the second prompt word. After receiving the training sample, the multimodal model will extract the visual features of the picture and the semantic features of the prompt word through its visual and language processing module, and perform cross-modal fusion to generate the corresponding output, that is, the type and location information of the transmission line foreign matter in the training sample. For the output result of each training sample, the model will compare it with its corresponding label. The label contains the true type and location of the foreign matter, such as in the form of a rectangular bounding box, a pixel-level segmentation mask, or other location information annotation. By calculating the error between the model output and the label, the overall loss function value corresponding to the sample is obtained using the designed loss function, cross entropy loss and IoU loss.

[0077] During the training process, the model is optimized through the back propagation algorithm. According to the calculated loss function value, the Adam optimizer is used to adjust the model parameters so that the loss function value gradually decreases. Each parameter update will gradually improve the model's prediction ability for input samples, while enhancing its ability to extract and understand the joint features of images and prompt words.

[0078] The training cycle will continue, and all training samples will be input into the model in turn and participate in the model parameter update. To accelerate the convergence of the model, batch training can be used to input multiple training samples each time, calculate the batch average loss value, and then update the model parameters. In addition, to improve the stability and effect of training, strategies can be added during the training process, such as learning rate adjustment, weight regularization, gradient clipping, etc.

[0079] The entire training process will be completed after reaching the preset number of training times. Ultimately, the trained multimodal model will be able to efficiently receive transmission line images and prompt words, accurately output the type and location information of foreign objects in the image, and provide a reliable foundation for subsequent applications.

[0080] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0081] like Figure 3 As shown, an embodiment of the present invention provides a transmission line foreign object recognition device, including: a sample acquisition module to be recognized, a foreign object recognition module and a model training module.

[0082] The to-be-identified sample acquisition module is used to acquire a first transmission line image to be identified and a first prompt word used to characterize the information content to be identified from the first transmission line image; wherein the information content includes: the type and location information of the foreign object;

[0083] The foreign object recognition module is used to input the first transmission line picture and the corresponding first prompt word into the multimodal model, so that the multimodal model generates the type and location information of the foreign object in the first transmission line picture;

[0084] The model training module is used to obtain a number of second transmission line pictures, second prompt words and picture labels corresponding to the second transmission line pictures for training; wherein the picture labels of the second transmission line pictures include the type of foreign matter and the location information of the foreign matter; construct a number of training samples based on the second transmission line pictures, the second prompt words and the picture labels corresponding to the second transmission line pictures; input each training sample into the multimodal model in turn, and train the multimodal model until a preset number of training times is reached; wherein, when the multimodal model receives a training sample each time, it outputs the type and location information of the foreign matter corresponding to the training sample; calculates a loss function value through a loss function based on the foreign matter information and the corresponding picture label; and updates the multimodal model based on the loss function value.

[0085] In a preferred embodiment, the transmission line foreign body identification device, the multimodal model, includes a large language model, a visual encoder and a language adapter;

[0086] Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including:

[0087] The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

[0088] In an optional embodiment, the transmission line foreign object identification device, when training a multimodal model, uses the weights of the Qwen-7B model to initialize the network parameters of the large language model; and uses the weights of the ViT-bigG model to initialize the network parameters of the visual encoder.

[0089] Optionally, the transmission line foreign object identification device further includes a foreign object information report generation module;

[0090] The foreign object information report generation module is used to generate a foreign object information report according to the type and location information of the foreign object in the first transmission line image, and send the foreign object report to the user end according to a preset push time.

[0091] It should be noted that the embodiments of the device described above correspond to the above embodiments of the present invention, and can implement any of the methods described above in the present invention. In addition, the embodiments of the above-mentioned device are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0092] Based on the above method embodiment of the present invention, a corresponding electronic device embodiment is provided.

[0093] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for identifying foreign objects in a power transmission line described in any one of the present invention is implemented, or when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented.

[0094] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0095] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0096] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0097] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0098] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0099] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned transmission line foreign object identification methods of the present invention.

[0100] The storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0102] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying foreign objects in a transmission line, characterized in that: include: Acquire a first transmission line image to be identified and a first prompt word for characterizing information content to be identified from the first transmission line image; wherein the information content includes: type and location information of the foreign object; Inputting the first transmission line image and the corresponding first prompt word into the multimodal model, so that the multimodal model generates type and location information of the foreign object in the first transmission line image; Wherein, the training of the multimodal model includes: Acquire a plurality of second transmission line images, second prompt words, and image labels corresponding to the second transmission line images for training; wherein the image labels of the second transmission line images include the type of foreign matter and the location information of the foreign matter; constructing a plurality of training samples according to the second transmission line picture, the second prompt word, and the picture label corresponding to the second transmission line picture; Input each training sample into the multimodal model in turn, and train the multimodal model until a preset number of training times is reached; wherein, when the multimodal model receives each training sample, it outputs the type and location information of the foreign body corresponding to the training sample; calculates the loss function value through the loss function according to the foreign body information and the corresponding image label; and updates the multimodal model according to the loss function value.

2. A method for identifying foreign matter in a power transmission line according to claim 1, characterized in that: The multimodal model includes a large language model, a visual encoder and a language adapter; Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including: The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

3. A method for identifying foreign matter in a power transmission line according to claim 2, characterized in that: When training the multimodal model, the weights of the Qwen-7B model are used to initialize the network parameters of the large language model; the weights of the ViT-bigG model are used to initialize the network parameters of the visual encoder.

4. A method for identifying foreign matter in a power transmission line according to claim 3, characterized in that: After inputting the first transmission line picture and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign matter in the first transmission line picture, the method further includes: A foreign object information report is generated according to the type and location information of the foreign object in the first transmission line image, and the foreign object report is sent to the user end according to a preset push time.

5. A transmission line foreign body identification device, characterized in that: include: Sample acquisition module to be identified, foreign body identification module and model training module; The to-be-identified sample acquisition module is used to acquire a first transmission line image to be identified and a first prompt word used to characterize the information content to be identified from the first transmission line image; wherein the information content includes: the type and location information of the foreign object; The foreign object recognition module is used to input the first transmission line picture and the corresponding first prompt word into the multimodal model, so that the multimodal model generates the type and location information of the foreign object in the first transmission line picture; The model training module is used to obtain a number of second transmission line pictures, second prompt words and picture labels corresponding to the second transmission line pictures for training; wherein the picture labels of the second transmission line pictures include the type of foreign matter and the location information of the foreign matter; construct a number of training samples based on the second transmission line pictures, the second prompt words and the picture labels corresponding to the second transmission line pictures; input each training sample into the multimodal model in turn, and train the multimodal model until a preset number of training times is reached; wherein, when the multimodal model receives a training sample each time, it outputs the type and location information of the foreign matter corresponding to the training sample; calculates a loss function value through a loss function based on the foreign matter information and the corresponding picture label; and updates the multimodal model based on the loss function value.

6. A transmission line foreign body identification device as claimed in claim 5, characterized in that: The multimodal model includes a large language model, a visual encoder and a language adapter; Inputting the first transmission line image and the corresponding first prompt word into the multimodal model so that the multimodal model generates the type and location information of the foreign object in the first transmission line image, including: The first transmission line image and the corresponding first prompt word are input into the multimodal model, so that the multimodal model extracts text features of the first prompt word through the large language model to generate corresponding text features; the visual encoder extracts visual features of the first transmission line image to be identified to generate corresponding visual features; and the language adapter generates the type and location information of the foreign object corresponding to the first transmission line image to be identified based on the text features and the visual features.

7. A transmission line foreign body identification device as claimed in claim 6, characterized in that: When training the multimodal model, the weights of the Qwen-7B model are used to initialize the network parameters of the large language model; the weights of the ViT-bigG model are used to initialize the network parameters of the visual encoder.

8. A transmission line foreign body identification device as claimed in claim 7, characterized in that: It also includes a foreign body information report generation module; The foreign object information report generation module is used to generate a foreign object information report according to the type and location information of the foreign object in the first transmission line image, and send the foreign object report to the user end according to a preset push time.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it can implement a method for identifying foreign objects in a power transmission line as described in any one of claims 1 to 4.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement a method for identifying foreign objects in a transmission line as described in any one of claims 1 to 4.

Images