Unmanned aerial vehicle inspection image intelligent sorting method based on multi-modal large model

Abstract

The embodiment of the invention provides an unmanned aerial vehicle inspection image intelligent sorting method based on a multi-modal large model, and belongs to the technical field of image processing. The sorting method comprises the steps that equipment state data are collected through unmanned aerial vehicle inspection and are preprocessed; extracting features capable of reflecting the state of the electric power facility from the preprocessed image data; on the basis of characteristics which are extracted from the preprocessed image data and can reflect the state of the electric power facility, high-precision matching of similar images is achieved through calculation; and according to an image encoder and a text encoder of the CLIP, encoding cross-modal image data and text data of features needing to be matched into a feature vector form, mapping the feature vector form to the same feature space, and calculating image-text cosine similarity so as to realize cross-modal matching of associated images and texts. According to the sorting method, the data demand intention of the user can be intelligently analyzed, the associated image data can be retrieved from the massive image database, and the effects of searching images by images and identifying the images by texts are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an intelligent sorting method for unmanned aerial vehicle inspection images based on a multimodal large model. Background Art

[0002] At present, drones are being applied on a large scale in various industries. The average daily data increment of a single provincial drone inspection mission exceeds 2TB, but the processing capacity of massive inspection images cannot match the growth of the operation scale, and image intelligent sorting applications are urgently needed. The current image data sorting methods mostly rely on manual pre-classification of image data, such as establishing indexes based on shooting date, line, voltage level, etc., and with the continuous addition of new data, it is necessary to continuously invest manpower to update and maintain the classification index of the image database, which is a large workload, inefficient, and unable to intelligently sort data based on image content. Summary of the invention

[0003] The purpose of an embodiment of the present invention is to provide an intelligent sorting method for drone inspection images based on a multimodal large model. The sorting method can intelligently analyze the user's data demand intentions, retrieve related image data from a massive image database, and achieve the effect of searching images by images and recognizing images by text.

[0004] In order to achieve the above object, an embodiment of the present invention provides a method for intelligent sorting of UAV inspection images based on a multimodal large model, the sorting method comprising:

[0005] Collect equipment status data through drone inspections and perform preprocessing;

[0006] Extracting features that can reflect the status of power facilities from the preprocessed image data;

[0007] Based on the features that can reflect the status of power facilities extracted from the pre-processed image data, high-precision matching of similar images is calculated;

[0008] According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vector form, mapped to the same feature space, and the cosine similarity of the image and text is calculated to achieve cross-modal matching of associated images and texts.

[0009] Optionally, the equipment status data is collected through drone inspections and preprocessed, including:

[0010] The drone is equipped with a camera and flies along a predetermined route to capture images or videos of the power facility;

[0011] Image data is transmitted to the ground control station in real time via wireless network;

[0012] Performing geometric correction and radiation correction on the image data;

[0013] Stitching a plurality of said image data into a panoramic image or an orthophoto;

[0014] Remove random noise from the image through filtering technology;

[0015] Standardize images to ensure that images collected at different times and under different conditions are consistent in color, brightness, etc.;

[0016] Through image enhancement technology, key features in the image are highlighted;

[0017] Separate different objects in the image through image segmentation algorithm;

[0018] Distributed storage technology is used to disperse the processed image data to different nodes, and compression technology is used to compress and store the image data.

[0019] Optionally, based on the features that can reflect the state of the electric power facility extracted from the pre-processed image data, high-precision matching of similar images is calculated, including:

[0020] Obtain image data after preprocessing and feature extraction;

[0021] Performing transformation processing of horizontal flipping, vertical flipping, 90 degree rotation, 180 degree rotation and 270 degree rotation on the acquired image data to obtain more initial images;

[0022] The initial image is processed by Gaussian filtering to obtain the width and height of the new image, which is half of the width and height of the initial image;

[0023] Upsampling the new image by linear interpolation to enlarge the new image to the same size as the initial image;

[0024] Subtract the corresponding enlarged new image from the initial image to obtain a residual image;

[0025] The residual image of the image to be matched is sent to the Siamese network structure to obtain a similar matching image.

[0026] Optionally, the residual image of the image to be matched is fed into the Siamese network structure to obtain a similar matching image, including:

[0027] Obtain the residual image of each pair of images that need to be matched;

[0028] The residual image of each pair of images is sent to the two sub-networks of the Siamese network, and after convolution, pooling and full connection processing, the corresponding embedding vector is obtained;

[0029] The similarity of the embedding vectors of each pair of images to be matched is calculated according to formula (1):

[0030] D=||f(I1)-f(I2)|| 2 , formula (1)

[0031] Where D represents the similarity, I1, I2 represent the first residual image and the second residual image of a pair of input images, and f(I1), f(I2) represent the corresponding embedding vectors;

[0032] When the similarity is greater than a preset threshold, the corresponding pair of images is determined to be similar matching images.

[0033] Optionally, the residual image of the image to be matched is fed into the Siamese network structure to obtain a similar matching image, including:

[0034] Get the similarity of each pair of images;

[0035] The loss function of the Siamese network is calculated according to formula (2):

[0036]

[0037] Where L is the loss function of the Siamese network, l represents the binary label, if the input I1 and I2 match, then l = 1, if not, then l = 0, margin represents the boundary value of the mismatched pair;

[0038] The gradient of the loss function of the Siamese network with respect to the network parameters is calculated through the back propagation algorithm;

[0039] According to the obtained gradient, the network parameters are updated through the optimization algorithm to minimize the loss function, and the embedding vector corresponding to each pair of images after optimization is calculated according to the network parameters when the loss function is minimized.

[0040] Optionally, according to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vector form, mapped to the same feature space, and the image-text cosine similarity is calculated to achieve cross-modal matching of associated images and texts, including:

[0041] Initialize CLIP's image encoder and text encoder;

[0042] Freeze the parameters of the image encoder to lock the image encoder, add power scene prompt words to the input layer of the text encoder through the prompt fine-tuning algorithm, integrate the existing semantic understanding ability of the general scene corpus of the model, and realize the knowledge transfer of the power scene corpus;

[0043] Freeze the parameters of the text encoder after preliminary fine-tuning to lock the text encoder, input a preset amount of power scene image data into the image encoder, perform self-supervised learning through MAE mask self-encoding, randomly cover part of the image blocks, and train the visual branch of the CLIP large model with the goal of restoring the image, thereby improving the adaptability of the image encoder to the power scene image data;

[0044] Fully unfreeze the text encoder and image encoder, carry out full parameter fine-tuning based on the visual and text branches' cognitive basis of power scene graphic data, and open up the collaborative capabilities of the graphic encoders to achieve alignment of vision and language.

[0045] Optionally, according to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vector form, mapped to the same feature space, and the image-text cosine similarity is calculated to achieve cross-modal matching of associated images and texts, including:

[0046] Obtain cross-modal image data and text data that need to be matched;

[0047] The cross-modal image data and text data are respectively fed into the image encoder and the text encoder to obtain corresponding feature vectors;

[0048] The corresponding feature vectors calculated by the image encoder and the text encoder are fed into the contrast loss function of CLIP, and the loss function is optimized according to the back-propagation algorithm to optimize and update the image encoder and text encoder of CLIP;

[0049] The cross-modal image data and text data to be matched are fed into the optimized image encoder and text encoder to obtain the corresponding feature vectors;

[0050] Calculate the image-text cosine similarity based on the corresponding feature vector;

[0051] When the cosine similarity is greater than a preset threshold, it is determined that the cross-modal image data and the text data match.

[0052] Optionally, the collected images include visible light images, infrared images and ultraviolet images.

[0053] Optionally, separating different objects in the image by an image segmentation algorithm includes distinguishing wires, insulators, and towers from the background.

[0054] Through the above technical scheme, the present invention provides a method for intelligent sorting of drone inspection images based on a multimodal large model. The device status data is collected through drone inspection, and preprocessing can be performed. Then, features that can reflect the status of power facilities can be extracted from the preprocessed image data. Based on the features that can reflect the status of power facilities extracted from the preprocessed image data, high-precision matching of similar images can be calculated. According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched can be encoded in the form of feature vectors, mapped to the same feature space, and the cosine similarity of the image and text can be calculated, so that cross-modal matching of associated images and texts can be achieved according to the cosine similarity of the image and text. The sorting method can intelligently analyze the user's data demand intention, retrieve related image data from a massive image database, and achieve the effect of searching images by images and recognizing images by text.

[0055] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0057] Figure 1 is a flow chart of a method for intelligent sorting of UAV inspection images based on a multimodal large model according to an embodiment of the present invention;

[0058] Figure 2 It is a flowchart of preprocessing of a method for intelligent sorting of UAV inspection images based on a multimodal large model according to an embodiment of the present invention;

[0059] Figure 3 A first flow chart of a method for intelligently sorting unmanned aerial vehicle inspection images based on a multimodal large model according to an embodiment of the present invention for achieving high-precision matching of similar images;

[0060] Figure 4 A second flow chart of a method for intelligently sorting unmanned aerial vehicle inspection images based on a multimodal large model according to an embodiment of the present invention for achieving high-precision matching of similar images;

[0061] Figure 5 A third flow chart of a method for intelligently sorting unmanned aerial vehicle inspection images based on a multimodal large model according to an embodiment of the present invention for achieving high-precision matching of similar images;

[0062] Figure 6It is a first flow chart of a method for intelligently sorting UAV inspection images based on a multimodal large model according to an embodiment of the present invention to achieve cross-modal matching of associated graphics and texts;

[0063] Figure 7 It is a second flow chart of a method for intelligent sorting of drone inspection images based on a multimodal large model according to an embodiment of the present invention to achieve cross-modal matching of associated graphics and texts. DETAILED DESCRIPTION

[0064] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0065] Figure 1 The flowchart of a method for intelligent sorting of drone inspection images based on a multimodal large model according to an embodiment of the present invention. In the present invention, the process of the sorting method may include:

[0066] In step S1, equipment status data is collected through drone inspection and preprocessed.

[0067] In step S2, features that can reflect the status of the power facilities are extracted from the preprocessed image data.

[0068] In step S3, based on the features that can reflect the status of the power facilities extracted from the pre-processed image data, high-precision matching of similar images is calculated.

[0069] In step S4, according to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vector form, mapped to the same feature space, and the cosine similarity of the image and text is calculated to achieve cross-modal matching of associated images and texts.

[0070] The current power drone inspection image sorting method mostly relies on manual pre-classification of image data, such as establishing indexes based on shooting date, line, voltage level, etc., which requires continuous investment in manpower updates. With the continuous addition of new power drone inspection image data, it is necessary to maintain the classification index of the image database at a high cost, which is labor-intensive and inefficient. The current power drone inspection image sorting method cannot intelligently sort data based on image content. Traditional image matching technology has problems such as insufficient flexibility of image matching rules, difficulty in quantifying similarity thresholds, and difficulty in taking into account subjective and objective judgment factors. The effect of using it in image sorting scenarios is not ideal. Relying on manual experience, potential problems are easily ignored. Therefore, the present invention provides a drone inspection image intelligent sorting method based on a multimodal large model, which can intelligently analyze user data demand intentions and realize intelligent sorting. When sorting, the status data of the equipment can be collected through drone inspection, and the data can be preprocessed. After preprocessing, features that can reflect the status of power facilities can be extracted from the preprocessed image data. Based on the features that can reflect the status of power facilities extracted from the preprocessed image data, high-precision matching of similar images can be calculated. The matching can be carried out by classifying and matching the images obtained from the drone inspection with the existing gallery, or by matching the images obtained from the drone inspection with each other. According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched can be encoded in the form of feature vectors, mapped to the same feature space, and the cosine similarity of the image and text can be calculated, so as to achieve cross-modal matching of associated images and texts. The matching can be to match the images obtained from the drone inspection according to the input text, or to match the images in the gallery according to the input text, or to match the corresponding text according to the images obtained from the inspection.

[0071] In one embodiment of the present invention, Figure 2 As shown, the pre-processing process may include:

[0072] In step S5, the drone is equipped with a camera and flies along a predetermined route to capture images or videos of the power facilities.

[0073] In step S6, the image data is transmitted to the ground control station in real time via a wireless network.

[0074] In step S7 , the image data is geometrically corrected and radiometrically corrected.

[0075] In step S8, the plurality of image data are stitched together into a panoramic image or an orthophoto.

[0076] In step S9, random noise in the image is removed by filtering technology.

[0077] In step S10, the images are standardized to ensure that the images collected at different times and under different conditions are consistent in terms of color, brightness, etc.

[0078] In step S11, key features in the image are highlighted by using image enhancement technology.

[0079] In step S12, different objects in the image are separated by an image segmentation algorithm.

[0080] In step S13, the processed image data is dispersed to different nodes using distributed storage technology, and the image data is compressed and stored using compression technology.

[0081] In the present invention, when processing the data collected by the drone, the image or video of the power facilities obtained during the drone inspection can be obtained first, and when the video is obtained, the video can be cut into an image. When the data collected by the drone is obtained, the image data can be transmitted to the ground control station in real time through a wireless network, or stored after preliminary processing on the drone, and uploaded after returning. Due to the movement of the drone, the original image may have problems such as blur, tilt or uneven exposure, so geometric correction and radiation correction are required. When the inspection area is large, it may be necessary to splice multiple images into a panorama or orthophoto for comprehensive analysis. Filtering technology is applied to remove random noise in the image and improve image clarity. The image can be standardized to ensure that the images collected at different times and under different conditions are consistent in color, brightness, etc. Through image enhancement techniques, such as contrast stretching, histogram equalization, etc., the key features in the image are highlighted to facilitate subsequent analysis, and then the different objects in the image can be separated by the image segmentation algorithm. After processing the image data collected by the drone, distributed storage technology can be used to disperse the processed image data to different nodes, and compression technology can be used to compress and store the image data at the same time to reduce the storage and response pressure of the data center.

[0082] In the present invention, after the data collected by the drone is preprocessed, the preprocessed data can be subjected to feature extraction. In order to improve the accuracy and efficiency of the model, it is sometimes necessary to convert the selected features. Different data preprocessing methods should be adopted for specific features. Including converting continuous features into discrete features, converting image features and text features into numerical features, data binning, logarithmic transformation, etc. When converting features, it is necessary to consider the characteristics of the power equipment and the specific image sorting scenario in order to select a suitable conversion method. For the massive power equipment image data after preprocessing, it is necessary to extract features that can represent the operating status, fault information, etc. of the power equipment. When extracting features, the data is deeply analyzed, and according to the specific distribution of the data and the specific application scenario of image sorting, the principal component analysis (PCA) is selected for unsupervised data and the linear discriminant analysis (LDA) is selected for supervised data. After extracting features, the original data contains a large number of features, and only a part of these features are useful for intelligent sorting of power drone inspection images. In order to solve this problem, the correlation coefficient between the features is calculated, the redundant features are removed, and the dimension of the feature space is reduced. And use models such as random forest and gradient boosting tree to rank the importance of features and select the most relevant features. Then use principal component analysis (PCA) to reduce the dimension, retain the principal components that best explain the variability of the data, and reduce the amount of calculation and noise. Then recursive feature elimination (RFE) gradually removes the features that contribute the least to the model prediction until the best model performance is achieved.

[0083] In one embodiment of the present invention, Figure 3 As shown, the first process for achieving high-precision matching of similar images may include:

[0084] In step S14, image data after preprocessing and feature extraction is obtained.

[0085] In step S15, the acquired image data is subjected to transformation processing such as horizontal flipping, vertical flipping, 90 degree rotation, 180 degree rotation and 270 degree rotation to obtain more initial images.

[0086] In step S16, the initial image is processed by Gaussian filtering to obtain the width and height of a new image, and the width and height of the new image are half of the initial image.

[0087] In step S17 , the new image is up-sampled by linear interpolation to enlarge the new image to the same size as the initial image.

[0088] In step S18, the corresponding enlarged new image is subtracted from the initial image to obtain a residual image.

[0089] In step S19, the residual image of the image to be matched is sent to the Siamese network structure to obtain a similar matching image.

[0090] In the present invention, after feature extraction is performed on the image data, the image data after feature extraction can be horizontally flipped, vertically flipped, rotated 90 degrees, 180 degrees, and 270 degrees. In this way, the image data is amplified, and the number of images in the data set is increased to 6 times the original data, so that more initial images can be obtained. After obtaining the initial image, the initial image can be processed by Gaussian filtering, so that the width and height of the new image can be obtained, and the width and height of the new image can be half of the width and height of the initial image. After obtaining the new image, the initial image can be upsampled by linear interpolation, so that the new image can be enlarged to the same size as the initial image. After enlarging the new image, the initial image can be subtracted from the corresponding enlarged new image to obtain a residual image. The residual image can contain high-frequency information in the initial image. Therefore, matching through the residual image can be more accurate and more convenient. After obtaining the residual image, the residual image of the image to be matched can be sent to the Siamese network structure to obtain a similar matching image. According to the matching of the residual image, similar images can be better matched.

[0091] In one embodiment of the present invention, Figure 4 As shown, the second process for achieving high-precision matching of similar images may include:

[0092] In step S20, a residual image of each pair of images to be matched is obtained.

[0093] In step S21, the residual image of each pair of images is sent to the two sub-networks of the Siamese network, and after convolution, pooling and full connection processing, the corresponding embedding vector is obtained.

[0094] In step S22, the similarity of the embedding vectors of each pair of images to be matched is calculated according to formula (1):

[0095] D=||f(I1)-f(I2)|| 2 , formula (1)

[0096] Where D represents the similarity, I1 and I2 represent the first residual image and the second residual image of a pair of input images, and f(I1) and f(I2) represent the corresponding embedding vectors.

[0097] In step S23, when the similarity is greater than a preset threshold, the corresponding pair of images is determined to be similar matching images.

[0098] In the present invention, when the residual image of each pair of images that need to be matched is obtained, the residual image of each pair of images can be sent to the two sub-networks of the Siamese network, and then after convolution, pooling and full connection processing, the corresponding embedding vector can be obtained. After obtaining the embedding vector, the similarity of the embedding vector of each pair of images that need to be matched can be calculated according to formula (1). According to the similarity, it can be judged whether the pair of images matches. When the similarity is greater than a preset threshold, it can be determined that the corresponding pair of images is a similar matching image, so that the pair of images can be matched. Each pair of images can be images obtained from a gallery or a drone inspection, and this method can accurately match the images.

[0099] In one embodiment of the present invention, Figure 5 As shown, the third process for achieving high-precision matching of similar images may include:

[0100] In step S24, the similarity between each pair of images is obtained.

[0101] In step S25, the loss function of the Siamese network is calculated according to formula (2):

[0102]

[0103] Where L is the loss function of the Siamese network, l represents the binary label, if the input I1 and I2 match, then l = 1, if not, then l = 0, and margin represents the boundary value of the mismatched pair.

[0104] In step S26, the gradient of the loss function of the Siamese network with respect to the network parameters is calculated by the back propagation algorithm.

[0105] In step S27, the network parameters are updated through an optimization algorithm according to the obtained gradient to minimize the loss function, and the embedding vector corresponding to each pair of images after optimization is calculated according to the network parameters when the loss function is minimized.

[0106] In the present invention, when calculating the similarity of a pair of images according to the Siamese network and then judging whether they match, the Siamese network can be trained so that the embedding vector distance of the same type of images converted by the optimized Siamese network is as small as possible, and the embedding vector distance of different types of images is as large as possible. Therefore, after constructing the loss function of the Siamese network, the gradient of the loss function of the Siamese network with respect to the network parameters can be calculated by the back propagation algorithm. According to the obtained gradient, the network parameters of the Siamese network can be updated by the optimization algorithm to minimize the loss function, and the embedding vector corresponding to each pair of images after optimization can be calculated according to the network parameters when minimizing the loss function, and then the similarity can be calculated according to the embedded vector and whether it matches can be evaluated.

[0107] In one embodiment of the present invention, Figure 6 As shown, the first process of implementing cross-modal matching of associated graphics and texts may include:

[0108] In step S28, the image encoder and text encoder of CLIP are initialized.

[0109] In step S29, the parameters of the image encoder are frozen to lock the image encoder, and the power scene prompt words are added to the input layer of the text encoder through the prompt fine-tuning algorithm, integrating the model's existing general scene corpus semantic understanding ability to achieve knowledge transfer of the power scene corpus.

[0110] In step S30, the parameters of the text encoder after preliminary fine-tuning are frozen to lock the text encoder, a preset amount of power scene image data is input into the image encoder, self-supervised learning is performed through MAE mask self-encoding, some image blocks are randomly masked, and the visual branch of the CLIP large model is trained with the goal of restoring the image, thereby improving the adaptability of the image encoder to power scene image data.

[0111] In step S31, the text encoder and image encoder are fully unfrozen, and based on the visual and text branches' cognitive basis for the graphic data of power scenes, all parameters are fine-tuned to open up the collaborative capabilities of the graphic encoders to achieve alignment of vision and language.

[0112] In the present invention, a three-stage fine-tuning method can be adopted, the core of which can be to first use LiT to make the text encoder read high-quality representations from the power scene corpus, and then transfer the entire model to the introduced fine-tuning data field. Before the three-stage pre-training, the present invention first initializes the image encoder and the text encoder with the pre-trained model. In the first stage, the image encoder can be "locked" by freezing the parameters of the image encoder during the fine-tuning process, and only the text encoder is fine-tuned. The power scene prompt words are added to the input layer through the prompt fine-tuning algorithm, and the semantic understanding ability of the general scene corpus existing in the model is integrated to realize the knowledge transfer of the power scene corpus. In the second stage, the present invention can "lock" the text encoder by freezing the parameters of the text encoder that has been preliminarily fine-tuned during the fine-tuning process, and only the image encoder is fine-tuned. A certain amount of power scene image data is input, and self-supervised learning is performed through the MAE mask self-encoding method, and part of the image blocks are randomly covered. The visual branch of the CLIP large model is trained with the restoration of the image as the goal, thereby improving the adaptability of the image encoder to the power scene image data. In the third stage, the present invention completely unfreezes the text encoder and the image encoder, and based on the cognitive basis of the visual and text branches on the graphic data of the power scene, conducts fine-tuning of all parameters, opens up the collaborative ability of the graphic encoders, and realizes vision-language alignment.

[0113] In one embodiment of the present invention, Figure 7 As shown, the second process of implementing cross-modal matching of associated graphics and texts may include:

[0114] In step S32, cross-modal image data and text data that need to be matched are obtained.

[0115] In step S33, the cross-modal image data and text data are respectively sent to the image encoder and the text encoder to obtain corresponding feature vectors.

[0116] In step S34, the corresponding feature vectors calculated by the image encoder and the text encoder are fed into the contrast loss function of CLIP, and the loss function is optimized according to the back propagation algorithm to optimize and update the image encoder and text encoder of CLIP.

[0117] In step S35, the cross-modal image data and text data to be matched are sent to the optimized image encoder and text encoder to obtain corresponding feature vectors.

[0118] In step S36, the image-text cosine similarity is calculated based on the corresponding feature vectors.

[0119] In step S37 , when the cosine similarity is greater than a preset threshold, it is determined that the cross-modal image data and the text data match.

[0120] In the present invention, when realizing cross-modal matching of associated images and texts, the cross-modal data and text data to be matched can be obtained first, and then the cross-modal image data and text data can be respectively sent to the image encoder and the text encoder, so that the corresponding feature vector can be obtained. The corresponding feature vector calculated by the image encoder and the text encoder can be sent to the contrast loss function of CLIP. The loss function can be optimized according to the back propagation algorithm, so that the image encoder and the text encoder of CLIP can be optimized and updated. After the image encoder and the text encoder are optimized, the cross-modal image data and text data to be matched can be sent to the optimized image encoder and the text encoder, so that the corresponding feature vector can be obtained. After obtaining the corresponding feature vector, the image-text cosine similarity can be calculated according to the corresponding feature vector, and then it can be determined whether the cosine similarity is greater than a preset threshold. When the cosine similarity is greater than the preset threshold, it can be determined that the cross-modal image data and the text data match. This method can match text by image or match image by text.

[0121] In one embodiment of the present invention, the collected images may include visible light images, infrared images, ultraviolet images and other images.

[0122] In one embodiment of the present invention, different objects in an image are separated by an image segmentation algorithm, so that the conductors, insulators and towers in the image can be separated from the background.

[0123] Through the above technical scheme, the present invention provides a method for intelligent sorting of drone inspection images based on a multimodal large model. The device status data is collected through drone inspection, and preprocessing can be performed. Then, features that can reflect the status of power facilities can be extracted from the preprocessed image data. Based on the features that can reflect the status of power facilities extracted from the preprocessed image data, high-precision matching of similar images can be calculated. According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched can be encoded in the form of feature vectors, mapped to the same feature space, and the cosine similarity of the image and text can be calculated, so that cross-modal matching of associated images and texts can be achieved according to the cosine similarity of the image and text. The sorting method can intelligently analyze the user's data demand intention, retrieve related image data from a massive image database, and achieve the effect of searching images by images and recognizing images by text.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for intelligent sorting of UAV inspection images based on a multimodal large model, characterized in that: The sorting method comprises: Collect equipment status data through drone inspections and perform preprocessing; Extracting features that can reflect the status of power facilities from the preprocessed image data; Based on the features that can reflect the status of power facilities extracted from the pre-processed image data, high-precision matching of similar images is calculated; According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vector form, mapped to the same feature space, and the cosine similarity of the image and text is calculated to achieve cross-modal matching of associated images and texts.

2. The sorting method according to claim 1, characterized in that: The equipment status data is collected through drone inspections and pre-processed, including: The drone is equipped with a camera and flies along a predetermined route to capture images or videos of the power facility; Image data is transmitted to the ground control station in real time via wireless network; Performing geometric correction and radiation correction on the image data; Stitching a plurality of said image data into a panoramic image or an orthophoto; Remove random noise from the image through filtering technology; Standardize images to ensure that images collected at different times and under different conditions are consistent in color, brightness, etc.; Through image enhancement technology, key features in the image are highlighted; Separate different objects in the image through image segmentation algorithm; Distributed storage technology is used to disperse the processed image data to different nodes, and compression technology is used to compress and store the image data.

3. The sorting method according to claim 1, characterized in that: Based on the features that reflect the status of power facilities extracted from the pre-processed image data, high-precision matching of similar images is calculated, including: Obtain image data after preprocessing and feature extraction; Performing transformation processing of horizontal flipping, vertical flipping, 90 degree rotation, 180 degree rotation and 270 degree rotation on the acquired image data to obtain more initial images; The initial image is processed by Gaussian filtering to obtain the width and height of the new image, which is half of the width and height of the initial image; Upsampling the new image by linear interpolation to enlarge the new image to the same size as the initial image; Subtract the corresponding enlarged new image from the initial image to obtain a residual image; The residual image of the image to be matched is sent to the Siamese network structure to obtain a similar matching image.

4. The sorting method according to claim 3, characterized in that: The residual image of the image to be matched is sent to the Siamese network structure to obtain a similar matching image, including: Obtain the residual image of each pair of images that need to be matched; The residual image of each pair of images is sent to the two sub-networks of the Siamese network, and after convolution, pooling and full connection processing, the corresponding embedding vector is obtained; The similarity of the embedding vectors of each pair of images to be matched is calculated according to formula (1): D=||f(I1)-f(I2)|| 2 , formula (1) Where D represents the similarity, I1, I2 represent the first residual image and the second residual image of a pair of input images, and f(I1), f(I2) represent the corresponding embedding vectors; When the similarity is greater than a preset threshold, the corresponding pair of images is determined to be similar matching images.

5. The sorting method according to claim 4, characterized in that: The residual image of the image to be matched is sent to the Siamese network structure to obtain a similar matching image, including: Get the similarity of each pair of images; The loss function of the Siamese network is calculated according to formula (2): Where L is the loss function of the Siamese network, l represents the binary label, if the input I1 and I2 match, then l = 1, if not, then l = 0, margin represents the boundary value of the mismatched pair; The gradient of the loss function of the Siamese network with respect to the network parameters is calculated through the back propagation algorithm; According to the obtained gradient, the network parameters are updated through the optimization algorithm to minimize the loss function, and the embedding vector corresponding to each pair of images after optimization is calculated according to the network parameters when the loss function is minimized.

6. The sorting method according to claim 1, characterized in that: According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vectors, mapped to the same feature space, and the image-text cosine similarity is calculated to achieve cross-modal matching of associated images and texts, including: Initialize CLIP's image encoder and text encoder; Freeze the parameters of the image encoder to lock the image encoder, add power scene prompt words to the input layer of the text encoder through the prompt fine-tuning algorithm, integrate the existing semantic understanding ability of the general scene corpus of the model, and realize the knowledge transfer of the power scene corpus; Freeze the parameters of the text encoder after preliminary fine-tuning to lock the text encoder, input a preset amount of power scene image data into the image encoder, perform self-supervised learning through MAE mask self-encoding, randomly cover part of the image blocks, and train the visual branch of the CLIP large model with the goal of restoring the image, thereby improving the adaptability of the image encoder to the power scene image data; Fully unfreeze the text encoder and image encoder, carry out full parameter fine-tuning based on the visual and text branches' cognitive basis of power scene graphic data, and open up the collaborative capabilities of the graphic encoders to achieve alignment of vision and language.

7. The sorting method according to claim 6, characterized in that: According to the image encoder and text encoder of CLIP, the cross-modal image data and text data of the features to be matched are encoded into feature vectors, mapped to the same feature space, and the image-text cosine similarity is calculated to achieve cross-modal matching of associated images and texts, including: Obtain cross-modal image data and text data that need to be matched; The cross-modal image data and text data are respectively fed into the image encoder and the text encoder to obtain corresponding feature vectors; The corresponding feature vectors calculated by the image encoder and the text encoder are fed into the contrast loss function of CLIP, and the loss function is optimized according to the back-propagation algorithm to optimize and update the image encoder and text encoder of CLIP; The cross-modal image data and text data to be matched are fed into the optimized image encoder and text encoder to obtain the corresponding feature vectors; Calculate the image-text cosine similarity based on the corresponding feature vector; When the cosine similarity is greater than a preset threshold, it is determined that the cross-modal image data and the text data match.

8. The sorting method according to claim 2, characterized in that: The collected images include visible light images, infrared images and ultraviolet images.

9. The sorting method according to claim 2, characterized in that: Image segmentation algorithms are used to separate different objects in the image, including distinguishing wires, insulators, and towers from the background.

Images