A method for pantograph state detection based on meta-learning and multi-scale feature fusion

Abstract

The application discloses a method for detecting the state of a pantograph by combining meta-learning and multi-scale feature fusion, which comprises the following steps: S1, obtaining and processing an original image; positioning the pantograph region by using a YOLOv4 algorithm and cutting in the original image; S2, training a meta-learning classification model, feeding the extracted pantograph ROI region image into a meta-learning classifier, and outputting a classification result. The application has the advantages of simple principle, wide application range, and the ability to effectively improve the detection precision.

Description

Technical Field

[0001] This invention mainly relates to the field of rail transit technology, specifically a pantograph status detection method that combines meta-learning with multi-scale feature fusion. Background Technology

[0002] In recent years, with the continuous development of rail transit, the safety of train operation has received increasing attention. The pantograph, mounted on the roof of an electric locomotive or electric multiple unit (EMU), is used to draw current from one or more contact lines and is a key piece of equipment for driving the electric locomotive. The pantograph operates in a complex environment, often subjected to continuous interference from electromagnetic fields, high temperatures, and vibrations during long-term train operation, making it highly susceptible to malfunctions and potentially leading to train safety accidents. Therefore, real-time online monitoring of the pantograph's status is an indispensable part of ensuring the safe and reliable operation of trains.

[0003] Currently, in actual train operation, pantograph condition detection is mainly divided into two types: contact-based manual inspection and non-contact image inspection. Manual inspection relies on professional maintenance personnel to inspect key components of the train one by one, which suffers from high workload, low efficiency, and high requirements for maintenance personnel, and is therefore gradually being replaced by non-contact image inspection technology. Existing non-contact image inspection methods mostly use deep learning-based object detection networks to locate the pantograph's Region of Interest (ROI). Based on this, traditional image processing methods are used for edge feature extraction, and geometric constraints are applied to the extracted edge features for fault classification. Alternatively, the ROI is used as input, and deep learning methods are used for feature extraction and fault classification. However, because pantograph datasets have a large number of normal samples and a small number of faulty samples, directly training the classifier on the ROI is not ideal. At the same time, compared to normal images, the faulty area of ​​the pantograph generally occupies a small area in the overall image, making it difficult to extract fault features directly using deep learning to train the classifier, resulting in poor fault classification performance.

[0004] In other words, traditional non-contact image detection has the following shortcomings:

[0005] 1. The pantograph fault types have the characteristic of uneven positive and negative samples, and the effect of directly using deep learning algorithms to classify and train them is poor.

[0006] 2. There are often only minor local differences between pantograph fault samples and normal sample images, and directly using deep learning algorithms to extract high-level features for classification results poorly. Summary of the Invention

[0007] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a pantograph state detection method that is simple in principle, has a wide range of applications, and can effectively improve detection accuracy by meta-learning and multi-scale feature fusion.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A pantograph state detection method based on meta-learning and multi-scale feature fusion includes:

[0010] Step S1: Acquire and process the original image; use the YOLOv4 algorithm to locate the pantograph area and crop the original image;

[0011] Step S2: After training with the meta-learning classification model, the extracted pantograph ROI region image is fed into the meta-learning classifier, and the classification result is output.

[0012] As a further improvement to the method of the present invention: the process of step S1 includes:

[0013] Step S101: Acquire the original image using the pantograph image acquisition device;

[0014] Step S102: The acquired original image is fed into the YOLOv4 target detection network to detect the pantograph location in the image and output the pantograph's position coordinates in the original image;

[0015] Step S103: The final output of the YOLOv4 model is (x, y, w, h) and the confidence score. The position of the output is compensated as follows:

[0016] (x,y,w,h)=(x,y,w+m,h+n)

[0017] In the above formula, x and y represent the coordinates of the center point of the detection frame, w and h represent the width and height of the detection frame, respectively, and m and n represent the amount of supplement to the original detection frame size for the trimmed area.

[0018] As a further improvement to the method of the present invention: in step S101, the YOLOv4 model is trained using the labeled dataset to obtain the target detection model for the pantograph ROI area.

[0019] As a further improvement to the method of the present invention: in step S102, the pantograph ROI region target detection model is used to locate the pantograph ROI region in the original image, and the original image is cropped according to the returned pantograph ROI coordinates.

[0020] As a further improvement to the method of the present invention, it also includes enhancing the ROI region image.

[0021] As a further improvement to the method of the present invention, it also includes generating negative samples of the pantograph using a generative adversarial network (GAN) to supplement the number of negative samples of the pantograph; that is, fine-tuning the input signal and the generator so that the distance between the image G(Z) generated by the generator G is minimized and the original image X is minimized.

[0022] As a further improvement to the method of the present invention: the process of generating negative samples of the pantograph using a generative adversarial network (GAN) includes:

[0023] Assuming the random noise vector is Z, the real sample is X, and the generator output is G(Z); the discriminator makes a judgment and obtains D(X) and D(G(Z)); the objective function V(D, G) is as follows:

[0024]

[0025] For the discriminator D, the expected value is D(X) = 1, D(G(Z)) = 0, that is, V(D, G) is maximized, denoted as For a generator G, the expected value is D(G(Z)) = 1, i.e., V(D, G) is minimized, denoted as...

[0026] Discriminator D training: Inputting X and G(Z), the discriminator classifies the two, calculates the error, and updates the parameters; the parameter update of the neural network discriminator is shown in the following formula:

[0027]

[0028]

[0029] In the above formula, W D and b D Here are the parameters of the discriminator; α is the learning rate. The objective is optimized using the stochastic gradient ascent method.

[0030] Generator G training: Virtual samples G(Z) are synthesized from random noise vector Z. Discriminator D classifies X and G(Z). The error is calculated and the generator training parameters are updated via backpropagation, as shown in the following formula:

[0031]

[0032]

[0033] In the above formula, W G and b G β represents the parameters of the discriminator; β is the learning rate.

[0034] As a further improvement to the method of the present invention: in the YOLOv4 target detection network, the YOLOv4 algorithm is used to train the pantograph ROI area localization, and the process includes:

[0035] The acquired pantograph images are labeled, and the labeled data is made into the COCO standard format and divided into training set, test set and validation set.

[0036] Let the training set be {(h1, t1), (h2, t2), ..., (h... n , t n )}, where h i For the input image, t i The corresponding label is denoted by , and n is the number of training samples. The input image h is processed by Backbone. j Feature extraction is performed as shown in the following formula:

[0037] x c =f(h i )

[0038] In the above formula, x c represents the generated feature values ​​used for prediction, and f represents the feature extraction process;

[0039] Calculate the loss generated during training:

[0040]

[0041] In the above formula, L(W, b; h, t) represents the loss generated during the training process; W and b represent the connection coefficient and bias coefficient in the neural network, respectively.

[0042] After solving the loss function, we first calculate the partial derivatives of the loss function with respect to the connection coefficients and bias coefficients. Then, we use the batch stochastic gradient descent algorithm to update the network coefficients to minimize the loss and complete the training of the neural network.

[0043] As a further improvement to the method of the present invention: In step S2, a meta-learning classifier based on metric learning is used to extract positive and negative sample features through a convolutional neural network, and learn a metric for comparing the similarity between samples. The similarity between the features of the ROI region to be detected and the features of the positive and negative sample images is calculated to determine the category.

[0044] As a further improvement to the method of this invention: meta-learning adopts an N-way K-shot classification method, where N represents the number of categories and K represents the number of samples used in each category; in the feature extraction process, the features of shallow networks and deep networks are fused; the classification model consists of a feature extraction module and a similarity measurement module, both of which are constructed using lightweight convolutional neural networks; the feature extraction module is used to extract features from the images of each sample set, and the similarity measurement module is used to learn the similarity measurement method.

[0045] As a further improvement to the method of the present invention: the backbone network of the feature extraction module consists of the first five convolutional modules of VGG16, with the output of the third convolutional module serving as an intermediate feature for multi-scale fusion.

[0046] As a further improvement to the method of the present invention: the training process in step S2 includes:

[0047] Step S601: Create a training set TR from all samples according to different categories;

[0048] Step S602: Randomly select k samples from each class in the training set TR to form the support set. Where, x i For the image, y i Let C be the label and C be the number of categories. Randomly select n images from the remaining images to form the query set. Suppose there are W negative samples enhanced by the GAN network, then the number of training iterations is increased from W times for the traditional classifier to... Second-rate;

[0049] Step S603: Input the support set and query set images into the feature extraction module. Feature maps are formed respectively in the middle. and Let the concatenation operation C(,) represent concatenating the feature maps along the depth direction. Then the concatenated feature map is: Let the extracted features be Concatenating it with the support set features yields the concatenated features.

[0050] Step S604: Input the concatenated features into the similarity measurement module g φ In this module, the output is a scalar between 0 and 1, representing x. i With x j The similarity between them is called the relationship score, as shown in the following formula:

[0051]

[0052] In the above formula, r i，jThis represents the similarity score between two features; the higher the score, the higher the similarity.

[0053] Step S605: Train the model using mean squared error loss, such that the similarity between correctly matched samples is 1, and the similarity between incorrectly matched samples is 0, as shown in the following formula:

[0054]

[0055] Compared with the prior art, the advantages of the present invention are as follows:

[0056] 1. The pantograph state detection method of this invention, which combines meta-learning and multi-scale feature fusion, is simple in principle, widely applicable, and effectively improves detection accuracy. It solves the problem of low accuracy in existing deep learning-based pantograph state detection methods due to uneven distribution of positive and negative pantograph samples and the lack of clear distinction between faulty and normal sample images. This invention first locates the pantograph in the image using YOLOv4, then crops the pantograph Region of Interest (ROI) from the original image based on the returned coordinates, and then feeds the ROI into a classifier for pantograph state classification. Secondly, it enhances the image data through operations such as geometric transformation, rotation, scaling, flipping, and noise injection to improve sample diversity. Simultaneously, in the feature extraction stage, it fuses features from shallow and deep networks to more fully preserve the features of small objects, achieving better differentiation between positive and negative sample features. This approach, through localization followed by classification, image enhancement, and feature fusion, can better extract features from small targets.

[0057] 2. The pantograph state detection method of the present invention, which integrates meta-learning and multi-scale feature fusion, utilizes a GAN network to generate negative samples to expand the negative sample dataset; and uses a meta-learning method to train the classifier network, effectively alleviating the problem of unsatisfactory classifier training results caused by uneven positive and negative samples.

[0058] 3. The pantograph status detection method of the present invention, which integrates meta-learning and multi-scale feature fusion, addresses the problem that the uneven distribution of positive and negative samples in pantograph fault types leads to poor classification training results when directly using deep learning algorithms. The present invention constructs a meta-learning method based on metric learning to train the classifier network, and simultaneously uses a GAN network to generate negative samples to expand the dataset. This effectively alleviates the impact of uneven positive and negative samples on classification accuracy and improves the accuracy of online real-time pantograph status detection. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the method of the present invention.

[0060] Figure 2 This is a schematic diagram illustrating the implementation principle in a specific application example.

[0061] Figure 3 This is a schematic diagram illustrating the model structure and principle of a similarity metric learning classifier in a specific application example.

[0062] Figure 4 This is a schematic diagram illustrating the principle of generating adversarial network model structures in specific application examples.

[0063] Figure 5 This is a schematic diagram illustrating the principle of the feature extraction module in a similarity measurement learning model in a specific application example.

[0064] Figure 6 This is a schematic diagram of the similarity calculation module in similarity metric learning in a specific application example.

[0065] Figure 7 This is a schematic diagram of image enhancement during the classification process of a meta-learning classifier in a specific application example. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] In the description of this application, in order to better illustrate the technical content of the present invention, the following technical terms will be explained:

[0068] ROI: Region of Interest, which represents the region where the target is located in an object detection task.

[0069] Image enhancement: By using a series of techniques such as rotation, cropping, and adding noise, the diversity of the original image is increased, thereby enabling the image to achieve better results in deep learning networks.

[0070] GAN Network: Generative Adversarial Network, is a deep learning method that uses random noise from the input to generate samples to supplement the dataset.

[0071] Meta-learning: A deep learning method for few-shot problems.

[0072] Backbone: The backbone network in a deep learning model, primarily used for feature extraction.

[0073] Bounding Box: The location of the target marked by the network in the target detection task.

[0074] The present invention provides a pantograph state detection method based on meta-learning and multi-scale feature fusion. The method first uses YOLOv4 to obtain the pantograph region of interest (ROI) and crops it in the original image. Then, it uses a meta-learning classifier based on metric learning to classify the ROI region image.

[0075] Take a specific application as an example, such as Figure 1 and Figure 2 As shown, the detailed process of the present invention may include:

[0076] Step S1: Acquire and process the original image;

[0077] The YOLOv4 algorithm is used to locate the pantograph region, and cropping is performed on the original image. Since the original image contains not only the pantograph but also a series of complex information such as the natural background environment, rooftop equipment, and overhead contact line suspension devices, direct target state classification yields poor results. Furthermore, the pantograph occupies a small area in the original image, and the differences between abnormal and normal pantograph samples are extremely subtle. Therefore, this invention innovatively crops out the pantograph ROI region for classification, which helps to reduce the detection range and improve classification accuracy.

[0078] In practical applications, the original data annotation is completed in step S1. That is, the original image is acquired by the camera, the pantograph area in the image is annotated using labelme software, and the annotated data is made into the COCO standard format for model training.

[0079] Step S2: Target detection model training: Train the YOLOv4 model using the labeled dataset to obtain the target detection model for the pantograph ROI area;

[0080] Step S3: Pantograph ROI region extraction: The pantograph ROI region is located in the original image using the pantograph ROI region target detection model, and the original image is cropped according to the returned pantograph ROI coordinates;

[0081] Step S4: ROI region image enhancement processing: Using methods such as contrast enhancement, color enhancement, sharpness enhancement, and brightness enhancement, the ROI region image is enhanced to improve the diversity of the samples;

[0082] Step S5: GAN network supplements negative samples: Generative adversarial network is used to generate negative samples of pantograph to supplement the number of negative pantograph samples;

[0083] Step S6: Meta-learning classification model training: Design a meta-learning classifier network, divide the enhanced and supplemented pantograph ROI images into training and test sets for classifier network training;

[0084] Step S7: Pantograph status classification: The extracted pantograph ROI region image is fed into the meta-learning classifier, and the classification result is output.

[0085] In a specific application example, in step S1, after the camera acquires the original image, it first performs data cleaning operations, namely, extracting key frame images and deduplicating images.

[0086] As a preferred embodiment, in a specific application example, step S1 may include the following process:

[0087] Step S101: Acquire the original image using the pantograph image acquisition device;

[0088] Step S102: The acquired original image is fed into the YOLOv4 target detection network to detect the pantograph location in the image and output the pantograph's position coordinates in the original image;

[0089] Step S103: The final output of the YOLOv4 model is (x, y, w, h) and the confidence score. To ensure that the cropped ROI region contains the complete pantograph outline, the output position is compensated as follows:

[0090] (x,y,w,h)=(x,y,w+m,h+n)

[0091] In the above formula, x and y represent the coordinates of the center point of the detection frame, w and h represent the width and height of the detection frame, respectively, and m and n represent the amount of supplement to the original detection frame size for the trimmed area.

[0092] In a specific application example, in step S2, the YOLOv4 algorithm is used to train the pantograph ROI area localization. The specific steps are as follows:

[0093] Step S201: Label the acquired pantograph images, convert the labeled data into COCO standard format, and divide them into training set, test set, and validation set;

[0094] Step S202: Let the training set be {(h1, t1), (h2, t2), ..., (h... n , t n )}, where h i For the input image, t i The corresponding label is denoted by , and n is the number of training samples. The input image h is processed by Backbone. j Feature extraction is performed as shown in the following formula:

[0095] x c =f(h i )

[0096] In the above formula, xc represents the generated feature values ​​used for prediction, and f represents the feature extraction process;

[0097] Step S203: Calculate the loss generated during training:

[0098]

[0099] In the above formula, L(W, b; h, t) represents the loss generated during the training process. W and b represent the connection coefficient and bias coefficient in the neural network, respectively.

[0100] Step S204: After solving the loss function, first calculate the partial derivatives of the loss function with respect to the connection coefficients and bias coefficients, and then use the batch stochastic gradient descent algorithm to update the network coefficients to minimize the loss, thereby completing the basic training of the neural network.

[0101] Because the pantograph anomaly classification problem exhibits significant differences in sample size across different categories, directly training the classification model on these samples would bias it towards normal samples, resulting in high overall classification accuracy but low anomaly detection rate. Therefore, in this specific application example, step S5 uses a GAN network to generate negative samples for sample supplementation; the input signal and generator are fine-tuned to minimize the distance between the image G(Z) generated by the generator G and the original image X.

[0102] As a preferred embodiment, in specific applications, the detailed process of step S5 may include:

[0103] Step S501: Assume the random noise vector is Z, the real sample is X, and the generator output is G(Z); the discriminator makes a judgment, obtaining D(X) and D(G(Z)). The objective function V(D, G) is as follows:

[0104]

[0105] For the discriminator D, the expected value is D(X) = 1, D(G(Z)) = 0, that is, V(D, G) is maximized, denoted as For a generator G, the expected value is D(G(Z)) = 1, i.e., V(D, G) is minimized, denoted as...

[0106] Step S502: Discriminator D Training: Input X and G(Z) into the discriminator, classify the two, calculate the error, and update the parameters. Taking a neural network discriminator as an example, the parameter update is shown in the following formula:

[0107]

[0108]

[0109] In the above formula, W D and b D α represents the parameters of the discriminator; α is the learning rate. The objective is optimized using the stochastic gradient ascent method.

[0110] Step S503: Generator G Training: Synthesize virtual samples G(Z) from the random noise vector Z. The discriminator D classifies X and G(Z). Calculate the error and update the generator training parameters using backpropagation, as shown in the following formula:

[0111]

[0112]

[0113] In the above formula, W G and b G β represents the parameters of the discriminator; β is the learning rate.

[0114] See Figures 3-7 In a specific application example, as a preferred solution, in step S6, the present invention further employs a meta-learning classifier based on metric learning. This classifier extracts positive and negative sample features through a convolutional neural network and learns a metric for comparing the similarity between samples. The similarity between the features of the ROI region to be detected and the features of the positive and negative sample images is calculated to determine the category. The meta-learning uses an N-way K-shot classification method, where N represents the number of categories and K represents the number of samples used in each category, effectively addressing the problem of uneven distribution of positive and negative samples. During feature extraction, features from shallow and deep networks are fused to more fully preserve the features of small objects, thus better distinguishing subtle differences between pantographs.

[0115] Furthermore, the classification model in this invention comprises a feature extraction module and a similarity measurement module, both of which are constructed using lightweight convolutional neural networks. The feature extraction module extracts features from the images of each sample set, while the similarity measurement module learns a similarity measurement method to replace traditional manually designed linear measurement methods, such as Euclidean distance or cosine similarity.

[0116] Furthermore, the backbone network of the feature extraction module consists of the first five convolutional modules of VGG16, and the output of the third convolutional module is retained as an intermediate feature for further multi-scale fusion.

[0117] Based on the needs of practical applications, the training process in this invention can preferably follow the following steps:

[0118] Step S601: Create a training set TR from all samples according to different categories;

[0119] Step S602: Randomly select k samples from each class in the training set TR to form the support set. Where, x i For the image, y i Let C be the label and C be the number of categories. Randomly select n images from the remaining images to form the query set. Suppose there are W negative samples enhanced by the GAN network, then the number of training iterations is increased from W times for the traditional classifier to... This allows for full utilization of information from negative samples;

[0120] Step S603: Considering the subtle differences between different pantographs, the feature extraction module uses an FPN+PAN structure to fuse high-level semantic information and low-level positional information. The support set and query set images are input into the feature extraction module. Feature maps are formed respectively in the middle. and Let the concatenation operation C(,) represent concatenating the feature maps along the depth direction. Then the concatenated feature map is: Since there are often only minor local differences between pantograph fault samples and normal sample images, this embodiment employs an FPN+PAN structure for feature fusion to better utilize low-level features. In the FPN module, the output of each stage is composed of the fusion of the upsampled feature map and the features from each stage of VGG16, such as... Figure 5 As shown, the feature map contains both a large amount of semantic information and low-level feature information. Let the extracted features be... Concatenating it with the support set features yields the concatenated features.

[0121] Step S604: Input the concatenated features into the similarity measurement module g φ In this module, the output is a scalar between 0 and 1, representing x. i With x j The similarity between them is called the relationship score, as shown in the following formula:

[0122]

[0123] In the above formula, r i，j This represents the similarity score between two features; a higher score indicates a higher similarity. The lightweight neural network constructed in this example forms the similarity measurement module, as shown below. Figure 6 As shown, after feature extraction and feature stitching of the pantograph ROI region are completed, the feature is sent to the similarity measurement module, and its category is determined based on the output score.

[0124] Step S605: Train the model using mean squared error loss, such that the similarity between correctly matched samples is 1, and the similarity between incorrectly matched samples is 0, as shown in the following formula:

[0125]

[0126] It should be noted that this application uses the YOLOv4 object detection model to determine the pantograph ROI region. Depending on the actual application needs and scenarios, this can be replaced by other object detection models such as Faster R-CNN and SSD. In constructing the meta-learning classifier based on metric learning, this application uses the first 5 layers of the VGG16 network to extract features of the ROI region. It can also be replaced by backbone networks such as GoogLeNet and AlexNet. Some convolutional layers in the network can be replaced by dilated convolutions to expand the receptive field and enhance the detection of small objects.

[0127] See Figure 7 The present invention enhances the sharpness and contrast of the located ROI region by means of other methods, so as to facilitate better classification of the ROI region and thus improve the classification accuracy.

[0128] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for pantograph condition detection based on meta-learning and multi-scale feature fusion, characterized in that, The utility model relates to a kind of pantograph detection method and device based on meta-learning classification, including: Step S1: obtaining original image and processing; Pantograph region positioning is carried out using YOLOv4 algorithm, and cropping is carried out in the original image; Step S2: meta-learning classification model training, the extracted pantograph ROI region image is sent into meta-learning classifier, and the classification result is output; The process of step S1 includes: Step S101: obtaining original image through pantograph image acquisition device; Step S102: the original image collected is sent into YOLOv4 target detection network, and the pantograph position existing in the image is detected and the position coordinates of pantograph in original image are output; The final output result of YOLOv4 model is (x, y, w, h) and confidence, and the following compensation is made to the output position: (x, y, w, h)=(x, y, w+m, h+n) In the formula, x and y represent the center point coordinates of the detection frame respectively, w and h represent the width and height of the detection frame respectively, and m and n represent the supplementary amount of the cropped region based on the original detection frame size; In the YOLOv4 target detection network, the training of pantograph ROI region positioning is carried out using YOLOv4 algorithm, and the process includes: The pantograph image obtained is labeled, the labeled data is made into COCO standard format, and the training set, test set and validation set are divided well; Let the training set be wherein is an input image, is a corresponding label, n is the number of training samples, the input image is extracted by the Backbone, the process is shown in the following formula: In the above formulae, denotes the generated feature values for predicting, f denotes a feature extraction process; The loss generated in the training process is calculated: In the above formula, denotes the loss generated by the training process; W and b denote the connection coefficients and bias coefficients in the neural network, respectively; After completing the loss function solution, first, the partial derivative of the loss function with respect to the connection coefficient and the bias coefficient is calculated, then the network coefficient is updated by using batch stochastic gradient descent algorithm to minimize the loss, and the training of neural network is completed; In step S2, the meta-learning classifier based on metric learning is adopted, the positive and negative sample features are extracted through convolutional neural network, and the similarity measurement method between samples is learned and compared, the similarity size of the features of the ROI region to be detected and the features of positive and negative sample images is calculated to judge the category; Meta-learning adopts N-way K-shot classification mode, wherein N represents the number of categories, and K represents the number of samples used for each category;In the feature extraction process, the features of the shallow network and the features of the deep network are fused;The composition of the classification model includes a feature extraction module and a similarity measurement module, both of which are composed of lightweight convolutional neural network;The feature extraction module is used to extract the features of each sample set image, and the similarity measurement module is used to learn the similarity measurement method; The backbone network of the feature extraction module is composed of the first five convolution modules of VGG16, and the output of the third convolution module is used as the intermediate feature for multi-scale fusion; The training process of step S2 includes: Step S601: all samples are made into training set TR according to different categories; Step S602: Randomly extract k samples of each class in the training set TR to form a support set , wherein x i is an image, y i is a label, C is the number of classes, and n images are randomly extracted from the remaining images to form a query set Suppose that there are W negative samples enhanced by the GAN network, and the training times are increased from W times of the traditional classifier to times. Step S603: input the support set and the query set image to the feature extraction module The feature maps are formed in the middle part respectively And Suppose the splicing operation Indicates that the feature maps are spliced in the depth direction, then the spliced features are Suppose the extracted features are Spliced with the support set features, the spliced features are ​ Step S604: Input the concatenated features into the similarity measurement module. In this module, the output is a scalar between 0 and 1, representing... and The similarity between them is called the relationship score, as shown in the following formula: In the above formula, r i,j represents the similarity score between two features, the greater the score, the higher the similarity. Step S605: the mean square error loss is used to train the model, so that the similarity between correctly matched samples is 1, and the similarity between incorrectly matched samples is 0, as shown in the following formula: 。 2. The method of claim 1, wherein, In step S101, the YOLOv4 model is trained using the labeled data set to obtain the pantograph ROI region target detection model.

3. The method of claim 1, wherein, In the step S102, the pantograph ROI region target detection model is used to locate the pantograph ROI region of the original image, and the original image is intercepted according to the returned pantograph ROI coordinates.

4. The method of claim 3, wherein, It also includes enhancing the ROI region image.

5. The method of claim 3, wherein, Also included is generating pantograph negative samples using a generative adversarial network (GAN) network to supplement the number of pantograph negative samples; that is, fine-tuning the input signal and the generator so that the generator G generated image G(Z) has the smallest distance from the original image X .

6. The method of claim 5, wherein the method further comprises: The process of generating the pantograph negative sample by using the generative adversarial network (GAN) network includes: Assume the random noise vector is Z , the real sample is X , and the generator output is ( ); the discriminator judges and obtains ( ) and ( ( )); the objective function ( , ) is as follows: For the discriminator D , we expect ( ) = 1, ( ( )) = 0, i.e. ( , ) max, denoted by ; for the generator G, we expect ( ( ))=1, i.e. ( , ) min, denoted by ; discriminator D Training: X is input into ( ) where the discriminator classifies both, computes error and updates parameters; the parameter update for the neural network discriminator is given by the following equation: In the above formula, and are parameters of the discriminator; is a learning rate; the objective is optimized using the method of stochastic gradient ascent. Generator G Training: synthetic samples from random noise vectors Z G ( ), Discriminator D Classify X and G ( ) Compute error and update generator training parameters in a backpropagation fashion, formula as follows:​ In the above formula, and are parameters of the discriminator; is the learning rate.

Images