Foreign body detection method and related equipment based on small sample learning

Through the foreign object detection method based on small sample learning, the weight sharing feature extraction and matching relationship learning of RPN network are used to solve the problem of poor foreign object detection effect under small sample data sets, and efficient foreign object detection in small sample situations is achieved.

CN115147406BActive Publication Date: 2024-11-26SHENZHEN INST OF ARTIFICIAL INTELLIGENCE & ROBOTICS FOR SOC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210976734.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-15
Publication Date
2024-11-26
Estimated Expiration
2042-08-15

AI Technical Summary

Technical Problem

Deep learning technology trained detection models under small sample data sets are prone to overfitting, resulting in poor foreign object detection effect. Especially in scenes such as transmission pole towers, it is difficult to obtain high quality due to the small number of samples and the small proportion of foreign object area. sample.

Method used

Using a foreign object detection method based on small sample learning, the pre-trained weight sharing feature extraction network performs feature extraction of the detected images and the supported image set, performs weighted fusion, and learns the matching between candidate area features and support features in the RPN network. Relationship, select the target matching relationship that meets the preset conditions to determine the foreign object type and region.

Benefits of technology

The performance of foreign object detection is significantly improved under the small sample data set, and it can achieve good detection results when there are only dozens of images supported in each category, which reduces the number of network parameters and improves the generalization ability, and reduces the over-sampling. Combination phenomenon.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115147406B_ABST
    Figure CN115147406B_ABST
Patent Text Reader

Abstract

This application discloses a foreign object detection method and related equipment based on few-shot learning, which is used to improve the foreign object detection effect based on few-shot learning. The method of this application includes: determining the image to be detected and a first set of supporting images; performing feature extraction based on a feature extraction network to obtain the features to be tested and a set of supporting features; weighting and fusing the features to be tested with each type of supporting features to obtain corresponding N types of weighted features to be tested; inputting each type of weighted features to be tested into an RPN network to determine at least one candidate region feature; learning the matching relationship between the candidate region features of each type of foreign object and the corresponding category of supporting features; selecting a target matching relationship that meets preset conditions from multiple matching relationships; determining the type of foreign object and the foreign object region in the image to be detected according to the candidate region features corresponding to the target matching relationship and the foreign object type corresponding to the candidate region features corresponding to the target matching relationship.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a foreign body detection method based on a small sample and related equipment. Background Art

[0002] In recent years, deep learning technology has been applied to various aspects of daily life, such as face recognition and license plate recognition, and has achieved great success. However, deep learning technology is data-driven and requires a large number of high-quality samples for training to obtain high-quality detection models.

[0003] When it is difficult to obtain high-quality samples, the detection model trained based on a small sample data set often suffers from serious overfitting, which in turn leads to poor detection effect of the obtained detection model. Summary of the invention

[0004] The embodiments of the present application provide a foreign body detection method based on small sample learning and related equipment, which are used to improve the foreign body detection effect based on small sample learning.

[0005] A first aspect of an embodiment of the present application provides a foreign body detection method based on small sample learning, comprising:

[0006] Determine an image to be detected and a first supporting image set, wherein the first supporting image set includes N types of first supporting images, each type of the first supporting images includes at least one first supporting image, each of the first supporting images has a pre-marked foreign body area and foreign body type, and each first supporting image in each type of the first supporting images has the same foreign body type;

[0007] Performing feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the features to be detected of the image to be detected and a support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of first support image;

[0008] The features to be tested are weightedly fused with each type of supporting features to obtain corresponding N types of weighted features to be tested;

[0009] Inputting each type of weighted features to be detected into the RPN network respectively, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter;

[0010] Learning the matching relationship between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category;

[0011] Among the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a target matching relationship that meets a preset condition is selected;

[0012] According to the candidate region features corresponding to the target matching relationship and the foreign matter types corresponding to the candidate region features corresponding to the target matching relationship, the foreign matter types and foreign matter regions in the image to be detected are determined.

[0013] In a specific implementation, the learning of the matching relationship between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category includes:

[0014] Performing region of interest pooling processing on at least one candidate region feature of each type of foreign body and the supporting features of the corresponding category;

[0015] The matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales is learned, and the matching relationship between each candidate region and the supporting features of the corresponding category is obtained by integrating the matching relationships at multiple pixel scales.

[0016] In a specific implementation, the learning of the matching relationship between each candidate region feature after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process at multiple pixel scales, and obtaining the matching relationship between each candidate region and the supporting features of the corresponding category by integrating the matching relationships at multiple pixel scales includes:

[0017] Connecting each of the candidate region features after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, and performing a first convolution process on the connected features to obtain a matching relationship at a first pixel scale;

[0018] Connecting each of the candidate region features after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, performing average pooling process on the connected features, and performing the first convolution process on the features after the average pooling process to obtain a matching relationship on a second pixel scale;

[0019] Performing a second convolution process on each of the candidate region features after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process in the corresponding channel dimension, and performing the first convolution process on the features after the second convolution process to obtain a matching relationship on a third pixel scale;

[0020] The matching relationship at the first pixel scale, the matching relationship at the second pixel scale, and the matching relationship at the third pixel scale are added together to obtain a matching relationship between each candidate region and a supporting feature of a corresponding category.

[0021] In a specific implementation, the method further includes:

[0022] Select each image from the preset training data set in turn as an image to be trained;

[0023] Selecting a plurality of images other than the to-be-trained images from a preset training data set as a second support image set;

[0024] Based on each image to be trained, the weight-shared initial feature extraction network is trained with each image to be trained and the corresponding second support image set to obtain the weight-shared feature extraction network.

[0025] In a specific implementation, the step of performing weighted fusion of the feature to be tested and each type of the supporting features to obtain corresponding N types of weighted features to be tested includes:

[0026] Determine the foreign matter region corresponding to each type of the supporting features based on the coordinate information of the foreign matter region of the supporting image corresponding to each type of the supporting features;

[0027] Separating the features of the foreign matter region corresponding to each type of the supporting features to obtain the foreign matter features corresponding to each type of the supporting features;

[0028] Each type of the foreign body feature is used to weight the feature to be measured to obtain corresponding N types of weighted features to be measured.

[0029] In a specific implementation, the step of determining the image to be detected includes:

[0030] The foreign object foreground detection is performed on the initial image to separate the image to be detected containing the foreign object and the foreign object foreground corresponding to the foreign object from the initial image.

[0031] In a specific implementation, the matching relationship includes matching confidence;

[0032] The selecting a target matching relationship that satisfies a preset condition from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category comprises:

[0033] Among the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a matching relationship corresponding to a matching confidence greater than a preset matching confidence is selected as a preliminary screening matching relationship;

[0034] According to the candidate area corresponding to each of the preliminary screening matching relationships, two preliminary screening matching relationships whose corresponding candidate areas in each of the preliminary screening matching relationships are greater than or equal to a preset intersection-and-union ratio are determined as a group of matching relationships to be selected;

[0035] Deleting the initial screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected;

[0036] Among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

[0037] A second aspect of an embodiment of the present application provides a foreign body detection device, including:

[0038] a determination unit, configured to determine an image to be detected and a first supporting image set, wherein the first supporting image set includes N types of first supporting images, each type of the first supporting images includes at least one first supporting image, each of the first supporting images has a pre-marked foreign body region and foreign body type, and each first supporting image in each type of the first supporting images has the same foreign body type;

[0039] an extraction unit, configured to perform feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the features to be detected of the image to be detected and a support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of first support image;

[0040] A weighting unit, used for weightedly fusing the feature to be tested with each type of supporting features to obtain corresponding N types of weighted features to be tested;

[0041] An input unit, used to input each type of the weighted features to be detected into the RPN network, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter;

[0042] A learning unit, used for learning the matching relationship between at least one candidate region feature of each type of foreign matter and a supporting feature of a corresponding category;

[0043] A selection unit, configured to select a target matching relationship that satisfies a preset condition from among the matching relationships between at least one candidate region feature of each type of foreign matter and a supporting feature of a corresponding category;

[0044] The determination unit is further used to determine the foreign body type and foreign body area in the image to be detected according to the candidate area features corresponding to the target matching relationship and the foreign body type corresponding to the candidate area features corresponding to the target matching relationship.

[0045] In a specific implementation, the learning unit is specifically used to perform region of interest pooling processing on at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category;

[0046] The matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales is learned, and the matching relationship between each candidate region and the supporting features of the corresponding category is obtained by integrating the matching relationships at multiple pixel scales.

[0047] In a specific implementation, the learning unit is specifically used to connect each of the candidate region features after the region of interest pooling processing with the supporting features of the corresponding category after the region of interest pooling processing in the channel dimension, and perform a first convolution processing on the connected features to obtain a matching relationship on a first pixel scale;

[0048] Connecting each of the candidate region features after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, performing average pooling process on the connected features, and performing the first convolution process on the features after the average pooling process to obtain a matching relationship on a second pixel scale;

[0049] Performing a second convolution process on each of the candidate region features after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process in the corresponding channel dimension, and performing the first convolution process on the features after the second convolution process to obtain a matching relationship on a third pixel scale;

[0050] The matching relationship at the first pixel scale, the matching relationship at the second pixel scale, and the matching relationship at the third pixel scale are added together to obtain a matching relationship between each candidate region and a supporting feature of a corresponding category.

[0051] In a specific implementation, the device further includes: a training unit;

[0052] The selection unit is further used to select each image from the preset training data set in turn as an image to be trained;

[0053] The selection unit is further used to select a plurality of images other than the to-be-trained images from a preset training data set as a second support image set;

[0054] The training unit is used to train the weight-shared initial feature extraction network based on each image to be trained with each image to be trained and the corresponding second support image set to obtain the weight-shared feature extraction network.

[0055] In a specific implementation, the weighting unit is specifically used to determine the foreign matter region corresponding to each type of the supporting features based on the coordinate information of the foreign matter region of the supporting image corresponding to each type of the supporting features;

[0056] Separating the features of the foreign matter region corresponding to each type of the supporting features to obtain the foreign matter features corresponding to each type of the supporting features;

[0057] Each type of the foreign body feature is used to weight the feature to be measured to obtain corresponding N types of weighted features to be measured.

[0058] In a specific implementation, the determination unit is specifically configured to perform foreign object foreground detection on an initial image, so as to separate an image to be detected containing a foreign object and a foreign object foreground corresponding to the foreign object from the initial image.

[0059] In a specific implementation, the matching relationship includes matching confidence;

[0060] The selection unit is specifically configured to select, from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a matching relationship corresponding to a matching confidence greater than a preset matching confidence as a preliminary screening matching relationship;

[0061] According to the candidate area corresponding to each of the preliminary screening matching relationships, two preliminary screening matching relationships whose corresponding candidate areas in each of the preliminary screening matching relationships are greater than or equal to a preset intersection-and-union ratio are determined as a group of matching relationships to be selected;

[0062] Deleting the initial screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected;

[0063] Among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

[0064] A third aspect of the present application provides a foreign body detection device, including:

[0065] CPU, memory and input / output interface;

[0066] The memory is a short-term storage memory or a persistent storage memory;

[0067] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0068] A fourth aspect of the embodiments of the present application provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method described in the first aspect.

[0069] A fifth aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0070] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: a foreign body detection network and a foreign body detection method suitable for small sample learning are provided. The input of the foreign body detection network includes a first support image set and an image to be detected, and a weight-sharing feature extraction network is used to extract basic features from the two inputs and weight them, and then matching relationship learning is performed to improve the detection performance of the foreign body detection network. Good foreign body detection performance can be achieved even when there are only dozens of support images of each type in the first support image set. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of a flow chart of a foreign body detection method based on small sample learning disclosed in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of a flow chart of a feature weighting method disclosed in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of a flow chart of matching relationship learning disclosed in an embodiment of the present application;

[0074] Figure 4 Another schematic diagram of the process of the foreign body detection method based on small sample learning disclosed in the embodiment of the present application;

[0075] Figure 5 A schematic diagram of the structure of the foreign body detection device disclosed in the embodiment of the present application;

[0076] Figure 6 This is another schematic diagram of the structure of the foreign object detection device disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] The safety of the power grid directly affects society, the economy, and people's livelihood. Transmission towers are key components of transmission lines, and the safety of transmission towers greatly affects the safe and stable operation of the power grid. Most transmission towers are in the wild for a long time, and are often attacked by foreign objects, such as beehives, bird nests, and animal carcasses. Foreign objects invading transmission towers pose a great safety hazard to the safety of the power grid. The common method of detecting foreign objects on transmission towers is manual on-site detection and identification, but this method is time-consuming, labor-intensive, and costly. With the development of drone technology, drone power inspections have come into view, but the images or videos collected by drones still need to be manually interpreted, which is inefficient.

[0079] In recent years, deep learning technology has been applied to various aspects of daily life, such as face recognition and license plate recognition, and has achieved great success. However, deep learning technology is data-driven and requires a large amount of high-quality data for training to obtain high-quality models. However, foreign body intrusion on power transmission lines is an occasional phenomenon, and the number of samples is relatively small. When drones collect images, there are often problems such as occlusion and the small proportion of the foreign body area in the image, making it difficult to obtain high-quality samples. Therefore, foreign body detection on transmission lines belongs to small sample and small target detection. Small sample learning applied to existing target detection neural networks often produces serious overfitting, which ultimately leads to poor detection results of the trained network.

[0080] In order to solve the above problems, the embodiments of the present application provide a foreign body detection method based on small sample learning and related equipment, which are used to improve the foreign body detection effect based on small sample learning.

[0081] See also Figure 1 In a first aspect, an embodiment of the present application provides a foreign body detection method based on small sample learning, comprising the following steps 101 to 106:

[0082] 101. Determine an image to be detected and a first supporting image set, where the first supporting image set includes N categories of first supporting images, each category of first supporting images includes at least one first supporting image, each first supporting image has a pre-marked foreign body area and foreign body type, and each first supporting image in each category of first supporting images has the same foreign body type.

[0083] When it is necessary to detect foreign matter in an image, it is first necessary to obtain the image and the preset first support image set as dual inputs of the foreign matter detection network. Among them, the preset first support image set contains multiple categories of first support images, each category of first support images contains at least one first support image, each first support image has a pre-marked foreign matter area and foreign matter type, and the foreign matter type of each first support image in each category of first support images is the same. It can be understood that a first support image in the preset first support image set can contain multiple pre-marked foreign matter areas and a unique foreign matter type corresponding to each foreign matter area, but in a detection process, a first support image in the preset first support image set can only select a foreign matter area in the image and the foreign matter type corresponding to the foreign matter area as the foreign matter area and foreign matter type of the first support image.

[0084] The aforementioned image that needs to be detected for foreign matter may be an image to be detected, and the image to be detected may refer to any acquired image that needs to be detected for foreign matter.

[0085] In some specific implementations, if foreign objects are usually attached to larger objects that are easier to detect, the image to be detected can be obtained by preprocessing the image to be detected. Specifically, the image to be detected can be subjected to foreign object background detection to separate the image to be detected containing the foreign object and the foreign object foreground of the foreign object from the image to be detected. For example, foreign objects on a transmission tower are usually attached to the tower. If foreign object detection of a transmission tower is to be performed, the image to be detected can first be subjected to tower area detection to separate the image of the tower area (i.e., the image to be detected), wherein the image of the tower area usually also contains foreign objects on the tower, so that the image to be detected containing the tower (foreign object foreground) and foreign objects on the tower (foreign objects) can be obtained. Because the shape of the tower is relatively fixed and the samples are sufficient, the detection of the tower area (foreground area) is relatively easy to achieve. Separating the foreign objects and the foreign object foreground from the image to be detected can effectively eliminate the influence of the complex background and solve the problem of small target foreign object detection.

[0086] 102. Perform feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the test features of the image to be detected and the support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of first support image.

[0087] A feature extraction network with pre-trained weight sharing is used for feature extraction to obtain the features to be detected of the image to be detected and the supporting feature set of the first supporting image set. Among them, the supporting feature set corresponding to the first supporting image set includes the supporting features corresponding to each type of the first supporting image. The feature extraction network with weight sharing is used to extract the common features of the first supporting image with the same type of foreign matter as the image to be detected, and to maintain the consistency of the extracted features (i.e., the features extracted from the foreign matter of the same type in the first supporting image and the image to be detected should be similar after the feature extraction network with weight sharing is used). At the same time, based on the supporting features different from the foreign matter type of the image to be detected in the first supporting image, it is convenient to distinguish the interference of the non-detection foreign matter type in the image to be detected (if the foreign matter type detected by the image to be detected is A, then the foreign matter types other than A in the image to be detected are all non-detection foreign matter types). At the same time, the parameter amount of the network is effectively reduced. In addition, the training method of the pre-trained feature extraction network please refer to the specific embodiment described later, which will not be repeated here.

[0088] It can be known that the feature extraction network in this step can be a Resnet50 network, a Resnet18 network, a Resnet101 network or a VGG series network, which is not limited here.

[0089] 103. Perform weighted fusion on the features to be tested and each type of supporting features to obtain corresponding N types of weighted features to be tested.

[0090] Furthermore, the features to be detected can be weighted based on the extracted multiple categories of supporting features, so that the features that may be foreign objects in the features to be detected in the image to be detected can be activated by supporting features that are the same type as the foreign objects in the image to be detected, and the features that may not be foreign objects in the features to be detected in the image to be detected can be suppressed by supporting features that are different from the type of foreign objects in the image to be detected.

[0091] Specifically, weighting can be achieved in the following ways: based on the coordinate information of the foreign matter area of ​​the supporting image corresponding to each type of supporting feature, determine the foreign matter area corresponding to each type of supporting feature; separate the features of the foreign matter area corresponding to each type of supporting feature to obtain the foreign matter features corresponding to each type of supporting feature; use each type of foreign matter feature to weight the features to be tested to obtain the corresponding N types of weighted features to be tested.

[0092] For details, please refer to Figure 2, according to the coordinate information of the foreign body area of ​​each first supporting image, the corresponding area in the supporting feature of the corresponding category is determined as the foreign body area. After the foreign body area corresponding to all the first supporting images in each type of supporting feature is determined, the foreign body area determination of each type of supporting feature is completed. The feature part corresponding to the foreign body area in each type of supporting feature is separated to obtain the foreign body feature corresponding to each type of supporting feature. Pooling processing (i.e., average pooling processing or region of interest pooling processing) is performed on each type of foreign body feature, and each type of foreign body feature after pooling processing is used to weight the feature to be tested respectively, and the foreign body feature after pooling processing is subjected to depth cross correlation processing (i.e., Depthwise Cross Correlation) with the feature to be tested to obtain the weighted feature to be tested (i.e., weighted feature to be tested). It can be known that each type of supporting feature can also be directly used to weight the feature to be tested to obtain the weighted feature to be tested. This embodiment does not specifically limit the way to weight the feature to be tested based on the extracted multiple types of supporting features.

[0093] 104. Input each type of weighted feature to be detected into the RPN network, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter.

[0094] Each type of weighted features to be tested in the embodiment described in step 103 is input into the region proposal network (RPN), and the RPN can determine at least one candidate region feature that may contain the corresponding type of foreign matter in each type of weighted features to be tested. In practical applications, the RPN network can determine hundreds of candidate region features from each type of weighted features to be tested, and some candidate region features can be selected from the multiple candidate region features determined by the RPN network from each type of weighted features to be tested as at least one candidate region feature that may contain foreign matter according to demand.

[0095] 105. Learn the matching relationship between at least one candidate region feature of each type of foreign object and the supporting features of the corresponding category.

[0096] The matching relationship between each candidate area feature in at least one candidate area feature of each category of weighted features to be tested that may contain the corresponding type of foreign matter and the supporting features of the corresponding category determined in the learning step 104, that is, the matching relationship between each candidate area feature and the supporting features of the corresponding category, (that is, determining that the type of foreign matter that may be contained in a candidate area feature in a matching relationship is the same as the type of foreign matter contained in the supporting image corresponding to the supporting feature in the matching relationship).

[0097] In some specific embodiments, this step can be achieved in the following ways: perform region of interest pooling processing on at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category respectively; learn the matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales, and obtain the matching relationship between each candidate region and the supporting features of the corresponding category by integrating the matching relationships at multiple pixel scales.

[0098] Specifically, after performing region of interest pooling processing on each candidate region feature and the supporting features of the corresponding category respectively (matching relationship learning can only be performed after the scales of the input candidate region features and the supporting features of the corresponding category are consistent), matching relationship learning at different pixel scales is performed for each candidate region feature and the supporting features of the corresponding category, and the matching relationship learning at different pixel scales of each candidate region feature and the supporting features of the corresponding category is integrated to obtain the matching relationship between the candidate region feature and the supporting features of the corresponding category.

[0099] In a specific implementation, the aforementioned learning of the matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales, and the step of obtaining the matching relationship between each candidate region and the supporting features of the corresponding category by integrating the matching relationships at multiple pixel scales, can be specifically implemented in the following manner: connecting each candidate region feature after the region of interest pooling processing with the supporting features of the corresponding category after the region of interest pooling processing in the channel dimension, performing a first convolution processing on the connected features to obtain a matching relationship at a first pixel scale; connecting each candidate region feature after the region of interest pooling processing with the supporting features of the corresponding category The supporting features after the pooling processing of the region of interest are connected in the channel dimension, the connected features are averaged pooled, and the features after the average pooling processing are subjected to a first convolution processing to obtain a matching relationship at a second pixel scale; the features of each candidate region after the pooling processing of the region of interest are subjected to a second convolution processing with the supporting features after the pooling processing of the region of interest of the corresponding category in the corresponding channel dimension, and the features after the second convolution processing are subjected to a first convolution processing to obtain a matching relationship at a third pixel scale; the matching relationship at the first pixel scale, the matching relationship at the second pixel scale, and the matching relationship at the third pixel scale are added to obtain a matching relationship between each candidate region and the supporting feature of the corresponding category.

[0100] Specifically, matching learning at different scales includes: local matching learning, global matching learning and pixel-level matching learning. Among them, local matching learning means: connecting each candidate region feature after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, and performing a first convolution process on the connected features to obtain a matching relationship on the first pixel scale; global matching learning means: connecting each candidate region feature after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, performing an average pooling process on the connected features, and performing a first convolution process on the features after the average pooling process to obtain a matching relationship on the second pixel scale; pixel-level matching learning means: performing a second convolution process on each candidate region feature after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the corresponding channel dimension, and performing a first convolution process on the features after the second convolution process to obtain a matching relationship on the third pixel scale. Finally, the matching relationship between each candidate region feature and the supporting feature of the corresponding category at three pixel scales is added (i.e., confidence is added) to obtain the confidence that the candidate region feature corresponds to the foreign object of the corresponding category appearing in the candidate region.

[0101] In practical applications, local matching learning can refer to: connecting each candidate region feature with the supporting feature of the corresponding category in the channel dimension, processing the connected features through a convolution kernel of 3*3, and then processing them through a fully connected convolution to obtain a matching relationship; global matching learning can refer to: connecting each candidate region feature with the supporting feature of the corresponding category in the channel dimension, performing average pooling on the connected features, and performing a fully connected convolution on the features after the average pooling to obtain a matching relationship; pixel-level matching learning can refer to: performing deep cross-correlation on each candidate region feature and the supporting feature of the corresponding category in the channel dimension, and then performing a fully connected convolution to obtain a matching relationship. Among them, the features obtained by global matching learning and pixel-level matching learning are only used to calculate the matching confidence, and the features obtained by local matching learning are used to calculate the candidate regions and matching confidence in the image to be detected. The final matching confidence score is obtained by weighted average of the matching confidences of the three scales. The final matching similarity score is used as the confidence that the corresponding category of foreign matter appears in the candidate region corresponding to the candidate region feature (that is, the foreign matter type of the supporting image corresponding to the supporting feature of the corresponding category for matching relationship learning with each candidate region feature).

[0102] It should be noted that one matching relationship corresponds to one candidate region feature and one type of supporting feature, and the candidate region features and supporting features corresponding to different matching relationships are not completely the same.

[0103] 106. Among the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, select a target matching relationship that meets a preset condition.

[0104] From the matching relationships between at least one candidate region feature of each type of foreign matter learned in step 105 and the supporting features of the corresponding category, a target matching relationship that meets preset conditions, that is, a matching relationship with the best matching degree among multiple matching relationships is selected as the target matching relationship.

[0105] In some specific implementations, the quality of the match can be determined by the match confidence in each matching relationship. The greater the match confidence, the better the match, and vice versa. Specifically, the matching relationship corresponding to the largest match confidence (i.e., the maximum match confidence) can be selected as the target matching relationship.

[0106] Furthermore, considering that the network will output a large number of matching relationships and the matching confidence of each matching relationship, the large number of matching relationships output by the network can be screened in advance, and the target matching relationship can be selected from the remaining matching relationships after screening to reduce the number of matching relationships that need to be traversed. Therefore, this step can be implemented in the following way: according to the candidate area corresponding to each preliminary screening matching relationship, the two preliminary screening matching relationships whose corresponding candidate areas in each preliminary screening matching relationship are greater than or equal to the preset intersection and union ratio are determined as a group of matching relationships to be selected; the preliminary screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected is deleted; and among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

[0107] Specifically, first, the matching relationships between each candidate region learned in step 105 and the supporting features of the corresponding category whose matching confidence is less than or equal to the preset matching confidence can be removed, that is, the matching relationships between each candidate region learned in step 105 and the supporting feature map of the corresponding category whose matching confidence is greater than the preset matching confidence are retained as the preliminary screening matching relationships. Further, the two preliminary screening matching relationships whose corresponding candidate regions in the preliminary screening matching relationships are greater than or equal to the preset intersection-and-union ratio are determined as a group of matching relationships to be selected, and the preliminary screening matching relationships with smaller values ​​of the corresponding matching confidence in each group of matching relationships to be selected are deleted, and finally the preliminary screening matching relationship with the largest value of the corresponding matching confidence is selected from the remaining preliminary screening matching relationships as the target matching relationship. Among them, the preset matching confidence can be set according to actual needs, and the degree of overlap between the two candidate regions can be determined by calculating any intersection-and-union ratio formula, which is not specifically limited in this embodiment.

[0108] In practical applications, a preset IoU ratio may be set in the initial screening of matching relationships, and matching relationships with smaller matching confidences among matching relationships with IoU ratios greater than the preset IoU ratio may be deleted in sequence through a non-maximum suppression (NMS) operation.

[0109] In another embodiment, the matching relationship with the highest confidence can be selected from the matching relationships corresponding to multiple candidate region features of each category as the preliminary screening matching relationship of the category, and then a preset intersection-and-union ratio is set for a preliminary screening matching relationship of each category, and through a non-maximum suppression (NMS) operation, the preliminary screening matching relationships with smaller matching confidences among the matching relationships with intersection-and-union ratios greater than the preset intersection-and-union ratio are deleted in turn, and finally, the matching relationship corresponding to the matching relationship with the highest matching confidence is selected from the remaining preliminary screening matching relationships as the target matching relationship.

[0110] It can be known that the screening criteria for the matching relationship in the embodiment of the present application include but are not limited to a preset matching confidence or an intersection-over-union ratio of corresponding candidate regions, which is not specifically limited here.

[0111] 107. Determine the type of foreign matter and the foreign matter region in the image to be detected according to the candidate region features corresponding to the target matching relationship and the foreign matter type corresponding to the candidate region features corresponding to the target matching relationship.

[0112] The area corresponding to the candidate area features corresponding to the target matching relationship in the image to be detected is the foreign body area in the image to be detected, and the foreign body type contained in the supporting image corresponding to the supporting features corresponding to the target matching relationship is the foreign body type of the foreign body in the foreign body area of ​​the image to be detected.

[0113] It should be noted that when the source of the image to be detected or the first supporting image is a user, the data related to the image to be detected or the first supporting image mentioned above and below in the embodiments of the present application are all obtained after the user's authorization. In addition, when the embodiments of the present application are applied to specific products or technologies, the data involved in the use need to obtain the user's permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0114] In an embodiment of the present application, a foreign body detection network and a foreign body detection method suitable for small sample learning are provided. The foreign body detection network input includes a first support image set and an image to be detected, and uses a weight-sharing feature extraction network to extract basic features from the two inputs and weight them, and then performs matching relationship learning to improve the detection performance of the foreign body detection network. Good foreign body detection performance can be achieved even when there are only dozens of support images of each type in the first support image set.

[0115] It can be understood that the convolution part of the RPN network, the feature extraction network and the convolution part of the matching learning used for detection in the aforementioned and subsequent embodiments are all pre-trained, and the specific training method is similar to the detection method, which will not be repeated here.

[0116] Specifically, each image can be selected from the preset training data set in turn as an image to be trained, and then multiple images other than the image to be trained can be selected from the preset training data set as a second support image set (which can be completely identical, partially identical, or completely different from the first support image set used in detection, which is not specifically limited here). Then, each image to be trained and the second support image set corresponding to the image to be trained are used to train the convolution part of the RPN network, the feature extraction network, and the convolution part of the matching learning used in the detection in the aforementioned and later embodiments.

[0117] Furthermore, the second support image set corresponding to each image to be trained can be selected from a preset training data set in the following manner.

[0118] Specifically, during training, the complete training data set and labels are read in to construct a data list. Each image in the data set and its corresponding label information are used as an item in the data list. Each item in the data list is classified according to the category of the target box in the label (the total number of categories is denoted as m, and the categories are denoted as A 1-m ) for splitting (if the label of the data contains three types of target frames, split the data into three data and update the corresponding labels so that each data label has A1, A2, and A3 target frames respectively), ensure that each data in the data list contains only one type of target frame in the corresponding label, and use this class as the category of the data; select one data from the data list in order as the image to be trained; select a total of non-repeated data from the data list , as the corresponding second support image set, where n represents the second support image of the target box containing n types of data labels in the second support image set, C i is the category of the second support image set, K is the number of second support image data of each category. The category of C1 data in the second support image set is the same as the image to be detected, C 2-n Class does not contain A 1-m Any of the categories.

[0119] In a specific implementation, the number of categories n of the second support image set is n=2 during training, that is, the category of C1 data is the same as the image to be trained, that is, C2 does not contain A 1-m Any one of the categories, and during reasoning, n is the total number of foreign body categories contained in the entire second data set image, so that all categories of foreign bodies can be detected during reasoning.

[0120] The foreign body detection method of the present application embodiment is described above. Figure 4 , the foreign object detection method and process of the embodiment of the present application are described below in the specific scenario of foreign object detection on a pole tower.

[0121] When using a foreign object detection network for detection, network training is required first to optimize the parameters of the foreign object detection network in order to achieve better foreign object detection performance.

[0122] Considering the small number of foreign body images (samples) of transmission lines, directly using foreign body images to train on a general pre-trained model will not only lead to slow convergence but also cause serious overfitting. However, first using a general data set (including but not limited to the coco data set or the pascal voc data set) to train the general pre-trained model, and then using foreign body images to train on the general pre-trained model can speed up parameter convergence, alleviate overfitting, and improve the generalization ability of the foreign body detection network for foreign body detection on poles and towers, thereby improving the detection performance of the foreign body detection network. The images in the data set are subjected to foreground (pole tower) detection before the data list is constructed, and foreign body detection is performed only on the foreground (pole tower) area in the image to reduce the interference of complex background and improve the detection accuracy.

[0123] Specifically, training is performed on a general data set with sufficient data volume to obtain a general pre-training model. During training, a certain class of objects in the Qurey image is marked as a class to be detected, and the remaining classes of objects are non-detected classes. The Support image supports image input of K categories, each category has N images, but the first category in the Support image should be the same as the class to be detected in the Query image, and the remaining K-1 categories are different from the class to be detected in the Query image. In the Qurey image, only the objects of the class to be detected are marked as foregrounds and need to be detected, and the non-detection objects are marked as backgrounds and do not need to be detected. During each training, after selecting the Qurey image from the general data set, multiple Support images are selected from the general data set according to the selected foreign objects of the class to be detected in the Qurey image. The training process is similar to the aforementioned embodiment and will not be repeated here. After each training network outputs the foreign body area and foreign body type, it is also necessary to calculate the loss according to the foreign body area and foreign body type corresponding to the Qurey image in the general data set to correct the foreign body detection network. Among them, the objects of the class to be detected in the Qurey image participate in the loss calculation, and the non-detection objects do not participate in the loss calculation.

[0124] Then, the general pre-trained model obtained by training is fine-tuned using the data input and training method described in 1 on the small sample data set of foreign objects on the transmission tower, thereby obtaining a foreign object detection network for the transmission tower. The obtained foreign object detection network for the transmission tower can detect foreign objects on the transmission tower corresponding to all foreign object types in the small sample data set of foreign objects on the transmission tower.

[0125] After the training is completed, foreign object detection on the tower can be realized based on the trained foreign object detection network.

[0126] For details, please refer to Figure 4 , and the following steps:

[0127] Since the original images (i.e., images to be detected) input to the network are mostly based on drone photography, the background is complex and the foreign body area accounts for a very small proportion of the entire image, which greatly increases the difficulty of detection. At the same time, considering that the shape of transmission towers is relatively fixed and the amount of data of transmission tower samples is sufficient, tower detection is relatively easy to implement. Therefore, the original image is first subjected to tower detection, and the tower detection result is used as the query image (i.e., target detection image) input to the network. After the tower detection step, the tower area is first separated from the original image, which effectively eliminates the influence of the complex background and solves the problem of small target detection.

[0128] Next, the Support image (sample image set) and the Query image are subjected to a weight-shared feature extraction network to extract features. An optional feature extraction network is the backbone network of Resnet50. The Support image contains both images of the to-be-detected class and images of the non-detected class. The use of a weight-shared feature extraction network is conducive to extracting the common features of the to-be-detected foreign objects in the Support image and the Query image, and is also conducive to distinguishing the to-be-detected foreign objects in the Support image and the non-detected foreign objects in the Query image, while effectively reducing the amount of network parameters.

[0129] Then, the features extracted from the Support image and the Query image are weighted. A feasible feature weighting method can be as follows: Figure 2As shown in the figure, according to the coordinate information of the foreign body area in the Support image label, the area corresponding to the support image feature (the second feature atlas) is separated for average pooling, and then Depthwise Cross Correlation is performed with the Query image feature (the first feature map), that is, the weighting of the Query image feature is completed. The Query image is an image after the pole tower area is detected, and the image contains the foreign body area and the entire pole tower area; therefore, the features of the Query image after being extracted by the feature extraction network include both the foreign body area features and the pole tower area background features. However, the Support image does not perform pole tower area detection, so the features of the foreign object area to be detected in the Support image can activate the foreign object area to be detected in the Query image, and the features of the non-detected foreign object area in the Support image can suppress the non-detected foreign object area in the Query image. However, the features other than the pole tower area and the foreign object area (including the non-detected foreign object area and the foreign object area to be detected) in the Support image will interfere with the activation or suppression. Therefore, according to the remaining coordinate information of the foreign object in the Support image, the area corresponding to the support image feature is separated and weighted with the Query image feature to activate the foreign object area to be detected in the Query image or suppress the non-detected foreign object area in the Query image. After the Support image feature is average pooled and dot-multiplied with the Query image feature, some channels in the Query image feature can be activated or suppressed, so that the subsequent positioning of the object to be detected is more accurate.

[0130] Finally, the weighted Query image features (i.e., weighted image features) are passed through the RPN network to extract the candidate regions, and then the regions of interest are pooled with the Support image features respectively. Subsequently, the results of the regions of interest pooling of the two are matched for relationship learning. A feasible matching relationship learning method can be as follows: Figure 3 As shown, matching learning is performed at three pixel scales, local, global and pixel, and the matching learning results of the three scales are added as the final output of the network. The matching relationship learning at different pixel scales is similar to the aforementioned embodiment and will not be repeated here. Matching relationship learning can fully compare and match the features of the candidate area of ​​the query image with the features of the support image, so as to detect foreign objects of the same category as the support image in the query image.

[0131] See also Figure 5 , the embodiment of the present application also provides a foreign body detection device, including:

[0132] A determination unit 501 is used to determine an image to be detected and a first supporting image set, wherein the first supporting image set includes N types of first supporting images, each type of first supporting images includes at least one first supporting image, each first supporting image has a pre-marked foreign body region and foreign body type, and each first supporting image in each type of first supporting images has the same foreign body type;

[0133] An extraction unit 502 is used to perform feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the features to be detected of the image to be detected and the support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of the first support image;

[0134] The weighting unit 503 is used to perform weighted fusion of the feature to be tested and each type of supporting feature to obtain corresponding N types of weighted features to be tested;

[0135] An input unit 504 is used to input each type of weighted features to be detected into the RPN network, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter;

[0136] A learning unit 505, configured to learn a matching relationship between at least one candidate region feature of each type of foreign matter and a supporting feature of a corresponding category;

[0137] A selection unit 506, configured to select a target matching relationship that satisfies a preset condition from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category;

[0138] The determination unit 501 is further used to determine the foreign body type and foreign body region in the image to be detected according to the candidate region features corresponding to the target matching relationship and the foreign body type corresponding to the candidate region features corresponding to the target matching relationship.

[0139] In a specific implementation, the learning unit 505 is specifically used to perform region of interest pooling processing on at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category;

[0140] The matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales is learned, and the matching relationship between each candidate region and the supporting features of the corresponding category is obtained by integrating the matching relationships at multiple pixel scales.

[0141] In a specific implementation, the learning unit 505 is specifically used to connect each candidate region feature after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, and perform a first convolution process on the connected features to obtain a matching relationship at a first pixel scale;

[0142] Connect each candidate region feature after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, perform average pooling process on the connected features, and perform a first convolution process on the features after the average pooling process to obtain a matching relationship on the second pixel scale;

[0143] Perform a second convolution process on each candidate region feature after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process in the corresponding channel dimension, and perform a first convolution process on the features after the second convolution process to obtain a matching relationship on a third pixel scale;

[0144] The matching relationship at the first pixel scale, the matching relationship at the second pixel scale, and the matching relationship at the third pixel scale are added together to obtain the matching relationship between each candidate region and the supporting features of the corresponding category.

[0145] In a specific implementation, the device further includes: a training unit;

[0146] The selection unit 506 is further used to select each image from the preset training data set in turn as an image to be trained;

[0147] The selection unit 506 is further configured to select a plurality of images other than the image to be trained from the preset training data set as a second support image set;

[0148] The training unit is used to train the weight-shared initial feature extraction network based on each image to be trained and the corresponding second support image set to obtain the weight-shared feature extraction network.

[0149] In a specific implementation, the weighting unit 503 is specifically configured to determine the foreign matter region corresponding to each type of support feature based on the coordinate information of the foreign matter region of the support image corresponding to each type of support feature;

[0150] Separate the features of the foreign body region corresponding to each type of supporting features to obtain the foreign body features corresponding to each type of supporting features;

[0151] Each type of foreign body feature is used to weight the feature to be tested to obtain the corresponding N types of weighted features to be tested.

[0152] In a specific implementation, the determination unit 501 is specifically configured to perform foreign object foreground detection on the initial image, so as to separate the image to be detected containing the foreign object and the foreign object foreground corresponding to the foreign object from the initial image.

[0153] In a specific implementation, the matching relationship includes a matching confidence;

[0154] The selection unit 506 is specifically configured to select, from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a matching relationship corresponding to a matching confidence greater than a preset matching confidence as a preliminary screening matching relationship;

[0155] According to the candidate area corresponding to each preliminary screening matching relationship, two preliminary screening matching relationships whose corresponding candidate areas in each preliminary screening matching relationship are greater than or equal to a preset intersection-and-union ratio are determined as a group of matching relationships to be selected;

[0156] Delete the initial screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected;

[0157] Among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

[0158] Figure 6 is a schematic diagram of the structure of a foreign body detection device provided in an embodiment of the present application. The foreign body detection device 600 may include one or more central processing units (CPU) 601 and a memory 605. The memory 605 stores one or more application programs or data.

[0159] The memory 605 may be a volatile storage or a persistent storage. The program stored in the memory 605 may include one or more modules, each of which may include a series of instruction operations in the foreign object detection device. Furthermore, the central processor 601 may be configured to communicate with the memory 605 and execute a series of instruction operations in the memory 605 on the foreign object detection device 600.

[0160] The foreign object detection device 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input and output interfaces 604, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0161] The CPU 601 can execute the aforementioned Figures 1 to 5 The operations performed by the foreign body detection device in the illustrated embodiment will not be described in detail here.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0167] An embodiment of the present application also provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the foreign body detection method based on small sample learning as described above.

Claims

1. A foreign body detection method based on small sample learning, It is characterized in that include: Determine an image to be detected and a first supporting image set, wherein the first supporting image set includes N types of first supporting images, each type of the first supporting images includes at least one first supporting image, each of the first supporting images has a pre-marked foreign body area and foreign body type, and each first supporting image in each type of the first supporting images has the same foreign body type; Performing feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the features to be detected of the image to be detected and a support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of first support image; Perform weighted fusion of the features to be tested and each type of supporting features to obtain corresponding N types of weighted features to be tested; Inputting each type of weighted features to be detected into the RPN network respectively, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter; Learning the matching relationship between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category; Among the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a target matching relationship that meets a preset condition is selected; Determine the type of foreign matter and the foreign matter region in the image to be detected according to the candidate region features corresponding to the target matching relationship and the foreign matter type corresponding to the candidate region features corresponding to the target matching relationship; The matching relationship includes matching confidence; The selecting a target matching relationship that satisfies a preset condition from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category comprises: Among the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a matching relationship corresponding to a matching confidence greater than a preset matching confidence is selected as a preliminary screening matching relationship; According to the candidate area corresponding to each of the preliminary screening matching relationships, two preliminary screening matching relationships whose corresponding candidate areas in each of the preliminary screening matching relationships are greater than or equal to a preset intersection-and-union ratio are determined as a group of matching relationships to be selected; Deleting the initial screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected; Among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

2. The method according to claim 1, It is characterized in that The learning of the matching relationship between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category includes: Performing region of interest pooling processing on at least one candidate region feature of each type of foreign body and the supporting features of the corresponding category; The matching relationship between each candidate region feature after the region of interest pooling processing and the supporting features of the corresponding category after the region of interest pooling processing at multiple pixel scales is learned, and the matching relationship between each candidate region and the supporting features of the corresponding category is obtained by integrating the matching relationships at multiple pixel scales.

3. The method according to claim 2, It is characterized in that The learning of the matching relationship between each candidate region feature after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process at multiple pixel scales, and combining the matching relationships at multiple pixel scales to obtain the matching relationship between each candidate region and the supporting features of the corresponding category, includes: Connecting each of the candidate region features after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, and performing a first convolution process on the connected features to obtain a matching relationship at a first pixel scale; Connecting each of the candidate region features after the region of interest pooling process with the supporting features of the corresponding category after the region of interest pooling process in the channel dimension, performing average pooling process on the connected features, and performing the first convolution process on the features after the average pooling process to obtain a matching relationship on a second pixel scale; Performing a second convolution process on each of the candidate region features after the region of interest pooling process and the supporting features of the corresponding category after the region of interest pooling process in the corresponding channel dimension, and performing the first convolution process on the features after the second convolution process to obtain a matching relationship on a third pixel scale; The matching relationship at the first pixel scale, the matching relationship at the second pixel scale, and the matching relationship at the third pixel scale are added together to obtain a matching relationship between each candidate region and a supporting feature of a corresponding category.

4. The method according to claim 1, It is characterized in that The method further comprises: Select each image from the preset training data set in turn as an image to be trained; Selecting a plurality of images other than the to-be-trained images from a preset training data set as a second support image set; Based on each image to be trained, the weight-shared initial feature extraction network is trained with each image to be trained and the corresponding second support image set to obtain the weight-shared feature extraction network.

5. The method according to claim 1, It is characterized in that The step of performing weighted fusion of the feature to be tested and each type of the supporting features to obtain corresponding N types of weighted features to be tested includes: Determine the foreign matter region corresponding to each type of the supporting features based on the coordinate information of the foreign matter region of the supporting image corresponding to each type of the supporting features; Separating the features of the foreign matter region corresponding to each type of the supporting features to obtain the foreign matter features corresponding to each type of the supporting features; Each type of the foreign body feature is used to weight the feature to be measured to obtain corresponding N types of weighted features to be measured.

6. The method according to claim 1, It is characterized in that The step of determining the image to be detected includes: The foreign object foreground detection is performed on the initial image to separate the image to be detected containing the foreign object and the foreign object foreground corresponding to the foreign object from the initial image.

7. A foreign body detection device, It is characterized in that include: a determination unit, configured to determine an image to be detected and a first supporting image set, wherein the first supporting image set includes N types of first supporting images, each type of the first supporting images includes at least one first supporting image, each of the first supporting images has a pre-marked foreign body region and foreign body type, and each first supporting image in each type of the first supporting images has the same foreign body type; an extraction unit, configured to perform feature extraction on the image to be detected and the first support image set based on a pre-trained weight-sharing feature extraction network to obtain the features to be detected of the image to be detected and a support feature set of the first support image set, wherein the support feature set includes support features corresponding to each type of first support image; A weighting unit, used for weightedly fusing the feature to be tested with each type of supporting features to obtain corresponding N types of weighted features to be tested; An input unit, used to input each type of the weighted features to be detected into the RPN network, so that the RPN network determines at least one candidate region feature of each type of foreign matter in the image to be detected that may contain N types of foreign matter; A learning unit, used for learning the matching relationship between at least one candidate region feature of each type of foreign matter and a supporting feature of a corresponding category; A selection unit, configured to select a target matching relationship that satisfies a preset condition from among the matching relationships between at least one candidate region feature of each type of foreign matter and a supporting feature of a corresponding category; The determination unit is further configured to determine the type of foreign matter and the foreign matter region in the image to be detected according to the candidate region features corresponding to the target matching relationship and the foreign matter type corresponding to the candidate region features corresponding to the target matching relationship; The matching relationship includes matching confidence; The selection unit is specifically configured to select, from the matching relationships between at least one candidate region feature of each type of foreign matter and the supporting features of the corresponding category, a matching relationship corresponding to a matching confidence greater than a preset matching confidence as a preliminary screening matching relationship; According to the candidate area corresponding to each of the preliminary screening matching relationships, two preliminary screening matching relationships whose corresponding candidate areas in each of the preliminary screening matching relationships are greater than or equal to a preset intersection-and-union ratio are determined as a group of matching relationships to be selected; and the preliminary screening matching relationship corresponding to the smaller matching confidence in each group of matching relationships to be selected is deleted; Among the remaining preliminary screening matching relationships, the preliminary screening matching relationship corresponding to the maximum matching confidence is selected as the target matching relationship.

8. A foreign body detection device, It is characterized in that include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 6.

9. A computer storage medium, It is characterized in that The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.