Small sample target detection method and system

Abstract

The invention discloses a small sample target detection method and system, and belongs to the technical field of computer vision, and the method comprises the steps: determining the information representation associated with a common and easy-to-recognize target and a special and difficult-to-recognize target in a support image based on the second image features of a plurality of support images which belong to the same category as a query image, according to the method, the distribution condition of intra-class samples is considered, the graph structure relation is established based on the information representation and the first image feature, and the aggregation enhancement feature of the query image and the information representation is determined, so that limited image information can be effectively utilized, the feature representation of positive and negative prototypes is improved, and the dependence on a large number of samples is reduced; and the to-be-detected target on the new category query image is determined by using the aggregation enhancement feature, so that the recognition accuracy is improved, and the problem of feature deviation existing in a current small sample target detection prototype is solved.

Description

Technical Field

[0001] The invention relates to a small sample target detection method and system, and belongs to the technical field of computer vision. Background Art

[0002] Model training through large-scale annotated data has achieved remarkable results. However, in practical work, we often face the problem of lack of large amounts of annotated data; most detectors have difficulty generalizing to new classes with only a small number of images.

[0003] FSOD (Few-Shot Object Detection) includes methods based on data enhancement, transfer learning, and meta-learning. The meta-learning method enables the model to learn generalized patterns and strategies by training on multiple tasks. However, the prototypes generated by the meta-learning method have feature bias and do not consider the distribution of samples within the class. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a small sample target detection method and system that can effectively utilize limited image information, take into account the distribution of samples within the class, and solve the problem of feature deviation in the current small sample target detection prototype.

[0005] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions: On one hand, the present invention provides a small sample target detection method, comprising: Obtaining a first image feature of a query image of a new category and second image features of a plurality of support images belonging to the same category as the query image; Based on the second image feature, determining a representation of information associated with a common, easily identifiable target and a special, less identifiable target in the supporting image; Establishing a graph structure relationship based on the information representation and the first image feature, and determining an aggregated enhanced feature of the query image and the information representation; According to the aggregated enhanced features, the target to be detected on the query image of the new category is determined.

[0006] Furthermore, the obtaining of the first image feature of the new category query image includes: using an attention region proposal network to generate a candidate box on the image based on the category of the query image, and using the region of interest to extract image features of the candidate box area to obtain the first image feature.

[0007] Furthermore, the step of obtaining second image features of a plurality of supporting images belonging to the same category as the query image includes: The image features of the supporting image are extracted using the region of interest to obtain the second image features.

[0008] Further, the determining, based on the second image feature, information representation associated with common, easily identifiable targets and special, difficult to identifiable targets in the supporting image includes: Based on the second image feature, respectively calculating the positive prototype representation associated with the common and easily identifiable target image in the support image and the negative prototype representation associated with the special and difficult to identify target image in the support image; Positive prototype representation and anti-prototype representation are used as representations of information associated with common, easily recognizable objects and special, less recognizable objects in supporting images.

[0009] Furthermore, the step of respectively calculating the positive prototype representation associated with the common and easily identifiable target image in the support image and the negative prototype representation associated with the special and difficult to identifiable target image in the support image based on the second image feature includes: Based on all the second image features, calculating the average support feature of the new category query image; Calculate the regularized cosine similarity and the regularized inverse cosine similarity between each second image feature and the average support feature; Calculate a positive prototype representation of the support image based on each second image feature and the regularized cosine similarity corresponding to each second image feature; Based on each second image feature and the regularized inverse cosine similarity corresponding to each second image feature, an inverse prototype representation of the support image is calculated.

[0010] Furthermore, the step of calculating the regularized cosine similarity and the regularized inverse cosine similarity between each second image feature and the average support feature includes: The cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: Indicates i The first query image of a new category k The cosine similarity between the second image feature of the support images and the average support feature; Indicates i The first query image of a new category k second image features of the supporting images; Indicates i The average support features of query images of new categories; Indicates calculation of absolute value; The regularized cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: express The normalized value; The exponential function representing the base e of the natural logarithm; Indicates i The first query image of a new category j The cosine similarity between the second image feature of the support images and the average support feature; K Indicates the number of supported images; The inverse cosine similarity of each second image feature with the average support feature is calculated by the following formula: ; in: Indicates i The first query image of a new category k The inverse cosine similarity between the second image feature of the support images and the average support feature; The regularized inverse cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: express The normalized value; Indicates i The first query image of a new category j The inverse cosine similarity between the second image feature of the support images and the average support feature.

[0011] Furthermore, the calculating of the positive prototype representation of the support image based on each second image feature and the regularized cosine similarity corresponding to each second image feature includes: The positive prototype representation of the support image is calculated by the following formula: ; in: Indicates i The positive prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; K Indicates the number of supported images; Indicates k The regularized cosine similarity corresponding to the second image features of the support images.

[0012] Furthermore, the calculating of the inverse prototype representation of the support image based on each second image feature and the regularized inverse cosine similarity corresponding to each second image feature includes: The inverse prototype representation of the support image is calculated by the following formula: ; in: Indicates iThe anti-prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; K Indicates the number of supported images; Indicates k The regularized inverse cosine similarity corresponding to the second image features of the support images.

[0013] Furthermore, the establishing of a graph structure relationship based on the information representation and the first image feature to determine the aggregated enhanced features of the query image and the information representation includes: Get the base class prototype of existing known category images; Construct a class-prototype-sample hierarchical graph convolutional network; In the class layer of the hierarchical graph convolutional network, a class prototype relationship matrix is ​​constructed according to the cosine similarity between the base class prototype and the positive prototype representation and the anti-prototype representation of the support image; Based on the class prototype relationship matrix, the positive prototype representation and the counter-prototype representation of the support image are updated to obtain an enhanced positive prototype representation and an enhanced counter-prototype representation; In the sample layer of the hierarchical graph convolutional network, the second image feature of each support image is split into a plurality of first local feature blocks, and the base class prototype of the existing known category image is split into a plurality of second local feature blocks; Establishing a connection between the first local feature block and the second local feature block to construct a feature relationship matrix; Update the first local feature block based on the feature relationship matrix to obtain a local enhanced feature; Aggregate the local enhanced features with the enhanced positive prototype representation and the enhanced anti-prototype representation at the prototype layer of the hierarchical graph convolutional network to obtain an updated positive prototype representation and an updated anti-prototype representation; A graph structure relationship is established between the updated positive prototype representation and the updated negative prototype representation and the first image feature to obtain an aggregated enhanced feature.

[0014] Another aspect of the present invention provides a small sample target detection system, comprising: A feature extraction module, used to obtain a first image feature of a query image of a new category and second image features of a plurality of support images belonging to the same category as the query image; a processing module for determining, based on the second image feature, information representation associated with common, easily identifiable objects and special, less identifiable objects in the supporting image; an aggregation module, configured to establish a graph structure relationship based on the information representation and the first image feature, and determine an aggregation enhancement feature of the query image and the information representation; The detection module is used to determine the target to be detected on the query image of the new category based on the aggregated enhanced features.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention determines the information representation associated with common and easily identifiable targets and special and difficult-to-identify targets in the supporting images based on the second image features of multiple supporting images that belong to the same category as the query image, takes into account the distribution of samples within the class, and establishes a graph structure relationship based on the information representation and the first image features, determines the aggregated enhanced features of the query image and the information representation, can effectively utilize limited image information, improves the feature representation of positive and negative prototypes, reduces dependence on a large number of samples, and utilizes the aggregated enhanced features to determine the target to be tested on the query image of the new category, thereby improving the recognition accuracy and solving the problem of feature deviation in the current small sample target detection prototype. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a small sample target detection method provided by an embodiment of the present invention; Figure 2 It is a principle framework diagram of a small sample target detection method provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a positive prototype representation and a negative prototype representation of a computing support image provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of disassembling the first local feature block and the second local feature block provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] In the description of the present invention, it is to be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more. Example

[0019] like Figure 1 and Figure 2 As shown, a small sample target detection method includes: The first image features of a query image of a new category and the second image features of multiple support images belonging to the same category as the query image are obtained; wherein, an attention region proposal network (Attention-RPN) is used to generate a candidate box on the image based on the category of the query image, and the region of interest (ROI) and the backbone network (Backbone) are used to extract the Proposal image features of the candidate box area to obtain the first image features; the ROI image features of the support image are extracted using the region of interest to obtain the second image features.

[0020] Based on the second image feature, information representation associated with common, easily identifiable targets and special, difficult to identify targets in the support image is determined, specifically: Based on the second image feature, respectively calculating the positive prototype representation associated with the common and easily identifiable target image in the support image and the negative prototype representation associated with the special and difficult to identify target image in the support image, including: like Figure 3 As shown, based on all second image features, the average support feature of the new category query image is calculated; Calculate the regularized cosine similarity and regularized inverse cosine similarity of each second image feature with the average support feature: The cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: Indicates i The first query image of a new category k The cosine similarity between the second image feature of the support images and the average support feature; Indicates i The first query image of a new category k second image features of the supporting images; Indicates i The average support features of query images of new categories; Indicates calculation of absolute value; The regularized cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: express The normalized value; The exponential function representing the base e of the natural logarithm; Indicates i The first query image of a new category j The cosine similarity between the second image feature of the support images and the average support feature; K Indicates the number of supported images; The inverse cosine similarity of each second image feature with the average support feature is calculated by the following formula: ; in: Indicates i The first query image of a new category k The inverse cosine similarity between the second image feature of the support images and the average support feature; The regularized inverse cosine similarity between each second image feature and the average support feature is calculated by the following formula: ; in: express The normalized value; Indicates i The first query image of a new category j The inverse cosine similarity between the second image feature of the support images and the average support feature; Based on each second image feature and the regularized cosine similarity corresponding to each second image feature, the positive prototype representation of the support image is calculated: The positive prototype representation of the support image is calculated by the following formula: ; in: Indicates i The positive prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; Indicates k The regularized cosine similarity corresponding to the second image features of the support images; Based on each second image feature and the regularized inverse cosine similarity corresponding to each second image feature, the inverse prototype representation of the support image is calculated: The inverse prototype representation of the support image is calculated by the following formula: ; in: Indicates i The anti-prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; Indicates k The regularized inverse cosine similarity corresponding to the second image features of the support images; Positive prototype representation and anti-prototype representation are used as representations of information associated with common, easily recognizable objects and special, less recognizable objects in supporting images.

[0021] A graph structure relationship is established based on the information representation and the first image feature, and an aggregated enhanced feature of the query image and the information representation is determined. Specifically: Get the base class prototype of existing known category images; Construct a class-prototype-sample hierarchical graph convolutional network; In the class layer of the hierarchical graph convolutional network, the class prototype relationship matrix is ​​constructed according to the cosine similarity between the base class prototype and the positive prototype representation and the anti-prototype representation of the support image. , express A two-dimensional matrix, C Indicates the number of image classes; Based on the class prototype relationship matrix, the positive prototype representation and the counter-prototype representation of the support image are updated to obtain an enhanced positive prototype representation and an enhanced counter-prototype representation; The enhanced positive prototype representation is calculated by the following formula: ; The enhanced anti-prototype representation is calculated by the following formula: ; in: is the class prototype relationship matrix The elements in represent the i New categories and j cosine similarity values ​​of other categories; like Figure 4 As shown in the figure, in the sample layer of the hierarchical graph convolutional network, the second image feature of each supporting image is split into multiple first local feature blocks, and the base class prototype of the existing known category image is split into multiple second local feature blocks. Specifically: each second image feature is used as an independent node, and the second image feature is further split into M Blocks, namely the second image features = , Indicates k The second image feature of the support image M The splitting form of the second local feature block is the same as that of the first local feature block, so it will not be repeated; The first local feature block is connected with the second local feature block and divided into M Group, Build * The characteristic relationship matrix of Indicates the number of base classes, K Represents the number of supporting images, updates the first local feature block based on the feature relationship matrix, and obtains local enhanced features, including: ; in: Indicates m Group k The first local feature block and the j The cosine similarity value of other local feature blocks; Indicates m Group k The first local feature block corresponds to the i Local enhancement features for new categories; Indicates m Group k The first local feature block corresponds to the i The original features of the new category; Indicates m Group j Other local feature blocks, m Other local feature blocks in the group include i A new class K The first local feature block and A second local feature block.

[0022] In the prototype layer of the hierarchical graph convolutional network, the local enhanced features are aggregated with the enhanced positive prototype representation and the enhanced anti-prototype representation to obtain an updated positive prototype representation and an updated anti-prototype representation; Establishing a graph structure relationship between the updated positive prototype representation and the updated anti-prototype representation and the first image feature to obtain an aggregated enhanced feature; It should be noted that Figure 4 1 represents the feature map constructed with the first local feature block. M Indicates the M A feature map constructed by the first local feature blocks; The aggregated features are input into the pairwise classifier to obtain the final detection results.

[0023] Example 2 A small sample target detection system, comprising: A feature extraction module, used to obtain a first image feature of a query image of a new category and second image features of a plurality of support images belonging to the same category as the query image; a processing module for determining, based on the second image feature, information representation associated with common, easily identifiable objects and special, less identifiable objects in the supporting image; an aggregation module, configured to establish a graph structure relationship based on the information representation and the first image feature, and determine an aggregation enhancement feature of the query image and the information representation; The detection module is used to determine the target to be detected on the query image of the new category based on the aggregated enhanced features.

[0024] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A small sample target detection method, characterized in that: include: Obtaining a first image feature of a query image of a new category and second image features of a plurality of support images belonging to the same category as the query image; Based on the second image feature, determining a representation of information associated with a common, easily identifiable target and a special, less identifiable target in the supporting image; Establishing a graph structure relationship based on the information representation and the first image feature, and determining an aggregated enhanced feature of the query image and the information representation; According to the aggregated enhanced features, the target to be detected on the query image of the new category is determined.

2. The small sample target detection method according to claim 1, characterized in that: The method of obtaining the first image feature of the query image of the new category includes: using an attention region proposal network to generate a candidate frame on the image based on the category of the query image, and using the region of interest to extract image features of the candidate frame area to obtain the first image feature.

3. The small sample target detection method according to claim 1, characterized in that: The step of obtaining second image features of a plurality of supporting images belonging to the same category as the query image comprises: The image features of the supporting image are extracted using the region of interest to obtain the second image features.

4. The small sample target detection method according to claim 1, characterized in that: The step of determining, based on the second image feature, information representation associated with common, easily identifiable targets and special, difficult to identifiable targets in the supporting image includes: Based on the second image feature, respectively calculating the positive prototype representation associated with the common and easily identifiable target image in the support image and the negative prototype representation associated with the special and difficult to identify target image in the support image; Positive prototype representation and anti-prototype representation are used as representations of information associated with common, easily recognizable objects and special, less recognizable objects in supporting images.

5. The small sample target detection method according to claim 4, characterized in that: The method of respectively calculating the positive prototype representation associated with the common and easily identifiable target image in the support image and the negative prototype representation associated with the special and difficult to identifiable target image in the support image based on the second image feature comprises: Based on all the second image features, calculating the average support feature of the new category query image; Calculate the regularized cosine similarity and the regularized inverse cosine similarity between each second image feature and the average support feature; Calculate a positive prototype representation of the support image based on each second image feature and the regularized cosine similarity corresponding to each second image feature; Based on each second image feature and the regularized inverse cosine similarity corresponding to each second image feature, an inverse prototype representation of the support image is calculated.

6. The small sample target detection method according to claim 5, characterized in that: The calculating of the regularized cosine similarity and the regularized inverse cosine similarity between each second image feature and the average support feature comprises: The cosine similarity between each second image feature and the average support feature is calculated by the following formula: ； in: Indicates i The first query image of a new category k The cosine similarity between the second image feature of the support images and the average support feature; Indicates i The first query image of a new category k second image features of the supporting images; Indicates i The average support features of query images of new categories; Indicates calculation of absolute value; The regularized cosine similarity between each second image feature and the average support feature is calculated by the following formula: ； in: express The normalized value; The exponential function representing the base e of the natural logarithm; Indicates i The first query image of a new category j The cosine similarity between the second image feature of the support images and the average support feature; K represents the number of support images; The inverse cosine similarity of each second image feature with the average support feature is calculated by the following formula: ； in: Indicates i The first query image of a new category k The inverse cosine similarity between the second image feature of the support images and the average support feature; The regularized inverse cosine similarity between each second image feature and the average support feature is calculated by the following formula: ； in: express The normalized value; Indicates i The first query image of a new category j The inverse cosine similarity between the second image feature of the support images and the average support feature.

7. The small sample target detection method according to claim 5, characterized in that: The calculating the positive prototype representation of the support image based on each second image feature and the regularized cosine similarity corresponding to each second image feature includes: The positive prototype representation of the support image is calculated by the following formula: ； in: Indicates i The positive prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; K Indicates the number of supported images; Indicates k The regularized cosine similarity corresponding to the second image features of the support images.

8. The small sample target detection method according to claim 5, characterized in that: The step of calculating the inverse prototype representation of the support image based on each second image feature and the regularized inverse cosine similarity corresponding to each second image feature includes: The inverse prototype representation of the support image is calculated by the following formula: ； in: Indicates i The anti-prototype representation of all support images corresponding to the query image of a new category; Indicates i The first query image of a new category k second image features of the supporting images; K Indicates the number of supported images; Indicates k The regularized inverse cosine similarity corresponding to the second image features of the support images.

9. The small sample target detection method according to claim 4, characterized in that: The step of establishing a graph structure relationship based on the information representation and the first image feature and determining the aggregated enhanced features of the query image and the information representation includes: Get the base class prototype of existing known category images; Construct a class-prototype-sample hierarchical graph convolutional network; In the class layer of the hierarchical graph convolutional network, a class prototype relationship matrix is ​​constructed according to the cosine similarity between the base class prototype and the positive prototype representation and the anti-prototype representation of the support image; Based on the class prototype relationship matrix, the positive prototype representation and the counter-prototype representation of the support image are updated to obtain an enhanced positive prototype representation and an enhanced counter-prototype representation; In the sample layer of the hierarchical graph convolutional network, the second image feature of each support image is split into a plurality of first local feature blocks, and the base class prototype of the existing known category image is split into a plurality of second local feature blocks; Establishing a connection between the first local feature block and the second local feature block to construct a feature relationship matrix; Update the first local feature block based on the feature relationship matrix to obtain a local enhanced feature; In the prototype layer of the hierarchical graph convolutional network, the local enhanced features are aggregated with the enhanced positive prototype representation and the enhanced anti-prototype representation to obtain an updated positive prototype representation and an updated anti-prototype representation; A graph structure relationship is established between the updated positive prototype representation and the updated negative prototype representation and the first image feature to obtain an aggregated enhanced feature.

10. A small sample target detection system, characterized in that: include: A feature extraction module, used to obtain a first image feature of a query image of a new category and second image features of a plurality of support images belonging to the same category as the query image; a processing module for determining, based on the second image feature, information representation associated with common, easily identifiable objects and special, less identifiable objects in the supporting image; an aggregation module, configured to establish a graph structure relationship based on the information representation and the first image feature, and determine an aggregation enhancement feature of the query image and the information representation; The detection module is used to determine the target to be detected on the query image of the new category based on the aggregated enhanced features.

Images