A visual-based object usability few-shot classification method

Abstract

The application discloses a visual-based object usability few-sample classification method, belongs to the field of computer vision and artificial intelligence, and comprises the following steps: constructing a data set for object usability classification; preprocessing the data set to obtain a preprocessed data set; dividing the preprocessed data set into a training set, a verification set and a test set which are mutually exclusive in category, and respectively inputting the training set and the test set; constructing an usability few-sample classification model; inputting the training set data into the usability few-sample classification model to train the usability few-sample classification model and obtain a trained usability few-sample classification model; and inputting the test set data into the trained usability few-sample classification model to obtain the classification of the object usability few-sample; and the output result of the method is the predicted category of an unlabeled query image, and the method reduces the overfitting phenomenon caused by the unbalanced category distribution and limited samples.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision and artificial intelligence, and relates to a visual-based object usability few-shot classification method. BACKGROUND

[0002] At present, there are many studies on function usability in the field of computer vision, which are mainly divided into two categories: pixel-level usability detection and object-level usability classification. The former focuses on predicting the label of image pixels and is mainly based on image semantic segmentation models; the latter focuses on predicting the function usability label of objects, and the present application belongs to the latter.

[0003] In the early stage, Farhadi A et al. [1] proposed to describe objects by attributes, T. Hermans et al. [2] considered that physical attributes and visual attributes provide a more suitable intermediate representation, and the function usability is more likely to be related to the attributes of objects. Y. Sun et al. [3] proposed a new method to enable intelligent robots to learn the interactive functions of objects from human demonstrations. Y. Zhu et al. [4] used a knowledge base representation to reason about the usability of objects, used three attributes (visual, object, and category) and used them as an intermediate representation, and performed knowledge transfer between objects to predict the function usability of objects that have not been seen, and the disadvantage is that its generalization performance is not high. Varadarajan and Vincze et al. [5] proposed a hybrid parallel architecture of deep learning and priming and activation (PDLSA) to overcome the problems of deep learning and single modality and serialization, and classified objects according to the features of function usability. Given the basic function usability, Ugur et al. [6] proposed a bootstrapping method to learn complex function usability through relational usability or so-called paired object usability. They used a real robot to evaluate objects of various shapes, such as boxes, spheres and cylinders. In addition, they also trained the robot to perform actions such as side-poking and stacking. M. Hjelm et al. [7] defined this as a simple classification problem to predict a function usability category for a whole picture. The work of Zheng, X et al. [8] proposed a high-level object usability detection method based on deep convolutional neural networks, which detects objects and then classifies them into a usability category. Deep learning has developed rapidly, but few of the above methods for object usability classification use feature extraction, because deep learning algorithms rely on a large amount of labeled data, and there is currently no large labeled dataset specifically for object function usability.

[0004] Few-shot classification model can classify new image categories with only a small amount of labeled training data, solving the overfitting phenomenon caused by too little training data. Existing few-shot classification methods have two main branches: optimization-based methods and metric-based methods. The goal of optimization-based methods is to effectively adapt model parameters to new tasks in a few-shot state; the method of the present application is more relevant to metric-based methods, which represent samples in a feature space and distinguish different categories of data using distance metrics. Metric learning-based methods mainly rely on learning a similarity metric, and representative works include Siamese networks [9] , matching networks

[10] , prototype networks

[11] , and relation networks

[12] . Specifically, to introduce metric-based methods into few-shot learning, G Koch et al. [9] initially used Siamese neural networks to learn powerful discriminative representations, and then extended them to unseen classes. This is the first method to introduce a distance metric approach to solve the few-shot classification problem, but it has great limitations, as it is very sensitive to position differences and color differences between two images. To solve this problem, a deeper convolutional neural network is used to make the extracted features more abstract. To this end, O. Vinyals et al.

[10] introduced an episode training mechanism into few-shot learning and proposed a matching network that combines attention and memory, which uses a deep neural network with attention and memory modules to map the feature information of the samples to a higher-dimensional, more abstract feature space. This method has good classification and strong migration ability, but due to the limitation of non-parametric algorithms, the computational load of each iteration will also increase rapidly as the support set increases, resulting in a decrease in computing speed. On this basis, J. Snell et al.

[11] proposed a prototype network that uses the mean of each class as its corresponding prototype representation to learn the metric space. It uses squared Euclidean distance as the distance metric, replacing the commonly used cosine function, making the prototype network perform well on multiple datasets. However, its neural network structure is too simple and cannot guarantee that the extracted feature information is abstract enough. Considering the relationship between the query image and the class image, and unlike the artificially set metric approach, Sung et al. proposed a relation network

[12] that completely learns the metric through a neural network and uses a meta-learning training approach, achieving good results on multiple datasets. Li et al.

[13] proposed a deep nearest neighbor network, which is different from other metric learning-based approaches by comparing the local descriptors between images and classes to find the closest class to the input image. Hou et al.

[14] It is found that the attention between the support image and the query image is often inconsistent, so a cross-attention module is used to alleviate this problem, and a model named cross-attention network is proposed. In addition, Hui

[15] A self-attention relationship network (SARN) is proposed, which can discover non-local information and allow remote dependencies. Unlike previous methods that map the entire image to a feature space, Zhang

[16] DeepEMD proposed by et al. splits the image into multiple patches and introduces a new metric called land moving distance to represent the similarity between the support set image and the query set image. In recent years, many models and methods have been proposed to solve the few-shot learning problem, such as Zhao

[17] A new domain adversarial prototypical network (DAPN) model is proposed, Hong

[18] An attention agent is equipped in the backbone network, and the attention agent is trained through reinforcement learning.

[0005] The relationship network is a typical few-shot classification model, but its classification effect on object availability images is limited. The reasons are twofold. First, the feature extractor of the relationship network may lose some detailed information we need. Second, the relationship network compares the entire input image, which is not very effective for object availability classification. We need the network to pay more attention to the areas with similar features in the class.

[0006] Common image information representation uses neural networks to extract image features. The features obtained have high-level semantic information but lack details, so they perform poorly in object availability classification. SUMMARY

[0007] To solve the above problems, the technical scheme adopted by the present application is as follows:

[0008] A visual-based object availability few-shot classification method, comprising the following steps:

[0009] Constructing a data set for object availability classification;

[0010] Preprocessing the data set to obtain a preprocessed data set;

[0011] Divide the preprocessed data set into mutually exclusive training set, validation set and test set;

[0012] Constructing an availability few-shot classification model;

[0013] Input the training set data into the availability few-shot classification model, train the availability few-shot classification model, and obtain the trained availability few-shot classification model;

[0014] inputting the test set data into the trained availability few-shot classification model to obtain the classification of the object availability few-shot sample.

[0015] Further, the availability few-shot classification model comprises a multi-scale feature extraction module, a dual attention module based on self-attention, and a correlation module.

[0016] The multi-scale feature extraction module is used for extracting sample features.

[0017] The dual attention module based on self-attention comprises a position attention sub-module and a channel attention sub-module.

[0018] The position attention sub-module adopts self-attention on the extracted sample features, selectively aggregates the features of each position through the weighted sum of the features at all positions, makes similar features related to each other, and obtains sample position features.

[0019] The channel attention sub-module selectively emphasizes mutually dependent channel mappings by integrating the correlation characteristics between all channel mappings, and further obtains sample channel features.

[0020] The sample position features and the sample channel features are added to obtain an added feature map.

[0021] The added feature map is input into the correlation module to obtain the similarity of the sample picture.

[0022] Further, the preprocessing refers to unifying the picture size in the data set to 448x448.

[0023] Further, the multi-scale feature extraction module adopts a multi-scale feature fusion convolutional neural network based on deep learning for feature extraction, removes the fully connected layer highly related to the classification task, and outputs image features.

[0024] Further, the multi-scale feature extraction module adopts a multi-scale feature fusion convolutional neural network based on deep learning for feature extraction, removes the fully connected layer highly related to the classification task, and outputs image features. The process is as follows:

[0025] The input image is subjected to a convolution operation by a first convolution layer, i.e. batch normalization, a nonlinear activation function Relu, and a maximum pooling operation are performed on the input image, and an image after convolution operation is output.

[0026] The image after convolution operation is sequentially subjected to a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer, respectively, to obtain different scale feature images output at three different stages.

[0027] The different scale feature images output in three different stages are fused in a top-down order to obtain fused features.

[0028] Further, the first residual layer, the second residual layer, the third residual layer and the fourth residual layer each comprise two basic block modules, and for each basic block of the residual layer, the input is connected to the input through a jump connection after being subjected to two 3*3 convolution, batch normalization and nonlinear activation function operations, and then the output of each basic block is obtained through a nonlinear activation function Relu.

[0029] Further, the first residual layer has 64 filters, the second residual layer has 128 filters, the third residual layer has 256 filters and the fourth residual layer has 512 filters.

[0030] A visual-based object usability few-shot classification device comprises:

[0031] A construction module I is configured to construct a data set for object usability classification.

[0032] A preprocessing module is configured to preprocess the data set to obtain a preprocessed data set, and divide the preprocessed data set into a training set, a validation set and a test set which are mutually exclusive in category.

[0033] A construction module II is configured to construct a usability few-shot classification model.

[0034] A training module is configured to input the training set data into the usability few-shot classification model, train the usability few-shot classification model, and obtain a trained usability few-shot classification model.

[0035] A classification module is configured to input the test set data into the trained usability few-shot classification model, and obtain a classification of the object usability few-shot.

[0036] A computer readable storage medium stores a computer program, and when the computer program runs, the visual-based object usability few-shot classification method is executed.

[0037] The visual-based object usability few-shot classification method provided by the application mainly proposes a visual-based object usability few-shot classification model, constructs a data set specially used for object usability classification, and realizes object-level functional usability classification according to a small amount of sample labeling in the case of lacking a large amount of labeled data. The method uses a multi-scale feature extraction and fusion method to improve the representation of image information and improve the classification effect caused by the large intra-class difference of object usability categories, and improves the classification accuracy. Meanwhile, the collected data set can provide a data basis for object usability classification.

[0038] Firstly, the deep neural network ResNet18 is used to extract image features of different levels of network output, and the optimal feature fusion method is found according to the experiment. For the problem of similar feature area of object availability category, a double attention mechanism (DANet) is introduced, which is based on the self-attention mechanism to capture the feature dependency in the spatial dimension and the channel dimension respectively, that is, to capture the common features in the object availability category, for example, the racket part of the "racket" in the "racket" category, "tennis racket" and "ping-pong racket".

[0039] The application takes a relationship network as a basic model, replaces the four-layer convolution feature extraction module with a multi-scale feature extraction module to extract more effective features, and combines with a double attention network to improve the poor classification effect caused by the large intra-class difference of the object availability dataset.

[0040] The support set and query set features extracted by the multi-scale fusion feature extraction are input into the double attention module, so that the model pays more attention to the similar feature part of the support set and the query set;

[0041] After the double attention module, each support set picture is spliced with the same query picture, and finally input into the correlation module to calculate the correlation score, and the highest score class is the category to which the query picture belongs.

[0042] The output result of the method is the predicted category of the unlabeled query image, and the method reduces the overfitting phenomenon caused by the unbalanced category distribution and limited samples. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 is a flowchart of the method;

[0045] Figure 2 is a structure diagram of the object availability few-shot classification model;

[0046] Figure 3 is a multi-scale feature extraction module structure diagram based on ResNet18;

[0047] Figure 4 is a structure diagram of the ResNet18 residual layer. DETAILED DESCRIPTION

[0048] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict, and the present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0049] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, but not all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0050] It should be noted that the terms used herein are only intended to describe specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0051] Unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions, and numerical values set forth in the various embodiments described herein are not limiting. It should be understood that the various parts shown in the drawings are not necessarily drawn to scale in proportion. The techniques, methods and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered as part of the authorized description. In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0052] In the description of the present application, it should be understood that the orientation words such as "front, back, up, down, left, right", "transverse, vertical, perpendicular, horizontal" and "top, bottom" and the like indicated orientation or position relationship is generally based on the orientation or position relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, without making the opposite statement, these orientation words do not indicate and imply that the device or element referred to must have a particular orientation or be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation on the scope of protection of the present application: the orientation words "inner, outer" refer to the inner and outer relative to the contour of each component itself.

[0053] For the convenience of description, spatial relative terms such as "over", "above", "upper surface", "upper" and the like can be used herein to describe the spatial positional relationship of one device or feature with other devices or features as shown in the drawings. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation of the device described in the drawings. For example, if the device in the drawing is inverted, the device described as "above" or "over" other devices or structures will be positioned "below" or "under" the other devices or structures. Thus, the exemplary term "above" can include both "above" and "below" orientations. The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein are interpreted accordingly.

[0054] In addition, it should be noted that the use of "first", "second" and the like to define parts is only for the convenience of distinguishing the corresponding parts, and the above words have no special meaning unless otherwise stated, and therefore cannot be understood as a limitation on the scope of protection of the present application.

[0055] Functional availability describes the interaction between an object and an actor, or what actions an actor can perform with the object in a certain scenario. For example, a knife can be used to cut, and a chair can be used to sit. When we want to sit down and rest, we can also find other objects to replace the chair, such as sofas, stools, etc. Therefore, a functional availability category contains multiple object categories.

[0056] Although the object affordance classification can be solved as a traditional classification problem, the traditional image classification task has a million-level dataset, and a classification model can be well trained using deep learning, but the existing functional availability dataset does not reach this level. Among the currently public datasets, there is an IIT-Affordance dataset for pixel-level availability detection, and a PADv2 dataset for availability detection in complex scenes, but none of them is suitable for object affordance classification. It is not only time-consuming and laborious to annotate a large dataset, but also economically expensive. In the case of unbalanced class distribution and limited samples, the traditional classification model is prone to overfitting. In addition, the traditional visual classification problem is to recognize by extracting features from visual perception, however, objects that provide the same object affordance may have great differences in appearance, shape and other features, for example, sofas, benches, swings and rocking chairs have the common functional affordance "sit".

[0057] Take a 5-class, 1-sample (5-way 1-shot) classification task as an example, 5 support set pictures (5 categories are randomly selected from the training set, and 1 picture from each category is taken to form the support set) and 75 query set pictures (15 pictures are selected from the remaining pictures of the selected 5 categories to form the query set);

[0058] Figure 1 is a flowchart of the method;

[0059] A visual-based object affordance few-shot classification method, characterized in that it comprises the following steps:

[0060] S1: Construct a dataset for object affordance classification;

[0061] S2: Preprocess the dataset to obtain a preprocessed dataset;

[0062] Divide the preprocessed dataset into mutually exclusive training set, validation set and test set, and divide the training set and test set respectively;

[0063] S3: Construct an affordance few-shot classification model;

[0064] S4: Input the training set data into the object affordance few-shot classification model to train the object affordance few-shot classification model, and obtain the trained object affordance few-shot classification model;

[0065] S5: Input the test set data into the trained object affordance few-shot classification model to obtain the object affordance classification.

[0066] S1, S2, S3, S4 and S5 are executed in sequence;

[0067] The dataset pictures proposed by the present application are mostly collected from the PADv2 dataset and the IIT_Affordance dataset. In order to increase the diversity of the dataset, some pictures are also collected from the Internet. Finally, the dataset proposed by the present application contains 4794 images, covers 38 functional availability categories and 88 object categories, and assigns a functional availability category label to each object category (such as "bicycle", "stool", "tennis racket", "sofa", "computer", "mug", "umbrella"), establishes a hierarchy for the dataset of the present application, and the descriptions of all the functional availability categories and the number of object categories contained therein are different. See Table 1 for specific information about the dataset.

[0068] Table 1 Dataset Category Information

[0069]

[0070]

[0071] One functional availability category can cover multiple object categories, for example, objects with the label "sitting" include "stool", "sofa", "bench", "swing", "wheelchair", "rocking chair", etc., which are different in size, geometry, material and appearance, but have the same intrinsic function.

[0072] According to the division ratio of the commonly used miniImageNet dataset in few-shot learning, the 38 functional availability categories of the dataset proposed by the present application are divided, of which 24 categories are used for training, 6 categories are used for verification, and 8 categories are used for testing, and the size of all the pictures in the dataset is adjusted to 448x448.

[0073] All networks are optimized using the Adam optimizer, and the object availability few-shot classification model is trained end-to-end and randomly initialized. For an N-class few-shot classification task with K samples per class,

[0074] The training set data is randomly sampled into a set of E few-shot classification tasks:

[0075]

[0076] D train consists of a support set and a query set ,

[0077] wherein: represents the kth image in the support image from the jth class in the ith few-shot task, represents the hth query image in the ith few-shot task (h query images are randomly selected from the remaining images in N categories), represents the category of the hth query image in the ith few-shot task.

[0078] During network training, each few-shot task includes N x K support images and h query images participate together;

[0079] For the test data set, according to similar steps as dividing the training set, E' few-shot test task groups are randomly sampled to form a set:

[0080]

[0081] wherein D test and D train There are two different points: (1) Different image sources, that is, the test set and the data set categories are disjoint; (2) D test does not include the category information of the query image , that is, this category information is only used as a standard to measure the pros and cons of the few-shot classification model, and does not participate in the calculation.

[0082] According to the standard setting adopted by the existing few-shot model, two classification tasks are adopted, the first one is 5-way 1-shot, that is, 5 categories are randomly selected in the training set, 1 image in each category is selected as the support set, and then 15 images are selected from the remaining images in each category to form the query set, to form an episode (that is, a few-shot classification task), 5 categories are randomly selected in the test set, 1 image in each category is selected as the support set, and then 1 image is selected from the remaining images in each category to form the query set, also forming an episode, and multiple random episodes are constructed in this way; The second one is 5-way 5-shot, that is, 5 categories are randomly selected in the training set, 5 images in each category are selected as the support set, and then 15 images are selected from the remaining images in each category to form the query set, 5 categories are randomly selected in the test set, 5 images in each category are selected as the support set, and then 1 image is selected from the remaining images in each category to form the query set, and multiple random episodes are constructed in this way;

[0083] The few-shot classification model includes a multi-scale feature extraction module, a self-attention-based double attention module, and a correlation module.

[0084] The multi-scale feature extraction module is used to extract sample features; the self-attention-based double attention module includes a position attention sub-module and a channel attention sub-module.

[0085] The position attention sub-module adopts self-attention on the extracted sample features, selectively aggregates the features of each position through the weighted sum of the features at all positions, makes similar features related to each other, and obtains sample position features;

[0086] The channel attention sub-module selectively emphasizes the interdependent channel mappings by integrating the relevant characteristics between all channel mappings, and further obtains sample channel features;

[0087] The sample position features and the sample channel features are added to obtain an added feature map;

[0088] The added feature map is input into the correlation module to obtain the similarity of the sample picture.

[0089] Further, the multi-scale feature extraction module is used to extract sample features; more effective features are extracted;

[0090] Taking a 5-way 1-shot classification task as an example, a support set (5 pictures) and a query set (75 pictures) are input into the multi-scale fusion feature extraction module,

[0091] The multi-scale fusion feature extraction module uses the structure of Resnet18, that is, an 18-layer residual network, which includes five convolution blocks: convolution block 1, convolution block 2, convolution block 3, convolution block 4, and convolution block 5. The output feature size of the convolution block 3 is 112x112, and the channel is 128; the output feature size of the convolution block 5 is 28x28, and the channel is 512. Next, the deconvolution operation is performed on the features extracted by the convolution block 5, and the size of the features output by the convolution block 5 is expanded to 112x112 through 128 deconvolutions with a step of 4, and the number of channels is reduced to 128, which is the same as the feature size and channel number of the convolution block 3. The expanded convolution block 5 and the convolution block 3 are added element by element, and finally the fused feature convolution block 3&5 is obtained. Therefore, each input picture will obtain a feature map with a size of 128x112x112, and then a convolution convolution block 6 is used to obtain a feature with a size of 64x112x112;

[0092] The dual attention module based on self-attention is composed of a position attention sub-module and a channel attention sub-module, which can simultaneously capture global feature dependency relationships in spatial and channel dimensions. The module inputs the feature map generated by the multi-scale fusion feature extraction module into the two parallel attention sub-modules, and finally combines the output features of the two attention sub-modules to obtain better feature representation.

[0093] The position attention sub-module part adopts self-attention, selectively aggregates the features of each position through the feature weighting sum of all positions, and makes similar features related to each other. The specific implementation is that the feature map A (64x112x112) extracted by the feature extraction is convolved to obtain three feature maps B (8x112x112), C (8x112x112) and D (64x112x112), wherein B is reshaped and transposed to obtain a feature map with a size of 12544x8, multiplied by the reshaped feature map C (8x12544), and a spatial attention map S (12544x12544) is obtained through a softmax function. The matrix multiplication is performed between the reshaped feature map D (64x12544) and S, multiplied by a scale factor a, reshaped to the original shape (64x112x112), and added to A to obtain the final output E1 (64x112x112). Wherein a is initialized to 0 and gradually learns to obtain a larger weight.

[0094] The channel attention sub-module part selectively emphasizes the interdependent channel mappings by integrating the related characteristics between all channel mappings, and the final feature of each channel is the weighted sum of all channels and the original feature. The specific implementation of this part is to respectively change the shape (64x12544) and reshape & transpose (12544x64) of the feature map A (64x112x112) extracted by the feature extraction, multiply the two obtained feature maps, obtain a channel attention map X (64x64) through a softmax function, then perform matrix multiplication between X and the reshaped A (64x12544), multiply the obtained result by a scale factor β, reshape to the original shape (64x112x112), and add to A to obtain the output E2 (64x112x112) of the channel attention sub-module, wherein β is initialized to 0 and gradually learns to obtain a larger weight.

[0095] The output E1 of the position attention sub-module is added to the output E2 of the channel attention sub-module to obtain the final output feature map (64x112x112) of the double attention module. The channel attention sub-module selectively emphasizes the interdependent channel mappings by integrating the related characteristics between all channel mappings, thereby obtaining sample channel features;

[0096] After passing through the double attention module, the query feature is spliced with the five support features, and the size of the spliced feature map is 128x112x112;

[0097] The obtained feature map with a size of 128x112x112 is input into a correlation module, which includes five convolution blocks: convolution block 7, convolution block 8, convolution block 9, convolution block 10, convolution block 11, and two fully connected layers, denoted as fully connected layer 1 (fc1) and fully connected layer 2 (fc2).

[0098] Convolution block 7, convolution block 8, convolution block 9, and convolution block 10 are each composed of a group of convolution, batch normalization, Relu activation function, and max pooling, and convolution block 11 is composed of convolution. The dimension of fully connected layer 1 is 8, and a Relu activation function is used. The dimension of fully connected layer 2 is 1, and a Sigmoid activation function is used. Finally, a relationship score is output, and the label with the largest score is assigned to the unlabeled query set picture.

[0099] The multi-scale feature extraction module adopts a deep learning-based feature extraction method, removes the fully connected layer highly related to the classification task, and uses the output of the convolution block as the image feature. For image feature extraction, ResNet18, an 18-layer residual network, is used.

[0100] Figure 3 is a multi-scale feature extraction module structure diagram based on ResNet18; first, a 7x7 convolution operation is performed on the input image, and batch normalization, a nonlinear activation function, and a max pooling operation are performed on the feature map, and then the four residual layers (Layer1, Layer2, Layer3, Layer4) are sequentially passed through.

[0101] The first residual layer Layer1 has 64 filters, the second residual layer Layer2 has 128 filters, the third residual layer Layer3 has 256 filters, and the fourth residual layer Layer4 has 512 filters. The network has a total of 17 convolutional layers, which greatly increases the depth and complexity of the ResNet18 network compared to the previous 4-layer convolution used in the few-shot network, which enables the network to extract deep abstract features. The detailed structure of each residual layer is shown in Figure 4 Each residual layer includes two basic blocks (Basicblock). For each basic block of the residual layer, the input is connected to the input through a jump connection after being processed by two 3x3 convolutions, batch normalization, and a nonlinear activation function, and then the output of each basic block is obtained through a nonlinear activation function Relu. The addition of the jump connection can suppress the overfitting phenomenon caused by the increase in network complexity and the lack of training samples.

[0102] However, if Resnet18 is used as a feature extraction network and only deep features are extracted, the final output feature information representation will be incomplete, and some image objects cannot be classified or recognized. In a neural network, different levels of features have different meanings. Shallow features have strong generalization, that is, more complex overall information, while deep features have strong semantics. Therefore, in order to extract more effective features, the application introduces a multi-scale feature extraction and fusion technology, which combines the detailed information of the shallow network and the advanced semantic information of the deep network to improve the network performance.

[0103] Figure 2 is a structure diagram of the object availability few-shot classification model;

[0104] The specific method is to combine features of different levels to find the optimal combination method to provide a more effective feature extractor for the object availability few-shot classification model. First, the picture size of the data set is unified to 448x448 as the input of the model, and after the Resnet18 network (remove the full connection layer), the outputs at the end of the second residual layer, the third residual layer and the fourth residual layer are taken as different scale features, respectively, to obtain three image feature maps output at different stages, represented as convolution block 3 (conv3), convolution block 4 (conv4) and convolution block 5 (conv5). The output feature size of convolution block 3 (conv3) is 112x112, and the channel is 128; the output feature size of convolution block 4 (conv4) is 56x56, and the channel is 256; the output feature size of convolution block 5 (conv5) is 28x28, and the channel is 512,

[0105] The feature fusion of the application refers to a feature pyramid fusion (FPN) method, which fuses features of different scales in a top-down order. For the merging of the features of the convolution block 5 and the features of the convolution block 4, the features extracted by the convolution block 5 are first processed to have the same size and channel number as the features extracted by the convolution block 4, and then the operation is performed. Specifically, for the output features of the convolution block 5 with a size of 28*28 and a channel number of 512, 256 deconvolutions with a step of 2 dimensions and a size of 512 are used to expand the features to the same size and channel number as the output features of the convolution block 4, and then the features are added element by element to obtain the fused new features, which are denoted as convolution block 4&5 (conv4&5). The fusion of the features of the convolution block 5 and the features of the convolution block 3 is similar. The features extracted by the convolution block 5 are also deconvoluted, and through 128 deconvolutions with a step of 4, the size of the features is expanded to 112*112, and the channel number is reduced to 128, which is the same as the size and channel number of the features of the convolution block 3. Similarly, the expanded features of the convolution block 5 and the features of the convolution block 3 are added element by element to obtain the fused features of the convolution block 3&5 (conv3&5). At the same time, the direct output of the convolution block 5 (conv5) is also reserved as an experimental control.

[0106] The loss function adopted in the application, also called a cost function, is used to evaluate the degree of inconsistency between the predicted value of the model and the true value, is an index for measuring the performance of the predicted model in predicting the expected result, and is also an objective function for optimization in the neural network. The process of neural network training or optimization is a process of minimizing the loss function. The smaller the loss function is, the closer the predicted value of the model is to the true value, and the better the robustness of the model is.

[0107] In the network training process, the sample x b in the query set Q in the training set Through the multi-scale feature extraction module Generation and Then input the two feature maps into the dual attention module h θ Generate a new feature map and Then combine Where C(·,·) is assumed to be a deep feature map cascade.

[0108] Next, the combined feature map is input into the correlation module , denoted as Finally, a scalar between 0 and 1 is generated to represent and x bsimilarity between them, also called relation score. The output relation score is regarded as a value between 0 and 1, where 0 means completely dissimilar and 1 means completely similar. Thus, in the C-way One-shot setting, C relation scores r can be generated to represent the relation between a query sample x and samples in the support set

[0109]

[0110] In the objective function and its optimization stage, the mean square error (MSE) is directly used as the loss function of network training, which can be expressed as:

[0111]

[0112] wherein, in the C-way K-shot setting, m = N x K, n is the number of query images in one few-shot task, representing the class of the support sample y b representing the class of the query sample x b The relation score is regressed to a label: 1 for matching and 0 for non-matching.

[0113] A visual-based object usability few-shot classification device, comprising:

[0114] A construction module I for constructing a data set for object usability classification;

[0115] A preprocessing module for preprocessing the data set to obtain a preprocessed data set; dividing the preprocessed data set into a training set, a validation set and a test set which are mutually exclusive in class

[0116] A construction module II for constructing a usability few-shot classification model;

[0117] A training module for inputting the training set data into the usability few-shot classification model to train the usability few-shot classification model, and obtaining a trained usability few-shot classification model;

[0118] A classification module for inputting the test set data into the trained usability few-shot classification model to obtain the classification of the object usability few-shot.

[0119] A computer readable storage medium, the computer readable storage medium stores a computer program, wherein the computer program runs to execute the visual-based object usability few-shot classification method.

[0120] ​To verify the performance of the proposed method in object availability image classification, experiments are performed on the constructed object availability dataset, and Table 2 summarizes the performance comparison of the proposed visual-based object availability few-shot classification method and other methods in two few-shot learning tasks.

[0121] Among them, the matching network proposed by O. Vinyals has an accuracy of 28.50% and 39.30% in the 5-way 1-shot and 5-way 5-shot tasks, respectively;

[0122] The accuracy of the prototype network proposed by J. Snell in the 5-way 1-shot and 5-way 5-shot tasks is 29.75% and 40.94%, respectively;

[0123] The accuracy of the relational network proposed by Sung (the base model of our method) in the 5-way 1-shot and 5-way 5-shot tasks is 35.69% and 45.05%, respectively;

[0124] For the deep nearest neighbor network proposed by Li, we conducted two sets of experiments according to the different feature extraction modules. The deep nearest neighbor network with four-layer convolution as the feature extractor has an accuracy of 30.45% and 41.45% in the 5-way 1-shot and 5-way 5-shot tasks, respectively;

[0125] The deep nearest neighbor network with ResNet-256 as the feature extractor has an accuracy of 30.79% and 40.97% in the 5-way 1-shot and 5-way 5-shot tasks, respectively;

[0126] The accuracy of the self-attention relational network proposed by Hui in the 5-way 1-shot and 5-way 5-shot tasks is 38.14% and 46.97%, respectively;

[0127] The accuracy of the proposed visual-based object availability few-shot classification method in the 5-way 1-shot and 5-way 5-shot tasks is 41.12% and 52.92%, respectively, which is better than other methods, thereby verifying the effectiveness of the method for object availability image classification.

[0128] Table 2 Accuracy comparison of the proposed method and other methods in two few-shot learning tasks

[0129] Model 5-way 1-shot (%) 5-way 5-shot (%) Matching network 28.50 39.30 Prototype network 29.75 40.94 Relation network 35.69 45.05 Deep nearest neighbor neural network (four-layer convolution) 30.45 41.45 Deep nearest neighbor neural network (ResNet256) 30.79 40.97 Self-attention relation network 38.14 46.97 Method of the invention 41.12 52.92

[0130] The application proposes a multi-scale feature fusion method to extract more efficient features, different experiments are conducted on the combination of each feature layer, and the combination results are analyzed to obtain a better combination method.

[0131] First, three feature layers are extracted, including convolution block 3, convolution block 4 and convolution block 5, the accuracy obtained by taking the features of the convolution block 5 as the input of the subsequent network is taken as the benchmark, the features of the convolution block 3, the features of the convolution block 4 and the features of the convolution block 5 are fused respectively, experiments are conducted, and the accuracy is calculated.

[0132] From the experimental results in Table 3, it can be seen that the accuracy obtained by taking the features of the convolution block 5 as the input of the subsequent network in the 5-way 1-shot and 5-way 5-shot tasks is 37.83% and 48.16% respectively, the accuracy obtained by taking the features combined by the convolution block 4 and the convolution block 5 as the input of the subsequent network in the 5-way 1-shot and 5-way 5-shot tasks is 38.02% and 49.23% respectively; and the accuracy obtained by taking the features combined by the convolution block 3 and the convolution block 5 as the input of the subsequent network in the 5-way 1-shot and 5-way 5-shot tasks is 39.67% and 51.06% respectively, analysis shows that compared with the experimental results of using the features of the convolution block 5, the combination of the convolution block 4 and the convolution block 5 (convolution block 4&5) does not achieve greater improvement in classification accuracy, while the feature combination of the convolution block 3 and the convolution block 5 (convolution block 3&5) has a more obvious improvement in experimental results, especially in 5-shot, which improves by about three percentage points. In summary, the best combination of the multi-scale feature extraction module is convolution block 5+convolution block 3 (convolution block 3&5).

[0133] From the experimental results (see Table 3), it can be seen that the fusion of the feature convolution block 3&5 as the output of the feature extraction is more effective for the classification of object availability.

[0134] Table 3 Multi-scale feature combination experiment

[0135]

[0136]

[0137] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0138] 1. Farhadi, A., et al. "Describing objects by their attributes." IEEE (2009).

[0139] 2. Hermans T, Rehg J M, Bobick A. Affordance Prediction via Learned Object Attributes. 2011.

[0140] 3. Ren, S., and S. Yu. "Human-object-object-interaction affordance." Robot Vision IEEE, 2013.

[0141] 4. Zhu, Y., A. Fathi, and F. F. Li. "Reasoning about Object Affordances in a Knowledge Base Representation." European Conference on Computer Vision Springer International Publishing, 2014.

[0142] 5. Karthik Mahesh Varadarajan and Markus Vincze. 2013. Parallel deep learning with suggestive activation for object category recognition. In Proceedings of the International Conference on Computer Vision Systems. Springer, 354-363.

[0143] 6. Emre Ugur, Sandor Szedmak, and Justus Piater. 2014. Bootstrapping paired-object affordance learning with learned single-affordance features. In Proceedings of the Joint IEEE International Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob' 14). IEEE, 476-481.

[0144] 7. Hjelm, M., et al. "Invariant Feature Mappings for Generalizing Affordance Understanding Using Regularized Metric Learning." (2019).

[0145] 8. Zheng, X., Z. Zeng, and J. Zhang. "High-level Object Affordance Recognition." 2018 IEEE International Conference on Information and Automation (ICIA) IEEE, 2018.

[0146] 9. G Koch, R Zemel, and R Salakhutdinov. "Siamese Neural Networks for One-shot Image Recognition."

[0147] 10. Vinyals, O., et al. "Matching Networks for One Shot Learning." 2016.

[0148] 11. Snell J, Swersky K, Zemel R S. Prototypical Networks for Few-shot Learning [J]. 2017.

[0149] 12. Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[0150] 13. Li, W., et al. "Revisiting Local Descriptor Based Image-To-Class Measure for Few-Shot Learning." 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) IEEE, 2019.

[0151] 14. Hou, R.; Chang, H.; Bingpeng, M.; Shan, S.; and Chen, X. 2019. Cross Attention Network for Few-shot Classification. In NeurIPS.

[0152] 15. Hui, B., et al. "Self-Attention Relation Network for Few-Shot Learning." 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) IEEE, 2019.

[0153] 16. Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2020. DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover’s Distance and Structured Classifiers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 12203-12213.

[0154] 17. Zhao, A., et al. "Domain-Adaptive Few-Shot Learning." 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) IEEE, 2021.

[0155] 18. Hong, J., et al. "Reinforced Attention for Few-Shot Learning and Beyond." (2021).

Claims

1. A vision-based few-shot classification method for object usability, characterized in that: The method comprises the following steps: constructing a data set for object availability classification, realizing object-level functional availability classification according to a small amount of sample annotation; preprocessing the data set to obtain a preprocessed data set; dividing the preprocessed data set into a training set, a validation set and a test set which are mutually exclusive in category; constructing an availability few-sample classification model; inputting the training set data into the availability few-sample classification model to train the availability few-sample classification model and obtaining a trained availability few-sample classification model; inputting the test set data into the trained availability few-sample classification model to obtain the classification of the object availability few samples; the availability few-sample classification model comprises a multi-scale feature extraction module, a self-attention-based double attention module and a correlation module; the multi-scale feature extraction module is used for extracting sample features; the self-attention-based double attention module comprises a position attention sub-module and a channel attention sub-module; the position attention sub-module adopts self-attention on the extracted sample features, selectively aggregates the features of each position through the weighted sum of the features at all positions, makes similar features related to each other, and obtains sample position features; the channel attention sub-module selectively emphasizes mutually dependent channel mappings by integrating the correlation characteristics between all channel mappings, and then obtains sample channel features; adding the sample position features and the sample channel features to obtain an added feature map; inputting the added feature map into the correlation module to obtain the similarity of the sample picture.

2. The method of claim 1, wherein: The preprocessing refers to unifying the picture size in the data set to 448x448.

3. The method of claim 1, wherein: The multi-scale feature extraction module adopts a deep learning-based multi-scale feature fusion convolutional neural network for feature extraction, removes the fully connected layer highly related to the classification task, and outputs image features.

4. The method of claim 1, wherein: The process of using the multi-scale feature extraction module to extract features by using the deep learning-based multi-scale feature fusion convolutional neural network and removing the fully connected layer highly related to the classification task and outputting image features is as follows: convolve the input image through the first convolutional layer, that is, perform batch normalization, nonlinear activation function Relu and maximum pooling operation on the input image, and output the picture after convolution operation; the picture after convolution operation is sequentially passed through the first residual layer, the second residual layer, the third residual layer and the fourth residual layer to obtain different scale feature images output by three different stages; fuse the different scale feature images output by the three different stages in top-down order to obtain fused features.

5. The method of claim 4, wherein: The first residual layer, the second residual layer, the third residual layer and the fourth residual layer each comprise two basic block modules, and for each basic block of the residual layer, the input is connected to the input through a jump connection after being subjected to two 3x3 convolution, batch normalization and nonlinear activation function operations, and then the output of each basic block is obtained through a nonlinear activation function Relu.

6. The method of claim 4, wherein: The first residual layer has 64 filters, the second residual layer has 128 filters, the third residual layer has 256 filters and the fourth residual layer has 512 filters.

7. A visual-based object usability few-shot classification device, characterized by: The method comprises the following steps: The construction module I is used for constructing a data set for object availability classification, and realizes object-level function availability classification according to a small amount of sample annotation. The preprocessing module is used for preprocessing the data set to obtain a preprocessed data set. The preprocessed data set is divided into a training set, a validation set and a test set which are mutually exclusive in category The construction module II is used for constructing an availability few-shot classification model; the availability few-shot classification model comprises a multi-scale feature extraction module, a self-attention-based double attention module and a correlation module. The multi-scale feature extraction module is used for extracting sample features. The self-attention-based double attention module comprises a position attention sub-module and a channel attention sub-module. The position attention sub-module adopts self-attention on the extracted sample features, selectively aggregates the features of each position through the weighted sum of the features at all positions, makes similar features related to each other, and obtains sample position features. The channel attention sub-module selectively emphasizes mutually dependent channel mappings by integrating the correlation characteristics between all channel mappings, and then obtains sample channel features. The sample position features and the sample channel features are added to obtain an added feature map. The added feature map is input into the correlation module to obtain the similarity of the sample picture. The training module is used for inputting the training set data into the availability few-shot classification model, training the availability few-shot classification model, and obtaining a trained availability few-shot classification model. The classification module is used for inputting the test set data into the trained availability few-shot classification model to obtain the classification of the object availability few-shot.

8. A computer-readable storage medium having stored therein a computer program comprising, When the computer program runs, the computer program executes the visual-based object availability few-shot classification method according to any one of claims 1-6.

Images