Image category prediction method, device, electronic device and storage medium

Abstract

The present application provides an image category prediction method, device, electronic device and storage medium, which are used to improve the problem of low accuracy in image category prediction. The method includes: obtaining an image to be processed, and randomly cropping the image to be processed to obtain a local image; extracting global features in the image to be processed and local features in the local image respectively; using a graph neural network to aggregate the global features and local features to obtain aggregate features of the image to be processed; predicting the category of the image to be processed based on the similarity between the aggregate features of the image to be processed and the class prototype aggregate features in the graph neural network, the class prototype aggregate features are obtained by training sample images corresponding to known category labels using the graph neural network.

Description

Technical Field

[0001] The present application relates to the technical fields of machine learning, image processing and image classification, and in particular to an image category prediction method, device, electronic device and storage medium. Background Art

[0002] At present, few-shot learning refers to the process of learning from a small number of samples (Few Examples) in the absence of model training data. For example, there are three categories A, B and C, and there are only 10 samples in each category to train the model. The model needs to learn to improve the classification accuracy as much as possible.

[0003] At present, few-shot learning is usually conducted based on metric learning. For example, sample images are obtained and projected into the same metric space using a neural network model. Then, the class similarity between the sample image and other images is determined by comparing the distance between the sample image and other images in the metric space, thereby determining the specific class of the sample image based on the class similarity. However, in actual practice, it is found that the metric learning-based approach only considers the global features of the image, and it is difficult to avoid the influence of local features such as the background area in the image on the image classification. Therefore, the accuracy of class prediction of images using the metric learning-based approach is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image category prediction method, device, electronic device and storage medium, which are used to improve the problem of low accuracy in image category prediction.

[0005] The embodiment of the present application provides an image category prediction method, including: obtaining an image to be processed, and randomly cropping the image to be processed to obtain a local image; extracting global features in the image to be processed and local features in the local image respectively; aggregating the global features and local features using a graph neural network to obtain aggregate features of the image to be processed; predicting the category of the image to be processed according to the similarity between the aggregate features of the image to be processed and the class prototype aggregate features in the graph neural network, the class prototype aggregate features are obtained by training sample images corresponding to known category labels using the graph neural network. In the implementation process of the above scheme, the global features corresponding to the image to be processed and the local features of the local image cropped from the image to be processed are aggregated by using the graph neural network to obtain the aggregate features of the image to be processed, and the category of the image to be processed is predicted according to the similarity between the aggregate features of the image to be processed and the class prototype aggregate features in the graph neural network, so that both the global features corresponding to the image to be processed and the local features of the local image cropped from the image to be processed are taken into account, so that the graph neural network can effectively determine the important local features in the category prediction process according to the correlation between the global features and the local features, effectively avoiding the interference of the background area in the image and other local areas on the image classification, thereby improving the accuracy of the image category prediction.

[0006] Optionally, in an embodiment of the present application, a graph neural network is used to aggregate global features and local features to obtain aggregate features of the image to be processed, including: obtaining the global nodes to be processed in the graph neural network for global features and the local nodes to be processed in the graph neural network for local features; extracting the edge features to be processed between the global nodes to be processed and the local nodes to be processed; and aggregating the global nodes to be processed, the local nodes to be processed, and the edge features to be processed to obtain the aggregate features of the image to be processed. In the implementation of the above scheme, by extracting the edge features to be processed between the global nodes to be processed and the local nodes to be processed, and aggregating the global nodes to be processed, the local nodes to be processed, and the edge features to be processed, the graph neural network avoids the interference of local background areas in the image on image classification based on the aggregate features, thereby improving the accuracy of image category prediction.

[0007] Optionally, in an embodiment of the present application, before using a graph neural network to aggregate global features and local features, it also includes: obtaining multiple sample images and multiple sample labels, the sample labels including category labels of the sample images; using multiple sample images and multiple sample labels to construct an initial graph structure; using multiple sample images as training data and multiple sample labels as training labels, the initial graph structure is trained to obtain a graph neural network. In the implementation process of the above scheme, an initial graph structure is constructed using multiple sample images and multiple sample labels, and the initial graph structure is trained to obtain a graph neural network that avoids local interference in image classification such as background areas in the image, thereby improving the accuracy of image category prediction through the graph neural network.

[0008] Optionally, in an embodiment of the present application, multiple sample images and multiple sample labels are used to construct an initial graph structure, including: randomly cropping each sample image in the multiple sample images to obtain multiple sample local images; extracting sample global features in the multiple sample images and sample local features in the multiple sample local images respectively; and constructing an initial graph structure using sample global features in the multiple sample images and sample local features in the multiple sample local images. In the implementation of the above scheme, an initial graph structure is constructed by using sample global features in multiple sample images and sample local features in multiple sample local images, thereby obtaining a graph neural network that avoids interference from background areas in the image based on the initial graph structure, thereby improving the accuracy of image category prediction through the graph neural network.

[0009] Optionally, in an embodiment of the present application, the sample global features in multiple sample images and the sample local features in multiple sample local images are used to construct an initial graph structure, including: using the sample global features in multiple sample images as global nodes, and connecting the global nodes corresponding to the multiple sample images to obtain a fully connected graph; using the sample local features in multiple sample local images as local nodes, and connecting the local nodes to the corresponding global nodes in the fully connected graph to obtain an initial graph structure. In the implementation process of the above scheme, the initial graph structure is obtained by connecting the local nodes to the corresponding global nodes in the fully connected graph, so as to obtain a graph neural network that avoids interference from background areas in the image based on the initial graph structure, thereby improving the accuracy of image category prediction through the graph neural network.

[0010] Optionally, in an embodiment of the present application, the initial graph structure is trained, including: aggregating and averaging the initial graph structure to obtain aggregate features of the class prototype; using the initial graph structure to extract the aggregate features of the sample image, and calculating the classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype; using the initial graph structure to extract the edge features of the sample image, and calculating the edge loss value between the edge features of the sample image and the edge features calculated by the sample label; training the initial graph structure according to the classification loss value and the edge loss value. In the implementation process of the above scheme, by training the initial graph structure according to the classification loss value and the edge loss value, a graph neural network is obtained that avoids local interference in image classification such as background areas in the image, thereby improving the accuracy of image category prediction through the graph neural network.

[0011] Optionally, in an embodiment of the present application, the initial graph structure is aggregated and averaged to obtain aggregate features of the class prototype, including: respectively calculating the first edge features between the global nodes and the global nodes in the initial graph structure, and the second edge features between the global nodes and the local nodes in the initial graph structure; for each global node in the initial graph structure, aggregating and updating the sample global features, the first edge features, and the second edge features corresponding to the global node to obtain global aggregate features; for each local node in the initial graph structure, aggregating and updating the sample local features and the second edge features corresponding to the local node to obtain local aggregate features; averaging the global aggregate features and the local aggregate features corresponding to each category to obtain the aggregate features of the class prototype.

[0012] The embodiment of the present application also provides an image category prediction device, including: an image acquisition and cropping module, used to acquire an image to be processed, and randomly crop the image to be processed to obtain a local image; an image feature extraction module, used to extract global features in the image to be processed and local features in the local image respectively; an aggregate feature acquisition module, used to aggregate global features and local features using a graph neural network to obtain aggregate features of the image to be processed; an image category prediction module, used to predict the category of the image to be processed based on the similarity between the aggregate features of the image to be processed and the class prototype aggregate features in the graph neural network, where the class prototype aggregate features are obtained by training sample images corresponding to known category labels using the graph neural network.

[0013] Optionally, in an embodiment of the present application, the aggregate feature acquisition module includes: a processing node acquisition submodule, used to obtain the global nodes to be processed in the graph neural network, and the local nodes to be processed in the graph neural network; a node feature extraction submodule, used to extract the edge features to be processed between the global nodes to be processed and the local nodes to be processed; a feature aggregation processing submodule, used to perform aggregation processing on the global nodes to be processed, the local nodes to be processed and the edge features to be processed to obtain the aggregate features of the image to be processed.

[0014] Optionally, in an embodiment of the present application, the image category prediction device also includes: an image label acquisition module, used to acquire multiple sample images and multiple sample labels, the sample labels including category labels of the sample images; an initial structure construction module, used to construct an initial graph structure using multiple sample images and multiple sample labels; a neural network training module, used to train the initial graph structure using multiple sample images as training data and multiple sample labels as training labels to obtain a graph neural network.

[0015] Optionally, in an embodiment of the present application, the initial structure construction module includes: an image random cropping sub-module, used to randomly crop each sample image in a plurality of sample images to obtain a plurality of sample local images; a sample feature extraction sub-module, used to respectively extract sample global features in a plurality of sample images and sample local features in a plurality of sample local images; and a feature initial construction sub-module, used to construct an initial graph structure using sample global features in a plurality of sample images and sample local features in a plurality of sample local images.

[0016] Optionally, in an embodiment of the present application, the feature initial construction submodule includes: a fully connected graph acquisition unit, which is used to use the sample global features in multiple sample images as global nodes, and interconnect the global nodes corresponding to the multiple sample images to obtain a fully connected graph; an initial structure acquisition unit, which is used to use the sample local features in multiple sample local images as local nodes, and interconnect the local nodes with the corresponding global nodes in the fully connected graph to obtain an initial graph structure.

[0017] Optionally, in an embodiment of the present application, the neural network training module includes: an aggregate feature acquisition submodule, which is used to aggregate and average the initial graph structure to obtain the aggregate features of the class prototype; a classification loss calculation submodule, which is used to extract the aggregate features of the sample image using the initial graph structure, and calculate the classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype; an edge loss value calculation submodule, which is used to extract the edge features of the sample image using the initial graph structure, and calculate the edge loss value between the edge features of the sample image and the edge features calculated by the sample label; an initial structure training submodule, which is used to train the initial graph structure according to the classification loss value and the edge loss value.

[0018] Optionally, in an embodiment of the present application, an aggregate feature acquisition submodule includes: a node edge feature calculation unit, which is used to respectively calculate the first edge features between global nodes and global nodes in the initial graph structure, and the second edge features between global nodes and local nodes in the initial graph structure; a global feature acquisition unit, which is used to aggregate and update the sample global features, first edge features and second edge features corresponding to each global node in the initial graph structure to obtain a global aggregate feature; a local feature acquisition unit, which is used to aggregate and update the sample local features and second edge features corresponding to each local node in the initial graph structure to obtain a local aggregate feature; an aggregate feature acquisition unit, which is used to average the global aggregate features and local aggregate features corresponding to each category to obtain the aggregate features of the class prototype.

[0019] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0020] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of a flow chart of an image category prediction method provided by an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of the structure of a graph neural network provided in an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the process of constructing and training a graph neural network provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of the structure of an image category prediction device provided in an embodiment of the present application is shown;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments in the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed for protection, but merely represents the selected embodiments in the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the embodiments of the present application.

[0028] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference.

[0029] Before introducing the image category prediction method provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0030] Metric Learning refers to a method of training a network model in which the input image data is projected into a metric space using the neural network model during the training process, and then the class similarity probability between images is determined by comparing the distances between the images in the metric space. Finally, the class similarity probability is used to predict the class of the image.

[0031] Graph Neural Network refers to a framework that mainly uses a domain aggregation framework for representation learning. The neighborhood aggregation framework is a framework that calculates node features by recursively aggregating and transforming the features of adjacent nodes.

[0032] It should be noted that the image category prediction method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer and a UNIX server.

[0033] The following introduces application scenarios in which the image category prediction method is applicable. The application scenarios here include but are not limited to: when using a learning method based on metric learning to predict image categories for an image classification model, the image category prediction method can be used to allow the image classification model to take into account both global features and local features, and determine important local features in the category prediction process based on the correlation between global features and local features, effectively avoiding interference from local areas such as background areas in the image on image classification, thereby improving the accuracy of category prediction for the image.

[0034] See also Figure 1 The flowchart of the image category prediction method provided by the embodiment of the present application is shown; the main idea of ​​the image category prediction method is to allow the image classification model to take into account both global features and local features, and determine the important local features in the category prediction process according to the correlation between the global features and the local features, so that the image classification model automatically predicts the category of the image according to these important local features. The above-mentioned image category prediction method may include:

[0035] Step S110: obtaining an image to be processed, and randomly cropping the image to be processed to obtain a local image.

[0036] The implementation of the above step S110 is, for example: the image to be processed may be received from other devices, or obtained from a file or database, etc. After obtaining the image to be processed, the Open Source Computer Vision Library (OpenCV) is used to randomly crop the image to be processed to obtain a partial image.

[0037] Step S120: extracting the global features in the image to be processed and the local features in the local image respectively.

[0038] The implementation of the above step S120 is, for example, to use a feature extraction network to extract features from the image to be processed to obtain global features, and use the feature extraction network to extract features from the local image to obtain local features. The feature extraction networks that can be used here include but are not limited to: ResNet (such as Resnet-12), DarkNet, DenseNetLeNet, AlexNet, VGG, GoogLeNet, etc.

[0039] Step S130: Use a graph neural network to aggregate global features and local features to obtain aggregated features of the image to be processed.

[0040] See also Figure 2A structural schematic diagram of a graph neural network provided by an embodiment of the present application is shown; the solid circles in the figure represent global feature corresponding nodes in the graph neural network, the dotted circles represent global feature corresponding nodes and local feature corresponding nodes obtained after processing the image to be processed, the boxes between the global feature corresponding nodes and the global feature corresponding nodes represent the first edge features between the two, the boxes between the global feature corresponding nodes and the local feature corresponding nodes represent the second edge features between the two, the one-way arrow from the box to the dotted circle represents an extraction operation, and the one-way arrow from the image to be processed to the local image represents a random cropping operation.

[0041] Step S140: predicting the category of the image to be processed based on the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network, where the class prototype aggregated features are obtained by training sample images corresponding to known category labels using the graph neural network.

[0042] For example, the implementation of the above step S140 is as follows: First, the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network can be calculated using cosine similarity, and the formula The similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network is calculated to obtain the predicted category of the image to be processed. Among them, p(y=c|x) represents the predicted category probability of the image to be processed. When the category probability is greater than the preset threshold, it is determined to be the category of the predicted image to be processed. Otherwise, it is determined not to be the category of the predicted image to be processed. The class prototype aggregated features are obtained by training sample images corresponding to known category labels using the graph neural network.

[0043] In the implementation process of the above scheme, the global features corresponding to the image to be processed and the local features of the local image cropped from the image to be processed are aggregated by using a graph neural network to obtain the aggregated features of the image to be processed, and the category of the image to be processed is predicted based on the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network. Therefore, both the global features corresponding to the image to be processed and the local features of the local image cropped from the image to be processed are taken into account. The graph neural network can effectively determine the important local features in the category prediction process based on the correlation between global features and local features, effectively avoiding the interference of local background areas in the image on image classification, thereby improving the accuracy of category prediction for images.

[0044] As an optional implementation of step S130, the edge features between nodes may be used to obtain the aggregate features of the image to be processed. This implementation may include:

[0045] Step S131: Obtain the global nodes to be processed in the graph neural network with global features, and the local nodes to be processed in the graph neural network with local features.

[0046] Step S132: extracting the edge features to be processed between the global node to be processed and the local node to be processed.

[0047] For example, the implementation of step S132 can calculate the feature similarity between the global node to be processed corresponding to the sample image and all the connected nodes in the initial graph structure, and use the feature similarity between all the connected nodes as the extracted edge features to be processed. Here, all the connected nodes include: global nodes and local nodes; for any node x1, x2 in the graph neural network, the formula can be used Calculate the feature similarity between the global node corresponding to the sample image and the connected global node. Similarly, the formula can be used Calculate the feature similarity between the global node corresponding to the sample image and the connected local node; where, represents the i-th layer feature similarity (i.e. edge feature) between the global node corresponding to the sample image and the connected global node, Represents the i-th layer feature similarity between the global node corresponding to the sample image and the connected local node, It is a convolutional network (for example: LeNet network model, AlexNet network model, VGG network model, GoogLeNet network model and ResNet network model, etc.), which can transform feature similarity to obtain edge features between global features. represents the global node corresponding to the sample image, represents the global node to which the global node corresponding to the sample image is connected, Indicates the local nodes connected to the global nodes corresponding to the sample image.

[0048] Step S133: Aggregate the features of the global nodes to be processed, the local nodes to be processed, and the edge features to be processed to obtain aggregate features of the image to be processed.

[0049] The implementation method of the above step S133 is, for example, to use the formula Aggregate the features of the global nodes to be processed, the local nodes to be processed, and the edge features to be processed to obtain aggregate features of the image to be processed; wherein, represents the aggregated features of the global nodes in the i-th layer of the graph neural network, S L (x) represents the set of local nodes corresponding to the global node, represents the similarity of the i-th layer features (i.e., edge features) between the global node x and the j-th global node it is connected to, represents the jth global node connected to the global node at the i-1th layer, represents the similarity of the i-th layer features (i.e., edge features) between the global node x and the j-th local node it is connected to, Represents the jth local node connected to the global node in the i-1th layer.

[0050] See also Figure 3 A schematic diagram of the process of constructing and training a graph neural network provided in an embodiment of the present application is shown; as an optional implementation, before using the graph neural network, a graph neural network may also be constructed and trained, and the construction and training process may include:

[0051] Step S210: Acquire multiple sample images and multiple sample labels, where the sample labels include category labels of the sample images.

[0052] It is understandable that the above-mentioned sample images and sample labels can be obtained separately, for example: manually collecting sample images and manually identifying sample labels of sample images; of course, sample images and sample labels can also be packaged as a training data set and obtained together, and here we take the acquisition of the training data set together as an example for explanation.

[0053] The implementation methods of the above-mentioned step S210 are, for example: a first acquisition method, receiving a training data set sent by other terminal devices, and storing the training data set in a file system, a database or a mobile storage device; a second acquisition method, obtaining a pre-stored training data set, specifically, for example: obtaining a training data set from a file system, a database or a mobile storage device; a third acquisition method, using a browser or other software to obtain a training data set on the Internet, or using other applications to access the Internet to obtain a training data set.

[0054] In the specific practice, multiple sample images and multiple sample labels can be used as a training data set, then the training data set can be divided into a training set, a support set and a query set; wherein, the training set is mainly used to construct the initial graph structure, and pre-train the initial graph structure to obtain the pre-trained graph structure. The support set is mainly used to train the pre-trained graph structure to obtain the graph neural network. The query set is mainly used to verify the accuracy, precision and recall rate of the graph neural network and other indicators. The specific process of dividing the training data set into the training set, support set and query set is as follows: randomly select n categories from the training set, and sample k images from each of the n categories as the support set of the current task. The support set here can be consistent with the image structure distribution of the support set in the data set; then, in the n categories selected above, sample q images from each category as the query set of the current task.

[0055] Step S220: construct an initial graph structure using multiple sample images and multiple sample labels.

[0056] As an optional implementation of step S220, the specific process of constructing the initial graph structure using the sample images and sample labels may include:

[0057] Step S221: randomly crop each sample image among the multiple sample images to obtain multiple sample partial images.

[0058] The implementation method of the above step S221 is, for example: using an open source computer vision library (OpenCV) to randomly crop each sample image in the multiple sample images to obtain multiple sample partial images.

[0059] Step S222: extracting sample global features from a plurality of sample images and sample local features from a plurality of sample local images respectively.

[0060] The implementation method of the above step S222 is, for example: using a feature extraction network to extract features from multiple sample images respectively to obtain sample global features in the multiple sample images, and using the feature extraction network to extract features from multiple sample local images respectively to obtain sample local features in the multiple sample local images. The feature extraction networks that can be used here include but are not limited to: ResNet (such as Resnet-12), DarkNet, DenseNetLeNet, AlexNet, VGG, GoogLeNet, etc.

[0061] Step S223: construct an initial graph structure using sample global features in multiple sample images and sample local features in multiple sample local images.

[0062] The specific process of the implementation of the above step S223 may include:

[0063] Step S223a: taking the sample global features in the multiple sample images as global nodes, and connecting the global nodes corresponding to the multiple sample images to obtain a fully connected graph.

[0064] In the specific practice process, in addition to constructing a fully connected graph based only on the global features of the samples as global nodes, we can also use the global features of the samples as global nodes and use the distribution information of known category labels and global features as auxiliary information of the fully connected graph, so that the constructed fully connected graph directly connects the nodes with strong correlation, and indirectly connects the nodes with weaker correlation through auxiliary information. The auxiliary information here can be understood as the sample global features in the graph structure as the propagation characteristic information between global nodes.

[0065] Step S223b: taking sample local features in multiple sample local images as local nodes, and connecting the local nodes with corresponding global nodes in the fully connected graph to obtain an initial graph structure.

[0066] The implementation of the above steps S223a to S223b is, for example: connecting the global nodes corresponding to the multiple sample images to obtain a fully connected graph, and connecting the local node to the corresponding global node in the fully connected graph. The obtained initial graph structure can be expressed as Among them, G0 represents the graph structure obtained by the aggregation update of the 0th layer (i.e., the initial graph structure), represents the global node set at level 0, represents the local node set of layer 0, represents the set of edges between global nodes at level 0, Represents the set of edges between local nodes at level 0.

[0067] Step S230: Using multiple sample images as training data and multiple sample labels as training labels, train the initial graph structure to obtain a graph neural network.

[0068] It is understandable that the initial graph structure obtained After that, each layer of the graph neural network can be The initial graph structure obtained by calculating the graph structure of the previous layer can be expressed as For the initial graph structure After aggregation and averaging, the graph neural network can include n-layer graph structures, and the graph structure obtained by aggregating and updating the i-th layer of the initial graph structure can be expressed as Among them, G i represents the graph structure obtained by the aggregation update of the i-th layer, i∈[0,n], represents the global node set of the i-th layer, represents the local node set of the i-th layer, represents the set of edges between global nodes at the i-th layer, Represents the set of edges between local nodes in the i-th layer.

[0069] As an optional implementation of step S230, the specific process of training the initial graph structure may include:

[0070] Step S231: Aggregate and average the initial graph structure to obtain aggregate features of the class prototype.

[0071] As an optional implementation of step S231, the specific process of aggregating and averaging the initial graph structure may include:

[0072] Step S231a: respectively calculating first edge features between global nodes and global nodes in the initial graph structure, and second edge features between global nodes and local nodes in the initial graph structure.

[0073] The implementation method of the above step S231a is, for example, to use the formula Calculate the first edge feature between the global nodes in the initial graph structure and use the formula Calculate the second edge feature between the global node and the local node in the initial graph structure; where, represents the feature similarity between the global node and the connected global nodes (i.e., edge features), Represents the feature similarity between the global node and the connected local nodes, It is a convolutional network that can transform feature similarity to obtain edge features between global features. represents the global node, represents the global nodes to which the global node is connected, Represents the local nodes to which the global node is connected.

[0074] Step S231b: For each global node in the initial graph structure, aggregate and perform feature conversion updates on the sample global features, first edge features, and second edge features corresponding to the global node to obtain global aggregate features.

[0075] Step S231c: For each local node in the initial graph structure, aggregate and update the sample local features and the second edge features corresponding to the local node to obtain local aggregate features.

[0076] The implementation of the above steps S231b to S231c is, for example, to use the formula Aggregate the sample global features, first edge features, and second edge features corresponding to the global node to obtain the global aggregate features. Similarly, the formula can be used Aggregate the sample local features and second edge features corresponding to the local node to obtain local aggregate features; where: represents the global aggregation feature, represents the local aggregation feature, S L (x) represents the set of local nodes corresponding to the global node, Represents the first edge feature between the global node x corresponding to the sample image and the jth global node connected to it, represents the jth global node connected to the global node corresponding to the sample image, Represents the second edge feature between the global node x corresponding to the sample image and the jth local node connected to it, Represents the jth local node connected to the global node corresponding to the sample image.

[0077] After obtaining the global aggregation features and the local aggregation features, the global aggregation features can be aggregated multiple times (for example, three times) to obtain the global features after the global aggregation features are aggregated three times. And use the formula The global features after three aggregations are updated through feature transformation to obtain updated global aggregation features; represents the updated global aggregation feature, Represent the global features after three aggregations, is a convolutional network. is a concatenation operation. Similarly, the local aggregation features can be aggregated multiple times (for example, three times) to obtain the local features after the local aggregation features are aggregated three times. And use the formula The local features after three aggregations are updated by feature transformation to obtain updated local aggregation features; represents the updated all-local aggregation features, Represent the local features after three aggregations, is a convolutional network. is a concatenation operation. Aggregate and update in the above manner in turn to obtain the final aggregate feature E of the global node G (x).

[0078] Step S231d: average the global aggregation features and local aggregation features corresponding to each category to obtain the aggregation features of the class prototype.

[0079] The implementation method of the above step S231d is, for example, to use the formula and The global aggregation features and local aggregation features corresponding to each category are averaged to obtain the aggregation features of the class prototype; where E w (c) represents the aggregated features of the class prototype, S L (x) represents the set of local nodes corresponding to the global node, E G (x) represents the final aggregated feature of the global node, Indicates the corresponding features of the global node x connected to the global node in the nth layer (i.e. the last layer), represents the n-th layer feature similarity (i.e. edge feature) between the global node x and the j-th local node it is connected to, Indicates the jth local node connected to the global node in the nth layer.

[0080] Step S232: extract edge features of the sample image using the initial graph structure.

[0081] For example, the implementation of step S232 can calculate the feature similarity between the global node corresponding to the sample image and all the connected nodes in the initial graph structure, and use the feature similarity between all the connected nodes as the edge feature of the extracted sample image. Here, all the connected nodes include: global nodes and local nodes; for any node x1, x2 in the graph neural network, the formula can be used Calculate the feature similarity between the global node corresponding to the sample image and the connected global node. Similarly, the formula can be used Calculate the feature similarity between the global node corresponding to the sample image and the connected local node; where, represents the i-th layer feature similarity (i.e. edge feature) between the global node corresponding to the sample image and the connected global node, Represents the i-th layer feature similarity between the global node corresponding to the sample image and the connected local node, It is a convolutional network that can transform feature similarity to obtain edge features between global features. represents the global node corresponding to the sample image, represents the global node to which the global node corresponding to the sample image is connected, Indicates the local nodes connected to the global nodes corresponding to the sample image.

[0082] Step S233: Calculate the edge loss value between the edge feature of the sample image and the edge feature calculated by the sample label.

[0083] The implementation of the above step S233 is as follows: It can be understood that the formula Calculate the edge loss value between the edge features of the sample image and the edge features calculated by the sample label; where, Represents the edge loss value between the edge features of the sample image and the edge features calculated by the sample label, represents the edge feature between node x1 and node x2, δ(a=b) is the Kronecker function, if (a=b) is true, then δ(a=b) is equal to 1, if (a=b) is false, then δ(a=b) is equal to 0.

[0084] Step S234: extracting aggregate features of the sample image using the initial graph structure.

[0085] For example, the implementation of the above step S234 is as follows: the global node of the i-th layer of the graph neural network can use the initial graph structure to extract the aggregate features of the sample image layer by layer, and the calculation process of the i-th layer is used as an example to illustrate, including: global nodes and local nodes; the formula can be used Extract the aggregate features of all global nodes corresponding to the sample image; among them, represents the aggregated features of the global nodes in the i-th layer of the graph neural network, S L (x) represents the set of local nodes corresponding to the global node, represents the similarity of the i-th layer features (i.e., edge features) between the global node x and the j-th global node it is connected to, represents the jth global node connected to the global node at the i-1th layer, represents the similarity of the i-th layer features (i.e., edge features) between the global node x and the j-th local node it is connected to, represents the jth local node connected to the global node at the i-1th layer. Similarly, the formula can be used Extract the aggregate features of all local nodes corresponding to the sample image; among them, represents the aggregated features of the local node x of the sample image in the i-th layer of the graph neural network, S G (x) represents the global node set corresponding to the local node x in the i-th layer of the graph neural network, represents the similarity of the i-th layer features (i.e., edge features) between the global node x and the j-th local node it is connected to, Indicates the jth global node connected to the global node corresponding to the sample image in the i-1th layer.

[0086] Step S235: Calculate the classification loss value between the aggregated features of the sample image and the aggregated features of the class prototype.

[0087] In the above step S235, the formula is used Calculate the classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype, that is, first use the cosine distance to measure the similarity between the aggregate features of each sample image and the aggregate features of the prototype, and then use the similarity to calculate the classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype. represents the classification loss value, cos is the cosine similarity, E w (x) is the aggregated feature of the sample image, E w (y) is the aggregate feature of the y-th class prototype, E w (k) is the aggregate feature of the k-th class prototype.

[0088] Step S236: Train the initial graph structure according to the classification loss value and the edge loss value until the total loss value of the sum of the classification loss value and the edge loss value is less than a preset threshold, thereby obtaining a trained graph neural network.

[0089] The implementation method of the above step S236 is as follows: first use the formula Calculate the total loss value of the sum of the classification loss value and the edge loss value, where represents the total loss value, λ c and λ e are the weight parameters of classification loss and edge loss, respectively. represents the classification loss value, It can be understood that the above total loss value is less than the preset threshold value, which means that the total loss value no longer drops sharply, that is, the total loss value tends to be stable, and the trained graph neural network can be obtained.

[0090] See also Figure 4 The structure diagram of the image category prediction device provided by the embodiment of the present application is shown; the embodiment of the present application provides an image category prediction device 300, including:

[0091] The image acquisition and cropping module 310 is used to acquire the image to be processed and randomly crop the image to be processed to obtain a partial image.

[0092] The image feature extraction module 320 is used to extract global features in the image to be processed and local features in the local image respectively.

[0093] The aggregated feature acquisition module 330 is used to aggregate global features and local features using a graph neural network to obtain aggregated features of the image to be processed.

[0094] The image category prediction module 340 is used to predict the category of the image to be processed based on the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network. The class prototype aggregated features are obtained by training sample images corresponding to known category labels using the graph neural network.

[0095] Optionally, in the embodiment of the present application, the aggregation feature acquisition module includes:

[0096] The processing node acquisition submodule is used to obtain the global nodes to be processed in the graph neural network with global features, and the local nodes to be processed in the graph neural network with local features.

[0097] The node feature extraction submodule is used to extract the edge features to be processed between the global node to be processed and the local node to be processed.

[0098] The feature aggregation processing submodule is used to aggregate the features of the global nodes to be processed, the local nodes to be processed and the edge features to be processed to obtain the aggregated features of the image to be processed.

[0099] Optionally, in the embodiment of the present application, the image category prediction device further includes:

[0100] The image label acquisition module is used to acquire multiple sample images and multiple sample labels, where the sample labels include category labels of the sample images.

[0101] The initial structure building module is used to build an initial graph structure using multiple sample images and multiple sample labels.

[0102] The neural network training module is used to train the initial graph structure using multiple sample images as training data and multiple sample labels as training labels to obtain a graph neural network.

[0103] Optionally, in an embodiment of the present application, the initial structure building module includes:

[0104] The image random cropping submodule is used to randomly crop each sample image among multiple sample images to obtain multiple sample local images.

[0105] The sample feature extraction submodule is used to extract the sample global features in a plurality of sample images and the sample local features in a plurality of sample local images respectively.

[0106] The feature initial construction submodule is used to construct an initial graph structure using sample global features in multiple sample images and sample local features in multiple sample local images.

[0107] Optionally, in the embodiment of the present application, the feature initial construction submodule includes:

[0108] The fully connected graph obtaining unit is used to use the sample global features in multiple sample images as global nodes, and connect the global nodes corresponding to the multiple sample images to obtain a fully connected graph.

[0109] The initial structure obtaining unit is used to use the sample local features in multiple sample local images as local nodes, and connect the local nodes with the corresponding global nodes in the fully connected graph to obtain the initial graph structure.

[0110] Optionally, in an embodiment of the present application, the neural network training module includes:

[0111] The aggregate feature acquisition submodule is used to aggregate and average the initial graph structure to obtain the aggregate features of the class prototype.

[0112] The classification loss calculation submodule is used to extract the aggregate features of the sample image using the initial graph structure, and calculate the classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype.

[0113] The edge loss value calculation submodule is used to extract the edge features of the sample image using the initial graph structure, and calculate the edge loss value between the edge features of the sample image and the edge features calculated by the sample label.

[0114] The initial structure training submodule is used to train the initial graph structure according to the classification loss value and the edge loss value.

[0115] Optionally, in the embodiment of the present application, the aggregation feature acquisition submodule includes:

[0116] The node edge feature calculation unit is used to calculate the first edge features between global nodes and global nodes in the initial graph structure, and the second edge features between global nodes and local nodes in the initial graph structure.

[0117] The global feature acquisition unit is used to aggregate and perform feature conversion updates on the sample global features, first edge features and second edge features corresponding to each global node in the initial graph structure to obtain a global aggregate feature.

[0118] The local feature acquisition unit is used to aggregate and perform feature conversion updates on the sample local features and second edge features corresponding to each local node in the initial graph structure to obtain local aggregated features.

[0119] The aggregate feature obtaining unit is used to average the global aggregate features and local aggregate features corresponding to each category to obtain the aggregate features of the class prototype.

[0120] It should be understood that the device corresponds to the above-mentioned image category prediction method embodiment and can perform the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be referred to in the above description. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0121] See also Figure 5 An electronic device 400 provided in an embodiment of the present application includes: a processor 410 and a memory 420, wherein the memory 420 stores machine-readable instructions executable by the processor 410, and when the machine-readable instructions are executed by the processor 410, the above method is executed.

[0122] The embodiment of the present application further provides a computer-readable storage medium 430 , on which a computer program is stored. When the computer program is executed by the processor 410 , the above method is executed.

[0123] Among them, the computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0124] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0126] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily have to be directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradicting each other.

[0127] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0128] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for predicting image categories, characterized in that: include: Acquire an image to be processed, and randomly crop the image to be processed to obtain a local image; Respectively extracting global features in the image to be processed and local features in the local image; Aggregating the global features and the local features using a graph neural network to obtain aggregated features of the image to be processed; Predicting the category of the image to be processed according to the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network, wherein the class prototype aggregated features are obtained by training the sample images corresponding to the known class labels using the graph neural network; The method of using a graph neural network to aggregate the global features and the local features to obtain aggregate features of the image to be processed includes: obtaining the global nodes to be processed of the global features in the graph neural network and the local nodes to be processed of the local features in the graph neural network; extracting the edge features to be processed between the global nodes to be processed and the local nodes to be processed; and aggregating the global nodes to be processed, the local nodes to be processed and the edge features to be processed to obtain aggregate features of the image to be processed. The method for obtaining the prototype-like aggregation feature comprises the following steps: respectively calculating the first edge feature between the global nodes and the global nodes in the initial graph structure, and the second edge feature between the global nodes and the local nodes in the initial graph structure; for each global node in the initial graph structure, using the formula The sample global features, the first edge features, and the second edge features corresponding to the global node are aggregated and feature transformed and updated to obtain global aggregate features; for each local node in the initial graph structure, the formula is used Aggregate and update the sample local features and second edge features corresponding to the local node to obtain local aggregate features; average the global aggregate features and the local aggregate features corresponding to each category to obtain the class prototype aggregate features; wherein, represents the set of global nodes, represents the global aggregation feature, Represents the set of local nodes corresponding to the global node, Represents the global node corresponding to the sample image x The connected j The first edge feature between global nodes, Indicates the first node connected to the global node corresponding to the sample image j A global node, Represents the global node corresponding to the sample image x The connected j The second edge feature between local nodes, Indicates the first node connected to the global node corresponding to the sample image j local nodes, represents the local aggregation feature, Represents the global node set corresponding to the local node x.

2. The method according to claim 1, characterized in that Before the graph neural network is used to aggregate the global features and the local features, the method further includes: Acquire a plurality of sample images and a plurality of sample labels, wherein the sample labels include category labels of the sample images; constructing an initial graph structure using the multiple sample images and the multiple sample labels; The initial graph structure is trained using the multiple sample images as training data and the multiple sample labels as training labels to obtain the graph neural network.

3. The method according to claim 2, characterized in that The constructing an initial graph structure using the multiple sample images and the multiple sample labels includes: Randomly cropping each sample image among the multiple sample images to obtain multiple sample partial images; Respectively extracting sample global features from the multiple sample images and sample local features from the multiple sample local images; The initial graph structure is constructed using sample global features in the multiple sample images and sample local features in the multiple sample local images.

4. The method according to claim 3, characterized in that The constructing the initial graph structure using the sample global features in the multiple sample images and the sample local features in the multiple sample local images comprises: Taking the sample global features in the multiple sample images as global nodes, and connecting the global nodes corresponding to the multiple sample images to obtain a fully connected graph; The sample local features in the multiple sample local images are used as local nodes, and the local nodes are connected to corresponding global nodes in the fully connected graph to obtain the initial graph structure.

5. The method according to claim 2, characterized in that: The training of the initial graph structure includes: Aggregating and averaging the initial graph structure to obtain the prototype-like aggregate features; Extracting aggregate features of the sample image using the initial graph structure, and calculating a classification loss value between the aggregate features of the sample image and the aggregate features of the class prototype; Extracting edge features of the sample image using the initial graph structure, and calculating edge loss values ​​between the edge features of the sample image and the edge features calculated by the sample label; The initial graph structure is trained according to the classification loss value and the edge loss value.

6. An image category prediction device, characterized in that: include: An image acquisition and cropping module is used to acquire an image to be processed and randomly crop the image to be processed to obtain a partial image; An image feature extraction module, used to extract global features in the image to be processed and local features in the local image respectively; An aggregated feature acquisition module, used to aggregate the global features and the local features using a graph neural network to obtain aggregated features of the image to be processed; An image category prediction module, used to predict the category of the image to be processed based on the similarity between the aggregated features of the image to be processed and the class prototype aggregated features in the graph neural network, wherein the class prototype aggregated features are obtained by training the sample images corresponding to the known category labels using the graph neural network; The method of using a graph neural network to aggregate the global features and the local features to obtain aggregate features of the image to be processed includes: obtaining the global nodes to be processed of the global features in the graph neural network and the local nodes to be processed of the local features in the graph neural network; extracting the edge features to be processed between the global nodes to be processed and the local nodes to be processed; and aggregating the global nodes to be processed, the local nodes to be processed and the edge features to be processed to obtain aggregate features of the image to be processed. The method for obtaining the prototype-like aggregation feature comprises the following steps: respectively calculating the first edge feature between the global nodes and the global nodes in the initial graph structure, and the second edge feature between the global nodes and the local nodes in the initial graph structure; for each global node in the initial graph structure, using the formula The sample global features, the first edge features, and the second edge features corresponding to the global node are aggregated and feature transformed and updated to obtain global aggregate features; for each local node in the initial graph structure, the formula is used Aggregate and update the sample local features and second edge features corresponding to the local node to obtain local aggregate features; average the global aggregate features and the local aggregate features corresponding to each category to obtain the class prototype aggregate features; wherein, represents the set of global nodes, represents the global aggregation feature, Represents the set of local nodes corresponding to the global node, Represents the global node corresponding to the sample image x The connected j The first edge feature between global nodes, Indicates the first node connected to the global node corresponding to the sample image j A global node, Represents the global node corresponding to the sample image x The connected j The second edge feature between local nodes, Indicates the first node connected to the global node corresponding to the sample image j local nodes, represents the local aggregation feature, Represents the global node set corresponding to the local node x.

7. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is executed.

Images