Image classification method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an image classification method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a first image; performing feature extraction processing on the first image through N convolutional layers of the first model to obtain an image feature vector of the first image, the image feature vector comprising N layers of feature information, and each layer of feature information comprising feature information extracted through at least two convolution kernels of different sizes in each convolutional layer; determining a scene category to which the first image belongs based on the image feature vector; wherein N is an integer greater than 1.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image classification method, device, electronic device, storage medium and program product. Background Art

[0002] At present, in order to facilitate the management and search of multiple images stored in the album application, the electronic device can usually classify and store the multiple images, so that the user can find the images he needs.

[0003] In the related technology, the user can input the scene classification folder control in the album application so that the electronic device can create a scene classification folder in the album application, and then the user can drag multiple images to the corresponding scene classification folders in turn, so that the electronic device can perform scene classification on multiple images in the album application.

[0004] However, since the above method requires users to manually create folders and manually add multiple images to corresponding scene classification folders, and when there are many images in the album application, it is difficult for users to avoid mistakes, the process of image classification in electronic devices is cumbersome and time-consuming, and the accuracy is poor. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image classification method, device, electronic device, storage medium and program product, which can improve the accuracy of image classification while improving the efficiency of image classification.

[0006] In a first aspect, an embodiment of the present application provides an image classification method, which includes: acquiring a first image; performing feature extraction processing on the first image through N convolutional layers of a first model to obtain an image feature vector of the first image, wherein the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolutional layer; based on the image feature vector, determining the scene category to which the first image belongs; wherein N is an integer greater than 1.

[0007] In a second aspect, an embodiment of the present application provides an image classification device, which includes: an acquisition module, a feature extraction module and a determination module. The acquisition module is used to acquire a first image. The feature extraction module is used to perform feature extraction processing on the first image acquired by the acquisition module through N layers of convolution layers of the first model to obtain an image feature vector of the first image, wherein the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolution layer. The determination module is used to determine the scene category to which the first image belongs based on the image feature vector; wherein N is an integer greater than 1.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device can extract feature information from the first image through multiple two convolution kernels of different sizes in each convolution layer of the N convolution layers of the first model, and then determine the scene category to which the first image belongs based on the image feature vector. In this solution, since image features of different scales can be extracted simultaneously through convolution kernels of different sizes in each convolution layer, the first model can capture more detailed information and global information, making scene recognition more accurate. In addition, the superposition of convolution operations can enable the first model to gradually extract higher-level semantic information at different levels, further improving the accuracy of recognition. Moreover, the first model is used to automatically classify images into scenes without the need for manual operation by the user. In this way, while improving the efficiency of scene classification by the electronic device, it also improves the accuracy of scene classification by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of an image classification method provided in an embodiment of the present application;

[0014] Figure 2 It is one of the example schematic diagrams of a first model provided in an embodiment of the present application;

[0015] Figure 3 This is a second example schematic diagram of a first model provided in an embodiment of the present application;

[0016] Figure 4 This is the second flowchart of an image classification method provided in an embodiment of the present application;

[0017] Figure 5 This is the third flowchart of an image classification method provided in an embodiment of the present application;

[0018] Figure 6 This is the fourth flowchart of an image classification method provided in an embodiment of the present application;

[0019] Figure 7 This is the fifth flowchart of an image classification method provided in an embodiment of the present application;

[0020] Figure 8 is a structural schematic diagram of an image classification device provided in an embodiment of the present application;

[0021] Fig. 9 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0022] Fig.10 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0026] The professional terms involved in the embodiments of the present application are explained below.

[0027] Convolution kernel: When processing an image, a convolution kernel is a weighted average of the pixel values ​​in a small area of ​​the input image, which becomes each corresponding pixel in the output image. The weights are defined by a function called a convolution kernel.

[0028] Pooling layer: The functions of the pooling layer mainly include downsampling, dimensionality reduction, removing redundant information, compressing features, simplifying network complexity, reducing computational effort, reducing memory consumption, achieving nonlinearity, expanding the perceptual field, and achieving invariance.

[0029] Specifically, the pooling layer can help us reduce the sensitivity of the convolutional layer to position and reduce the sensitivity to spatial downsampling representation. In addition, the pooling layer also helps to achieve spatial invariance, that is, even if the input image is slightly translated, the output of the model will not change much. In terms of preventing overfitting, since the pooling layer has no parameters that need to be learned, it can prevent overfitting to a certain extent during the training process. In addition, the use of the pooling algorithm, while reducing the width and height of the image, will also bring better performance improvements to the training and reasoning of the model.

[0030] Principal Component Analysis (PCA) algorithm: A commonly used data dimensionality reduction algorithm that aims to retain most of the variance of the data. PCA finds new coordinate axes by eigenvalue decomposition or singular value decomposition of the covariance matrix, which corresponds to the direction of maximum data variability. Dimensionality reduction helps to simplify data processing, improve computational efficiency, and remove noise. The two main implementation methods of PCA are based on eigenvalue decomposition of the covariance matrix and based on SVD decomposition of the covariance matrix. In PCA, choosing the appropriate number of principal components K is a key step, which is usually determined by the variance contribution rate or the amount of information retained.

[0031] Linear Discriminant Analysis (LDA) algorithm: also known as Fisher Linear Discriminant (FLD), is a classic algorithm for pattern recognition. It was introduced into the field of pattern recognition and artificial intelligence by Belhumeur in 1996. The basic idea of ​​linear discriminant analysis is to project high-dimensional pattern samples into the optimal discriminant vector space to achieve the effect of extracting classification information and compressing the dimension of feature space. After projection, the pattern samples are guaranteed to have the largest inter-class distance and the smallest intra-class distance in the new subspace, that is, the pattern has the best separability in the space. Therefore, it is an effective feature extraction method. Using this method can maximize the inter-class scatter matrix of the projected pattern samples, and at the same time minimize the intra-class scatter matrix. In other words, it can ensure that the projected pattern samples have the smallest intra-class distance and the largest inter-class distance in the new space, that is, the pattern has the best separability in the space.

[0032] Support Vector Machine (SVM) model: It is one of the most influential methods in supervised learning. The SVM model does not output probability, but only outputs categories.

[0033] K-nearest neighbor (k-NN): K-nearest neighbor learning is a commonly used supervised learning method. When K is 1, it is also called the nearest neighbor method, i.e. 1NN. The working mechanism of K-nearest neighbor is: given a test sample, find the K training samples closest to it in the training set based on a certain distance metric, and then make predictions based on the information of these K neighbors.

[0034] The image classification method, device, electronic device, storage medium and program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0035] In today's information society, with the popularity of smartphones and mobile devices, the amount of photo data taken and stored by users is increasing. How to efficiently and accurately classify and manage these photos has become an urgent problem to be solved. Traditional photo management methods usually rely on users to manually create albums and classification tags, or perform simple classification based on metadata such as time and geographic location. These methods are feasible for processing a small number of photos, but they are unable to cope with hundreds or even thousands of photos, and it is difficult to meet users' needs for efficient and accurate photo management.

[0036] In the prior art, methods for automatic classification based on image content have been applied, mainly including image retrieval and classification methods based on low-level features such as color, texture, shape, and image classification algorithms based on machine learning. However, these methods have the following major defects:

[0037] Traditional image classification methods based on low-level features such as color, texture, and shape cannot fully capture the high-level semantic information in the image. These methods can usually only handle simple image classification tasks and have poor classification effects on diverse images in complex scenes.

[0038] Early machine learning models, such as SVM and k-nearest neighbor (k-NN), have improved the accuracy of image classification to a certain extent, but their feature extraction process still relies on manually designed features and cannot automatically learn complex patterns and semantic information in images. In addition, the classification performance of these models shows problems such as low computational efficiency and insufficient classification accuracy when facing large-scale, high-dimensional image data.

[0039] In recent years, deep learning models, especially convolutional neural networks (CNNs), have made significant progress in the field of image classification. However, applying these models to photo classification tasks on mobile devices still faces some challenges. First, deep learning models usually require a lot of computing resources and storage space, making it difficult to run efficiently on mobile devices. Second, the training process of deep learning models requires a large amount of labeled data, which is very costly for ordinary users to obtain and label. In addition, existing deep learning models are mostly used for image classification in specific scenarios, and lack special optimization for automatic classification of mobile phone albums.

[0040] In the image classification method, device, electronic device, storage medium and program product provided in the embodiments of the present application, since image features of different scales can be extracted simultaneously through convolution kernels of different sizes in each convolution layer, the first model can capture more detailed information and global information, making scene recognition more accurate. In addition, the superposition of convolution operations can enable the first model to gradually extract higher-level semantic information at different levels, further improving the accuracy of recognition. Moreover, the first model is used to automatically classify images into scenes without the need for manual operation by the user. In this way, while improving the efficiency of scene classification by the electronic device, it also improves the accuracy of scene classification by the electronic device.

[0041] The image classification method provided in the embodiment of the present application may be executed by an image classification device, which may be an electronic device or a functional module in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0042] The present application embodiment provides an image classification method. Figure 1 FIG. 1 is a flowchart of an image classification method provided by an embodiment of the present application. Figure 1As shown, the image classification method provided in the embodiment of the present application may include the following steps 201 to 203.

[0043] Step 201: An electronic device acquires a first image.

[0044] Optionally, in an embodiment of the present application, the first image may be an image in any application in the electronic device; or, the first image may be an image captured in real time by the electronic device.

[0045] Also exemplarily, the first image may be a photo.

[0046] In the embodiment of the present application, the first image may include image content and image data.

[0047] Exemplarily, the above-mentioned image data may include at least one of the following: resolution, color temperature, size, shooting time and shooting location, etc.

[0048] Optionally, in the embodiment of the present application, the first image may be an image selected by a user or determined by the electronic device itself.

[0049] Exemplarily, the user may manually select the first image in an album application of the electronic device. For example, the user may select photos by time range, or by geographic location, or by album name.

[0050] As another example, the electronic device may determine all images in the photo album application as first images to perform scene classification processing on all images in the photo album application.

[0051] Also exemplarily, the first image may be one or more.

[0052] Optionally, in the embodiment of the present application, when there are multiple first images, the electronic device can acquire the first images in batches.

[0053] For example, the electronic device can perform a batch read operation on the photo data selected by the user, import the photos from the hard disk storage into the memory, and verify the format of the photos to ensure that the photos are in a supported image format. For unsupported formats, the electronic device can perform a format conversion operation. Then, the electronic device can check whether each photo is intact, including file size check and image integrity check, to ensure that each photo file can be read and processed normally.

[0054] For example, the image format may be any of the following: JPEG format, PNG format, or HEIC format.

[0055] For example, the above-mentioned image integrity check may include image content check and image data check.

[0056] Optionally, in the embodiment of the present application, the electronic device may perform image processing on the second image to obtain the first image.

[0057] Exemplarily, the above-mentioned image processing may include at least one of the following: denoising processing, color enhancement processing, and resolution adjustment processing, etc. The specific processing may be determined according to actual usage, and the embodiment of the present application does not limit this.

[0058] Optionally, in the embodiment of the present application, the second image may be an image selected by the user or determined by the electronic device itself.

[0059] Exemplarily, the user can manually select the second image in the album application of the electronic device. For example, the user can select photos according to a time range; or, select photos according to a geographical location; or select photos according to an album name.

[0060] Step 202: The electronic device performs feature extraction processing on the first image through N convolution layers of the first model to obtain an image feature vector of the first image; the image feature vector includes N layers of feature information, and each layer of feature information in the N layers of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolution layer; wherein N is an integer greater than 1.

[0061] In an embodiment of the present application, the above-mentioned first model is used to extract feature information in the first image.

[0062] Exemplarily, the first model may be a neural network model.

[0063] As another example, Figure 2 As shown, the above-mentioned first model may include an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, an activation function layer, a fully connected layer and an output layer.

[0064] Among them, the input layer is used to obtain the first image. The first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer, and the third pooling layer are used to perform feature extraction processing on the first image to obtain a feature vector. The activation function layer is used to perform nonlinear transformation processing on the feature vector to obtain a feature vector after nonlinear transformation processing. The fully connected layer is used to perform weighted fusion on the feature vector after nonlinear transformation processing to obtain a fused feature vector. The output layer is used to output the fused feature vector.

[0065] For example, taking the at least two convolution kernels (i.e., the multi-scale convolution kernels described below) as three as an example, the size of the first convolution kernel of the three convolution kernels may be 3×3, the size of the second convolution kernel of the three convolution kernels may be 5×5, and the size of the third convolution kernel of the three convolution kernels may be 7×7. The number and size of the convolution kernels may be determined according to actual usage requirements, and are not limited in the embodiments of the present application.

[0066] Optionally, in an embodiment of the present application, the number and size of multi-scale convolution kernels included in each convolution layer may be the same or different.

[0067] Again illustratively, in combination Figure 2 ,like Figure 3 As shown, the first convolution layer includes: a multi-scale convolution kernel and an activation function; the second convolution layer includes: a multi-scale convolution kernel and an activation function; the third convolution layer includes: a multi-scale convolution kernel and an activation function.

[0068] Optionally, in the embodiment of the present application, the above-mentioned characteristic information may be a characteristic graph or a characteristic value.

[0069] Step 203: The electronic device determines the scene category to which the first image belongs based on the image feature vector.

[0070] Optionally, in the embodiment of the present application, the above scene categories may include at least one of the following: running scene, movie watching scene, painting scene, and play scene, etc. The specifics may be determined according to actual use requirements, and the embodiment of the present application does not limit them.

[0071] In an embodiment of the present application, the electronic device can attach the scene category identifier to which the first image belongs to the first image, and then the electronic device can automatically classify the first image into a corresponding folder according to the scene category identifier, and the name of the folder corresponds to the scene category.

[0072] Exemplarily, the electronic device may update the data structure of the first image to {I, L}, where I represents the photo image content and L represents the classification label identifier.

[0073] As another example, the scene category identifier may include at least one of the following: a text identifier, a special symbol identifier, an English identifier, or a digital identifier, etc. The specific identifier may be determined according to actual use requirements, and the embodiment of the present application does not limit the identifier.

[0074] In an embodiment of the present application, after the electronic device updates the classification label of the above-mentioned image, the image can be automatically moved to a folder corresponding to the updated classification label.

[0075] In the image classification method provided in the embodiment of the present application, since image features of different scales can be extracted simultaneously through convolution kernels of different sizes in each convolution layer, the first model can capture more detailed information and global information, making scene recognition more accurate. In addition, the superposition of convolution operations can enable the first model to gradually extract higher-level semantic information at different levels, further improving the accuracy of recognition. Moreover, the first model is used to automatically classify images into scenes without the need for manual operation by the user. In this way, while improving the efficiency of scene classification by electronic devices, it also improves the accuracy of scene classification by electronic devices.

[0076] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 4 As shown, the above step 202 can be specifically implemented through the following steps 202a to 202c.

[0077] Step 202a: The electronic device performs feature extraction processing on the first image through at least two convolution kernels in the first convolution layer of the first model to obtain a feature vector corresponding to the first convolution layer.

[0078] In an embodiment of the present application, the electronic device may perform convolution processing on the first image in sequence through at least two convolution kernels in the first convolution layer to obtain a feature vector corresponding to the first convolution layer.

[0079] Optionally, in an embodiment of the present application, taking the at least two convolution kernels as three as an example, the electronic device may perform convolution processing on the first image through the first convolution kernel in the first convolution layer to obtain a first vector, and then perform convolution processing on the first vector through the second convolution kernel to obtain a second vector, and finally perform convolution processing on the second vector through the third convolution kernel to obtain a feature vector corresponding to the first convolution layer.

[0080] Optionally, in the embodiment of the present application, combined with Figure 4 ,like Figure 5 As shown, the above step 202a can be specifically implemented through the following steps 202a1 and 202a2.

[0081] Step 202a1: The electronic device calculates a feature vector corresponding to each convolution kernel in the first convolution layer based on weights and feature map values ​​corresponding to at least two convolution kernels in the first convolution layer.

[0082] In an embodiment of the present application, the above-mentioned feature map value is obtained based on the weighted sum of pixel values ​​of at least two image blocks, and each of the at least two image blocks is an image block corresponding to a convolution kernel in the first convolution layer in the first image.

[0083] It can be understood that the above image block is an image area corresponding to a convolution kernel in the first image.

[0084] Step 202a2: The electronic device performs feature fusion on the feature vector corresponding to each convolution kernel in the first convolution layer to obtain the feature vector corresponding to the first convolution layer.

[0085] In an embodiment of the present application, the electronic device can calculate the feature vector corresponding to each convolution kernel in the first convolution layer through the weights corresponding to at least two convolution kernels in the first convolution layer, the feature map values, the biases corresponding to at least two convolution kernels, and the activation function, and perform feature fusion to obtain the feature vector corresponding to the first convolution layer, which can be specifically implemented by the following formula 1.

[0086]

[0087] Among them, F l+1 is the feature vector corresponding to the first convolutional layer, W l.K (i, j) is the weight of the kth convolution kernel at the position (i, j) of the lth layer, F l (p+i,q+j) is the value of the feature map of the lth layer at the position (p+i,q+j), b l,K is the bias corresponding to the kth convolution kernel of the lth layer.

[0088] In the embodiment of the present application, the bias is a parameter in the convolution layer, which is used to add a fixed value to the result after the convolution operation to help the first model better fit the data.

[0089] In the embodiment of the present application, the electronic device performs convolution operation by using multi-scale convolution kernels. Convolution kernels of different sizes can capture features of different scales. The 3x3 convolution kernel is more suitable for capturing small local features, while the 7x7 convolution kernel is suitable for capturing larger structural features. In this way, the model can simultaneously obtain information of different sizes and scales in the image, thereby more accurately identifying the scene in the photo.

[0090] In the embodiment of the present application, the above scale refers to the feature size in the image. Different convolution kernel sizes (such as 3x3, 5x5, 7x7) can capture image features of different sizes.

[0091] For example, in a picture containing trees and buildings, a smaller convolution kernel (such as 3x3) will capture the details of the leaves, while a larger convolution kernel (such as 7x7) will capture the outline of the entire building. Through multi-scale convolution operations, the neural network can simultaneously obtain detailed features and global features in the image.

[0092] Step 202b: The electronic device performs feature extraction processing on the first feature vector through at least two convolution kernels in the j-th convolution layer of the first model to obtain a feature vector corresponding to the j-th convolution layer.

[0093] In the embodiment of the present application, the above-mentioned first feature vector is the feature vector corresponding to the j-1th convolutional layer, j∈[2,N].

[0094] In the embodiment of the present application, the electronic device can obtain the feature vector corresponding to the j-th convolutional layer through the above formula 1. For details, please refer to the above embodiment. To avoid repetition, it will not be repeated here.

[0095] It should be noted that the specific values ​​of the multiple convolution kernels and biases in each layer of the above N convolution layers are different. The specific values ​​of the multiple convolution kernels and biases in each layer can be adjusted according to specific training data so that each layer can capture different features. The above weights are learned through the training process. The training process of the convolutional neural network adjusts the weights through the back propagation algorithm so that the entire network can better classify or recognize images. The weight calculation is based on the gradient descent of the loss function and is updated in each training iteration to minimize the difference between the predicted results and the actual labels.

[0096] Step 202c: After obtaining the feature vector corresponding to the Nth convolutional layer, the electronic device performs feature fusion on the feature vectors corresponding to the Nth convolutional layer based on the feature fusion layer in the first model to obtain an image feature vector.

[0097] Exemplarily, the electronic device may perform feature fusion through a cascade operation of feature vectors to obtain an image feature vector, which may be specifically implemented through the following formula 2.

[0098]

[0099] Among them, V1 is the image feature vector, F l+1 is the feature vector corresponding to the first convolutional layer, F l+2 is the feature vector corresponding to the second convolutional layer, F l+n is the feature vector corresponding to the Nth convolutional layer, and concat represents the cascade operation of the feature vector. Represents a cascade operation.

[0100] In the embodiment of the present application, the fusion of feature vectors can enable the first model to obtain feature information at different levels. That is, the electronic device can obtain more detailed information and global information, making scene recognition more accurate and further improving the accuracy of recognition.

[0101] Optionally, in the embodiment of the present application, the image classification method provided in the embodiment of the present application further includes the following steps 301 to 303, and the above step 202a2 can be specifically implemented by the following step 304.

[0102] Step 301: After obtaining the feature vector corresponding to each convolution layer, the electronic device performs pooling processing on the feature vector corresponding to each convolution layer based on the pooling parameters in each pooling layer of the first model.

[0103] In the embodiment of the present application, the pooling parameters in each of the above-mentioned pooling layers are obtained based on the feature vector corresponding to the previous convolutional layer.

[0104] Optionally, in an embodiment of the present application, the above-mentioned pooling parameters may include at least one of the following: weight and bias of the convolution kernel.

[0105] In an embodiment of the present application, the above-mentioned pooling parameters are calculated through a training process using a back-propagation algorithm. The back-propagation algorithm updates the parameters of the model by calculating the gradient of the loss function. The model calculates the error between the output result and the actual result in each forward propagation, and then calculates the gradient through back-propagation. The parameters of the model are adjusted using the gradient descent method to minimize the loss function. The process is repeated many times until the model converges, that is, the error reaches the lowest point.

[0106] In the embodiment of the present application, the number of the above pooling layers corresponds to the number of the convolutional layers. That is, there are as many pooling layers as there are convolutional layers.

[0107] In an embodiment of the present application, after obtaining the feature vector corresponding to the first convolutional layer, the electronic device performs pooling processing on the feature vector corresponding to the first convolutional layer based on the pooling parameters in the first pooling layer of the first model.

[0108] Exemplarily, the first model may input the first vector into a first pooling layer, perform pooling processing on the first vector through the first pooling layer, and output a first pooled vector.

[0109] In an embodiment of the present application, the electronic device inputs the feature vector output by the first convolution layer into the jth pooling layer, performs pooling processing on the feature vector output by the first convolution layer through the jth pooling layer, and outputs a second pooling vector.

[0110] Exemplarily, the first model may input the first pooled vector into the jth convolution layer, perform convolution processing on the first pooled vector through the jth convolution layer, and output a fourth vector. Then, the fourth vector is input into the jth pooling layer, and the jth pooling layer performs pooling processing on the fourth vector and outputs a second pooled vector.

[0111] In an embodiment of the present application, the electronic device inputs the feature vector output by the Nth convolutional layer into the Nth pooling layer, performs pooling processing on the feature vector output by the Nth convolutional layer through the Nth pooling layer, and outputs a third pooling vector, j∈[2,N].

[0112] Exemplarily, the first model may input the second pooled vector into the Nth convolution layer, perform convolution processing on the third pooled vector through the Nth convolution layer, and output the fifth vector. Then, the fifth vector is input into the Nth pooling layer, and the Nth pooling layer performs pooling processing on the fourth vector and outputs the third pooled vector.

[0113] Optionally, in an embodiment of the present application, the above-mentioned pooling processing includes at least one of the following: downsampling, dimensionality reduction and removal of redundant information.

[0114] Exemplarily, taking the first model including 3 convolution layers and 3 pooling layers as an example, after the first model performs convolution processing on the first convolution layer to obtain the first vector, the first model can input the first vector into the first pooling layer, and the first pooling layer can perform pooling processing on the first vector to obtain the first pooled vector; then, the first model can input the first pooled vector into the first convolution layer for convolution processing to obtain the fourth vector; then, the first model can input the fourth vector into the second pooling layer, and the second pooling layer can perform pooling processing on the fourth vector to obtain the second pooled vector; then, the first model can input the second pooled vector into the third convolution layer for convolution processing to obtain the fifth vector; finally, the first model can input the fifth vector into the third pooling layer, and the third pooling layer can perform pooling processing on the fifth vector to obtain the third pooled vector.

[0115] In the embodiment of the present application, the electronic device can perform pooling processing on the feature vectors corresponding to each convolution layer through the adaptive pooling function and pooling parameters in each pooling layer. Specifically, it can be implemented through the following formula 3.

[0116]

[0117] Among them, P l is the pooling result of the lth layer, adaptive_pool is the adaptive pooling function, and α1 is the dynamically adjusted pooling parameter.

[0118] It should be noted that each of the N pooling layers above performs the same operation, and to avoid repetition, it will not be described again here.

[0119] In the embodiment of the present application, the pooling window size can be dynamically adjusted according to the image content through the adaptive pooling method, which can better retain important image details and improve the flexibility of the pooling operation.

[0120] Step 302: The electronic device determines a target activation function from at least one activation function of the activation function layer through the activation function layer of the first model based on feature vectors corresponding to the N convolutional layers after pooling processing.

[0121] Optionally, in the embodiment of the present application, the activation function may include at least one of the following: ReLU, Sigmoid, Tanh, etc. The specific activation function may be determined according to actual use requirements, and the embodiment of the present application does not limit it.

[0122] Step 303: The electronic device performs nonlinear transformation processing on the feature vectors corresponding to the N convolutional layers after the pooling processing based on the target activation function to obtain feature vectors corresponding to the N convolutional layers after the nonlinear transformation.

[0123] In the embodiment of the present application, the electronic device can determine the target activation function from at least one activation function of the activation function layer through the weight of at least one activation function and at least one activation function, and perform nonlinear transformation processing on the feature vectors corresponding to the N convolutional layers to obtain the feature vectors corresponding to the N convolutional layers after nonlinear transformation. This can be specifically implemented through the following formula 4.

[0124]

[0125] Among them, A l is the activation result corresponding to the lth convolutional layer, β l,j is the weight of the jth activation function in the lth layer, γ l,j and δ l.j are the weights of other activation functions, and J represents the number of activation functions.

[0126] In the embodiment of the present application, by mixing activation functions and combining the advantages of multiple activation functions, the activation function is dynamically selected according to image features to improve the flexibility and adaptability of the activation function.

[0127] Step 304: The electronic device performs feature weighted fusion on the feature vectors corresponding to the N convolutional layers after the nonlinear transformation based on the feature fusion layer in the first model to obtain an image feature vector.

[0128] In the embodiment of the present application, the electronic device may use a multi-level feature fusion method to fuse features at different levels to generate an image feature vector, which may be specifically implemented by the following formula 5.

[0129]

[0130] Among them, V is the image feature vector, is the feature vector corresponding to the convolutional layer after the nonlinear transformation of the j-th convolutional layer, It is a cascade operation, and concat represents the cascade operation of feature vectors.

[0131] In the embodiment of the present application, multi-scale convolution kernels, adaptive pooling methods and mixed activation function layers are used to efficiently extract image features and perform scene classification. The multi-level feature fusion method of the feature fusion layer enables features at different levels to be effectively fused, thereby improving the accuracy of scene classification.

[0132] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 6 As shown, the above step 203 can be specifically implemented through the following steps 203a and 203b.

[0133] Step 203a: The electronic device performs feature extraction processing on the image feature vector through the second model to obtain key feature information.

[0134] In the embodiment of the present application, the above key feature information represents at least one object in the first image.

[0135] For example, suppose there are two types of pictures: one is pictures of cats, and the other is pictures of dogs. In the pictures, there may be a large number of different features, but not all of them are key features. For example, the color of fur may have many similarities between different cats and dogs, and it is not possible to distinguish the two types of pictures well. However, features such as the shape of the ears, the size of the nose, and the position of the eyes may be key features to distinguish between cats and dogs. Key features can help the model more accurately identify whether the object in the image is a cat or a dog.

[0136] In the embodiment of the present application, the electronic device can perform feature extraction processing on the image feature vector by using the PCA algorithm and the LDA algorithm to obtain key feature information, which can be specifically shown in the following formula 6.

[0137] V'=PCA(V)+LDA(V) (6)

[0138] Among them, V' represents the analyzed eigenvector, PCA represents the principal component analysis algorithm, and LDA represents the linear discriminant analysis algorithm.

[0139] Step 203b: The electronic device determines the scene category of the first image based on the key feature information.

[0140] In an embodiment of the present application, the electronic device may input key feature information into a third model to determine the scene category of the first image through the third model.

[0141] In the embodiment of the present application, the third model is used to select the target model.

[0142] Optionally, in an embodiment of the present application, the third model may be a neural network model or an artificial intelligence (AI) model.

[0143] In the embodiment of the present application, the electronic device ensures the accuracy of feature extraction and the adaptability of the classification model through the combination of the feature analysis module and the model selection module.

[0144] Optionally, in the embodiment of the present application, combined with Figure 6 ,like Figure 7 As shown, the above step 203b can be specifically performed through the following steps 203b1 to 203b3.

[0145] Step 203b1: The electronic device performs feature extraction processing on the first image based on the key feature information through the third model to obtain second feature information.

[0146] In the embodiment of the present application, the model parameters of the third model mentioned above match the key feature information.

[0147] In an embodiment of the present application, the above-mentioned third model is an integrated multi-model architecture or feature selection model. Different from the standard convolutional neural network, it not only relies on the deep feature extraction of the neural network, but also combines the feature analysis results of other statistical methods. The core idea is to improve the accuracy and robustness of classification by selecting and integrating the advantages of multiple pre-trained models.

[0148] Step 203b2: The electronic device performs scene classification processing on the second feature information through the fourth model to obtain at least one scene category value.

[0149] In an embodiment of the present application, the fourth model includes at least one scene category, and each scene category value in the at least one scene category value represents the probability that the first image belongs to a scene category.

[0150] In the embodiment of the present application, the fourth model mentioned above is a scene classification model.

[0151] Optionally, in an embodiment of the present application, the above-mentioned scene classification model may be a neural network model or an AI model.

[0152] In the embodiment of the present application, the electronic device may input the second feature information into the fourth model, and in the classification layer, use the Softmax function to convert the output of the last layer into a scene classification probability, which may be specifically shown in the following formula 7:

[0153]

[0154] Among them, P(c i|V”) is the probability that the second feature information belongs to the i-th scene category, H L,i is the output of the i-th neuron in the last layer, and C represents the total number of scene categories.

[0155] Step 203b3: The electronic device determines the scene category corresponding to the maximum value of at least one scene category value as the scene category to which the first image belongs.

[0156] In the embodiment of the present application, the electronic device may select the scene category with the highest probability according to the scene classification probability as the scene classification result of the first object, which may be specifically shown in the following formula 8.

[0157]

[0158] Among them, c * is the scene category to which the first image belongs.

[0159] In an embodiment of the present application, the electronic device may determine the scene category corresponding to the maximum value of at least one scene category value as the scene category to which the first image belongs, thereby improving the accuracy of the electronic device in determining the scene category.

[0160] Optionally, in the embodiment of the present application, before the above step 203b1, the image classification method provided in the embodiment of the present application further includes the following steps 501 and 502.

[0161] Step 501: The electronic device performs feature analysis on key feature information to obtain feature analysis results.

[0162] In the embodiment of the present application, the above-mentioned feature analysis result represents the similarity between the key feature information and the model parameters in the preset model set.

[0163] Optionally, in the implementation of the present application, the electronic device may perform feature analysis on the key feature information through the above-mentioned PCA algorithm and LDA algorithm to obtain feature analysis results.

[0164] Step 502: Based on the feature analysis result, the electronic device determines the model with the greatest similarity between the model parameters and the key feature information in the preset model set as the third model.

[0165] Exemplarily, the electronic device may obtain the third model through the following formula 9.

[0166] Z=argmax m∈M (W m ·Sim(V',θ m )) (9)

[0167] Among them, Z represents the most suitable model selected, M is the set of pre-trained models, and W mis the weight of model m, Sim(V',θ m ) is the feature vector V' and the model parameter θ m The similarity.

[0168] In an embodiment of the present application, the electronic device obtains a model that best matches the feature analysis results through the feature analysis results. When the electronic device extracts feature information through the model, it can accurately extract scene category information, thereby improving the accuracy of the electronic device in scene category recognition.

[0169] Exemplarily, taking the electronic device as a mobile phone as an example, the image classification method provided in the embodiment of the present application is explained below through specific examples, which can be specifically through the following S81 to S810.

[0170] S81. Obtain a large amount of photo data to be processed, and store the photo data in a mobile phone memory.

[0171] S82, performing a batch reading operation on the large amount of photo data obtained, importing the photo data into the memory of the application, wherein each photo data is represented as a data structure including image content and metadata thereof;

[0172] S83, using a preprocessing module to perform denoising, color enhancement, and resolution adjustment on the batch-read photo data;

[0173] S84, inputting the preprocessed photo data into a pre-trained convolutional neural network model to extract image features and generate an image feature vector for each photo;

[0174] S85, inputting the generated image feature vector into a scene classification model to perform scene recognition and generate a scene classification result;

[0175] S86, generating corresponding classification labels according to the scene classification results, and attaching the classification labels to corresponding photos;

[0176] S87, automatically classifying the photos into corresponding folders or albums according to the generated classification labels, where the names of the folders or albums correspond to the classification labels;

[0177] S88. During the storage process, check whether each photo is correctly classified into a corresponding folder or album to ensure the accuracy of the classification process;

[0178] S89. In the mobile phone album application, provide an intuitive display of batch classification results, so that users can quickly browse and manage photos of different scene categories;

[0179] S810: Allow the user to manually adjust and correct the batch classification results, update the classification labels and reclassify the photos.

[0180] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0181] It should be noted that the image classification method provided in the embodiment of the present application can be executed by an image classification device. In the embodiment of the present application, an image classification device executing the image classification method is taken as an example to illustrate the image classification device provided in the embodiment of the present application.

[0182] Figure 8 A possible structural diagram of an image classification device involved in an embodiment of the present application is shown. Figure 8 As shown, the image classification device 70 may include: an acquisition module 71 , a feature extraction module 72 and a determination module 73 .

[0183] The acquisition module 71 is used to acquire the first image. The feature extraction module 72 is used to perform feature extraction processing on the first image acquired by the acquisition module through N convolution layers of the first model to obtain an image feature vector of the first image, wherein the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolution layer. The determination module 73 is used to determine the scene category to which the first image belongs based on the image feature vector; wherein N is an integer greater than 1.

[0184] In a possible implementation, the feature extraction module 72 is specifically used to perform feature extraction processing on the first image through at least two convolution kernels in the first convolution layer of the first model to obtain a feature vector corresponding to the first convolution layer; perform feature extraction processing on the first feature vector through at least two convolution kernels in the jth convolution layer of the first model to obtain a feature vector corresponding to the jth convolution layer; the first feature vector is the feature vector corresponding to the j-1th convolution layer; j∈[2,N]; after obtaining the feature vector corresponding to the Nth convolution layer, feature fusion is performed on the feature vectors corresponding to the Nth convolution layer based on the feature fusion layer in the first model to obtain an image feature vector.

[0185] In a possible implementation, the image classification device further includes: a fusion module; the feature extraction module 72 is specifically used to: calculate the feature vector corresponding to each convolution kernel in the first convolution layer based on the weights and feature map values ​​corresponding to at least two convolution kernels in the first convolution layer; the feature map value is obtained by weighted summation of pixel values ​​of at least two image blocks, each image block is an image block corresponding to a convolution kernel in the first convolution layer in the first image. The fusion module is used to perform feature fusion on the feature vector corresponding to each convolution kernel in the first convolution layer to obtain the feature vector corresponding to the first convolution layer.

[0186] In a possible implementation, the above-mentioned image classification device also includes: a processing module; a processing module, which is used to, after obtaining the feature vector corresponding to each convolution layer, perform pooling processing on the feature vector corresponding to each convolution layer based on the pooling parameters in each pooling layer of the first model, and the pooling parameters in each pooling layer are obtained based on the feature vector corresponding to the previous convolution layer; the above-mentioned determination module 73 is also used to determine the target activation function from at least one activation function of the activation function layer through the activation function layer of the first model based on the feature vector corresponding to the N convolution layers after the pooling processing; the processing module is also used to perform nonlinear transformation processing on the feature vector corresponding to the N convolution layers after the pooling processing based on the target activation function, and obtain the feature vector corresponding to the N convolution layers after the nonlinear transformation; a fusion module, which is specifically used to perform feature weighted fusion on the feature vector corresponding to the N convolution layers after the nonlinear transformation based on the feature fusion layer in the first model to obtain the image feature vector.

[0187] In a possible implementation, the feature extraction module 72 is further used to perform feature extraction processing on the image feature vector through the second model to obtain key feature information. The determination module 73 is specifically used to determine the scene category of the first image based on the key feature information.

[0188] In a possible implementation, the image classification device further includes: a processing module; the feature extraction module 72 is further used to perform feature extraction processing on the first image through a third model based on the key feature information to obtain second feature information, and the model parameters of the third model match the key feature information. The processing module is used to perform scene classification processing on the second feature information through a fourth model to obtain at least one scene category value, and the fourth model includes at least one scene category, and each scene category value represents the probability that the first image belongs to a scene category. The determination module 73 is specifically used to determine the scene category corresponding to the maximum value of at least one scene category value as the scene category to which the first image belongs.

[0189] In a possible implementation, the above-mentioned processing module is also used to: based on the key feature information, perform feature extraction processing on the first image through the third model, and before obtaining the second feature information, perform feature analysis on the key feature information to obtain feature analysis results, and the feature analysis results represent the similarity between the key feature information and the model parameters in the preset model set; based on the feature analysis results, determine the model with the greatest similarity between the model parameters in the preset model set and the key feature information as the third model.

[0190] The embodiment of the present application provides an image classification device. Since image features of different scales can be extracted simultaneously through convolution kernels of different sizes in each convolution layer, the first model can capture more detailed information and global information, making scene recognition more accurate. In addition, the superposition of convolution operations can enable the first model to gradually extract higher-level semantic information at different levels, further improving the accuracy of recognition. Moreover, the first model is used to automatically classify images into scenes without the need for manual operation by the user. In this way, while improving the efficiency of the image classification device in scene classification, it also improves the accuracy of the image classification device in scene classification.

[0191] The image classification device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0192] The image classification device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0193] The image classification device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0194] Alternatively, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned image classification method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0195] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0196] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0197] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0198] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0199] The input unit 104 is used to obtain a first image. The processor 110 is used to perform feature extraction processing on the first image through N convolution layers of the first model to obtain an image feature vector of the first image, where the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolution layer; based on the image feature vector, determine the scene category to which the first image belongs; where N is an integer greater than 1.

[0200] Optionally, in an embodiment of the present application, the processor 110 is specifically used to perform feature extraction processing on the first image through at least two convolution kernels in the first convolution layer of the first model to obtain a feature vector corresponding to the first convolution layer; perform feature extraction processing on the first feature vector through at least two convolution kernels in the jth convolution layer of the first model to obtain a feature vector corresponding to the jth convolution layer; the first feature vector is the feature vector corresponding to the j-1th convolution layer; j∈[2,N]; after obtaining the feature vector corresponding to the Nth convolution layer, perform feature fusion on the feature vectors corresponding to the Nth convolution layer based on the feature fusion layer in the first model to obtain an image feature vector.

[0201] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to calculate the feature vector corresponding to each convolution kernel in the first convolution layer based on the weights and feature map values ​​corresponding to at least two convolution kernels in the first convolution layer; the feature map value is obtained by weighted summation of pixel values ​​of at least two image blocks, each image block is an image block corresponding to a convolution kernel in the first convolution layer in the first image; feature fusion is performed on the feature vector corresponding to each convolution kernel in the first convolution layer to obtain the feature vector corresponding to the first convolution layer.

[0202] Optionally, in an embodiment of the present application, the processor 110 is further used to, after obtaining the feature vector corresponding to each convolution layer, perform pooling processing on the feature vector corresponding to each convolution layer based on the pooling parameters in each pooling layer of the first model, and the pooling parameters in each pooling layer are obtained based on the feature vector corresponding to the previous convolution layer; through the activation function layer of the first model, based on the feature vectors corresponding to the N convolution layers after the pooling processing, determine the target activation function from at least one activation function of the activation function layer; based on the target activation function, perform nonlinear transformation processing on the feature vectors corresponding to the N convolution layers after the pooling processing to obtain the feature vectors corresponding to the N convolution layers after the nonlinear transformation; the processor 110 is specifically used to perform feature weighted fusion on the feature vectors corresponding to the N convolution layers after the nonlinear transformation based on the feature fusion layer in the first model to obtain the image feature vector.

[0203] Optionally, in an embodiment of the present application, the processor 110 is specifically used to perform feature extraction processing on the image feature vector through a second model to obtain key feature information; and determine the scene category of the first image based on the key feature information.

[0204] Optionally, in an embodiment of the present application, the processor 110 is specifically used to perform feature extraction processing on the first image based on the key feature information through a third model to obtain second feature information, and the model parameters of the third model match the key feature information; perform scene classification processing on the second feature information through a fourth model to obtain at least one scene category value, the fourth model includes at least one scene category, and each scene category value represents the probability that the first image belongs to a scene category; and determine the scene category corresponding to the maximum value of at least one scene category value as the scene category to which the first image belongs.

[0205] Optionally, in an embodiment of the present application, the processor 110 is further used to perform feature extraction processing on the first image based on the key feature information through a third model, and perform feature analysis on the key feature information to obtain a feature analysis result before obtaining the second feature information, wherein the feature analysis result represents the similarity between the key feature information and the model parameters in the preset model set; based on the feature analysis result, the model in which the model parameters in the preset model set have the greatest similarity with the key feature information is used as the third model.

[0206] An embodiment of the present application provides an electronic device. Since image features of different scales can be extracted simultaneously through convolution kernels of different sizes in each convolution layer, the first model can capture more detailed information and global information, making scene recognition more accurate. In addition, the superposition of convolution operations can enable the first model to gradually extract higher-level semantic information at different levels, further improving the accuracy of recognition. Moreover, the first model is used to automatically classify images into scenes without the need for manual operation by the user. In this way, the efficiency of scene classification by the electronic device is improved while also improving the accuracy of scene classification by the electronic device.

[0207] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0208] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0209] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0210] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0211] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0212] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0213] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0214] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0215] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0216] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image classification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0217] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0219] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image classification method, characterized in that: The method comprises: acquiring a first image; Performing feature extraction processing on the first image through N convolutional layers of the first model to obtain an image feature vector of the first image, wherein the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolutional layer; Determining, based on the image feature vector, a scene category to which the first image belongs; Wherein, N is an integer greater than 1.

2. The method according to claim 1, characterized in that The step of performing feature extraction processing on the first image through the N convolutional layers of the first model to obtain an image feature vector of the first image includes: Performing feature extraction processing on the first image through at least two convolution kernels in the first convolution layer of the first model to obtain a feature vector corresponding to the first convolution layer; Perform feature extraction processing on the first feature vector through at least two convolution kernels in the j-th convolution layer of the first model to obtain a feature vector corresponding to the j-th convolution layer; the first feature vector is a feature vector corresponding to the j-1-th convolution layer; j∈[2,N]; After obtaining the feature vector corresponding to the Nth convolutional layer, feature fusion is performed on the feature vectors corresponding to the Nth convolutional layer based on the feature fusion layer in the first model to obtain the image feature vector.

3. The method according to claim 2, characterized in that The step of performing feature extraction processing on the first image by using at least two convolution kernels in the first convolution layer of the first model to obtain a feature vector corresponding to the first convolution layer includes: Based on the weights and feature map values ​​corresponding to at least two convolution kernels in the first convolution layer, a feature vector corresponding to each convolution kernel in the first convolution layer is calculated; the feature map value is obtained based on the weighted sum of pixel values ​​of at least two image blocks, each image block being an image block corresponding to a convolution kernel in the first convolution layer in the first image; Feature fusion is performed on the feature vector corresponding to each convolution kernel in the first convolution layer to obtain the feature vector corresponding to the first convolution layer.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: After obtaining the feature vector corresponding to each convolution layer, based on the pooling parameters in each pooling layer of the first model, the feature vector corresponding to each convolution layer is pooled, and the pooling parameters in each pooling layer are obtained based on the feature vector corresponding to the previous convolution layer; Determine, through the activation function layer of the first model, a target activation function from at least one activation function of the activation function layer based on feature vectors corresponding to the N convolutional layers after pooling; Based on the target activation function, a nonlinear transformation is performed on the feature vectors corresponding to the N convolutional layers after the pooling process to obtain the feature vectors corresponding to the N convolutional layers after the nonlinear transformation; The step of performing feature fusion on the feature vectors obtained by the N convolutional layers based on the feature fusion layer in the first model to obtain the image feature vector comprises: Based on the feature fusion layer in the first model, feature weighted fusion is performed on the feature vectors corresponding to the N convolutional layers after nonlinear transformation to obtain the image feature vector.

5. The method according to any one of claims 1 to 3, characterized in that: The determining, based on the image feature vector, the scene category to which the first image belongs includes: Performing feature extraction processing on the image feature vector through a second model to obtain key feature information; Based on the key feature information, a scene category of the first image is determined.

6. The method according to claim 5, characterized in that The determining the scene category of the first image based on the key feature information includes: Based on the key feature information, a feature extraction process is performed on the first image through a third model to obtain second feature information, and a model parameter of the third model matches the key feature information; performing scene classification processing on the second feature information through a fourth model to obtain at least one scene category value, wherein the fourth model includes at least one scene category, and each scene category value represents a probability that the first image belongs to a scene category; The scene category corresponding to the maximum value of the at least one scene category value is determined as the scene category to which the first image belongs.

7. The method according to claim 6, characterized in that Before performing feature extraction processing on the first image based on the key feature information and obtaining the second feature information through a third model, the method further includes: Performing feature analysis on the key feature information to obtain a feature analysis result, wherein the feature analysis result represents the similarity between the key feature information and the model parameters in the preset model set; Based on the feature analysis result, the model whose model parameters in the preset model set have the greatest similarity with the key feature information is determined as the third model.

8. An image classification device, characterized in that: The image classification device comprises: an acquisition module, a feature extraction module and a determination module; The acquisition module is used to acquire a first image; The feature extraction module is used to perform feature extraction processing on the first image acquired by the acquisition module through N convolution layers of the first model to obtain an image feature vector of the first image, wherein the image feature vector includes N layers of feature information, and each layer of feature information includes feature information extracted by at least two convolution kernels of different sizes in each convolution layer; The determination module is used to determine the scene category to which the first image belongs based on the image feature vector; wherein N is an integer greater than 1.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image classification method as claimed in any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image classification method according to any one of claims 1 to 8 are implemented.

11. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the image classification method according to any one of claims 1 to 8.

Images