Image classification method, device, equipment and storage medium

Abstract

The present application is about an image classification method, device, equipment and storage medium, and specifically relates to the field of image recognition. The method includes: obtaining a training sample image; processing the training sample image through a target classification model to obtain an output value corresponding to the training sample image; obtaining a first loss function value according to the output value of the training sample image and the label value of the training sample image; obtaining a second loss function value according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model; updating the target classification model based on the first loss function value and the second loss function value, so that the updated target classification model can classify the input target image. The above scheme improves the discrimination of the parameters of different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from training sample images, and improving the accuracy of image classification.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an image classification method, device, equipment and storage medium. Background Art

[0002] Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms.

[0003] At present, when classifying images, deep learning models are usually used for data processing, that is, the input image is processed through the trained deep learning model, and the type of the input image is determined according to the processing result. In the training process of the deep learning model, specific category numbers of different categories of images can be given in the training data with supervised information, generally using 0 to N numbers. These numbers are used as labels for supervised training during the training process, and the output value and label value of the fully connected layer FC of the deep learning model are calculated through the loss function, and the model parameters are updated through the loss function value to realize the constraints on the output of the fully connected layer.

[0004] In the above scheme, the deep learning model where the output result of FC is directly constrained for training through the classification loss function is not dispersed enough when extracting the feature space of the image, and the classification accuracy is low. Summary of the invention

[0005] The present application provides an image classification method, apparatus, device and storage medium, which improve the accuracy of image classification. The technical solution is as follows.

[0006] In one aspect, a method for image classification is provided, the method comprising:

[0007] Obtain training sample images;

[0008] Processing the training sample image through the target classification model to obtain an output value corresponding to the training sample image;

[0009] Obtaining a first loss function value according to the output value of the training sample image and the label value of the training sample image;

[0010] Obtaining the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model;

[0011] The target classification model is updated based on the first loss function value and the second loss function value, so that the updated target classification model can classify the input target image.

[0012] In another aspect, a method for image classification is provided, the method comprising:

[0013] Get the target image;

[0014] Processing the target image through the target classification model to obtain a classification result of the target image;

[0015] Among them, the target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model.

[0016] In another aspect, an image classification device is provided, the device comprising:

[0017] A sample acquisition module is used to acquire training sample images;

[0018] A sample image processing module is used to process the training sample image through a target classification model to obtain an output value corresponding to the training sample image;

[0019] A first loss function value acquisition module, used to acquire a first loss function value according to the output value of the training sample image and the label value of the training sample image;

[0020] A second loss function value acquisition module, used to acquire the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model;

[0021] A model updating module is used to update the target classification model based on the first loss function value and the second loss function value, so that the updated target classification model can classify the input target image.

[0022] In another aspect, an image classification device is provided, the device comprising:

[0023] A target image acquisition module, used for acquiring a target image;

[0024] A target image processing module is used to process the target image through a target classification model to obtain a classification result of the target image;

[0025] Among them, the target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model.

[0026] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the above-mentioned image classification method.

[0027] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned image classification method.

[0028] The technical solution provided by this application may have the following beneficial effects:

[0029] When it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After the target classification model is updated by the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 The figure is a schematic diagram showing the structure of an image classification system according to an exemplary embodiment.

[0032] Figure 2The figure is a flowchart of an image classification method according to an exemplary embodiment.

[0033] Figure 3 The figure is a flowchart of an image classification method according to an exemplary embodiment.

[0034] Figure 4 The figure is a flowchart of an image classification method according to an exemplary embodiment.

[0035] Figure 5 A schematic diagram of image mirror flipping involved in an embodiment of the present application is shown.

[0036] Figure 6 A schematic diagram of pixel sampling involved in an embodiment of the present application is shown.

[0037] Figure 7 A schematic diagram of pixel replacement involved in an embodiment of the present application is shown.

[0038] Figure 8 A schematic diagram of Gaussian blur involved in an embodiment of the present application is shown.

[0039] Fig. 9 A schematic diagram of random rotation involved in an embodiment of the present application is shown.

[0040] Fig.10 A schematic diagram of a second loss function value acquisition process involved in an embodiment of the present application is shown.

[0041] Fig.11 is a structural block diagram of an image classification device according to an exemplary embodiment.

[0042] Fig.12 is a structural block diagram of an image classification device according to an exemplary embodiment.

[0043] Fig.13 It is a schematic diagram of a computer device provided according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0045] It should be understood that the "indication" mentioned in the embodiments of the present application can be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association relationship between A and B.

[0046] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0047] In an embodiment of the present application, "predefinition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including a terminal device and a network device). The present application does not limit its specific implementation method.

[0048] The image classification method provided in the embodiment of the present application can be applied to a computer device with strong data processing capabilities. The image classification method can be a training method for an image classification model, and the image classification model can process the input image to obtain an image category corresponding to the input image. In a possible implementation, the image classification method provided in the embodiment of the present application can be applied to a personal computer, a workstation or a server, that is, the image classification model is trained by at least one of a personal computer, a workstation and a server.

[0049] Figure 1 1 is a schematic diagram of a structure of an image classification system according to an exemplary embodiment. The image classification system includes a server 110 and a terminal 120. The terminal 120 and the server 110 perform data communication via a communication network, which may be a wired network or a wireless network.

[0050] Optionally, an application with an image processing function is installed in the terminal 120. The application may be a professional image processing application, a social application, a virtual reality application, or an AI application with an image processing function, which is not limited in the embodiments of the present application.

[0051] Optionally, the terminal 120 may be a terminal device having an image acquisition component, which is used to acquire images and store them in a data storage module in the terminal 120; the terminal 120 may also be a terminal device having a data transmission interface, which is used to receive image data acquired by an image acquisition device having an image acquisition component.

[0052] Optionally, the terminal 120 may be a mobile terminal such as a smart phone, a tablet computer, a laptop computer, or the like, or a terminal such as a desktop computer, a projection computer, or an intelligent terminal with a data processing component, and the embodiments of the present application do not impose any restrictions on this.

[0053] The server 110 may be implemented as a single server or a server cluster consisting of a group of servers, which may be a physical server or a cloud server. In a possible implementation, the server 110 is a background server of an application in the terminal 120 .

[0054] In a possible implementation of the embodiment of the present application, the server 110 trains the image classification model through a pre-set training sample set (including each training sample image), wherein the training sample set may contain training sample images of different categories, and each training sample image has its own category annotation information (i.e., label value). After the server 110 completes the training process of the image classification model, the trained image classification model is sent to the terminal 120 via a wired network or a wireless network.

[0055] The terminal 120 receives the trained image classification model and sends the data information (such as weight information) corresponding to the image classification model to an application with an image classification function, so that when the user uses the application, the input image can be classified.

[0056] Optionally, the above-mentioned server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms and other technical computing services.

[0057] Optionally, the system may further include a management device, which is used to manage the system (such as managing the connection status between each module and the server, etc.), and the management device is connected to the server via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0058] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any other network, including but not limited to any combination of local area network, metropolitan area network, wide area network, mobile, limited or wireless network, private network or virtual private network. In some embodiments, the technology and / or format including hypertext markup language, extensible markup language, etc. are used to represent the data exchanged through the network. In addition, conventional encryption technologies such as secure socket layer, transport layer security, virtual private network, Internet protocol security, etc. can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology.

[0059] Figure 2 is a method flow chart of an image classification method according to an exemplary embodiment. The method is executed by a computer device, which may be Figure 1 The server or terminal in the image classification system shown in FIG. Figure 2 As shown, the image classification method may include the following steps:

[0060] Step 201: Obtain training sample images.

[0061] In the embodiment of the present application, the training sample image may be an image of a specified pixel size.

[0062] Step 202: Process the training sample image through the target classification model to obtain an output value corresponding to the training sample image.

[0063] When the target classification model needs to be trained, the training sample images can be processed by the target classification model to obtain output values ​​corresponding to the training sample images.

[0064] In one possible implementation, the target classification model includes a feature extraction module and a fully connected layer. The computer device performs feature extraction on an input training sample image of a specified pixel size through the feature extraction module in the target classification model to obtain a feature map of the training sample image. The feature map represents the image features obtained by extracting features from the training sample image through the feature extraction module.

[0065] The computer device then processes the feature map of the training sample image through the fully connected layer in the target classification model to obtain the output value of the training sample image.

[0066] In a possible implementation of the embodiment of the present application, there are fully connected layers of each category in the target classification model, and the fully connected layers of each category are used to map the feature map to each category to obtain the probability value of the training sample image being each category. For example, when the dimension of the feature map of the training sample image is 512 dimensions, the fully connected layer parameter of the fully connected layer is C*512, which can be intuitively understood as including mapping the feature map to C categories through C 512-dimensional FC parameters respectively.

[0067] Step 203: Obtain a first loss function value according to the output value of the training sample image and the label value of the training sample image.

[0068] After the training sample image is processed by the target classification model, the output value of the training sample image can be obtained. At this time, the output value of the training sample is the classification probability obtained after the target classification model processes the training sample data.

[0069] For example, when a 512-dimensional feature map is extracted after processing the training sample image, the 512-dimensional feature map can be processed through a C*512 fully connected layer to obtain the probabilities of the training sample image corresponding to C categories respectively.

[0070] At this time, the training sample set where the training sample image is located includes the label value of the training sample image, and the label value is used to indicate the category of the training sample image. For example, when the training sample image belongs to a certain category, the label value in the dimensional data corresponding to the category is 1; if the training sample image does not belong to a certain category, the label value in the dimensional data corresponding to the category is 0.

[0071] The output of the training sample image and the label value of the training sample image are processed through the loss function to obtain the first loss function value, which is used to represent the error when the training sample image is classified by the target classification model this time.

[0072] Step 204, obtaining the second loss function value according to the normalized inner products of the parameters of at least two types of fully connected layers in the target classification model.

[0073] During the training process, the computer device will also obtain the parameters of at least two types of fully connected layers in the target classification model, that is, at least two types of parameters are arbitrarily selected from the C-type fully connected layer parameters, and normalized as vectors, and then the vectors of at least two types of parameters obtained after normalization are inner-product processed to obtain the second loss function value. The closer the parameters of at least two types of fully connected layers are, the larger the second loss function value is, and the more different the parameters of at least two types of fully connected layers are, the smaller the second loss function value is. Therefore, the second loss function value can represent the difference between any two layers of parameters in the C-type fully connected layer.

[0074] Step 205: Update the target classification model based on the first loss function value and the second loss function value, so that the updated target classification model can classify the input target image.

[0075] After obtaining the first loss function value and the second loss function value, the target classification model can be updated through the back propagation algorithm according to the first loss function value and the second loss function value. Since the goal of the model update is to reduce the first loss function value and the second loss function value as much as possible, the updated target classification model obtained by training with the first loss function value and the second loss function value can improve the classification and recognition ability of the training sample images, and also increase the specificity between the parameters of various fully connected layers, so that the recognition ability of the target classification model for different categories of training sample images is enhanced, and the accuracy of image classification of the target classification model is improved.

[0076] To summarize, when it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After updating the target classification model through the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification.

[0077] Figure 3 is a method flow chart of an image classification method according to an exemplary embodiment. The method is executed by a computer device, which may be Figure 1 The server or terminal in the image classification system shown in FIG. Figure 3 As shown, the image classification method may include the following steps:

[0078] Step 301, acquiring a target image.

[0079] The target image is an image that needs to be recognized.

[0080] Taking a face recognition scenario as an example, the target image may include a face image, and the identity information of the face image included in the target image may be determined by recognizing the target image.

[0081] In a possible implementation, the target image is an image with a pixel size, that is, when performing image recognition on the target image, the target image is scaled to a specified pixel size to meet the requirements of the image recognition process.

[0082] Step 302: Process the target image through the target classification model to obtain a classification result of the target image.

[0083] Among them, the target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model.

[0084] After the target image is processed by the target classification model, the target classification model extracts the image features of the target image through the feature extraction module, and then processes the image features of the target image through several types of fully connected layers to obtain several probability results of the target image.

[0085] Taking the face recognition scenario as an example, the probability results of the target image are the probabilities of the faces in the target image. In a possible implementation, the target classification model can output the identity corresponding to the largest probability result among the probability results as the face recognition result of the target image.

[0086] In another possible implementation, when several probability results obtained by the target classification model are all smaller than a preset confidence threshold, an abnormal prompt information indicating that the target image cannot be normally identified is input.

[0087] In addition, the training process of the target classification model of the embodiment of the present application, in addition to training through training sample images and the label values ​​corresponding to the training sample images, also takes into account the similarities between the fully connected layers of multiple classifications. During the training process, the parameters of each fully connected layer are kept as far away from each other as possible, making it difficult for the target classification model to identify a certain type of image as a similar image type, thereby improving the accuracy of target classification in multi-classification problems.

[0088] To summarize, when it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After updating the target classification model through the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification.

[0089] Figure 4 is a method flow chart of an image classification method according to an exemplary embodiment. The method is executed by a computer device, which may be Figure 1 The server or terminal in the image classification system shown in FIG. Figure 4 As shown, the image classification method may include the following steps:

[0090] Step 401: Obtain training sample images.

[0091] In a possible implementation, a target sample image is acquired; data enhancement processing is performed on the target sample image to obtain the training sample image; the data enhancement processing includes at least one operation of rotation, translation, cropping, flipping, and overlaying.

[0092] The target sample image may be any one in a training sample set, and the training sample set contains various sample images and category labels of various sample images.

[0093] When data enhancement processing is performed on the target sample image, that is, the target sample image is processed by at least one of rotation, translation, cropping, flipping, and overlaying to obtain a training sample image, the category of the training sample image will not change, that is, the category of the training sample image should be consistent with the category of the target sample image, so the category label of the target sample image is determined as the category label of the training sample image.

[0094] Taking the face recognition scenario as an example, the computer device obtains any face sample image from the training sample set, and rotates the face sample image to generate a training sample image, and then determines the identity information corresponding to the face sample image as the identity information of the training sample image as the label data of the training sample image.

[0095] In a possible implementation of the embodiment of the present application, the data enhancement method that can be adopted by the computer device includes at least one of the following methods:

[0096] 1) Grayscale conversion, that is, the computer converts the three-channel color RGB format image into a GRAY grayscale image according to probability p.

[0097] 2) Random flip Random left and right mirror flip. Figure 5 A schematic diagram of image mirror flipping involved in an embodiment of the present application is shown.

[0098] 3) Random resize: The computer device resizes the original image to a certain scale (0.5 to 1.0 of the original width and height) according to probability p, and then resizes it back to the original scale. Figure 6 A schematic diagram of random size change involved in an embodiment of the present application is shown.

[0099] 4) Random erase randomly generates a black block of size h*w to replace a block in the image with probability p. Figure 7 A schematic diagram of pixel replacement involved in an embodiment of the present application is shown.

[0100] 5) Colorjitter: Computer equipment can enhance color data by color jitter, that is, change the brightness, saturation, and contrast of the image. For example, the size relationship between the pixels of the three RGB channels can be adjusted while keeping the sum of the pixels of the three RGB channels unchanged.

[0101] 6) Gaussian blur: The computer device adds Gaussian noise to each pixel on the image. Figure 8 A schematic diagram of Gaussian blur involved in an embodiment of the present application is shown.

[0102] 7) Random rotation: The computer device will randomly rotate the direction of the image. Fig. 9 A schematic diagram of random rotation involved in an embodiment of the present application is shown.

[0103] Through the above method, the computer device can expand the number of training samples to obtain more training sample images to train the image classification model, and train the image classification model with the training sample images obtained through data enhancement processing, so that the image classification model still has good recognition ability for images that have undergone operations such as rotation, translation, cropping, flipping, and overlaying, thereby further improving the recognition ability of the image classification model.

[0104] In a possible implementation, data enhancement processing is performed on the target sample image to obtain an enhanced sample image; the enhanced sample image is converted to a target size and normalized to obtain the training sample image.

[0105] For example, the computer device transforms the enhanced sample image obtained after data enhancement processing to a target size, such as 160*144, and then normalizes each pixel in the enhanced sample image. The normalization formula is src=(src-127.5) / 128, where src is a three-channel RGB image (i.e., the enhanced sample image mentioned above).

[0106] The enhanced sample images obtained after data augmentation processing of the target sample image may not be consistent with the pixel size that can be recognized by the image classification model. Therefore, in order to enable the image classification model to accurately perform image processing, the enhanced sample images can be converted to the target size and normalized to obtain training sample images in a format that is convenient for the image classification model to perform data processing.

[0107] In a possible implementation of the embodiment of the present application, a computer device aligns the normalized images to obtain a sample image for input into a feature extraction model.

[0108] Usually, computer equipment needs to perform alignment processing before inputting images into feature extraction networks to ensure that the feature extraction model can perform normal feature extraction on the input images;

[0109] In a possible implementation, the computer device outputs a 1*6 dimensional matrix through STN (Spatial Transformer Networks), that is, using the network structure of mobilenet_v2; and then reshapes the 1*6 result into a 2*3 affine transformation matrix structure.

[0110] The computer device then uses the spatialTransformer of mxnet to perform spatial transformation based on the obtained affine transformation matrix and the 160*144 input image to obtain a 160*144 aligned image.

[0111] Finally, the obtained 160*144 image is centrally cropped to obtain a 128*112 image (i.e., a training sample image that conforms to the processing format of the feature extraction network).

[0112] Step 402: Process the training sample image through the target classification model to obtain an output value corresponding to the training sample image.

[0113] In one possible implementation, the target classification model includes a feature extraction module and a fully connected layer module.

[0114] For example, the feature extraction module may be a resnet101 structure, in which case the feature extraction module may perform feature extraction on an image with an input of 128 (Height)*112 (Width)*3 (Channel), and output a 512-dimensional feature.

[0115] The fully connected layer module may include a C-type 512-dimensional fully connected layer. The 512-dimensional features output by the above-mentioned feature extraction module can be processed by the C-type 512-dimensional fully connected layer on the fully connected layer module to obtain C-type output values, which respectively represent the probability that the training sample image belongs to each category.

[0116] In one possible implementation, the weight decay parameter of the FC layer in the model training process involved in the embodiment of the present application is set to 0.1 times the weight decay parameter of the feature network, the FC layer learning rate is the same as the feature network learning rate, and the FC parameter size is C*512.

[0117] Step 403: Obtain a first loss function value according to the output value of the training sample image and the label value of the training sample image.

[0118] In a possible implementation method of an embodiment of the present application, when an image classification model is trained through training sample images, a batch of training sample images is usually used to train the image classification model, that is, a batch of training sample images is first processed through the image classification model to obtain output values ​​corresponding to each training sample image in the batch, and then the output values ​​corresponding to each training sample image in a batch and the label values ​​corresponding to each training sample image in a batch are input into the first loss function to obtain the first loss function value.

[0119] For example, in an embodiment of the present application, the first loss function may be arcface loss, and the specific formula is as follows:

[0120]

[0121] Where m represents the number of samples, i represents the i-th sample, y represents the y-th category, θj represents the softmax result of the j-th sample in the j-th category, and s, m1, m2, and m3 represent margin parameters. For example, during the training process, the four parameters are set to 128, 1.0, 0.3, and 0.2 respectively.

[0122] Through the above-mentioned first loss function, the error between the output value and the label value obtained for each training sample image in a batch can be determined.

[0123] Step 404, obtaining the second loss function value according to the normalized inner products of the parameters of at least two types of fully connected layers in the target classification model.

[0124] In this target classification model, the parameters of each type of fully connected layer are 512-dimensional parameters. The parameters of at least two types of fully connected layers are respectively regarded as 512-dimensional vectors and normalized, so that the directions of the parameters of at least two types of fully connected layers in the vector space can be determined. When the directions of the parameters of the fully connected layers in the vector space are closer, it can be considered that the similarity of the parameters of the fully connected layers is higher; when the directions of the parameters of the fully connected layers in the vector space are farther apart, it can be considered that the similarity of the parameters of the fully connected layers is lower, and the features paid attention to by the fully connected layers are more different.

[0125] In a possible implementation, parameters of 2M types of fully connected layers are obtained, and the parameters of each type of fully connected layer are normalized to construct normalized vectors corresponding to the 2M types of fully connected layers respectively;

[0126] Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix;

[0127] The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model;

[0128] Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as the second loss function value corresponding to each training sample image in a training batch of the target classification model.

[0129] Please refer to Fig.10 , which shows a schematic diagram of a second loss function value acquisition process involved in an embodiment of the present application. Fig.10As shown, when 2M types of fully connected layers are selected, the M types of fully connected layers are constructed as a 1*M dimensional matrix 1001, and the elements in the matrix 501 are L1 to LM, respectively, and each element can be considered as a 1012-dimensional vector value. Similarly, the remaining M types of fully connected layers are constructed as an M*1 dimensional matrix 1002, and the elements in each dimension of the matrix 502 are LN+1 to LN+M, respectively, and each element can also be considered as a 512-dimensional vector value.

[0130] After constructing the matrix 1001 and the matrix 1002, the matrix 1001 is dot-multiplied with the matrix 1002. At this time, each element in the matrix 1001 is inner-producted with each element in the matrix 1002, and the structure is constructed as an M*M matrix 1003.

[0131] In the M*M matrix 1003, the element (1,1) is the inner product value of the vector L1 and the vector LN+1, and the element (M,M) is the inner product value of the vector LM and the vector LM+1. Therefore, each element in the M*M matrix 1003 represents the inner product value between each vector in the matrix 1001 and each vector in the matrix 1002.

[0132] After obtaining the M*M matrix 1003, the matrix 1003 can be linearly transformed. The number of rows of the transformed matrix 1004 is the same as the number of samples in a training batch of the image classification model. At this time, the elements in the matrix 1004 are accumulated column by column to obtain a matrix 1005 with 1 row and a column number equal to the number of samples in a training batch. The elements in the matrix 1005 can be considered as the second loss function value obtained for each sample in the training batch.

[0133] The second loss function value represents the similarity between the fully connected layer parameters of different categories of the image classification model during the training process of each sample in the training batch.

[0134] For example, in the embodiment of the present application, the second loss function is FC-uniform loss. In classification training, the loss function is generally the result after supervising FC mapping, and generally does not directly supervise FC parameters. The FC-uniform loss function used in the embodiment of the present application directly acts on the FC parameters themselves, and supervises the classification training task together with arcface loss.

[0135] Among them, the above FC-uniform loss definition: L fc = normlize(W1) normlize(W2) T, where W1 and W2 are 2*M category parameters randomly selected from the C category parameters of FC, and M should be divisible by batchsize. The first M parameters are W1 and the last M parameters are W2. The calculation process of FC-uniform loss is as follows:

[0136] (1) W1 and W2 are normalized and then dot-multiplied to obtain an M*M matrix

[0137] (2) Resize the matrix to a new matrix of [batchsize, M*M / batchsize]

[0138] (3) The resized new matrix is ​​summed up by column to get a matrix of [batchsize, 1]

[0139] The second loss function value obtained in the above manner characterizes, to a certain extent, the similarity of the parameters between the fully connected layers of different categories.

[0140] Step 405: weight the first loss function value and the second loss function value to obtain a target loss function value.

[0141] After obtaining the first loss function value and the second loss function value, since in the training process of the target classification model, the sample images of a training batch are usually processed before the parameters are updated, in a possible implementation, the first loss function value includes the loss function values ​​corresponding to the sample images of a training batch, and the second loss function value is as follows: Fig.10 As shown in the figure, there is also a loss function value for the number of samples in a training batch, so the second loss function value can be added to the first loss function value one by one according to the weight, so as to obtain the target loss function value. At this time, the target loss function value represents the error between the prediction result and the annotation result of each sample image, and also represents the similarity between the parameters of various fully connected layers.

[0142] Step 406: Perform back propagation update on the target classification model according to the target loss function value.

[0143] When the target classification model is back-propagated and updated through the target loss function value, the update goal of the back-propagation update is to minimize the target loss function value. When the target loss function value decreases, the error between the prediction results and the annotation results of each sample image is gradually reduced, and the similarity between the parameters of various fully connected layers is also gradually reduced. Therefore, the target classification model trained by the above scheme not only improves the recognition effect of sample images, but also improves the recognition ability of the target classification model for different categories, and has better image recognition effect on multi-classification problems.

[0144] That is, in a possible implementation of the embodiment of the present application, the first loss function Arcface is combined with the second loss function FC-uniform as a new loss in the embodiment of the present application to supervise the training of the model:

[0145] L=L arc +λL fc , where λ is a coefficient, generally ranging from 0.001 to 0.01.

[0146] And in a possible training process of an embodiment of the present application, the training data used by the computer device to train the target classification model is 4 million IDs and 20 million images;

[0147] During the training process, the optimizer uses the SGD stochastic gradient descent algorithm, with FC layer parameters wd=1e-7, feature network wd=1e-6, and momentum=0.9;

[0148] The learning rate strategy in the embodiment of the present application is: initial learning rate lr = 0.01, training 15 epochs, and lr drops to the original 0.1 every 5 epochs;

[0149] The training machine used in the embodiment of the present application is 8*2080Ti, batchsize=640. That is, category calculation can be performed on 8 CPUs at the same time, and each GPU is responsible for a part of the category calculation. For example, for a classification task of 1000 categories, there are 8 GPUs in total, and the FC parameters on each GPU are only 1000 / 8=125 categories of parameters, card 0 is responsible for 0-124, card 1 is responsible for 125-249, and card 7 is responsible for 750-999, which significantly improves the batchsize in training and speeds up the training.

[0150] After the image classification model is trained through the above scheme, the computer device can process the input target image with the image classification model and generate a recognition result corresponding to the target image, which is used to indicate the most likely identity information of the face image contained in the target image.

[0151] To summarize, when it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After updating the target classification model through the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification.

[0152] Fig.11 : is a structural block diagram of an image classification device according to an exemplary embodiment. The device includes:

[0153] The sample acquisition module 1101 is used to acquire training sample images;

[0154] The sample image processing module 1102 is used to process the training sample image through the target classification model to obtain the output value corresponding to the training sample image;

[0155] A first loss function value acquisition module 1103, configured to acquire a first loss function value according to the output value of the training sample image and the label value of the training sample image;

[0156] A second loss function value acquisition module 1104 is used to acquire the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model;

[0157] The model updating module 1105 is used to update the target classification model based on the first loss function value and the second loss function value, so that the updated target classification model can classify the input target image.

[0158] In a possible implementation, the model updating module includes:

[0159] A loss function weighting unit, used to weight the first loss function value and the second loss function value to obtain a target loss function value;

[0160] A model updating unit is used to perform back-propagation update on the target classification model according to the target loss function value.

[0161] In a possible implementation, the second loss function value acquisition module is further used to:

[0162] Obtain the parameters of 2M types of fully connected layers, and normalize the parameters of each type of fully connected layer to construct normalized vectors corresponding to the 2M types of fully connected layers;

[0163] Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix;

[0164] The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model;

[0165] Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as a second loss function value corresponding to each of the training sample images in a training batch of the target classification model.

[0166] In a possible implementation, the sample acquisition module further includes:

[0167] A target sample image acquisition unit, used for acquiring a target sample image;

[0168] The data enhancement unit is used to perform data enhancement processing on the target sample image to obtain the training sample image; the data enhancement processing includes at least one operation of rotation, translation, cropping, flipping, and overlaying.

[0169] In a possible implementation, the data enhancement unit is further configured to:

[0170] Performing data enhancement processing on the target sample image to obtain an enhanced sample image;

[0171] The enhanced sample image is converted to a target size and normalized to obtain the training sample image.

[0172] To summarize, when it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After updating the target classification model through the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification.

[0173] Fig.12: is a structural block diagram of an image classification device according to an exemplary embodiment. The device includes:

[0174] The target image acquisition module 1201 is used to acquire the target image;

[0175] The target image processing module 1202 is used to process the target image through the target classification model to obtain the classification result of the target image;

[0176] Among them, the target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model.

[0177] To summarize, when it is necessary to train the target classification model, data processing can be performed through the target classification model according to the training sample image to obtain the output value corresponding to the training sample image, and the first loss function value between the output value and the label value can be calculated through the loss function; and in the training process, the second loss function value can be obtained according to the product of the normalized parameters of at least two types of fully connected layers in the target classification model. After updating the target classification model through the first loss function value and the second loss function value, the target classification model improves the recognition ability of the training sample image while improving the discrimination of the parameters between different fully connected layers, thereby improving the discrimination of different fully connected layers when extracting features from the training sample image, and improving the accuracy of image classification.

[0178] See also Fig.13 , which is a schematic diagram of a computer device provided according to an exemplary embodiment of the present application, the computer device includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0179] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0180] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0181] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] In an exemplary embodiment, a computer-readable storage medium is also provided, which is used to store at least one computer program, and the at least one computer program is loaded and executed by a processor to implement all or part of the steps in the above method. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0183] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0184] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image classification method, characterized in that: The method comprises: Obtain training sample images; Processing the training sample image through the target classification model to obtain an output value corresponding to the training sample image; Obtaining a first loss function value according to the output value of the training sample image and the label value of the training sample image; Obtaining a second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model; The target classification model is updated based on the first loss function value and the second loss function value, so that the updated target classification model performs classification processing on the input target image; The step of obtaining the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model comprises: Obtain the parameters of 2M types of fully connected layers, and normalize the parameters of each type of fully connected layer to construct normalized vectors corresponding to the 2M types of fully connected layers; Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix; The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model; Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as a second loss function value corresponding to each of the training sample images in a training batch of the target classification model.

2. The method according to claim 1, characterized in that The updating of the target classification model based on the first loss function value and the second loss function value includes: Weighting the first loss function value and the second loss function value to obtain a target loss function value; According to the target loss function value, the target classification model is back-propagated and updated.

3. The method according to claim 1 or 2, characterized in that: The step of acquiring a training sample image comprises: Acquire a target sample image; Performing data enhancement processing on the target sample image to obtain the training sample image; the data enhancement processing includes at least one operation of rotation, translation, cropping, flipping, and overlaying.

4. The method according to claim 3, characterized in that The performing data enhancement processing on the target sample image to obtain the training sample image includes: Performing data enhancement processing on the target sample image to obtain an enhanced sample image; The enhanced sample image is converted to a target size and normalized to obtain the training sample image.

5. An image classification method, characterized in that: The method comprises: Get the target image; Processing the target image through the target classification model to obtain a classification result of the target image; The target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model; Obtaining the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model, including: Obtain the parameters of 2M types of fully connected layers, and normalize the parameters of each type of fully connected layer to construct normalized vectors corresponding to the 2M types of fully connected layers; Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix; The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model; Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as a second loss function value corresponding to each of the training sample images in a training batch of the target classification model.

6. An image classification device, characterized in that: The device comprises: A sample acquisition module is used to acquire training sample images; A sample image processing module is used to process the training sample image through a target classification model to obtain an output value corresponding to the training sample image; A first loss function value acquisition module, used to acquire a first loss function value according to the output value of the training sample image and the label value of the training sample image; A second loss function value acquisition module, used to acquire a second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model; A model updating module, used for updating the target classification model based on the first loss function value and the second loss function value, so that the updated target classification model performs classification processing on the input target image; The second loss function value acquisition module is used to: Obtain the parameters of 2M types of fully connected layers, and normalize the parameters of each type of fully connected layer to construct normalized vectors corresponding to the 2M types of fully connected layers; Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix; The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model; Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as a second loss function value corresponding to each of the training sample images in a training batch of the target classification model.

7. An image classification device, characterized in that: The device comprises: A target image acquisition module, used for acquiring a target image; A target image processing module is used to process the target image through a target classification model to obtain a classification result of the target image; The target classification model is updated based on the first loss function value and the second loss function value; the first loss function value is obtained by processing the training sample image according to the target classification model, and the label value of the training sample image; the second loss function value is obtained according to the normalized inner product of the parameters of at least two types of fully connected layers in the target classification model; Obtaining the second loss function value according to normalized inner products of parameters of at least two types of fully connected layers in the target classification model, including: Obtain the parameters of 2M types of fully connected layers, and normalize the parameters of each type of fully connected layer to construct normalized vectors corresponding to the 2M types of fully connected layers; Select the normalized vectors corresponding to the fully connected layers of the M categories, make inner products with the normalized vectors corresponding to the fully connected layers of the remaining M categories, and construct the inner product results into an inner product matrix; The inner product matrix is ​​linearly transformed into a target matrix; the number of columns of the target matrix is ​​the number of training samples in a training batch of the target classification model; Each element in the target matrix is ​​accumulated column by column, and the accumulated result is determined as a second loss function value corresponding to each of the training sample images in a training batch of the target classification model.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the image classification method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image classification method according to any one of claims 1 to 5.

Images