Method and device for training image classification model and for image classification

Abstract

The embodiments of the present disclosure disclose methods, devices, electronic devices and media for training image classification models. A specific implementation of the method includes: inputting sample images in a pre-acquired initial sample set and a new sample set into the initial image classification model, generating first loss values ​​corresponding to the initial sample set and the new sample set respectively, wherein the initial image classification model includes a fully connected layer corresponding to the new sample set and a backbone network trained based on the initial sample set and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the results of the fully connected layer output corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; adjusting the weights associated with the generated first loss value in the initial image classification model to train the initial image classification model. This implementation improves the scalability of training image classification models.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to methods, devices, electronic devices, and media for training image classification models and for image classification. Background Art

[0002] With the development of machine learning technology, various image classification models such as target recognition and face recognition have been increasingly widely used.

[0003] In the process of model training, when it is necessary to use training data acquired in new scenes or new periods to iteratively upgrade the model, there are usually two ways to adopt: the first way is to mix the newly acquired training data with the original training data into a new training data set to retrain the model, and the other way is to only use the new training data to fine-tune the original model and discard the old training data. Therefore, a more effective image classification model training method is needed. Summary of the invention

[0004] Embodiments of the present disclosure provide methods, devices, electronic devices, and media for training image classification models and for image classification.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for training an image classification model, the method comprising: obtaining an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information; inputting the sample images in the initial sample set and the new sample set into the initial image classification model, generating first loss values ​​corresponding to the initial sample set and the new sample set, respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; adjusting the weights associated with the generated first loss values ​​in the initial image classification model to train the initial image classification model.

[0006] In some embodiments, the sample annotation information included in the new sample set includes sample annotation information that is not included in the initial sample set.

[0007] In some embodiments, the initial sample set is divided into a first number of batches, and the new sample set is divided into a second number of batches; and the sample images in the initial sample set and the new sample set are input into the initial image classification model to generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively, including: integrating the sample images in the initial sample set and the new sample set according to the batch dimension to generate a total sample image set; inputting the total sample image set into the backbone network to generate feature vector sets corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively; inputting the feature vectors in the generated feature vector sets into the fully connected layers corresponding to the first number of batches and the second number of batches, respectively, to generate fully connected layer output results; based on the generated fully connected layer output results and the corresponding sample annotation information, generating first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively.

[0008] In some embodiments, the above-mentioned generating first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively based on the generated fully connected layer output results and the corresponding sample annotation information includes: for samples in the initial sample set of the first number of batches and the new sample set of the second number of batches, using the loss function corresponding to the samples of the batch, generating the first loss value corresponding to the sample according to the fully connected layer output results and sample annotation information corresponding to the samples of the batch.

[0009] In some embodiments, the above-mentioned adjustment of the weights associated with the generated first loss value in the initial image classification model includes: fusing the generated first loss values ​​to generate a first total loss value; and adjusting the weights in the initial image classification model according to the first total loss value.

[0010] In some embodiments, the batch size (batchsize) of the new sample set is larger than the batch size of the initial sample set; and adjusting the weights in the initial image classification model according to the first total loss value includes: adjusting the weights of the fully connected layer corresponding to the new sample set in the initial image classification model at a first learning rate according to the first total loss value.

[0011] In some embodiments, the method further includes: in response to determining that the training completion condition of the initial image classification model is met, determining the initial image classification model after weight adjustment as a quasi-image classification model; inputting the sample images in the initial sample set and the new sample set into the quasi-image classification model to generate second loss values ​​corresponding to the initial sample set and the new sample set respectively; fusing the generated second loss values ​​to generate a second total loss value; and adjusting the weights of the backbone network and the fully connected layer included in the quasi-image classification model at a second learning rate according to the second total loss value.

[0012] In some embodiments, the second learning rate is less than the first learning rate.

[0013] In a second aspect, an embodiment of the present disclosure provides a method for image classification, the method comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to any of the methods described in the first aspect.

[0014] In a third aspect, an embodiment of the present disclosure provides a device for training an image classification model, the device comprising: a first acquisition unit, configured to acquire an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information; a first generation unit, configured to input the sample images in the initial sample set and the new sample set into the initial image classification model, and generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the results of the output of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; an adjustment unit, configured to adjust the weights associated with the generated first loss values ​​in the initial image classification model to train the initial image classification model.

[0015] In some embodiments, the sample annotation information included in the new sample set includes sample annotation information that is not included in the initial sample set.

[0016] In some embodiments, the above-mentioned initial sample set is divided into a first number of batches, and the above-mentioned new sample set is divided into a second number of batches; and the above-mentioned first generation unit includes: an integration module, configured to integrate the sample images in the initial sample set and the new sample set according to the batch dimension to generate a total sample image set; a feature extraction module, configured to input the total sample image set into the backbone network, and generate feature vector sets corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively; the first generation module, configured to input the feature vectors in the generated feature vector set into the fully connected layers corresponding to the first number of batches and the second number of batches respectively, to generate fully connected layer output results; the second generation module, configured to generate first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively based on the generated fully connected layer output results and the corresponding sample annotation information.

[0017] In some embodiments, the second generation module is further configured to: for samples in the initial sample set of the first number of batches and the new sample set of the second number of batches, use the loss function corresponding to the samples of the batch to generate a first loss value corresponding to the sample according to the fully connected layer output results and sample annotation information corresponding to the samples of the batch.

[0018] In some embodiments, the adjustment unit includes: a fusion module configured to fuse the generated first loss values ​​to generate a first total loss value; and an adjustment module configured to adjust the weights in the initial image classification model according to the first total loss value.

[0019] In some embodiments, the batch size of the new sample set is larger than the batch size of the initial sample set; and the adjustment module is further configured to: adjust the weights of the fully connected layer corresponding to the new sample set in the initial image classification model at a first learning rate according to the first total loss value.

[0020] In some embodiments, the device is further configured to: in response to determining that the training completion condition of the initial image classification model is met, determine the initial image classification model with adjusted weights as a quasi-image classification model; input the sample images in the initial sample set and the new sample set into the quasi-image classification model to generate second loss values ​​corresponding to the initial sample set and the new sample set respectively; fuse the generated second loss values ​​to generate a second total loss value; and adjust the weights of the backbone network and the fully connected layer included in the quasi-image classification model at a second learning rate according to the second total loss value.

[0021] In some embodiments, the second learning rate is less than the first learning rate.

[0022] In a fourth aspect, an embodiment of the present disclosure provides a device for image classification, the device comprising: a second acquisition unit, configured to acquire an image to be processed; a second generation unit, configured to input the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to any of the methods described in the first aspect.

[0023] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0024] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0025] The embodiments of the present disclosure provide methods, devices, electronic devices and media for training image classification models and for image classification. By dividing the sample images of the initial sample set and the new sample set into different data branches and inputting them into the model, the data branches are independent of each other. Based on the model trained with the initial sample set, there is no need to check whether the initial sample set and the new sample set overlap. In addition, by sharing the backbone network and setting corresponding fully connected layers for the initial sample set and the new sample set respectively, it is possible to fine-tune the model with new data without changing the network model structure, without the need for large-scale retraining. This provides a scalable method that supports large-scale data batch training of image classification models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0028] Figure 2a is a flowchart of an embodiment of a method for training an image classification model according to the present disclosure;

[0029] Figure 2b is a flowchart of yet another embodiment of a method for training an image classification model according to the present disclosure;

[0030] Figure 3 is a schematic diagram of an application scenario of a method for training an image classification model according to an embodiment of the present disclosure;

[0031] Figure 4 is a flow chart of an embodiment of a method for image classification according to the present disclosure;

[0032] Figure 5 is a structural schematic diagram of an embodiment of a device for training an image classification model according to the present disclosure;

[0033] Figure 6 is a structural schematic diagram of an embodiment of a device for image classification according to the present disclosure;

[0034] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0036] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] Figure 1 An exemplary architecture 100 is shown to which the method for training an image classification model and an image classification method or an apparatus for training an image classification model and an image classification method of the present disclosure can be applied.

[0038] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0039] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as web browser applications, image processing applications, search applications, instant messaging tools, email clients, etc.

[0040] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting human-computer interaction, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0041] The server 105 may be a server that provides various services, such as a background server that provides support for image processing applications on the terminal devices 101, 102, and 103. The background server may train an image classification model, and then send the trained image classification model to the terminal device so that the terminal device performs an image classification task.

[0042] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0043] It should be noted that the method for training the image classification model provided in the embodiments of the present disclosure is generally performed by the server 105, and accordingly, the apparatus for training the image classification model is generally set in the server 105. The method for image classification provided in the embodiments of the present disclosure can be performed by the terminal device 101, 102, 103 or the server 105, and accordingly, the apparatus for image classification can be set in the terminal device 101, 102, 103 or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] Continue to refer Figure 2a , shows a process 200 of an embodiment of a method for training an image classification model according to the present disclosure. The method for training an image classification model comprises the following steps:

[0046] Step 201: Acquire an initial sample set and a new sample set.

[0047] In this embodiment, the execution subject of the method for training the image classification model (such as Figure 1 The server 105 shown in the figure can obtain the initial sample set and the new sample set through a wired connection or a wireless connection. The initial sample set and the new sample set may include sample images and corresponding sample annotation information. Usually, the sample images and corresponding sample annotation information included in the initial sample set may be used as old training data. The sample images and corresponding sample annotation information included in the new sample set may be used as new training data. The new training data may be used to characterize the training data obtained and applied to a new scene or a new period.

[0048] In this embodiment, the execution subject may obtain the initial sample set and the new sample set pre-stored locally, or may obtain the initial sample set and the new sample set sent by an electronic device (eg, a database server) to which it is communicatively connected.

[0049] As an example, the above-mentioned initial sample set may include various public face recognition data sets. The above-mentioned new sample set may include, for example, face recognition data collected by actual application scenarios (such as access control systems). As another example, the above-mentioned initial sample set may include, for example, various public target detection training data sets. The above-mentioned new sample set may include, for example, road images collected by actual application scenarios (such as autonomous driving scenarios).

[0050] In some optional implementations of this embodiment, the sample annotation information included in the new sample set may include sample annotation information not included in the initial sample set.

[0051] In these implementations, it means that the category of the image classification model trained by the above-mentioned new sample set is different from the category of the image classification model trained by the above-mentioned initial sample set (such as face ID, obstacle type for autonomous driving, etc.).

[0052] Based on the above optional implementation methods, this solution can be applied to the training of image classification models when the categories corresponding to the original training data and the new training data are different. Since there is no need to change the backbone network and structure of the model, and the training results of the original training data can still be used, the model training efficiency in this case can be significantly improved.

[0053] In some optional implementations of this embodiment, the initial sample set may be divided into a first number of batches, and the new sample set may be divided into a second number of batches.

[0054] In these implementations, the execution entity may input the sample images in the initial sample set and the new sample set into the model to be trained according to the divided batches.

[0055] Step 202: input the sample images in the initial sample set and the new sample set into the initial image classification model to generate first loss values ​​corresponding to the initial sample set and the new sample set respectively.

[0056] In this embodiment, the execution subject may input the sample images in the initial sample set and the new sample set obtained in step 201 into the initial image classification model in various ways to generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively. The initial image classification model may include an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set. The first loss value may be generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set, respectively, and the corresponding sample annotation information.

[0057] It should be noted that, before the above step 202, the execution entity for training the first image classification model can use the above initial sample set to train the first image classification model including the backbone network and the fully connected layer. Among them, the above backbone network can be used to characterize a feature extraction network, such as a multi-layer convolutional neural network. The above backbone network can output feature vectors corresponding to the sample images in the input initial sample set and the new sample set, respectively. Then, the backbone network in the trained first image classification model is connected in parallel to the fully connected layer with initialized weights corresponding to the above new sample set, thereby forming the above initial image classification model.

[0058] It should be noted that those skilled in the art know that the above-mentioned execution entity can use the loss function to determine the difference between the results of the fully connected layer output corresponding to the initial sample set (usually after passing through the softmax function) and the sample annotation information corresponding to the sample image input in the initial sample set, so as to generate a first loss value corresponding to the initial sample set.

[0059] Optionally, if the above-mentioned initial sample set is input into the above-mentioned initial image classification model in batches of a first number, then the above-mentioned fully connected layers corresponding to the initial sample set are also the first number, and thus the first loss value corresponding to the above-mentioned initial sample set is also the first number.

[0060] Similarly, the first loss value corresponding to the above new sample set can be obtained, which will not be repeated here.

[0061] In some optional implementations of this embodiment, see Figure 2b , based on the initial sample set being divided into a first number (e.g. Figure 2b The new sample set is divided into the second number (e.g. Figure 2bThe execution subject may input the sample images in the initial sample set and the new sample set into the initial image classification model according to the following steps to generate the first loss values ​​corresponding to the initial sample set and the new sample set respectively:

[0062] In the first step, the sample images in the initial sample set and the new sample set are integrated according to the batch dimension to generate a total sample image set.

[0063] In these implementations, the execution entity may integrate the sample images in the initial sample set and the new sample set according to the batch dimension to generate a total sample image set.

[0064] In the second step, the total sample image set is input into the backbone network to generate feature vector sets corresponding to the initial sample sets of the first number of batches and the new sample sets of the second number of batches, respectively.

[0065] In these implementations, the feature vector set may include feature vector subsets corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively, wherein the feature vector subsets may correspond to the sample images of each batch of input.

[0066] In the third step, the feature vectors in the generated feature vector set are respectively input into the fully connected layers corresponding to the first number of batches and the second number of batches to generate the fully connected layer output results.

[0067] In these implementations, the execution subject may input the feature vectors in the generated feature vector set into the fully connected layers corresponding to the first number of batches and the second number of batches, respectively, to generate fully connected layer output results. It can be seen that each batch of sample images may correspond to the feature vectors of the batch, and then to the fully connected layer output results obtained by the fully connected layer input with the feature vectors of the batch.

[0068] The fourth step is to generate first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively, based on the generated fully connected layer output results and the corresponding sample annotation information.

[0069] In these implementations, as an example, the above-mentioned execution entity can use a preset loss function to determine the difference between the output of the fully connected layer output result generated in the third step and the corresponding sample annotation information obtained by the softmax function, thereby generating a first loss value corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively.

[0070] Based on the above optional implementation methods, this solution can relatively isolate the sample data of each batch through multiple data branches, so that without changing the original input data of the model, the new data can be directly input as a new branch, and then by sharing the weights of the backbone network and accessing their respective fully connected layers, it is possible to continue training with new samples while retaining the knowledge obtained from the original data training.

[0071] Optionally, the execution subject may also generate first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively based on the generated fully connected layer output result and the corresponding sample annotation information in the following manner:

[0072] For samples in the initial sample set of the first number of batches and the new sample set of the second number of batches, the loss function corresponding to the samples of the batch is used to generate the first loss value corresponding to the sample according to the fully connected layer output results and sample annotation information corresponding to the samples of the batch.

[0073] In these implementations, the above loss function can correspond to the batch, that is, it is not necessary for all batches of samples to correspond to the same loss function, but the loss function matching the training samples of the batch can be flexibly selected (for example, the training samples of the batch used for face recognition can correspond to the arcface loss function).

[0074] Based on the above optional implementation methods, this solution can use different parts (batches) of training data to access different loss functions, and is therefore more compatible with the differences between different parts of data, simplifying the learning process of the model, making the training process easier, and improving the efficiency of model training.

[0075] Step 203: adjust the weights associated with the generated first loss value in the initial image classification model to train the initial image classification model.

[0076] In this embodiment, the execution subject may adjust the weights associated with the first loss value generated in step 202 in the initial image classification model in various ways to train the initial image classification model. As an example, the execution subject may adjust the weights involved in the first loss value generated in step 202 by mini-batch gradient descent (MBGD), and then train the initial image classification model by back propagation. Alternatively, the execution subject may also only adjust the weights of the backbone network and the weights of the full connection corresponding to the new sample set.

[0077] In some optional implementations of this embodiment, the execution subject may adjust the weight associated with the generated first loss value in the initial image classification model according to the following steps:

[0078] In the first step, the generated first loss values ​​are integrated to generate a first total loss value.

[0079] In these implementations, the execution subject may fuse the first loss values ​​generated in step 202 in various ways to generate a first total loss value. As an example, the execution subject may perform weighted summation on the generated first loss values ​​to generate a first total loss value. The weights of the weighted summation may be pre-set according to the actual application scenario, or may be obtained through training as a learning variable, which is not limited here.

[0080] In the second step, the weights in the initial image classification model are adjusted according to the first total loss value.

[0081] In these implementations, based on the first total loss value generated in the first step, the execution entity may adjust the weights in the initial image classification model in various ways. As an example, the execution entity may adjust the trainable weights in the initial image classification model using a small batch gradient descent method, and then train the initial image classification model through back propagation. Among them, the trainable weights may include all the adjustable weights in the initial image classification model, or may only include some weights selected from all the adjustable weights, which is not limited here. Optionally, the execution entity may also only adjust the weights of the backbone network and the fully connected weights corresponding to the new sample set.

[0082] Based on the above optional implementation method, this solution can fuse the multiple first loss values ​​generated so that the obtained loss value contains information from both the initial sample set and the new sample set, thereby improving data utilization and helping to improve model training effects.

[0083] Optionally, based on the fact that the batch size of the new sample set is larger than the batch size of the initial sample set, the execution entity may also adjust the weight of the fully connected layer corresponding to the new sample set in the initial image classification model at a first learning rate according to the first total loss value.

[0084] In these implementations, the batch size of the new sample set may be generally larger than the batch size of the initial sample set. As an example, the batch size of the initial sample set is generally smaller than a preset threshold. The preset threshold may even be set to 2.

[0085] Based on the above optional implementation method, since the backbone network and the fully connected layer corresponding to the initial sample set in the initial image classification model are trained based on the initial sample set, when using the new sample set to adjust only the weights of the fully connected layer corresponding to the new sample set in the initial image classification model at the first learning rate, the training speed of the model can be improved by reducing the batch size in the initial sample set, thereby taking into account both the effectiveness and speed of model training.

[0086] Optionally, after adjusting the weight of the fully connected layer corresponding to the new sample set in the initial image classification model at the first learning rate according to the first total loss value, the execution body may further continue to perform the following steps:

[0087] S1. In response to determining that a training completion condition of the initial image classification model is met, determining the initial image classification model after weight adjustment as a quasi image classification model.

[0088] In these implementations, in response to determining that the training completion condition of the initial image classification model is met, the execution subject may determine the initial image classification model after weight adjustment as a quasi-image classification model. The training completion condition may be set according to actual needs, such as reaching the training time, reaching the number of iterations, and the accuracy of the test set meeting the requirements.

[0089] S2. Input the sample images in the initial sample set and the new sample set into the quasi-image classification model to generate second loss values ​​corresponding to the initial sample set and the new sample set respectively.

[0090] In these implementations, the execution entity may input the sample images in the initial sample set and the new sample set into the quasi-image classification model determined in the above step S1 by a method similar to the above step 202 and its optional implementations, and generate second loss values ​​corresponding to the initial sample set and the new sample set, respectively.

[0091] It should be noted that the relationship between the batch size of the new sample set input here and the batch size of the initial sample set is usually not specifically limited.

[0092] S3. Fusing the generated second loss values ​​to generate a second total loss value.

[0093] In these implementations, the execution subject may fuse the second loss values ​​generated in step S2 in various ways to generate a second total loss value. As an example, the execution subject may perform weighted summation on the generated second loss values ​​to generate a second total loss value. The weights of the weighted summation may be pre-set according to the actual application scenario, or may be obtained through training as a learning variable, which is not limited here.

[0094] S4. According to the second total loss value, adjust the weights of the backbone network and the fully connected layer included in the quasi-image classification model with a second learning rate.

[0095] In these implementations, according to the second total loss value generated in step S3, the execution subject may adjust the weights of the backbone network and the fully connected layer included in the quasi-image classification model at a second learning rate. The fully connected layer may generally include all fully connected layers corresponding to the initial sample set and the new sample set.

[0096] It should be noted that the execution subject may iterate by executing the steps S2-S4 for multiple times before the training is finished, and finally determine the quasi-image classification model that meets the training end condition as the trained image classification model.

[0097] Based on the above optional implementation methods, this scheme adjusts the weights of the fully connected layer corresponding to the new sample set in the quasi-image classification model determined based on the initial image classification model with a first learning rate, and adjusts the weights of the backbone network and the fully connected layer in the above quasi-image classification model with a second learning rate, thereby forming a complete image classification model training scheme and improving the training efficiency of the image classification model.

[0098] Optionally, the second learning rate is usually smaller than the first learning rate.

[0099] Based on the above optional implementation methods, this solution can further improve the training effect of the image classification model by setting different learning rates in the above two processes.

[0100] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of a method for training an image classification model according to an embodiment of the present disclosure. Figure 3In the application scenario, the server can first obtain the initial sample set 301 and the new sample set 302. Then, the sample images in the initial sample set 301 and the sample images in the new sample set 302 are respectively input into the initial image classification model 303. The initial image classification model 303 includes a backbone network 3031 and a fully connected layer 3032 obtained by training based on the initial sample set 301, and an initialized fully connected layer 3033. The fully connected layer 3032 and the fully connected layer 3033 correspond to the initial sample set 301 and the new sample set 302, respectively. Afterwards, the first loss value 304 corresponding to the initial sample set 301 is generated according to the output result of the fully connected layer 3032 corresponding to the initial sample set 301 and the sample annotation information. Similarly, the first loss value 305 corresponding to the new sample set 302 is generated. Afterwards, the network weights included in the initial image classification model 303 are adjusted to train the initial image classification model 303.

[0101] At present, one of the existing technologies is usually to retrain the model based on the new and old data, but it is time-consuming and laborious because it is necessary to check whether there are repeated samples in the new and old data, and the entire model needs to change the structure and weight when the number of classifications changes. When the difference between the new and old data is large, it will cause the model training to be difficult and the fitting is not robust enough for some data scenes with a small number. One of the existing technologies also adopts the method of fine-tuning the model based on the new data and discarding the previous data, which causes the model to gradually forget the knowledge learned in the previous data, and there is also a risk of overfitting when only fitting the new data, resulting in a decrease in the generalization ability of the model. The method provided by the above-mentioned embodiment of the present disclosure divides the sample images of the initial sample set and the new sample set into different data branches and inputs them into the model, so that each data branch is independent of each other. On the basis of the model trained by the initial sample set, there is no need to check whether the initial sample set and the new sample set overlap, and by sharing the backbone network and setting corresponding fully connected layers for the initial sample set and the new sample set, it is realized to fine-tune the model with new data without changing the network model structure, without large-scale retraining, and provides a scalable method for supporting large-scale data batch training of image classification models.

[0102] Further references Figure 4 , which shows a process 400 of another embodiment of a method for image classification. The process 400 of the method for image classification includes the following steps:

[0103] Step 401: Obtain an image to be processed.

[0104] In this embodiment, the execution subject of the method for image classification (eg Figure 1The server 105 or the terminal devices 101, 102, 103) shown in the figure can obtain the image to be processed in various ways. The above-mentioned image to be processed may include various images that can be used for image classification, which is not limited here.

[0105] In this embodiment, as an example, the image to be processed may be an image containing a face. As another example, the image to be processed may also be an image containing a lane indicator line. The execution subject may obtain the image to be processed from a local or communication-connected electronic device.

[0106] Step 402: input the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed.

[0107] In this embodiment, the execution subject may input the image to be processed obtained in step 401 into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed. The image processing model may be trained by a method for training an image classification model as described in any of the implementations in the aforementioned embodiments. The image classification result may correspond to the image classification model. As an example, when the image classification model is a face recognition model, the image classification result may be used to characterize the identification information of a person displayed in a face image. As another example, when the image classification model is an autonomous driving obstacle detection model, the image classification result may be used to indicate whether it belongs to a pedestrian, a vehicle, a traffic light, etc.

[0108] from Figure 4 It can be seen that the process 400 of the method for image classification in this embodiment embodies the steps of processing the image using the image classification model trained based on the method for training the image classification model. Therefore, the image classification model used in the solution described in this embodiment can improve the accuracy of image classification because the knowledge obtained by training with the initial sample set is not forgotten while training with the new sample set.

[0109] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training an image classification model. Figure 2a or Figure 2b Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] like Figure 5As shown, the apparatus 500 for training an image classification model provided by this embodiment includes a first acquisition unit 501, a first generation unit 502, and an adjustment unit 503. The first acquisition unit 501 is configured to acquire an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information; the first generation unit 502 is configured to input the sample images in the initial sample set and the new sample set into the initial image classification model, and generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network obtained by training based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; the adjustment unit 503 is configured to adjust the weights associated with the generated first loss values ​​in the initial image classification model to train the initial image classification model.

[0111] In the apparatus 500 for training an image classification model in this embodiment, the specific processing of the first acquisition unit 501, the first generation unit 502 and the adjustment unit 503 and the technical effects thereof can be referred to in Figure 2a The relevant descriptions of step 201, step 202 and step 203 in the corresponding embodiment are not repeated here.

[0112] In some optional implementations of this embodiment, the sample annotation information included in the new sample set may include sample annotation information not included in the initial sample set.

[0113] In some optional implementations of this embodiment, the above-mentioned initial sample set can be divided into a first number of batches. The above-mentioned new sample set can be divided into a second number of batches. The above-mentioned first generation unit 502 may include: an integration module (not shown in the figure), configured to integrate the sample images in the initial sample set and the new sample set according to the batch dimension to generate a total sample image set; a feature extraction module (not shown in the figure), configured to input the total sample image set into the backbone network, and generate feature vector sets corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively; the first generation module (not shown in the figure), configured to input the feature vectors in the generated feature vector set into the fully connected layers corresponding to the first number of batches and the second number of batches, respectively, to generate the fully connected layer output results; the second generation module (not shown in the figure), configured to generate the first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively based on the generated fully connected layer output results and the corresponding sample annotation information.

[0114] In some optional implementations of this embodiment, the above-mentioned second generation module can be further configured to: for samples in the initial sample set of the first number of batches and the new sample set of the second number of batches, use the loss function corresponding to the samples of the batch to generate the first loss value corresponding to the sample according to the fully connected layer output results and sample annotation information corresponding to the samples of the batch.

[0115] In some optional implementations of this embodiment, the above-mentioned adjustment unit 503 may include: a fusion module (not shown in the figure), configured to fuse the generated first loss values ​​to generate a first total loss value; an adjustment module (not shown in the figure), configured to adjust the weights in the initial image classification model according to the first total loss value.

[0116] In some optional implementations of this embodiment, the batch size of the new sample set may be larger than the batch size of the initial sample set. The adjustment module may be further configured to: adjust the weight of the fully connected layer corresponding to the new sample set in the initial image classification model at a first learning rate according to the first total loss value.

[0117] In some optional implementations of the present embodiment, the above-mentioned device 500 for training an image classification model may be further configured to: in response to determining that the training completion condition of the initial image classification model is met, determine the initial image classification model after weight adjustment as a quasi-image classification model; input the sample images in the initial sample set and the new sample set into the quasi-image classification model to generate second loss values ​​corresponding to the initial sample set and the new sample set respectively; fuse the generated second loss values ​​to generate a second total loss value; and adjust the weights of the backbone network and the fully connected layer included in the quasi-image classification model at a second learning rate according to the second total loss value.

[0118] In some optional implementations of this embodiment, the second learning rate may be smaller than the first learning rate.

[0119] The device provided by the above-mentioned embodiment of the present disclosure divides the sample images of the initial sample set and the new sample set acquired by the first acquisition unit 501 into different data branches for input into the model through the first generation unit 502, so that each data branch is independent of each other. On the basis of the model trained with the initial sample set, there is no need to check whether the initial sample set and the new sample set overlap. In addition, by sharing the backbone network and setting corresponding fully connected layers for the initial sample set and the new sample set respectively, it is possible to fine-tune the model with new data without changing the network model structure, without the need for large-scale retraining, and provides a scalable method for supporting large-scale data batch training of image classification models.

[0120] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for image classification, and the device embodiment is Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 6 As shown, the apparatus 600 for image classification provided in this embodiment includes a second acquisition unit 601 and a second generation unit 602. The second acquisition unit 601 is configured to acquire an image to be processed; the second generation unit 602 is configured to input the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to any method for training an image classification model described in the above embodiments.

[0122] The apparatus provided by the above-mentioned embodiment of the present disclosure includes a step of processing the image acquired by the second acquisition unit 601 by using the image classification model trained by the method for training the image classification model by the second generation unit 602. Therefore, the image classification model used in the scheme described in this embodiment can improve the accuracy of image classification because the knowledge acquired by training with the initial sample set is not forgotten while training with the new sample set.

[0123] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of server 105)700. Figure 7 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0125] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0126] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0127] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0128] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information; inputs the sample images in the initial sample set and the new sample set into the initial image classification model, and generates first loss values ​​corresponding to the initial sample set and the new sample set, respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; adjusts the weights associated with the generated first loss values ​​in the initial image classification model to train the initial image classification model; or

[0129] The electronic device is configured to: obtain an image to be processed; input the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to the aforementioned method for training an image classification model.

[0130] Computer program code for performing the operations of the embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C", Python, or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including a first acquisition unit, a first generation unit and an adjustment unit; or a processor including a second acquisition unit and a second generation unit. The names of these units do not constitute a limitation on the units themselves in certain cases. For example, the first acquisition unit may also be described as "a unit for acquiring an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information".

[0133] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A method for training an image classification model, comprising: Acquire an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information, the initial sample set is divided into a first number of batches, and the new sample set is divided into a second number of batches; Inputting sample images in the initial sample set and the new sample set into an initial image classification model, generating first loss values ​​corresponding to the initial sample set and the new sample set respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information; adjusting a weight associated with the generated first loss value in the initial image classification model to train the initial image classification model; The step of inputting the sample images in the initial sample set and the new sample set into the initial image classification model to generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively, includes: Integrate the sample images in the initial sample set and the new sample set according to the batch dimension to generate a total sample image set; Inputting the total sample image set into the backbone network, generating feature vector sets corresponding to the initial sample sets of the first number of batches and the new sample sets of the second number of batches respectively; Inputting the feature vectors in the generated feature vector set to the fully connected layers corresponding to the first number of batches and the second number of batches respectively, to generate a fully connected layer output result; Based on the generated fully connected layer output result and the corresponding sample annotation information, first loss values ​​corresponding to the initial sample sets of the first number of batches and the new sample sets of the second number of batches are generated.

2. The method according to claim 1, wherein: The sample labeling information included in the new sample set includes sample labeling information not included in the initial sample set.

3. The method according to claim 1, wherein: The generating, based on the generated fully connected layer output result and the corresponding sample annotation information, first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches, respectively, comprises: For the samples in the initial sample set of the first number of batches and the new sample set of the second number of batches, the loss function corresponding to the samples of the batch is used to generate a first loss value corresponding to the sample according to the fully connected layer output results and sample annotation information corresponding to the samples of the batch.

4. The method according to any one of claims 1 to 3, wherein: The step of adjusting a weight associated with the generated first loss value in the initial image classification model comprises: The generated first loss values ​​are integrated to generate a first total loss value; According to the first total loss value, weights in the initial image classification model are adjusted.

5. The method according to claim 4, wherein: The batch size of the new sample set is larger than the batch size of the initial sample set; as well as The step of adjusting the weights in the initial image classification model according to the first total loss value includes: According to the first total loss value, the weight of the fully connected layer corresponding to the new sample set in the initial image classification model is adjusted at a first learning rate.

6. The method according to claim 5, wherein: The method further comprises: In response to determining that a training completion condition of the initial image classification model is satisfied, determining the initial image classification model after weight adjustment as a quasi image classification model; Inputting sample images in the initial sample set and the new sample set into the quasi-image classification model to generate second loss values ​​corresponding to the initial sample set and the new sample set respectively; The generated second loss values ​​are integrated to generate a second total loss value; According to the second total loss value, the weights of the backbone network and the fully connected layer included in the quasi-image classification model are adjusted at a second learning rate.

7. The method according to claim 6, wherein: The second learning rate is smaller than the first learning rate.

8. A method for image classification, comprising: Get the image to be processed; The image to be processed is input into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to one of the methods of claims 1-7.

9. A device for training an image classification model, comprising: A first acquisition unit is configured to acquire an initial sample set and a new sample set, wherein the initial sample set and the new sample set include sample images and corresponding sample annotation information, the initial sample set is divided into a first number of batches, and the new sample set is divided into a second number of batches; a first generating unit, configured to input the sample images in the initial sample set and the new sample set into an initial image classification model, and generate first loss values ​​corresponding to the initial sample set and the new sample set, respectively, wherein the initial image classification model includes an initialized fully connected layer corresponding to the new sample set, a backbone network trained based on the initial sample set, and a fully connected layer corresponding to the initial sample set, and the first loss value is generated based on the output results of the fully connected layers corresponding to the initial sample set and the new sample set and the corresponding sample annotation information, respectively; an adjusting unit configured to adjust a weight associated with the generated first loss value in the initial image classification model to train the initial image classification model; Wherein, the first generating unit is further configured to: The sample images in the initial sample set and the new sample set are integrated according to the batch dimension to generate a total sample image set; the total sample image set is input into the backbone network to generate feature vector sets corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches respectively; the feature vectors in the generated feature vector sets are respectively input into the fully connected layers corresponding to the first number of batches and the second number of batches to generate fully connected layer output results; based on the generated fully connected layer output results and the corresponding sample annotation information, the first loss values ​​corresponding to the initial sample set of the first number of batches and the new sample set of the second number of batches are generated.

10. A device for image classification, comprising: A second acquisition unit is configured to acquire an image to be processed; The second generating unit is configured to input the image to be processed into a pre-trained image classification model to obtain an image classification result corresponding to the image to be processed, wherein the image classification model is trained according to one of the methods according to claims 1-7.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images