Classification model training method, image classification method, device and electronic equipment

Abstract

The application discloses a kind of classification model training method, image classification method, device and electronic equipment, by obtaining multiple target training images from first sample image set, using initial model extracts the first feature of the first image region of each target training image, and extract the second image region corresponding to the object under multiple scaling ratios of each target training image, the second feature corresponding to each second image region and image label, according to the first image region in each target training image, the first feature of the first image region and sample label, the second image region in target training image, the second feature corresponding to each second image region and image label, obtain feature loss value, according to feature loss value adjustment initial model's parameter, until reach training end condition when obtaining target classification model.More accurate target classification model is obtained, and more accurate classification is realized using target classification model to image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology in artificial intelligence technology, and more specifically, to a classification model training method, an image classification method, an apparatus, and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, neural network models can provide users with various services, such as speech recognition, image recognition, search, and image classification.

[0003] Image classification technology refers to the technique of identifying and classifying objects within an image, and it is a common image processing method. Among related technologies, machine learning models can be used to identify and classify objects in images. When training a machine learning model, images containing the target objects can be acquired for training. However, in existing technologies, when using the above-mentioned methods to obtain classification models for image classification, the classification results obtained are often inaccurate. Summary of the Invention

[0004] In view of the above problems, embodiments of this application propose a classification model training method, an image classification method, an apparatus, and an electronic device to improve the above problems.

[0005] In a first aspect, embodiments of this application provide a classification model training method, the method comprising: acquiring multiple target training images from a first sample image set, the first sample image set including training images corresponding to multiple categories of objects, each training image being labeled with a first image region including an object image and a sample label corresponding to the first image region, the multiple target training images respectively belonging to training images corresponding to at least two categories of objects; extracting a first feature of the first image region of each target training image in the multiple target training images using an initial model; extracting a second image region corresponding to an object in each target training image at multiple scaling ratios, a second feature corresponding to each second image region, and an image label corresponding to each second image region using the initial model; obtaining a feature loss value based on the first image region, the first feature of the first image region, and the sample label of the first image region in each target training image, the second image region in the target training image, the second feature corresponding to each second image region, and the image label corresponding to each second image region; and adjusting the parameters of the initial model based on the feature loss value until a training termination condition is met to obtain a target classification model.

[0006] Secondly, embodiments of this application provide an image classification method, which includes: acquiring an image to be classified; calling a target classification model obtained by the above-mentioned classification model training method to classify the image to be classified, and obtaining a classification result.

[0007] Thirdly, embodiments of this application provide a classification model training device, which includes: a training image acquisition module, a feature extraction module, an information extraction module, a loss value acquisition module, and a model acquisition module. The training image acquisition module is used to acquire multiple target training images from a first sample image set, which includes training images corresponding to multiple categories of objects. Each training image is labeled with a first image region containing an object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to at least two categories of objects. The feature extraction module is used to extract a first feature of the first image region of each target training image from the multiple target training images using an initial model. The information extraction module is used to extract a second image region corresponding to an object in each target training image at multiple scaling ratios, a second feature corresponding to each second image region, and an image label corresponding to each second image region using the initial model. The loss value acquisition module is used to obtain a feature loss value based on the first image region, the first feature, and the sample label of the first image region in each target training image, the second image region, the second feature, and the image label corresponding to each second image region in the target training image, and the image label corresponding to each second image region. The model acquisition module is used to adjust the parameters of the initial model based on the feature loss value until the training termination condition is met to obtain a target classification model.

[0008] In one possible implementation, the loss value acquisition module includes a region selection submodule, a first loss value acquisition submodule, a second loss value acquisition submodule, and a feature loss value acquisition submodule. The region selection submodule is used to select a target image region from a plurality of second image regions corresponding to the target training image based on a first image region in the target training image. The first loss value acquisition submodule is used to obtain a first loss value based on a first feature corresponding to each target training image, a second feature of the target image region of each target training image, and the category to which each target training image belongs. The second loss value acquisition submodule is used to obtain a second loss value based on the first image region corresponding to each target training image and the sample label corresponding to the first image region, and each second image region corresponding to each target training image and the image label corresponding to each second image region. The feature loss value acquisition submodule is used to accumulate the first loss value and the second loss value to obtain a feature loss value.

[0009] In one possible implementation, the first loss value acquisition submodule includes a target feature acquisition unit, a first distance calculation unit, a second distance calculation unit, and a first loss value acquisition unit. The target feature acquisition unit is used to calculate the average of a first feature corresponding to the target training image and a second feature of the target image region of the target training image to obtain the target features of the target training image. The first distance calculation unit is used to calculate the intra-class distance by performing feature distance calculation on the target features of target training images belonging to the same category. The second distance calculation unit is used to calculate the inter-class distance by performing feature distance calculation on the target features of target training images belonging to different categories. The first loss value acquisition unit is used to obtain a first loss value based on the ratio between the intra-class distance and the inter-class distance.

[0010] In one possible implementation, the first loss value acquisition submodule includes a target feature acquisition unit, a first distance calculation unit, a second distance calculation unit, and a first loss value acquisition unit. The target feature acquisition unit is used to calculate the average of a first feature corresponding to the target training image and a second feature of the target image region of the target training image to obtain the target features of the target training image. The first distance calculation unit is used to calculate the intra-class distance by performing feature distance calculation on the target features of target training images belonging to the same category. The second distance calculation unit is used to calculate the inter-class distance by performing feature distance calculation on the target features of target training images belonging to different categories. The first loss value acquisition unit is used to obtain a first loss value based on the ratio between the intra-class distance and the inter-class distance.

[0011] In one possible implementation, the region selection submodule includes a matching degree calculation unit and a region determination unit. The matching degree calculation unit is used to calculate the matching degree between a first image region in the target training image and each corresponding second image region of the target training image to obtain the matching degree value of each second image region corresponding to the target training image; the region determination unit is used to obtain the second image region whose matching degree value is greater than the matching degree threshold as the target image region of the target training image.

[0012] In one possible implementation, the second loss value acquisition submodule includes a first parameter acquisition unit, a second parameter acquisition unit, an inference probability acquisition unit, and a second loss value acquisition unit. The first parameter acquisition unit is used to obtain a quality balance parameter corresponding to each second image region in the target training image based on a first image region and each second image region corresponding to each target training image. The second parameter acquisition unit is used to obtain a sample balance parameter for the target training image based on the number of second image regions and the number of target image regions in each target training image. The inference probability acquisition unit is used to obtain the true label inference probability of the target training image based on the sample labels corresponding to the first image regions and the image labels corresponding to each second image region in the target training image. The second loss value acquisition unit is used to obtain a second loss value based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability.

[0013] In one possible implementation, the first parameter obtaining unit is further configured to calculate the overlap between the first image region corresponding to each target training image and each second image, thereby obtaining the overlap value corresponding to each second image in each target training image, and the overlap value corresponding to each second image is the quality balance parameter corresponding to the second image region.

[0014] In one possible implementation, the second parameter obtaining unit is further configured to obtain the quantity difference between the number of second image regions corresponding to the target training image and the number of target image regions; and to obtain the sample balance parameter of the target training image based on the ratio between the quantity difference corresponding to the target training image and the number of target region images corresponding to the target training image.

[0015] In one possible implementation, the inference probability acquisition unit is further configured to acquire the number of image labels in the image labels corresponding to each second image region in the target training image that are the same as the sample labels corresponding to the first image region; and to use the ratio obtained by comparing the number with the number of second image regions corresponding to the target training image as the true label inference probability of the target training image.

[0016] In one possible implementation, the second loss value acquisition unit is further configured to: calculate the average value of the quality balance parameters corresponding to each second image in the target training image to obtain the quality balance parameters of the target training image; calculate the relative entropy of the true label inference probability of the target training image; multiply the calculated relative entropy by the quality balance parameters and sample balance parameters of the target training image to obtain the loss value corresponding to the target training image; and sum the loss values ​​of each target training image to obtain the second loss value.

[0017] In one possible implementation, the classification model training device further includes a sample acquisition module and a model training module. The sample acquisition module is used to acquire a second set of sample images, which includes sample images corresponding to at least two categories of objects. Each sample image is labeled with a first image region containing the object image and a sample label corresponding to that first image region. The model training module is used to input each sample image from the second set of sample images into a convolutional neural network model, and train the convolutional neural network model based on each sample image to obtain an initial model.

[0018] Fourthly, embodiments of this application also provide an image classification apparatus, which includes a module for acquiring an image to be classified and an image classification module. The module for acquiring the image to be classified is used to acquire the image to be classified; the image classification module is used to call a target classification model obtained by the classification model training apparatus described above to classify the image to be classified and obtain a classification result.

[0019] Fifthly, embodiments of this application also provide an electronic device, including: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the video text extraction method described above is implemented.

[0020] Sixthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the video text extraction method described above.

[0021] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device retrieves the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0022] This application provides a classification model training method, image classification method, apparatus, and electronic device. It acquires multiple target training images from a first set of sample images, extracts a first feature of a first image region in each target training image using an initial model, and extracts a second image region corresponding to an object in each target training image at multiple scaling ratios, along with a second feature and image label for each second image region. Based on the first image region, the first feature and sample label of the first image region in each target training image, the second image region in the target training image, and the second feature and image label for each second image region, a feature loss value is obtained. The parameters of the initial model are adjusted based on the feature loss value until the training termination condition is met, resulting in a target classification model. This leads to a more accurate target classification model and enables more precise image classification using the target classification model. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0025] Figure 2 This is a flowchart illustrating a classification model training method according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of extracting the second image region corresponding to the object after scaling the target training image using the initial model;

[0027] Figure 4 This is a flowchart illustrating a classification model training method according to another embodiment of this application;

[0028] Figure 5 yes Figure 4 A flowchart illustrating step S250;

[0029] Figure 6 yes Figure 4 A flowchart illustrating step S260;

[0030] Figure 7 This is a flowchart illustrating a classification model training method according to another embodiment of this application;

[0031] Figure 8This is a schematic diagram of a network structure proposed according to an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of an initial model with a prototype-guided comparison framework added according to an embodiment of this application;

[0033] Figure 10 This is a schematic diagram illustrating a prototype-guided comparison framework according to a specific embodiment of this application;

[0034] Figure 11 This is a schematic diagram illustrating the classification results obtained by using an initial model and a target classification model according to a specific embodiment of this application;

[0035] Figure 12 This is a flowchart illustrating an image classification method according to an embodiment of this application;

[0036] Figure 13 This is a connection block diagram of a classification model training device according to a specific embodiment of this application;

[0037] Figure 14 This is a connection block diagram of an image classification apparatus according to a specific embodiment of this application;

[0038] Figure 15 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of this application is shown. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] It should be noted that "multiple" as mentioned in this article refers to two or more.

[0044] Before proceeding with specific explanations, the terms used in this application are explained as follows:

[0045] The first sample image set includes training images corresponding to multiple categories of objects, such as vehicles, people, trees, or animals. The training images are labeled with the image regions containing the object images and the category to which the image regions belong. The number of images for each category is limited, which can be a few, dozens, or hundreds of images.

[0046] Image region: An image region refers to the area occupied by the object corresponding to the sample label in the training image. The image region can be a rectangle or a polygon, as long as it can identify the area occupied by the object in the training image. The image region can include only the local area occupied by the object (i.e., only the local image of the object) or it can include the entire area occupied by the object (i.e., only the complete image of the object).

[0047] Sample labels: Sample labels are used to identify objects in an image region. Specifically, they can be used to identify the name of an object in an image region, the category of an object in an image region, or both the name and category of an object in an image region.

[0048] The initial model refers to the model obtained by inputting a large number of sample images from the second sample image set into the algorithm model and training the algorithm model based on each sample image. The aforementioned algorithm model can be a linear regression algorithm model, a deep neural network algorithm model, a decision tree algorithm model, a random forest algorithm model, a support vector machine algorithm model, and so on.

[0049] Second sample image set: The second sample image set includes images of multiple categories, and the number of images in each category is huge, which can be thousands, tens of thousands or even hundreds of thousands. For example, it can include images of at least two categories such as human images, tree images, vehicle images, land animal images, marine life images and bird images.

[0050] The training images in the first sample image set may be partially derived from the second sample image set, with the remainder derived from new class images. New class images are those that differ from the categories included in the second sample image set. For example, if the second sample image set includes a large number of images of humans, trees, and vehicles, but excludes images of land animals and birds, then the new class images may include at least one of land animal images, marine life images, and bird images. In other words, the first sample image set includes images of at least some categories from the second sample image set (e.g., at least one category of images of humans, trees, and vehicles), as well as new class images (at least one of land animal images, marine life images, and bird images). Furthermore, the number of training images in the first sample image set is far less than the number of sample images in the second sample image set.

[0051] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0052] Figure 1 This diagram illustrates the application environment of the classification model training methods provided in some embodiments. For example... Figure 1 As shown, this application environment includes terminal 10 and server 20.

[0053] Terminal 10 can communicate with server 20 via a network. Terminal 10 sends a model training instruction to server 20. Server 20 obtains multiple target training images from a first sample image set according to the instruction. The first sample image set includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region containing the object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories. Server 20 uses an initial model to extract the first feature of the first image region of each target training image and extracts the second image region corresponding to the object at multiple scaling ratios, the second feature of each second image region, and the image label of each second image region. Server 20 obtains a feature loss value based on the first image region, the first feature, and the sample label of each target training image, the second image region, the second feature, and the image label of each second image region, and adjusts the parameters of the initial model according to the feature loss value until the training termination condition is met, thus obtaining the target classification model. After the server 20 completes the training of the target classification model, it can use the target classification model to classify the image to be classified sent by the client terminal 10 and feed back the classification result to the terminal 10.

[0054] In some embodiments, the initial model may be stored in the terminal 10, and the terminal 10 may execute the steps of training the initial model based on the first sample image set to obtain a target classification model, and using the target classification model to classify the image to be classified.

[0055] Server 20 can be a standalone physical server or a server cluster consisting of multiple physical servers. It can also be a cloud server providing basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN. Terminal 10 can be a smartphone, tablet, laptop, desktop computer, medical device, vehicle terminal, etc., but is not limited to these. Terminal 10 and server 20 can be connected via communication methods such as networks, which are not limited herein.

[0056] Figure 2 This is a flowchart illustrating a classification model training method according to an embodiment of this application. This method can be executed by an electronic device with processing capabilities, such as a server, a terminal device, or an interaction between a server and a terminal device to implement this solution, etc., without specific limitations herein. (Refer to...) Figure 2 As shown, the method includes at least steps S110 to S140, which are described in detail below:

[0057] Step S110: Obtain multiple target training images from the first sample image set.

[0058] The first sample image set includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region containing the object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories.

[0059] Specifically, the electronic device can obtain the first set of sample images from a database. Alternatively, the electronic device can obtain the first set of sample images from a third party, which refers to a service provider that stores training samples. The electronic device can also collect the first set of sample images from the Internet.

[0060] Step S120: Use the initial model to extract the first feature of the first image region of each of the multiple target training images.

[0061] The initial model can be a model trained using an artificial intelligence algorithm on a massive number of sample images from the second sample image set. This artificial intelligence algorithm can be linear regression, neural network, decision tree, random forest, support vector machine, etc.

[0062] Among them, neural network algorithms can be CNN (Convolutional Neural Network) algorithms, DNN (Deep Neural Network) algorithms, and RNN (Recurrent Neural Network) algorithms, etc.

[0063] Convolutional neural networks (CNNs) consist of convolutional layers and pooling layers. There are various CNN algorithms, such as VGG (Visual Geometry Group), GoogleNet, and ResNet. Deep neural networks consist of input layers, hidden layers, and output layers, with fully connected layers. Recurrent neural networks (RNNs) model sequential data, meaning the current output of a sequence is related to previous outputs. Specifically, the network memorizes previous information and applies it to the calculation of the current output; nodes in the hidden layers are no longer disconnected but connected, and the input to the hidden layer includes not only the output of the input layer but also the output of the hidden layer from the previous time step. Examples of RNN algorithms include LSTM (Long Short-Term Memory Neural Network).

[0064] In one possible implementation of this application, the aforementioned artificial intelligence algorithm can be a convolutional neural network algorithm. That is, each sample image in the second sample image set can be input into the convolutional neural network model, and the convolutional neural network model can be trained based on each sample image to obtain an initial model.

[0065] When using the initial model to extract the first feature of the first image region of each of the multiple target training images, the first feature of the extracted first image region can specifically refer to a feature vector, which is used to characterize the image type corresponding to the object in the first image region.

[0066] Step S130: Using the initial model, extract the second image region corresponding to the object in each target training image at multiple scaling ratios, the second feature corresponding to each second image region, and the image label corresponding to each second image region.

[0067] The initial model can scale each target training image and perform image recognition on the scaled images to obtain a second image region containing objects. This second image region is then classified to obtain an image label. The image label includes at least the category of the object in the second image region; it should also include the name of the object (recognition result). This process yields the second image region and its corresponding image label for each target training image at each scaling ratio. It should be understood that the second image region may include only a portion of the first image region or the entire first image region; that is, the first image region can be a part of the second image region.

[0068] Taking a target training image of size 416×416 pixels as an example, when scaling the target sample to obtain target training images at multiple scaling ratios, specifically, the target training image can be reduced to a size of 208×208 pixels or 172×172 pixels, or it can be enlarged to a size of 832×832 pixels or 1248×1248 pixels, and so on. In this way, target training images at multiple scaling ratios can be obtained. Figure 3 As shown, taking a vehicle image as the target training image as an example, the first image region corresponding to the vehicle is marked in the target training image. By scaling the target training image, the following can be extracted: Figure 3 The second image region corresponding to the vehicle shown.

[0069] In some embodiments of this application, the initial model may include at least two convolutional layers and a pooling layer, wherein the length of the convolutional window corresponding to each of the at least two convolutional layers is different. In this embodiment, the initial model can output a first feature and a second feature of the target training image through the following process: the at least two convolutional layers include a first convolutional layer and a second convolutional layer; the first convolutional layer extracts features from the target training image to obtain a first feature corresponding to a first image region; the pooling layer processes the scaled target training image to obtain a second image region in the scaled training image; and the second convolutional layer extracts features from the second image region in each scaled training image to obtain the second feature.

[0070] In this implementation, the initial model may further include an output layer, which is used to output the probability of the category corresponding to the object in each second image region based on the features extracted from each second image region, that is, to obtain the image label corresponding to the second image region of each target training image at each scaling ratio.

[0071] In one embodiment, the output layer can be a softmax layer. The softmax layer has the ability to obtain the final classification result of the image classification. Specifically, the softmax layer can take a second feature as input and perform matrix multiplication, vector addition, and normalization processing sequentially based on the second feature to obtain the final classification result of the object in the second image region of the scaled target training image, where the object in the second image region has a first probability of belonging to a predetermined classification category. That is, the image label corresponding to the second image region in the scaled target training image can be obtained through the output layer.

[0072] Step S140: Based on the first image region, the first feature of the first image region, and the sample label of the first image region in each target training image, the second image region in the target training image, the second feature corresponding to each second image region, and the image label corresponding to each second image region, obtain the feature loss value.

[0073] In one possible implementation, step S140 may be: obtaining a first loss value based on a first feature corresponding to a first image region and a second feature corresponding to a second image region in the target training image, and the category to which each target training image belongs; obtaining a second loss value based on a first image region corresponding to each target training image and the sample label corresponding to that first image region, and each second image region corresponding to each target training image and the image label corresponding to that second image region; and obtaining a feature loss value based on the first loss value and the second loss value.

[0074] In this implementation, the first loss value can be obtained during the fine-tuning stage of the initial model by processing the prototype-guided contrastive framework added to the initial model, and the second loss value can be calculated using the loss function used for classification in the initial model. The aforementioned prototype-guided contrastive framework is used to propose a prototype-guided contrastive loss function to optimize the initial model's ability to represent object features. This ensures that, when performing feature comparison analysis, the optimized initial model makes features of images of the same type closer (smaller feature distance) and features of images of different types farther apart (larger feature distance).

[0075] Specifically, the target features of each target training image can be obtained by taking the mean or median value of the first and second features corresponding to the target training image. The intra-class distance is obtained by calculating the distance between the target features of target training images belonging to the same category, and the inter-class distance is obtained by calculating the distance between the target features of target training images belonging to different categories. The ratio of the intra-class distance to the inter-class distance is used as the first loss value.

[0076] In another possible implementation, the above steps may further include: obtaining target regions in the second region corresponding to each target training image that have a similarity or overlap greater than a preset value with the first region and have the same label; obtaining target features of the training image based on the second features and first features corresponding to each target region of the same training image; calculating the intra-class distance by performing feature distance calculation on the target features of each target training image belonging to the same category, and calculating the inter-class distance by performing distance calculation on the features of each training image not belonging to different categories; and obtaining the feature loss value based on the ratio between the intra-class distance and the inter-class distance.

[0077] It should be understood that, based on the first image region, the first feature of the first image region, and the sample label of the first image region in each target training image, the second image region in the target training image, the second feature corresponding to each second image region, and the image label corresponding to each second image region, there are many other ways to calculate the feature loss value, which will not be elaborated here.

[0078] Step S150: Adjust the parameters of the initial model based on the feature loss value until the training termination condition is met to obtain the target classification model.

[0079] In one implementation, when adjusting the parameters of the initial model based on the feature loss, the gradient backpropagation method can be used to adjust the parameters of the initial model. After one adjustment, the execution steps S110-S150 can be returned until the training reaches the termination condition to obtain the target classification model.

[0080] The training termination condition mentioned above can be either when the feature loss value is less than a preset threshold, or when every training image in the first sample image set has been acquired to perform the above steps S110-S150.

[0081] Based on the above description, in the scheme of this application, the classification model can be trained through two training stages, which can be called the pre-training stage and the fine-tuning stage. The initial model is first trained in the pre-training stage, and then the initial model is trained using the target training images in the fine-tuning stage to finally obtain the target classification model.

[0082] By employing the classification model training method of this application, the first feature of the first image region of each target training image is extracted using the initial model, and the second image region corresponding to the object in each target training image at multiple scaling ratios, along with the second feature and image label corresponding to each second image region, are extracted. Based on the first image region, the first feature and sample label of the first image region in each target training image, the second image region in the target training image, and the second feature and image label corresponding to each second image region, a feature loss value is obtained. The parameters of the initial model are adjusted according to the feature loss value until the training termination condition is met, resulting in a target classification model. This method allows for optimization of the initial model using a small number of training images. Furthermore, during the optimization process, feature extraction after scaling each target training image ensures the accuracy of the extracted second features, thereby obtaining a more accurate target classification model and enabling more precise image classification using this model.

[0083] Please refer to 4. This application embodiment also provides a classification model training method, the method including:

[0084] Step S210: Obtain multiple target training images from the first sample image set.

[0085] The first sample image set includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region containing the object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories.

[0086] Step S220: Use the initial model to extract the first feature of the first image region of each target training image from multiple target training images.

[0087] Step S230: Using the initial model, extract the second image region corresponding to the object in each target training image at multiple scaling ratios, the second feature corresponding to each second image region, and the image label corresponding to each second image region.

[0088] Step S240: Select a target image region from multiple second image regions corresponding to the target training image based on the first image region in the target training image.

[0089] There are several ways to select the target image region from its corresponding multiple second image regions based on the first image region in the target training image. For example, it can be selected according to a preset algorithm.

[0090] The aforementioned preset algorithms can be similarity calculation methods, matching algorithms, etc.

[0091] In one possible implementation, step S240 may involve calculating the matching degree between a first image region in the target training image and each corresponding second image region of the target training image to obtain a matching degree value for each second image region corresponding to the target training image; and selecting a second image region with a matching degree value greater than a matching degree threshold as the target image region of the target training image. The specific value of the matching degree threshold can be 0.5, 0.7, 0.85, 0.9, or 0.95, etc., and can be selected according to actual needs.

[0092] When the aforementioned matching degree specifically refers to the overlap degree, step S240 can also be: selecting an image region from the second image region corresponding to the first image region in the target training image, whose overlap degree with the first image region is greater than a preset overlap degree threshold. The specific value of the preset overlap degree threshold can be 0.5, 0.7, 0.8, 0.85, 0.9, or 0.95, etc., selected according to actual needs.

[0093] Wherein, the overlap (IOU) is expressed as IOU = (A∩B) / (A∪B) for the area A corresponding to the first image region and the area B corresponding to the second image region.

[0094] In one optional implementation, the aforementioned preset overlap threshold can be 0.5. That is, the image region in the second image region corresponding to the target training image with an overlap greater than 0.5 with the first image region corresponding to the target training image is taken as the target image region.

[0095] Step S250: Obtain the first loss value based on the first feature corresponding to each target training image, the second feature of the target image region of each target training image, and the category to which each target training image belongs.

[0096] Specifically, step S250 may involve determining the target features of a target training image based on the first feature corresponding to each target training image and the second feature of the target image region of that target training image. A first distance is calculated between the target features of target training images belonging to the same category, and a second distance is calculated between the target features of target training images belonging to different categories. The ratio of the first distance to the second distance is then used as the first loss value.

[0097] The method for determining the target feature of a target training image based on the first feature corresponding to the target training image and the second feature of the corresponding target region can be to take the mean or median value of the first feature of the target training image and the second feature of the corresponding second image region as the target feature of the target training image.

[0098] The above-mentioned method for calculating the first distance between training images of targets belonging to the same category can be to calculate the feature distance of the target features of the target images belonging to the same category and then obtain the mean or median to get the first distance, or it can be to calculate the feature distance of the target features of the target images belonging to the same category and then sum them up to get the first distance.

[0099] The above-mentioned method for calculating the second distance between target training images belonging to different categories can be: calculating the feature distance of target features of target training images belonging to different categories and then taking the mean or median to obtain the second distance; or it can be that calculating the feature distance of target features of target images belonging to different categories and then summing them to obtain the second distance.

[0100] The above step S250 may include, for example: Figure 5 The steps shown are as follows:

[0101] Step S251: Calculate the average value of the first feature corresponding to the target training image and the second feature of the target image region of the target training image to obtain the target feature of the target training image.

[0102] Step S252: Calculate the intra-class distance of the target features of the target training images belonging to the same category.

[0103] Step S253: Calculate the inter-class distance of the target features of the training images belonging to different categories.

[0104] Step S254: Obtain the first loss value based on the ratio between the intra-class distance and the inter-class distance.

[0105] Specifically, step S254 can be used as the ratio of the intra-class distance to the inter-class distance as the first loss value.

[0106] For example, if the target training images include images a, b, c, d, and e, and images a and b belong to category A, image c belongs to category B, and images d and e belong to category C, then step S252 can be: calculating the feature distance between images a and b and calculating the feature distance between images d and e, and then summing or averaging the results to obtain the intra-class distance of the samples.

[0107] Accordingly, step S253 may be: calculating the feature distance between image a and images c, d and e respectively; calculating the feature distance between image b and images c, d and e respectively; calculating the distance between image c and images d and e respectively; and summing or averaging the results obtained from the above distance calculations to obtain the inter-class distance of the samples.

[0108] Step S253 above can also be: calculating the mean feature value of image a and image b to obtain the feature value corresponding to category A, and calculating the mean feature value of image d and image e to obtain the feature value corresponding to category C, and calculating the feature distance between each pair of categories A, B and C and then summing or calculating the mean to obtain the inter-class distance of the samples.

[0109] Since the distance between target features in training images of the same class is small, while the distance between target features in training images of different classes is large, the smaller the ratio obtained by comparing the target feature distance (intra-class distance) between training images of the same class with the target feature distance (inter-class distance) between training images of different classes, the more accurate the image labels and second features of the target image regions obtained by the initial model are.

[0110] For example, the first loss value can be calculated using the following formula: Among them, L PC N is the first loss value. f The number of training images for the target. u is the number of samples of each class in the first set. i The matching degree of the target training image, β is the proportion coefficient of the intra-class distance of the samples, C is the number of classes in the first sample set, and y i Refers to category, Train target features for the target image. For y i The first feature of the training image of the j-th target in the category, where τ is the temperature coefficient (specifically a constant). Let be the first feature of the m-th target training image of the k-th class, and n be the number of target training images. In the above formula, the numerator is the calculated intra-class distance, and the denominator is the calculated inter-class distance.

[0111] Step S260: Obtain the second loss value based on the first image region corresponding to each target training image and the sample label corresponding to the first image region, as well as each second image region corresponding to each target training image and the image label corresponding to each second image region.

[0112] Specifically, step S260 can be as follows: substituting the second loss value obtained by taking the first image region corresponding to the target training image and the sample label corresponding to the first image region, and each second image region corresponding to each target training image and the image label corresponding to each second image region, into a preset loss function for calculation, thereby obtaining the second loss value. The preset loss function can be the cross-entropy loss function.

[0113] In one possible implementation, step S260 may include, for example: Figure 6 The steps shown are as follows:

[0114] Step S261: Based on the first image region and each second image region corresponding to each target training image, obtain the quality balance parameter corresponding to each second image region in the target training image.

[0115] Specifically, step S261 can be performed by calculating the overlap between the first image region corresponding to each target training image and each second image, to obtain the overlap value corresponding to each second image in each target training image, and the overlap value corresponding to each second image is the quality balance parameter corresponding to the second image region.

[0116] Alternatively, the similarity between the first image region corresponding to each target training image and each second image can be calculated to obtain the similarity value of each second image in each target training image. The similarity value of each second image is the quality balance parameter corresponding to that second image region.

[0117] Step S262: Obtain the sample balance parameter of the target training image based on the number of second image regions corresponding to each target training image and the number of target image regions.

[0118] The target image region corresponding to the aforementioned target training image can be a second image region corresponding to the target training image, wherein the overlap or similarity with the first image region is greater than a preset threshold.

[0119] Accordingly, the sample balance parameter mentioned above can be the ratio of the number of target image regions to the number of second image regions in each target training image. Alternatively, the sample balance parameter can be the ratio of the number of target image regions to the difference between the number of second image regions and the number of target image regions in each target training image.

[0120] In one possible implementation, step S262 may specifically be: obtaining the difference between the number of second image regions corresponding to the target training image and the number of target image regions; and obtaining the sample balance parameter of the target training image based on the ratio between the difference between the number of second image regions corresponding to the target training image and the number of target region images corresponding to the target training image.

[0121] The method of obtaining the sample balance parameter of the target training image based on the ratio between the quantity difference corresponding to the target training image and the quantity of the target region image corresponding to the target training image can be achieved by multiplying the ratio between the quantity difference corresponding to the target training image and the quantity of the target region image corresponding to the target training image by a preset coefficient, or by using the ratio between the quantity difference corresponding to the target training image and the quantity of the target region image corresponding to the target training image as the sample balance parameter of the target training image.

[0122] Step S263: Based on the sample labels corresponding to the first image region in the target training image and the image labels corresponding to each second image region, obtain the true label inference probability of the target training image.

[0123] Specifically, step S263 can be: obtaining the number of samples in the second image region of the target image that are identical to the samples in the first image; comparing this number with the number of samples in the second image region of the target image to obtain the true label inference probability.

[0124] That is, in one possible implementation, step S263 includes: obtaining the number of image labels in the image labels corresponding to each second image region in the target training image that are the same as the sample labels corresponding to the first image region; and using the ratio obtained by comparing the number with the number of second image regions corresponding to the target training image as the true label inference probability of the target training image.

[0125] Step S264: Obtain the second loss value based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability.

[0126] Specifically, step S264 can be: obtaining the mean or median of the quality balance parameters corresponding to each second image region in each training image to obtain the quality balance parameters of the target training image, and summing the products obtained by multiplying the quality balance parameters, sample balance parameters, and true label inference probabilities of each target training image to obtain the second loss value.

[0127] In one possible implementation, step S264 may specifically be: obtaining the average value of the quality balance parameters corresponding to each second image in the target training image to obtain the quality balance parameters of the target training image; obtaining the relative entropy of the true label inference probability of the target training image, and multiplying the obtained relative entropy with the quality balance parameters and sample balance parameters of the target training image to obtain the loss value corresponding to the target training image; and summing the loss values ​​of each target training image to obtain the second loss value.

[0128] For example, the second loss value can be calculated using the following formula:

[0129] Wherein, H(u) i ) represents the mass balance parameter. N is the sample balancing parameter. b The sum of the number of second features corresponding to each target training image. The number of target image regions in the target training image. γ is the difference between the number of second image regions in the i-th target training image and the number of target image regions, and γ is a hyperparameter, specifically a constant.

[0130] Step S270: Add the first loss value and the second loss value to obtain the feature loss value.

[0131] Step S280: Adjust the parameters of the initial model according to the feature loss value until the training termination condition is met to obtain the target classification model.

[0132] By employing the classification model training method of this application, a target image region is selected from multiple second image regions corresponding to a first image region in the target training image. A first loss value is obtained based on the first feature corresponding to each target training image, the second feature of the target image region of each target training image, and the category to which each target training image belongs. A second loss value is obtained based on the first image region and the sample label corresponding to each target training image, and each second image region and the image label corresponding to each second image region. The initial model is adjusted using the feature loss value obtained by accumulating the first and second loss values. This allows for the simultaneous use of two loss values ​​to adjust the initial model during training. Specifically, the first loss value, which characterizes the differences between features within the same category and features between different categories (i.e., features within the same category have small differences while features between different categories have large differences), and the second loss value, which characterizes the classification loss, are used to jointly optimize the initial model. This results in a more accurate target classification model and enables more precise image classification using the target classification model.

[0133] For example, please refer to Figure 7 This application also provides a classification model training method, which may include the following steps:

[0134] Step S310: Input the sample images of multiple categories in the second sample image set into the network structure of the convolutional neural network, and use the network structure to train to obtain the initial model.

[0135] Among them, such as Figure 8The network structure shown may include a first convolutional layer, a second convolutional layer, a fully connected layer, a feature mapping layer, and a linear mapping layer. The first convolutional layer extracts the first feature of the image, the second convolutional layer extracts the second feature of the image at multiple scaling ratios, the fully connected layer selects the regions of objects in the image at multiple scaling ratios to obtain the second image regions, the feature mapping layer matches the first feature, the second feature, and the second image region corresponding to each image to establish a correspondence, and the linear mapping layer performs vector transformation on the features. During the training phase, the network structure can output a localization loss function value L. reg A classification loss function value L BC By adjusting the parameters of the network structure according to the bounding box loss function and the classification loss function until the loss functions (bounding box loss function and classification loss function) converge, the initial model can be obtained.

[0136] like Figure 9 As shown, after training the initial model and entering the fine-tuning stage, a prototype repository can be introduced, along with a prototype-guided comparison framework added to the initial model.

[0137] The prototype repository is used to store training images in the first sample image set. The number of training images in the first sample image set is small. It can select a set number of sample images from each category of sample images in the second sample image set, and select a set number of images from each new category in the new category image set. The first sample image set obtained in this way includes training images of multiple categories, and the training images of each category can be the same.

[0138] like Figure 11 As shown, the prototype-guided contrast framework can include a feature mapping layer and a linear mapping layer. The feature mapping layer can be used to extract and encode the first and second features corresponding to each target training image. The linear mapping layer can perform feature mapping for subsequent feature processing. Specifically, the prototype-guided contrast framework can be used to propose a prototype-guided contrast loss function, which can better optimize the network's ability to represent object features, making the feature distances of training images of the same category closer and the feature distances of training images of different categories farther.

[0139] Specifically, when using a prototype-guided contrastive framework to optimize the network's ability to represent object features, the following steps can be performed:

[0140] The following steps can be performed:

[0141] Step S320: Use the first convolutional layer of the initial model to extract the first feature of the first image region of each of the multiple target training images.

[0142] Step S330: Use the pooling layer of the initial model to extract the second image region corresponding to the object in each target training image at multiple scaling ratios, use the second convolutional layer of the initial model to extract the second feature corresponding to each second image region, and use the output layer of the initial model to output the image label corresponding to each second image region.

[0143] Step S340: Using the prototype-guided comparison framework, select a target image region from multiple second image regions corresponding to the target training image based on the first image region in the target training image, and obtain a first loss value based on the first feature corresponding to each target training image, the second feature of the target image region of each target training image, and the category to which each target training image belongs.

[0144] Specifically, the prototype-guided comparison framework can utilize a linear mapping layer to calculate the matching degree between a first image region in the target training image and each corresponding second image region of the target training image, obtaining the matching degree value of each second image region corresponding to the target training image; second image regions with matching degree values ​​greater than a matching degree threshold are selected as target image regions of the target training image. The target features of the target training image are obtained by averaging the first features corresponding to the target training image and the second features of the target image regions of the target training image; the intra-class distance is obtained by calculating the feature distance between target features of target training images belonging to the same category; the inter-class distance is obtained by calculating the feature distance between target features of target training images belonging to different categories; and the first loss value is obtained based on the ratio between the intra-class distance and the inter-class distance.

[0145] It should be noted that the first feature of the target training image is the prototype feature of the target training image in the prototype-guided comparison framework. This prototype feature, like the corresponding second feature, is encoded by the ROI feature extractor (feature mapping layer) into a one-dimensional feature vector. Due to the presence of the ReLU activation function in the ROI feature extractor, feature mapping (using a linear mapping layer) is required to avoid spurious similarities caused by zeros in many feature dimensions affecting the measurement of feature distance. Then, the ROI feature corresponding to the second feature and the ROI feature corresponding to the prototype feature can be compared, i.e., the calculation process described above is performed to obtain the first loss value.

[0146] For example, the following prototype can be used to guide the comparison loss function and calculate the first loss value: The physical meaning of each string in the prototype-guided contrastive loss function can be found in the description of the foregoing embodiments. It should be noted that in the process of calculating the first loss value, the numerator of the prototype-guided contrastive loss function calculates the intra-class distance between target training images belonging to the same category, while the denominator calculates the inter-class distance between target training images of different categories. Therefore, after adjusting the initial model based on this first loss value, it can move closer to prototype features of the same category while moving further away from prototype features of different categories. This better optimizes the network's ability to represent object features.

[0147] While using prototype-guided comparative framework diagrams to learn the differences between features of different categories (either before or after), the classification loss value for classifying the target training image can also be calculated. Specifically, the following steps can be used to calculate the classification loss value:

[0148] Step S350: Obtain the second loss value based on the first image region corresponding to each target training image and the sample label corresponding to the first image region, as well as each second image region corresponding to each target training image and the image label corresponding to each second image region.

[0149] In step S350, two balance parameters are introduced into the original cross-entropy loss function: a quality balance parameter and a sample balance parameter, to calculate the classification loss. Specifically, the overlap between the first image region corresponding to each target training image and each second image is calculated to obtain the overlap value of each second image in each target training image. The overlap value of each second image is the quality balance parameter corresponding to that second image region. The difference between the number of second image regions corresponding to the target training image and the number of target image regions is obtained. The sample balance parameter of the target training image is obtained based on the ratio between the difference in the number of second image regions corresponding to the target training image and the number of target region images corresponding to the target training image. The number of image labels in the image labels corresponding to each second image region in the target training image that are the same as the sample labels corresponding to the first image region is obtained. The ratio of this number to the number of second image regions corresponding to the target training image is used as the true label inference probability of the target training image.

[0150] Thus, by employing a quality balance parameter, the loss values ​​of each selected second image region in the target training image can be balanced. Since the number of second image regions with matching scores greater than the matching score threshold is usually small when used as target image regions in the target training image, and the matching score scores are usually less than 1, a sample balance parameter is introduced when calculating the second loss value. As training progresses, the number of target image regions will gradually increase, and correspondingly, the quality balance parameter will also increase. Therefore, by employing a sample balance parameter, the influence of the quality balance parameter on the weight of the loss generated by the target training image is reduced, thereby making the classification loss value obtained based on the above step S350 more accurate.

[0151] Specifically, step S350 above can be calculated using the following formula: For the physical meaning of this formula, please refer to the specific description in the foregoing embodiments, which will not be repeated here.

[0152] The second loss value is obtained based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability.

[0153] After obtaining the first and second loss values, the following steps can be performed:

[0154] Step S360: Add the first loss value and the second loss value to obtain the feature loss value.

[0155] Step S370: Adjust the parameters of the initial model based on the feature loss value.

[0156] After adjusting the parameters of the initial model, it can be checked whether the training termination condition has been met. If not, return to step S310 until the training termination condition is met to obtain the target classification model.

[0157] It should be noted that, in order to avoid inconsistencies between the prototype features of the target training image and the second features of the extracted second image region caused by network parameter updates, each target image needs to be input into the model in a first-in-first-out update manner during each iteration to ensure the accuracy of the obtained target classification model.

[0158] like Figure 11 As shown, where, Figure 11 In this context, A refers to the classification result obtained using the initial model. Figure 11In the diagram, B represents the classification result obtained using the target classification model. It can be seen that this application obtains a feature loss value by accumulating the first loss value calculated using the prototype-guided contrastive loss function proposed using the prototype-guided contrastive framework and the second loss value obtained using the classification function. The feature loss value is then used to adjust the parameters of the initial model to obtain the target classification model. This results in target training images belonging to the same category having short feature distances, and target training images belonging to different categories having long feature distances.

[0159] Based on the above, this application also provides an image classification method that can be applied to electronic devices, the method comprising:

[0160] Step S410: Obtain the image to be classified.

[0161] The images to be classified can be obtained from web pages, from a preset database, or from the memory of an electronic device or a storage device associated with that electronic device.

[0162] The number of images to be classified can be one or more. In this embodiment, there are multiple images to be classified.

[0163] Step S420: Call the target classification model to classify the image to be classified and obtain the classification result.

[0164] The target classification model described above is the target classification model obtained using the above embodiments. The training process of the target classification model can be referred to the specific description of the foregoing embodiments, and will not be repeated in this embodiment.

[0165] By calling the target classification model to classify the image when it is acquired, the accuracy of the classification result can be effectively ensured.

[0166] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0167] Figure 13 The classification model training device 500 shown in one embodiment includes: a training image acquisition module 510, a feature extraction module 520, an information extraction module 530, a loss value acquisition module 540, and a model acquisition module 550.

[0168] The training image acquisition module 510 is used to acquire multiple target training images from a first sample image set. The first sample image set includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region containing an object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories. The feature extraction module 520 is used to extract the first feature of the first image region of each target training image from the multiple target training images using an initial model. The information extraction module 530 is used to extract the second image region corresponding to the object in each target training image at multiple scaling ratios, the second feature corresponding to each second image region, and the image label corresponding to each second image region using an initial model. The loss value acquisition module 540 is used to obtain a feature loss value based on the first image region, the first feature of the first image region, and the sample label of the first image region in each target training image, the second image region in the target training image, the second feature corresponding to each second image region, and the image label corresponding to each second image region. The model acquisition module 550 is used to adjust the parameters of the initial model based on the feature loss value until the training termination condition is met to obtain the target classification model.

[0169] In one possible implementation, the loss value acquisition module 540 includes a region selection submodule, a first loss value acquisition submodule, a second loss value acquisition submodule, and a feature loss value acquisition submodule.

[0170] The region selection submodule is used to select a target image region from multiple second image regions corresponding to the target training image based on a first image region in the target training image; the first loss value acquisition submodule is used to obtain a first loss value based on a first feature corresponding to each target training image, a second feature of the target image region of each target training image, and the category to which each target training image belongs; the second loss value acquisition submodule is used to obtain a second loss value based on the first image region and the sample label corresponding to the first image region corresponding to each target training image, and each second image region and the image label corresponding to each second image region corresponding to each target training image; the feature loss value acquisition submodule is used to accumulate the first loss value and the second loss value to obtain a feature loss value.

[0171] In this implementation, the first loss value acquisition submodule includes a target feature acquisition unit, a first distance calculation unit, a second distance calculation unit, and a first loss value acquisition unit.

[0172] The target feature acquisition unit is used to obtain the target feature of the target training image by averaging the first feature corresponding to the target training image and the second feature of the target image region of the target training image; the first distance calculation unit is used to calculate the intra-class distance of the target features of the target training images belonging to the same category; the second distance calculation unit is used to calculate the inter-class distance of the target features of the target training images belonging to different categories; and the first loss value acquisition unit is used to obtain the first loss value based on the ratio between the intra-class distance and the inter-class distance.

[0173] In this implementation, the region selection submodule includes a matching degree calculation unit and a region determination unit.

[0174] The matching degree calculation unit is used to calculate the matching degree between the first image region in the target training image and each corresponding second image region of the target training image to obtain the matching degree value of each second image region corresponding to the target training image; the region determination unit is used to obtain the second image region with a matching degree value greater than the matching degree threshold as the target image region of the target training image.

[0175] In this implementation, the second loss value acquisition submodule includes a first parameter acquisition unit, a second parameter acquisition unit, an inference probability acquisition unit, and a second loss value acquisition unit.

[0176] The first parameter acquisition unit is used to obtain the quality balance parameter corresponding to each second image region in the target training image based on the first image region and each second image region corresponding to each target training image; the second parameter acquisition unit is used to obtain the sample balance parameter of the target training image based on the number of second image regions and the number of target image regions corresponding to each target training image; the inference probability acquisition unit is used to obtain the true label inference probability of the target training image based on the sample label corresponding to the first image region and the image label corresponding to each second image region in the target training image; the second loss value acquisition unit is used to obtain the second loss value based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability.

[0177] In one possible implementation, the first parameter obtaining unit is further configured to calculate the overlap between the first image region corresponding to each target training image and each second image, thereby obtaining the overlap value corresponding to each second image in each target training image, and the overlap value corresponding to each second image is the quality balance parameter corresponding to the second image region.

[0178] In one possible implementation, the second parameter obtaining unit is further configured to obtain the quantity difference between the number of second image regions corresponding to the target training image and the number of target image regions; and to obtain the sample balance parameter of the target training image based on the ratio between the quantity difference corresponding to the target training image and the number of target region images corresponding to the target training image.

[0179] In one possible implementation, the inference probability acquisition unit is further configured to acquire the number of image labels in the image labels corresponding to each second image region in the target training image that are the same as the sample labels corresponding to the first image region; and to use the ratio obtained by comparing the number with the number of second image regions corresponding to the target training image as the true label inference probability of the target training image.

[0180] In one possible implementation, the second loss value acquisition unit is further configured to obtain the quality balance parameter of the target training image by calculating the average value of the quality balance parameter corresponding to each second image in the target training image; calculate the relative entropy of the true label inference probability of the target training image; multiply the calculated relative entropy by the quality balance parameter and the sample balance parameter of the target training image to obtain the loss value corresponding to the target training image; and sum the loss values ​​of each target training image to obtain the second loss value.

[0181] In one possible implementation, the classification model training device 500 further includes a sample acquisition module and a model training module.

[0182] The sample acquisition module is used to acquire a second sample image set, which includes sample images corresponding to at least two categories of objects. Each sample image is labeled with a first image region including the object image and the sample label corresponding to the first image region. The model training module is used to input each sample image in the second sample image set into the convolutional neural network model, and train the convolutional neural network model based on each sample image to obtain an initial model.

[0183] Please combine Figure 14 Another embodiment of this application also provides an image classification device 600, which includes an image acquisition module 610 and an image classification module 620.

[0184] The image acquisition module 610 is used to acquire images to be classified.

[0185] The image classification module 620 is used to call the target classification model to classify the image to be classified and obtain the classification result.

[0186] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0187] The following will combine Figure 15 This application provides a description of an electronic device 100.

[0188] Please see Figure 15 Based on the information group synchronization method provided in the above embodiments, this application embodiment also provides another electronic device 100 including a processor 102 capable of executing the aforementioned method. The electronic device 100 can be a server 10 or a terminal device. The terminal device can be a smartphone, tablet computer, computer, or portable computer, etc., which can install a first client and / or a second client.

[0189] The electronic device 100 also includes a memory 104. The memory 104 stores a program that can execute the contents of the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104.

[0190] The processor 102 may include one or more cores for data processing and message matrix units. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0191] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data acquired by the electronic device 100 during use (e.g., recommended data and operating methods).

[0192] The electronic device 100 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and facilitate data interaction.

[0193] In some embodiments, the electronic device 100 may further include a peripheral interface 106 and at least one peripheral device. The processor 102, memory 104, and peripheral interface 106 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118, and a power supply 122.

[0194] Peripheral interface 106 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 102 and memory 104. In some embodiments, processor 102, memory 104 and peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 102, memory 104 and peripheral interface 106 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0195] The radio frequency (RF) component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF component 108 communicates with communication networks and other communication devices via electromagnetic signals. The RF component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF component 108 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF component 108 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0196] The positioning component 112 is used to locate the current geographical location of the electronic device 100 in order to enable navigation or LBS (Location Based Service). The positioning component 112 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0197] Camera 114 is used to capture images or videos. Optionally, camera 114 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 100, and the rear-facing camera is located on the back of the electronic device 100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera 114 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0198] Audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to processor 102 for processing, or input to radio frequency component 108 for voice communication. For stereo acquisition or noise reduction purposes, there may be multiple microphones, each located at a different part of electronic device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 102 or radio frequency component 108 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as ranging. In some embodiments, audio component 114 may also include a headphone jack.

[0199] Display screen 118 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 118 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 102 for processing. In this case, display screen 118 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 118, which serves as the front panel of electronic device 100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of electronic device 100 or in a folded design; in still other embodiments, display screen 118 may be a flexible display screen, disposed on a curved or folded surface of electronic device 100. Furthermore, display screen 118 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 118 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0200] Power supply 122 is used to supply power to various components in electronic device 100. Power supply 122 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0201] This application also provides a computer-readable storage medium. This computer-readable medium stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0202] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0203] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various optional implementations above.

[0204] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0205] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0206] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0207] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A classification model training method, characterized in that, include: Multiple target training images are obtained from a first sample image set, which includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region including the object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories. The first feature of the first image region of each of the multiple target training images is extracted using the initial model; The initial model is used to extract the second image region corresponding to the object in each target training image at multiple scaling ratios, the second feature corresponding to each second image region, and the image label corresponding to each second image region; Based on the first image region in the target training image, a target image region is selected from a plurality of second image regions corresponding to the target training image; A first loss value is obtained based on the first feature corresponding to each target training image, the second feature of the target image region of each target training image, and the category to which each target training image belongs; Based on the first image region and each second image region corresponding to each target training image, the quality balance parameter corresponding to each second image region in the target training image is obtained; The sample balance parameter of the target training image is obtained based on the number of second image regions corresponding to each target training image and the number of target image regions. Based on the sample labels corresponding to the first image region and the image labels corresponding to each second image region in the target training image, the true label inference probability of the target training image is obtained; The second loss value is obtained based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability. The first loss value and the second loss value are summed to obtain the feature loss value; The parameters of the initial model are adjusted based on the feature loss value until the training termination condition is met, at which point the target classification model is obtained.

2. The method according to claim 1, characterized in that, The step of obtaining a first loss value based on a first feature corresponding to each target training image, a second feature of the target image region of each target training image, and the category to which each target training image belongs includes: The target feature of the target training image is obtained by taking the average value of the first feature corresponding to the target training image and the second feature of the target image region of the target training image. The intra-class distance of samples is obtained by calculating the feature distance of target features in training images belonging to the same category. The inter-class distance of samples is obtained by calculating the feature distance of target features in training images belonging to different categories. The first loss value is obtained based on the ratio between the intra-class distance and the inter-class distance of the samples.

3. The method according to claim 1, characterized in that, The step of selecting a target image region from a plurality of second image regions corresponding to each target training image based on a first image region in each target training image includes: The matching degree of the first image region in the target training image is calculated with each of the corresponding second image regions of the target training image to obtain the matching degree value of each of the corresponding second image regions of the target training image; The second image region with a matching degree value greater than the matching degree threshold is obtained and used as the target image region of the target training image.

4. The method according to claim 1, characterized in that, Based on the first image region and each second image region corresponding to each target training image, the quality balance parameters corresponding to each second image region in the target training image are obtained, including: The overlap between the first image region corresponding to each target training image and each second image is calculated to obtain the overlap value corresponding to each second image in each target training image. The overlap value corresponding to each second image is the quality balance parameter corresponding to the second image region.

5. The method according to claim 1, characterized in that, The step of obtaining the sample balance parameters of the target training image based on the number of second image regions corresponding to each target training image and the number of target image regions includes: Obtain the difference between the number of second image regions corresponding to the target training image and the number of target image regions; The sample balance parameter of the target training image is obtained by the ratio between the difference in the number of images corresponding to the target training image and the number of images of the target region corresponding to the target training image.

6. The method according to claim 1, characterized in that, The step of obtaining the true label inference probability of the target training image based on the sample labels corresponding to the first image region and the image labels corresponding to each second image region in the target training image includes: Obtain the number of image labels in the image labels corresponding to each second image region in the target training image that are the same as the sample labels corresponding to the first image region; The ratio obtained by comparing this number with the number of second image regions corresponding to the target training image is used as the true label inference probability of the target training image.

7. The method according to claim 1, characterized in that, The step of obtaining the second loss value based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability includes: The average value of the quality balance parameters corresponding to each second image in the target training image is used to obtain the quality balance parameters of the target training image. The relative entropy of the true label inference probability of the target training image is calculated, and the calculated relative entropy is multiplied by the quality balance parameter and sample balance parameter of the target training image to obtain the loss value corresponding to the target training image. The loss values ​​of each target training image are summed to obtain a second loss value.

8. The method according to any one of claims 1 to 7, characterized in that, The initial model was trained in the following manner: Obtain a second set of sample images, which includes sample images corresponding to objects of at least two categories. Each sample image is labeled with a first image region including the object image and a sample label corresponding to the first image region. Each sample image in the second sample image set is input into the convolutional neural network model, and the convolutional neural network model is trained based on each sample image to obtain the initial model.

9. An image classification method, characterized in that, The method includes: Obtain the image to be classified; The target classification model obtained as described in any one of claims 1-8 is invoked to classify the image to be classified, and the classification result is obtained.

10. A classification model training device, characterized in that, The device includes: The training image acquisition module is used to acquire multiple target training images from a first sample image set. The first sample image set includes training images corresponding to objects of multiple categories. Each training image is labeled with a first image region including the object image and a sample label corresponding to the first image region. The multiple target training images belong to training images corresponding to objects of at least two categories. The feature extraction module is used to extract the first feature of the first image region of each of the multiple target training images using the initial model; The information extraction module is used to extract, using the initial model, a second image region corresponding to an object in each target training image at multiple scaling ratios, a second feature corresponding to each second image region, and an image label corresponding to each second image region; The loss value acquisition module is configured to: select a target image region from multiple second image regions corresponding to the target training image based on a first image region in the target training image; obtain a first loss value based on a first feature corresponding to each target training image, a second feature of the target image region in each target training image, and the category to which each target training image belongs; obtain a quality balance parameter corresponding to each second image region in the target training image based on the first image region and each second image region corresponding to each target training image; obtain a sample balance parameter for the target training image based on the number of second image regions and the number of target image regions corresponding to each target training image; obtain the true label inference probability of the target training image based on the sample label corresponding to the first image region and the image label corresponding to each second image region in the target training image; obtain a second loss value based on the quality balance parameter corresponding to each second image region in each target training image, the sample balance parameter of the target training image, and the true label inference probability; and accumulate the first loss value and the second loss value to obtain a feature loss value. The model acquisition module is used to adjust the parameters of the initial model based on the feature loss value until the training termination condition is met to obtain the target classification model.

11. An image classification device, characterized in that, The method includes: The image acquisition module is used to acquire images to be classified. The image classification module is used to call the target classification model obtained as described in any one of claims 1-8 to classify the image to be classified and obtain the classification result.

12. An electronic device, characterized in that, It includes a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-8 or 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method described in any one of claims 1-8 or 9.

14. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method described in any one of claims 1-8 or 9.

Images