Image classification method, image classification model training method and device

By identifying the scene of live images in the image classification model and performing weighted fusion, the problem of low image category recognition accuracy in the live broadcast environment is solved, and higher image classification accuracy is achieved.

CN112488218BActive Publication Date: 2024-11-29BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202011414786.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-04
Publication Date
2024-11-29
Estimated Expiration
2040-12-04

AI Technical Summary

Technical Problem

In the complex and changing live broadcast environment, existing live broadcast image classification methods are difficult to accurately identify image categories, resulting in low classification accuracy.

Method used

The image classification model is adopted to determine the final image category by identifying the scenes contained in the image to be processed and image category recognition is performed in multiple preset scenarios, and weighted fusion is performed based on the scene recognition results and the initial classification results. The model includes a feature extraction network, a scene classification network and an image classification network of multiple preset scenes. The final image classification model is obtained through machine learning training.

Benefits of technology

It effectively avoids the interference of scene differences on image category recognition, improves the accuracy of image classification, and is suitable for image classification in various live broadcast scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112488218B_ABST
    Figure CN112488218B_ABST
Patent Text Reader

Abstract

The present invention provides an image classification method, a training method and a device for an image classification model, comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained image classification model so that the image classification model recognizes the scene contained in the image to be processed and obtains a scene recognition result; identifying the image category of the image to be processed under multiple preset scenes and obtaining an initial classification result corresponding to each preset scene; outputting the image category of the image to be processed based on the initial classification result corresponding to each preset scene and the scene recognition result. In this method, the image classification model can recognize the scene contained in the image, and determine the final category of the image based on the scene recognition result and the initial classification result of the image category obtained under each scene; because the scene contained in the image is taken into consideration when determining the image category, the interference of the difference in scene factors on the image category recognition is avoided, thereby improving the accuracy of image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image classification method, and a training method and device for an image classification model. Background Art

[0002] With the development of network technology and smart mobile platforms, the live broadcast industry has an increasingly greater impact on people's lives. In order to avoid the harm of bad information on live broadcast platforms, it is necessary to supervise the live broadcast content of live broadcast platforms. However, due to the huge number of live broadcast platforms, human supervision is often time-consuming, labor-intensive and costly. Therefore, an algorithm is needed to automatically classify live broadcast images in order to determine whether the content of the live broadcast images contains bad information through the classification results.

[0003] The live image classification method in the related art usually classifies the live images through a pre-established mapping relationship between the input image and the image category. However, due to the diversity of the live broadcast environment, the scenes contained in the live images are also complex and changeable, which interferes with the recognition and determination of the image category, resulting in a low classification accuracy of this method. Summary of the invention

[0004] The purpose of the present invention is to provide an image classification method, an image classification model training method and a device to avoid scene differences interfering with the recognition and determination of image categories, thereby improving the accuracy of image classification.

[0005] In a first aspect, an embodiment of the present invention provides an image classification method, the method comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained image classification model, and outputting the image category of the image to be processed; wherein the image classification model is used to: identify the scene contained in the image to be processed and obtain a scene recognition result; identify the image category of the image to be processed under multiple preset scenes and obtain an initial classification result corresponding to each preset scene; and obtain the image category of the image to be processed based on the initial classification result and scene recognition result corresponding to each preset scene.

[0006] In an optional embodiment, the scene recognition result includes the probability that the image to be processed belongs to each preset scene; the initial classification result includes the probability that the image to be processed belongs to each preset category in multiple preset categories; the step of obtaining the image category of the image to be processed based on the initial classification result corresponding to each preset scene and the scene recognition result includes: determining the weighted weight of the initial classification result corresponding to each preset scene based on the probability that the image to be processed belongs to each preset scene in the scene recognition result; determining the final probability that the image to be processed belongs to each preset category based on the weighted weight and the initial classification result corresponding to each preset scene, and determining the preset category corresponding to the largest final probability as the image category of the image to be processed.

[0007] In an optional embodiment, the step of determining the weighted weight of the initial classification result corresponding to each preset scene based on the probability that the image to be processed in the scene recognition result belongs to each preset scene includes: for each preset scene, determining the probability that the image to be processed in the scene recognition result belongs to the current scene as the weighted weight of the initial classification result corresponding to the current scene; the step of determining the final probability that the image to be processed belongs to each preset category based on the weighted weight and the initial classification result corresponding to each preset scene includes: for each preset scene, multiplying the weighted weight corresponding to the current scene by the probability of each preset category in the initial classification result corresponding to the current scene, respectively, to obtain the probability product corresponding to each preset category in the current scene; for each preset category, adding the probability products corresponding to the current category in each preset scene to obtain the final probability that the image to be processed belongs to the current category.

[0008] In an optional embodiment, the image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; the feature extraction network is used to extract image features from the image to be processed, and input the image features into the scene classification network and the image classification networks corresponding to multiple preset scenes; the scene classification network is used to identify the scene contained in the image to be processed according to the image features, and obtain a scene recognition result; the image classification network corresponding to each preset scene is used to identify the image category of the image to be processed in the preset scene according to the image features, and obtain an initial classification result corresponding to the preset scene; the classification output network is used to obtain the image category of the image to be processed based on the initial classification result and scene recognition result corresponding to each preset scene.

[0009] In second aspect, an embodiment of the present invention provides a training method for an image classification model, the training method comprising: obtaining a sample set; the sample set comprising multiple samples, each sample comprising a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one scene among multiple preset scenes; based on the sample set, performing machine learning training on an initial model of the image classification model to obtain an image classification model.

[0010] In an optional embodiment, the initial model of the above-mentioned image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; the above-mentioned step of performing machine learning training on the initial model of the image classification model based on the sample set to obtain the image classification model includes: training the feature extraction network and the scene classification network based on the sample set to obtain a trained feature extraction network and a trained scene classification network; for the image classification network corresponding to each preset scene, training the image classification network corresponding to the current preset scene through samples in the sample set whose scene category label is the current preset scene to obtain a trained image classification network; training the trained feature extraction network, the trained scene classification network, multiple trained image classification networks, and the classification output network based on the sample set to obtain the final image classification model.

[0011] In an optional embodiment, the above-mentioned step of training an image classification network corresponding to the current preset scene through samples in the sample set whose scene category label is the current preset scene to obtain a trained image classification network includes: determining a target sample whose scene category label is the current preset scene from the sample set; inputting the sample image in the target sample into the trained feature extraction network to obtain a first output result; inputting the first output result into the image classification network corresponding to the current preset scene to obtain a second output result; determining a network loss value based on the second output result and the image category label corresponding to the sample image; updating the weight parameters of the image classification network based on the network loss value; and continuing to execute the step of determining a target sample whose scene category label is the current preset scene from the sample set until the image classification network converges to obtain a trained image classification network corresponding to the current preset scene.

[0012] In an optional embodiment, the step of training the trained feature extraction network, the trained scene classification network, the multiple trained image classification networks and the classification output network based on the sample set to obtain the final image classification model includes: determining the training sample based on the sample set; the training sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; inputting the sample image into the trained feature extraction network to output a first result, inputting the first result into the trained scene classification network to output a second result; inputting the first result into multiple trained image classification networks respectively to obtain third results corresponding to multiple preset scenes; inputting the second result and the third result corresponding to each preset scene into the classification output network to obtain an output result; determining the loss amount based on the output result and the image category label; updating the weight parameters of the trained feature extraction network, the trained scene classification network, the multiple trained image classification networks and the classification output network based on the loss amount; and continuing to execute the step of determining the training sample based on the sample set until each network converges to obtain the image classification model.

[0013] In an optional embodiment, the step of determining the loss amount based on the output result and the image category label includes: the loss amount is determined by the following formula: L = -log (c y ), where L represents the loss amount, and c represents the output result, which includes the final probability that the sample image belongs to each of the multiple preset categories; c y It indicates the final probability that the sample image belongs to the preset category corresponding to the image category label in the output result, and log indicates logarithmic operation.

[0014] In a third aspect, an embodiment of the present invention provides an image classification device, which includes: an image acquisition module, used to acquire an image to be processed; an image classification module, used to input the image to be processed into a pre-trained image classification model, and output the image category of the image to be processed; wherein the image classification model is used to: identify the scene contained in the image to be processed and obtain a scene recognition result; identify the image category of the image to be processed under multiple preset scenes and obtain an initial classification result corresponding to each preset scene; obtain the image category of the image to be processed based on the initial classification result and scene recognition result corresponding to each preset scene.

[0015] In a fourth aspect, an embodiment of the present invention provides a training device for an image classification model, the training device comprising: a sample set acquisition module, used to acquire a sample set; the sample set comprises a plurality of samples, each sample comprises a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one of a plurality of preset scenes; and a model training module, used to perform machine learning training on an initial model of the image classification model based on the sample set to obtain an image classification model.

[0016] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image classification method described in any one of the aforementioned embodiments or the training method of the image classification model described in any one of the aforementioned embodiments.

[0017] In a sixth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image classification method described in any one of the aforementioned embodiments or the training method of the image classification model described in any one of the aforementioned embodiments.

[0018] The embodiments of the present invention bring the following beneficial effects:

[0019] The present invention provides an image classification method, an image classification model training method and a device, which first obtain an image to be processed; then input the image to be processed into a pre-trained image classification model, the image classification model identifies the scene contained in the image to be processed, and obtains a scene recognition result; identifies the image category of the image to be processed under multiple preset scenes, and obtains an initial classification result corresponding to each preset scene; based on the initial classification result and scene recognition result corresponding to each preset scene, outputs the image category of the image to be processed. In this method, the image classification model can identify the scene contained in the image, and determine the final category of the image based on the scene recognition result and the initial classification result of the image category obtained under each scene; because the scene contained in the image is taken into account when determining the image category, the interference of the difference in scene factors on the image category recognition is avoided, thereby improving the accuracy of image classification.

[0020] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flowchart of an image classification method provided by an embodiment of the present invention;

[0024] Figure 2 A flowchart of another image classification method provided by an embodiment of the present invention;

[0025] Figure 3 A flowchart of another image classification method provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the network structure of an image classification model provided by an embodiment of the present invention;

[0027] Figure 5 A flowchart of a method for training an image classification model provided by an embodiment of the present invention;

[0028] Figure 6 A flowchart of another method for training an image classification model provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of the structure of an image classification device provided by an embodiment of the present invention;

[0030] Figure 8 A schematic diagram of the structure of a training device for an image classification model provided by an embodiment of the present invention;

[0031] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In the related art, there are two methods for classifying live images. The first is the traditional image classification algorithm, which generally requires manual extraction of image features (such as shape, color, texture, etc.), and then training a classifier that converts image features to categories. The live images can be classified through this classifier, but this method lacks high-level semantic information of the image, resulting in low image classification accuracy.

[0035] The second method is to classify live images based on a neural network. The neural network establishes a mapping relationship between input images and image categories through machine learning training. Compared with traditional image classification algorithms, this method can consider high-level semantic information of images and has relatively high accuracy. However, the image sources of the live broadcast platform are diverse, such as outdoor scenes, indoor scenes, film and television scenes, animation scenes, comic scenes, game scenes, etc., and the images of different scenes are quite different, which increases the difficulty of classifying live images and interferes with the recognition and judgment of image categories, resulting in a low classification accuracy of this method.

[0036] Based on the above problems, the embodiments of the present invention provide an image classification method, an image classification model training method and a device. The technology can be applied to various image classification scenarios, especially image classification scenarios in various live broadcast scenarios. To facilitate understanding of the present embodiment, an image classification method disclosed in the embodiment of the present invention is first introduced in detail. Figure 1 As shown, the method comprises the following steps:

[0037] Step S102, obtaining an image to be processed.

[0038] The image can be acquired by shooting with a camera or other device connected by communication and then transmitting the image, or by acquiring the image from a storage device storing the shot images to be processed.

[0039] The above-mentioned images to be processed may be images in various live broadcast scenes, which may be outdoor scenes, indoor scenes, film and television scenes, animation scenes, comic scenes or game scenes, etc. The images to be processed may contain specific people, objects or events, etc.

[0040] Step S104, input the above-mentioned image to be processed into a pre-trained image classification model, and output the image category of the image to be processed; wherein the image classification model is used to: identify the scene contained in the image to be processed and obtain a scene recognition result; identify the image category of the image to be processed under multiple preset scenes, and obtain an initial classification result corresponding to each preset scene; based on the initial classification result and scene recognition result corresponding to each preset scene, obtain the image category of the image to be processed.

[0041] The above preset scenes can be any live broadcast scenes. For example, the multiple preset scenes can include: outdoor scenes, indoor scenes, film and television scenes, animation scenes, comic scenes, and game scenes, etc. The specific live broadcast scenes included in the multiple preset scenes can be set according to user needs and are not specifically limited here.

[0042] The above-mentioned image classification model can be a deep learning model or a neural network model, etc. The image classification model is obtained by machine learning training using a preset sample set. In specific implementation, the image classification model can identify the scene contained in the image to be processed, that is, to identify which preset scene the image to be processed comes from, or the probability of each preset scene from multiple preset scenes; it can also identify the image category of the image to be processed in multiple preset scenes respectively, and obtain the initial classification result in each preset scene. This method explicitly considers the difference in image appearance in different scenes, so as to eliminate the influence of scene factors on the image classification result. The initial classification result can be used to indicate the image category of the image to be processed in a certain preset scene; then the initial classification result and the scene recognition result corresponding to each preset scene are fused to obtain a more accurate image category of the image to be processed.

[0043] An image classification method provided by an embodiment of the present invention first obtains an image to be processed; then the image to be processed is input into a pre-trained image classification model, the image classification model identifies the scene contained in the image to be processed and obtains a scene recognition result; the image category of the image to be processed is identified under multiple preset scenes, and an initial classification result corresponding to each preset scene is obtained; based on the initial classification result and scene recognition result corresponding to each preset scene, the image category of the image to be processed is output. In this method, the image classification model can identify the scene contained in the image, and determine the final category of the image based on the scene recognition result and the initial classification result of the image category obtained under each scene; because the scene contained in the image is taken into account when determining the image category, the interference of the difference in scene factors on the image category recognition is avoided, thereby improving the accuracy of image classification.

[0044] The embodiment of the present invention also provides another image classification method, which is implemented on the basis of the method of the above embodiment; the method focuses on describing the specific process of obtaining the image category of the image to be processed based on the initial classification result and scene recognition result corresponding to each preset scene (implemented by the following step S208); Figure 2 As shown, the method comprises the following steps:

[0045] Step S202, obtaining an image to be processed.

[0046] Step S204: input the above-mentioned image to be processed into a pre-trained image classification model.

[0047] Step S206, using the image classification model, identifying the scene contained in the image to be processed to obtain a scene recognition result; identifying the image category of the image to be processed under multiple preset scenes to obtain an initial classification result corresponding to each preset scene; wherein the scene recognition result includes the probability that the image to be processed belongs to each preset scene; the initial classification result includes the probability that the image to be processed belongs to each preset category in multiple preset categories.

[0048] In a specific implementation, the scene recognition result includes the probability that the image to be processed belongs to each preset scene under multiple preset scenes; the initial classification result corresponding to each preset scene includes the probability that the image to be processed belongs to each preset category in multiple preset categories under the current preset scene. The multiple preset categories can be image categories pre-set according to user needs, and the preset categories can include normal, vulgar, pornographic and other image categories.

[0049] Step S208, through the image classification model, based on the probability that the image to be processed belongs to each preset scene in the scene recognition result, the weighted weight of the initial classification result corresponding to each preset scene is determined; based on the weighted weight and the initial classification result corresponding to each preset scene, the final probability that the image to be processed belongs to each preset category is determined, and the preset category corresponding to the largest final probability is determined as the image category of the image to be processed.

[0050] The weighted weight of the initial classification result corresponding to each of the above-mentioned preset scenes is determined based on the probability that the image to be processed belongs to each preset scene in the scene recognition result. Generally, the greater the probability that the image to be processed belongs to a preset scene, the greater the weighted weight of the initial classification result corresponding to the preset scene. For example, the probability value that the image to be processed belongs to a preset scene can be determined as the weighted weight of the initial classification result corresponding to the preset scene, or a plurality of values ​​of different sizes can be pre-set, and then a value is selected from the pre-set values ​​according to the size of the probability that the image to be processed belongs to each preset scene as the weighted weight of the initial classification result corresponding to each preset scene.

[0051] In specific implementation, the above step S208 can be implemented by the following steps 10-12:

[0052] Step 10: For each preset scene, the probability that the image to be processed in the scene recognition result belongs to the current scene is determined as the weighted weight of the initial classification result corresponding to the current scene.

[0053] Step 11, for each preset scene, multiply the weighted weight corresponding to the current scene by the probability of each preset category in the initial classification result corresponding to the current scene, to obtain the probability product corresponding to each preset category in the current scene.

[0054] Step 12: for each preset category, add the probability products corresponding to the current category in each preset scene to obtain the final probability that the image to be processed belongs to the current category.

[0055] For example, the scene recognition result includes three preset scenes, namely animation scene, comic scene and game scene. The probability that the image to be processed in the scene recognition result belongs to the animation scene is 0.2, the probability that it belongs to the comic scene is 0.1, and the probability that it belongs to the game scene is 0.7; then the weighted weight of the initial classification result corresponding to the animation scene is 0.2, the weighted weight of the initial classification result corresponding to the comic scene is 0.1, and the weighted weight of the initial classification result corresponding to the game scene is 0.7.

[0056] Assume that each initial classification result contains three preset categories, namely normal, vulgar and pornographic; among them, the probability that the image to be processed belongs to the three preset categories in the initial classification result corresponding to the animation scene is [0.6, 0.3, 0.2], the probability that the image to be processed belongs to the three preset categories in the initial classification result corresponding to the comic scene is [0.8, 0.1, 0.1], and the probability that the image to be processed belongs to the three preset categories in the initial classification result corresponding to the game scene is [0.5, 0.2, 0.3]; then the probability product of the image category of the image to be processed being normal is 0.2*0.6+0.1*0.8+0.7*0.5=0.55, the probability product of the image category of the image to be processed being vulgar is 0.2*0.3+0.1*0.1+0.7*0.2=0.20, and the probability product of the image category of the image to be processed being pornographic is 0.2*0.2+0.1*0.1+0.7*0.3=0.26. It can be seen that the maximum value of the probability product is 0.55, so the image category of the image to be processed is normal.

[0057] In the above-mentioned image classification method, the image classification model can identify the scene contained in the image and the initial classification results under different scenes, thereby eliminating the influence of scene factors on the image recognition results. Finally, the recognized scene results and each initial classification result are integrated to obtain the final image category. Therefore, this method explicitly considers the differences between the appearances of images under different scenes, eliminates the interference of different scene factors on image recognition, and improves the accuracy of image classification and recognition. At the same time, since this method can accurately identify the image category of live broadcast images, it can achieve the purpose of intelligent monitoring of network live broadcast rooms and greatly reduce the cost of human supervision.

[0058] The embodiment of the present invention also provides another image classification method, which is implemented on the basis of the method of the above embodiment; the method focuses on describing the image classification method when the image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; Figure 3As shown, the method comprises the following steps:

[0059] Step S302, obtaining an image to be processed.

[0060] Step S304: input the above-mentioned image to be processed into a pre-trained image classification model, which includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network.

[0061] Specifically, the feature extraction network is connected to the scene classification network and the image classification network corresponding to the plurality of preset scenes, and the classification output network is connected to the scene classification network and the image classification network corresponding to the plurality of preset scenes. Figure 4 The following is a schematic diagram of the network structure of the image classification model. Figure 4 The image classification model in contains image classification networks corresponding to three preset scenes, namely the animation scene sub-network, the comic scene sub-network and the game scene sub-network.

[0062] In specific implementation, the feature extraction network includes sequentially connected convolution layers, batch normalization layers and activation function layers. The feature extraction network can extract image features of the input image to obtain high-level semantic information of the image features. In order to improve the performance of the feature extraction network, the feature extraction network usually includes multiple groups of sequentially connected convolution layers, batch normalization layers and activation function layers. The activation function layer can specifically include Sigmoid function, tanh function, Relu function, softmax function, etc. The scene classification network can be composed of a series of "convolution layers + batch normalization layers + activation function layers", global average pooling layers and fully connected layers; the animation scene sub-network, comic scene sub-network and game scene sub-network can also be composed of a series of "convolution layers + batch normalization layers + activation function layers", global average pooling layers and fully connected layers. The classification output network fuses the outputs of the scene classification network, the animation scene sub-network, the comic scene sub-network and the game scene sub-network to obtain the final image recognition result.

[0063] Step S306, extracting image features from the image to be processed through a feature extraction network, and inputting the image features into a scene classification network and image classification networks corresponding to a plurality of preset scenes.

[0064] Step S308, using a scene classification network, based on image features, identifying the scene contained in the image to be processed, and obtaining a scene recognition result.

[0065] Step S310, using the image classification network corresponding to each preset scene, according to the image features, the image category of the image to be processed is identified in the preset scene, and an initial classification result corresponding to the preset scene is obtained.

[0066] Step S312, obtaining the image category of the image to be processed based on the initial classification result and the scene recognition result corresponding to each preset scene through the classification output network.

[0067] The above-mentioned image classification method can identify the scenes contained in the image and identify the initial classification results under different scenes, thereby eliminating the influence of scene factors on the image recognition results. Finally, the recognized scene results and the initial classification results are integrated to obtain the final image category. Therefore, this method explicitly considers the differences between image appearances in different scenes, eliminates the interference of different scene factors on image recognition, and improves the accuracy of image classification and recognition.

[0068] In view of the above-mentioned image classification method, the embodiment of the present invention also provides a training method for an image classification model, such as Figure 5 As shown, the training method includes the following specific steps:

[0069] Step S502, obtaining a sample set; the sample set includes a plurality of samples, each sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one scene among a plurality of preset scenes.

[0070] The above sample set contains a large number of samples, each of which contains a sample image, as well as a scene category label and an image category label corresponding to the sample. The scene category label is used to indicate the scene contained in the sample image, which is one of multiple preset scenes. The number of multiple preset scenes and the included scenes are set according to user needs. The preset scenes may include outdoor scenes, indoor scenes, film and television scenes, animation scenes, comic scenes or game scenes, etc.; the above image category label is used to indicate the image category to which the sample image belongs. The image category is one of multiple preset categories. The number of multiple preset categories and the included preset categories are set according to user needs. The preset categories may include normal, vulgar, pornographic, etc.

[0071] Step S504: Based on the sample set, perform machine learning training on the initial model of the image classification model to obtain the image classification model.

[0072] During the training process of the image classification model, the sample images in the sample set are input into the initial model to obtain the category prediction results of the sample images. Then, the loss amount is obtained through the image category label and scene category label corresponding to the sample image. Then, the weight parameters of the initial model are updated based on the loss amount. When the loss amount converges or reaches the preset number of training times, the image classification model is obtained.

[0073] The training method of the above-mentioned image classification model first obtains a sample set, which includes multiple samples, each of which includes a sample image and a scene category label and an image category label corresponding to the sample image; then, based on the sample set, the initial model of the image classification model is trained by machine learning to obtain an image classification model. When training the image classification model, this method takes into account the differences in image appearance in different scenes, so that when the trained image classification model is used for image classification, the influence of scene factors on image recognition results can be eliminated, thereby improving the generalization ability and classification accuracy of the image classification model.

[0074] The embodiment of the present invention also provides another training method for an image classification model, which is implemented on the basis of the above training method embodiment; the training method focuses on describing the specific process of the steps of performing machine learning training on the initial model of the image classification model based on a sample set to obtain the image classification model when the initial model of the image classification model includes a feature extraction network, a scene classification network, an image classification network corresponding to multiple preset scenes, and a classification output network (implemented by the following steps S604-S608); Figure 6 As shown, the training method includes the following steps:

[0075] Step S602, obtaining a sample set; the sample set includes a plurality of samples, each sample including a sample image, and a scene category label and an image category label corresponding to the sample image.

[0076] When acquiring the sample set, first collect the live images As a sample image, it is assumed that a total of N images are collected, and the i-th image is X i For each image X i , first of all, the image category labels are annotated. For example, suppose there are three preset categories y i ∈{1, 2, 3}, 1 represents normal, 2 represents vulgar, and 3 represents pornography. Then mark X i The scene category label is assumed to be in three preset scenes s i ∈{1, 2, 3}, 1 represents animation scene, 2 represents comic scene, and 3 represents game scene. The final sample set is obtained

[0077] Assume that there are N1 animation samples in the sample set D, that is, there are N1 sample images with scene category labels of animation scenes; N2 comic samples, that is, there are N2 sample images with scene category labels of comic scenes; N3 game samples, that is, there are N3 sample images with scene category labels of game scenes; where N1+N2+N3=N, assuming that the animation data set is Comics Dataset Game Dataset

[0078] Step S604: training a feature extraction network and a scene classification network based on the sample set to obtain a trained feature extraction network and a trained scene classification network.

[0079] In specific implementation, the above step S604 can be implemented by the following steps 20-24:

[0080] Step 20, randomly initialize the weight parameters of the feature extraction network and the weight parameters of the scene classification network.

[0081] Step 21, determining a learning sample based on the sample set; the learning sample includes a sample image, and a scene category label and an image category label corresponding to the sample image.

[0082] Step 22: input the sample images in the above learning samples into the above feature extraction network to obtain feature extraction results.

[0083] Step 23, input the above feature extraction result into the above scene recognition network, and output the recognition result; based on the recognition result and the scene category label corresponding to the sample image, determine the first loss value.

[0084] Step 24, based on the first loss value, update the weight parameters of the feature extraction network and the scene classification network; continue to execute the step of determining the learning sample based on the sample set until the weight parameters of the feature extraction network and the weight parameters of the scene recognition network converge, and obtain the trained feature extraction network and the trained scene classification network.

[0085] For example, the weight parameters W1 of the feature extraction network and W2 of the scene classification network are randomly initialized respectively; then the sample image of the learning sample is input into the feature extraction network and the scene classification network to obtain the output result z, and the output result z is sent to the softmax function of the activation function layer to obtain the recognition result P:

[0086]

[0087] Among them, p j represents the probability corresponding to the jth preset scene in the recognition result P, z j Indicates the output value corresponding to the jth element in the output result, z m represents the output value corresponding to the mth element in the output result z, and M represents the total number of elements in the output result and the recognition result, that is, the total number of categories of the preset scene. Then, based on the scene category label corresponding to the recognition result and the sample image, the first loss value L1 is obtained:

[0088] L1=-log(ps );

[0089] Among them, s represents the scene category label corresponding to the sample image, p s Indicates the probability of the scene category corresponding to the scene category label of the sample image in the recognition result. The derivative of L1 with respect to the weight parameter W1 and weight W2 is calculated by the back propagation algorithm. Next, we update W1 and W2 through the stochastic gradient descent algorithm to obtain the updated weight parameters W1' of the feature extraction network and the updated weight parameters W2' of the scene classification network:

[0090]

[0091]

[0092] Then, W1' and W2' are continuously updated iteratively until the two weight parameters converge, and a trained feature extraction network and a trained scene classification network are obtained.

[0093] Step S606, for the image classification network corresponding to each preset scene, the image classification network corresponding to the current preset scene is trained using samples in the sample set whose scene category labels are the current preset scene, to obtain a trained image classification network.

[0094] In a specific implementation, each of the multiple preset scenes can be used as the current preset scene, and the image classification network corresponding to the current preset scene can be trained through the following steps 30-33:

[0095] Step 30, determine the target sample whose scene category label is the current preset scene from the sample set; that is, determine the target sample from the samples in the above sample set whose scene category label is the current preset scene. For example, if the current preset scene is a comic scene, the target sample can be determined from the comic data set D2.

[0096] Step 31: input the sample image in the target sample into the trained feature extraction network to obtain a first output result.

[0097] Step 32, input the first output result to the image classification network corresponding to the current preset scene to obtain a second output result; based on the second output result and the image category label corresponding to the sample image, determine the network loss value.

[0098] Step 33, update the weight parameters of the image classification network based on the network loss value; continue to perform the step of determining the target sample whose scene category label is the current preset scene from the sample set until the image classification network converges, and obtain the trained image classification network corresponding to the current preset scene.

[0099] For example, the weight parameters of the trained feature network are fixed, and the weight parameters W3 of the image classification network corresponding to the current preset scene are randomly initialized; then the sample image of the target sample is input into the feature extraction network and the image classification network corresponding to the current preset scene to obtain the classification result A, and the classification result is sent to the softmax function of the activation function layer to obtain the second output result B:

[0100]

[0101] Among them, B j represents the probability corresponding to the jth preset category in the second output result, A j Indicates the output value corresponding to the jth element in the classification result, A h represents the output value corresponding to the hth element in the classification result, and H represents the total number of elements contained in the second output result and the classification result, that is, the total number of categories of the preset category. Then, based on the image category label corresponding to the second output result and the sample image, the network loss value L2 is determined:

[0102] L2=-log(B y )

[0103] Among them, y represents the image category label corresponding to the sample image, B y Represents the probability of the image category corresponding to the image category label of the sample image in the second recognition result. The derivative of L2 with respect to the weight parameter W3 is calculated through the back propagation algorithm Next, we update W3 through the stochastic gradient descent algorithm to obtain the updated weight parameter W3' of the image classification network corresponding to the current preset scene:

[0104]

[0105] Then, W3' is continuously updated iteratively until the weight parameter converges, and the trained image classification network corresponding to the current preset scene is obtained.

[0106] Step S608, training the trained feature extraction network, the trained scene classification network, the multiple trained image classification networks and the classification output network based on the sample set to obtain a final image classification model.

[0107] The purpose of the above step S608 is to fine-tune the weight parameters of each network in the image classification model to obtain an image classification model that can more accurately predict the image category. In specific implementation, the above step S608 can be implemented by the following steps 40-43:

[0108] Step 40, determining a training sample based on the sample set; the training sample includes a sample image, and a scene category label and an image category label corresponding to the sample image.

[0109] Step 41, input the sample image into the trained feature extraction network, output a first result, input the first result into the trained scene classification network, and output a second result.

[0110] Step 42, input the first result into a plurality of trained image classification networks respectively to obtain third results corresponding to a plurality of preset scenes; input the second result and the third result corresponding to each preset scene into a classification output network to obtain an output result.

[0111] Step 43, based on the above output results and image category labels, determine the loss amount; based on the loss amount, update the weight parameters of the trained feature extraction network, the trained scene classification network, multiple trained image classification networks and the classification output network; continue to execute the step of determining the training samples based on the sample set until each network converges to obtain an image classification model.

[0112] Taking multiple preset scenes including animation scenes, comic scenes and game scenes as examples, the above steps 40-43 are introduced in detail. First, the sample image of the determined training sample is input into the trained feature extraction network to obtain a first result, and the first result is input into the trained scene classification model to obtain a classification output result Z, and the classification output result is sent to the softmax function of the activation function layer to obtain a second result P';

[0113]

[0114] Among them, p′ j represents the probability corresponding to the jth preset scene in the second result P', the value range of j is 1-3, the first preset scene represents the animation scene, the second preset scene represents the comic scene, and the third preset scene represents the game scene; Z j Represents the output value corresponding to the jth element in the classification output result Z, Z m Represents the output value corresponding to the mth element in the classification output result Z.

[0115] Then, the first result is respectively input into the image classification network corresponding to the trained animation scene, the image classification network corresponding to the trained comic scene, and the image classification network corresponding to the trained game scene, to obtain the classification result A1 corresponding to the animation scene, the classification result A2 corresponding to the comic scene, and the classification result A3 corresponding to the game scene. The above classification results are respectively input into the softmax function to obtain the third result B1 corresponding to the animation scene, the third result B2 corresponding to the comic scene, and the third result B3 corresponding to the game scene:

[0116]

[0117]

[0118]

[0119] Among them, B1 j 、B2 j 、B3 j A1 represents the probability corresponding to the jth preset category in the third result corresponding to the animation scene, the third result corresponding to the comic scene, and the third result corresponding to the game scene, respectively. j 、A2 j 、A3 j A represents the output value corresponding to the jth element in the classification results corresponding to the animation scene, the comic scene, and the game scene, respectively. 1h 、A2 h 、A3 h It represents the output value corresponding to the hth element in the classification results corresponding to the animation scene, the classification results corresponding to the comic scene, and the classification results corresponding to the game scene. H represents the total number of elements in the third result and the classification results, that is, the total number of categories of the preset categories.

[0120] Then the second result, the third result corresponding to the animation scene, the third result corresponding to the comic scene, and the third result corresponding to the game scene are input into the classification output network to obtain the output result c:

[0121] c=p′1×A1+p′2×A2+p′3×P3;

[0122] Furthermore, it is necessary to determine the loss amount L based on the output results and image category labels:

[0123] L = -log(c y );

[0124] Wherein, L represents the loss amount, c represents the output result, and the output result includes the final probability that the sample image belongs to each preset category in multiple preset categories; c y Indicates the final probability that the sample image in the output result belongs to the preset category corresponding to the image category label, and log represents the logarithmic operation. Then, the back propagation algorithm is used to find the derivative of L with respect to the weight parameter w1 of the trained feature extraction network, the weight parameter w2 of the trained scene classification network, the weight parameter w3 of the image classification network corresponding to the trained animation scene, the weight parameter w4 of the image classification network corresponding to the trained comic scene, and the weight parameter w5 of the image classification network corresponding to the trained game scene. Next, we update w1, w2, w3, w4, and w5 through the stochastic gradient descent algorithm to obtain the updated w1′, w2′, w3′, w4′, and w5′:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] The weight parameters of the trained feature extraction network, the trained scene classification network, the weight parameters of the image classification network corresponding to the trained animation scenes, the weight parameters of the image classification network corresponding to the trained comic scenes, and the weight parameters of the image classification network corresponding to the trained game scenes are continuously and iteratively updated until convergence to obtain the final image classification model.

[0131] In specific implementation, the recognition accuracy of the trained image classification model can be determined by a test set used to verify the model; usually, a test sample can be determined from the test set, and the test sample includes a sample image and an image category label corresponding to the sample image. The image category of the sample image can be obtained by inputting the sample image in the test sample into the image classification model, and the image category is compared with the image category label of the sample image to determine whether the image category is the same as the image category label. If they are the same, it indicates that the test result is correct; then continue to determine test samples from the test set until all samples in the test set are selected; count the correctness of the test result corresponding to each test sample to obtain the prediction accuracy of the trained image classification model.

[0132] Compared with the traditional image classification algorithm, the training method of the above image classification model can automatically learn the multi-level semantic features of the image related to the image category, thereby improving the image classification accuracy. Compared with the method of classifying live images based on neural networks, this method explicitly considers the differences between the appearances of different scene images. In order to eliminate the interference of different scene factors on live image recognition, the core idea of ​​the image classification model provided by this method is divide and conquer. First, a scene classification network is used to determine the scene category of the input image, and then an image classification network is designed for different scenes. This can eliminate the influence of scene factors on image recognition results. Finally, the outputs of each classification network are fused to obtain the final image classification result.

[0133] In view of the above-mentioned embodiment of the image classification method, the embodiment of the present invention provides an image classification device, such as Figure 7 As shown, the device comprises:

[0134] The image acquisition module 70 is used to acquire the image to be processed.

[0135] The image classification module 71 is used to input the image to be processed into a pre-trained image classification model and output the image category of the image to be processed. The image classification model is used to: identify the scene contained in the image to be processed and obtain the scene recognition result; identify the image category of the image to be processed under multiple preset scenes and obtain the initial classification result corresponding to each preset scene; obtain the image category of the image to be processed based on the initial classification result and scene recognition result corresponding to each preset scene.

[0136] The above-mentioned image classification device first obtains the image to be processed; then inputs the image to be processed into a pre-trained image classification model, the image classification model identifies the scene contained in the image to be processed, and obtains a scene recognition result; identifies the image category of the image to be processed under multiple preset scenes, and obtains the initial classification result corresponding to each preset scene; based on the initial classification result and scene recognition result corresponding to each preset scene, outputs the image category of the image to be processed. In this method, the image classification model can identify the scene contained in the image, and determine the final category of the image based on the scene recognition result and the initial classification result of the image category obtained under each scene; because the scene contained in the image is taken into account when determining the image category, the interference of the difference in scene factors on the image category recognition is avoided, thereby improving the accuracy of image classification.

[0137] Specifically, the scene recognition result includes the probability that the image to be processed belongs to each preset scene; the initial classification result includes the probability that the image to be processed belongs to each preset category in multiple preset categories; the image classification module 71 is used to: determine the weighted weight of the initial classification result corresponding to each preset scene based on the probability that the image to be processed in the scene recognition result belongs to each preset scene through the image classification model; determine the final probability that the image to be processed belongs to each preset category based on the weighted weight and the initial classification result corresponding to each preset scene, and determine the preset category corresponding to the largest final probability as the image category of the image to be processed.

[0138] Furthermore, the above-mentioned image classification module 71 is also used to: for each preset scene, determine the probability that the image to be processed in the scene recognition result belongs to the current scene as the weighted weight of the initial classification result corresponding to the current scene; for each preset scene, multiply the weighted weight corresponding to the current scene by the probability of each preset category in the initial classification result corresponding to the current scene, respectively, to obtain the probability product corresponding to each preset category in the current scene; for each preset category, add the probability products corresponding to the current category in each preset scene to obtain the final probability that the image to be processed belongs to the current category.

[0139] Furthermore, the above-mentioned image classification model includes a feature extraction network, a scene classification network, an image classification network corresponding to multiple preset scenes, and a classification output network; the feature extraction network is used to extract image features from the image to be processed, and input the image features into the scene classification network and the image classification networks corresponding to multiple preset scenes; the scene classification network is used to identify the scene contained in the image to be processed according to the image features, and obtain the scene recognition result; the image classification network corresponding to each preset scene is used to identify the image category of the image to be processed in the preset scene according to the image features, and obtain the initial classification result corresponding to the preset scene; the classification output network is used to obtain the image category of the image to be processed based on the initial classification result and the scene recognition result corresponding to each preset scene.

[0140] The image classification device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned image classification method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0141] In view of the above-mentioned embodiment of the training method of the image classification model, the embodiment of the present invention provides a training device for the image classification model, such as Figure 8 As shown, the training device comprises:

[0142] The sample set acquisition module 80 is used to acquire a sample set; the sample set includes multiple samples, each sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one scene among multiple preset scenes.

[0143] The model training module 81 is used to perform machine learning training on the initial model of the image classification model based on the sample set to obtain the image classification model.

[0144] The training device of the above-mentioned image classification model first obtains a sample set, which includes multiple samples, each sample includes a sample image and a scene category label and an image category label corresponding to the sample image; then based on the sample set, the initial model of the image classification model is trained by machine learning to obtain an image classification model. This method takes into account the differences in images in different scenes when training the image classification model, so that when the trained image classification model is used for image classification, the influence of scene factors on image recognition results can be eliminated, thereby improving the generalization ability and classification accuracy of the image classification model.

[0145] Specifically, the initial model of the above-mentioned image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; the above-mentioned model training module 81 includes: a first training unit, which is used to train the feature extraction network and the scene classification network based on the sample set to obtain a trained feature extraction network and a trained scene classification network; a second training unit, which is used to train the image classification network corresponding to each preset scene through samples in the sample set whose scene category labels are the current preset scenes, to obtain a trained image classification network; a third training unit, which is used to train the trained feature extraction network, the trained scene classification network, multiple trained image classification networks, and a classification output network based on the sample set to obtain a final image classification model.

[0146] Furthermore, the above-mentioned second training unit is also used to: determine a target sample whose scene category label is the current preset scene from the sample set; input the sample image in the target sample into the trained feature extraction network to obtain a first output result; input the first output result into the image classification network corresponding to the current preset scene to obtain a second output result; determine the network loss value based on the second output result and the image category label corresponding to the sample image; update the weight parameters of the image classification network based on the network loss value; continue to execute the step of determining the target sample whose scene category label is the current preset scene from the sample set until the image classification network converges to obtain the trained image classification network corresponding to the current preset scene.

[0147] Furthermore, the above-mentioned third training unit is also used to: determine training samples based on the sample set; the training samples include sample images, and scene category labels and image category labels corresponding to the sample images; input the sample images into the trained feature extraction network, output a first result, input the first result into the trained scene classification network, and output a second result; input the first result into multiple trained image classification networks respectively to obtain third results corresponding to multiple preset scenes; input the second result and the third result corresponding to each preset scene into the classification output network to obtain an output result; determine the loss amount based on the output result and the image category label; update the weight parameters of the trained feature extraction network, the trained scene classification network, the multiple trained image classification networks and the classification output network based on the loss amount; continue to execute the steps of determining the training samples based on the sample set until each network converges to obtain an image classification model.

[0148] In specific implementation, the above loss amount is determined by the following formula:

[0149] L = -log(c y );

[0150] Wherein, L represents the loss amount, c represents the output result, and the output result includes the final probability that the sample image belongs to each preset category in multiple preset categories; c y It represents the final probability that the sample image in the output result belongs to the preset category corresponding to the image category label, and log represents logarithmic operation.

[0151] The training device for the image classification model provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned training method embodiment for the image classification model. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0152] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, the electronic device includes a processor 101 and a memory 100, wherein the memory 100 stores machine executable instructions that can be executed by the processor 101, and the processor 101 executes the machine executable instructions to implement the above-mentioned image classification method or the above-mentioned image classification model training method.

[0153] Further, Fig. 9 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0154] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0155] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0156] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image classification method or the training method of the above-mentioned image classification model. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0157] The computer program product of the image classification method, image classification model training method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0158] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0159] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image classification method, It is characterized in that The method comprises: Acquire an image to be processed; the image to be processed is an image in various live broadcast scenarios; Inputting the image to be processed into a pre-trained image classification model, and outputting the image category of the image to be processed; The image classification model is used to: identify the scene contained in the image to be processed to obtain a scene recognition result; identify the image category of the image to be processed under multiple preset scenes to obtain an initial classification result corresponding to each preset scene; obtain the image category of the image to be processed based on the initial classification result corresponding to each preset scene and the scene recognition result; The scene recognition result includes the probability that the image to be processed belongs to each of the preset scenes; the initial classification result includes the probability that the image to be processed belongs to each of the preset categories in a plurality of preset categories; The step of obtaining the image category of the image to be processed based on the initial classification result corresponding to each of the preset scenes and the scene recognition result comprises: Determining a weighted weight of an initial classification result corresponding to each of the preset scenes based on a probability that the image to be processed in the scene recognition result belongs to each of the preset scenes; Based on the weighted weights and the initial classification results corresponding to each of the preset scenes, the final probability that the image to be processed belongs to each of the preset categories is determined, and the preset category corresponding to the largest final probability is determined as the image category of the image to be processed.

2. The method according to claim 1, It is characterized in that The step of determining the weighted weight of the initial classification result corresponding to each preset scene based on the probability that the image to be processed in the scene recognition result belongs to each preset scene comprises: For each of the preset scenes, determining the probability that the image to be processed in the scene recognition result belongs to the current scene as the weighted weight of the initial classification result corresponding to the current scene; The step of determining the final probability that the image to be processed belongs to each of the preset categories based on the weighted weights and the initial classification results corresponding to each of the preset scenes comprises: For each of the preset scenes, the weighted weight corresponding to the current scene is multiplied by the probability of each preset category in the initial classification result corresponding to the current scene, to obtain the probability product corresponding to each of the preset categories in the current scene; For each of the preset categories, the probability products corresponding to the current category in each of the preset scenarios are added together to obtain a final probability that the image to be processed belongs to the current category.

3. The method according to claim 1, It is characterized in that The image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to a plurality of preset scenes, and a classification output network; The feature extraction network is used to extract image features from the image to be processed, and input the image features into the scene classification network and the image classification networks corresponding to the plurality of preset scenes; The scene classification network is used to identify the scene contained in the image to be processed according to the image features, and obtain a scene recognition result; the image classification network corresponding to each of the preset scenes is used to identify the image category of the image to be processed in the preset scene according to the image features, and obtain an initial classification result corresponding to the preset scene; The classification output network is used to obtain the image category of the image to be processed based on the initial classification result corresponding to each of the preset scenes and the scene recognition result.

4. A training method for an image classification model, It is characterized in that The training method comprises: Acquire a sample set; the sample set includes a plurality of samples, each sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one of a plurality of preset scenes; the sample image is a collected live broadcast image; Based on the sample set, performing machine learning training on the initial model of the image classification model to obtain the image classification model; The initial model of the image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; The step of performing machine learning training on the initial model of the image classification model based on the sample set to obtain the image classification model includes: Training the feature extraction network and the scene classification network based on the sample set to obtain a trained feature extraction network and a trained scene classification network; For each image classification network corresponding to a preset scene, the image classification network corresponding to the current preset scene is trained by using samples in the sample set whose scene category label is the current preset scene to obtain a trained image classification network; Based on the sample set, the trained feature extraction network, the trained scene classification network, a plurality of the trained image classification networks and the classification output network are trained to obtain a final image classification model; The step of training the trained feature extraction network, the trained scene classification network, a plurality of trained image classification networks and the classification output network based on the sample set to obtain a final image classification model includes: Determining a training sample based on the sample set; the training sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; Inputting the sample image into the trained feature extraction network to output a first result, and inputting the first result into the trained scene classification network to output a second result; Inputting the first result into the plurality of trained image classification networks respectively to obtain third results corresponding to a plurality of preset scenes; inputting the second result and the third result corresponding to each preset scene into the classification output network to obtain an output result; Based on the output result and the image category label, determine the loss amount; based on the loss amount, update the weight parameters of the trained feature extraction network, the trained scene classification network, multiple trained image classification networks and the classification output network; continue to perform the step of determining training samples based on the sample set until each network converges to obtain an image classification model.

5. The training method according to claim 4, It is characterized in that The step of training an image classification network corresponding to the current preset scene by using the samples in the sample set whose scene category label is the current preset scene to obtain the trained image classification network comprises: Determine, from the sample set, a target sample whose scene category label is a current preset scene; Inputting the sample image in the target sample into the trained feature extraction network to obtain a first output result; Inputting the first output result into the image classification network corresponding to the current preset scene to obtain a second output result; determining a network loss value based on the second output result and the image category label corresponding to the sample image; The weight parameters of the image classification network are updated based on the network loss value; and the step of determining the target sample whose scene category label is the current preset scene from the sample set is continued until the image classification network converges to obtain the trained image classification network corresponding to the current preset scene.

6. The training method according to claim 4, It is characterized in that The step of determining the loss amount based on the output result and the image category label comprises: The loss amount is determined by the following formula: ; Wherein, L represents the loss amount, c represents the output result, and the output result includes the final probability that the sample image belongs to each preset category in multiple preset categories; represents the final probability that the sample image in the output result belongs to the preset category corresponding to the image category label, Represents logarithmic operation.

7. An image classification device, It is characterized in that The device comprises: An image acquisition module is used to acquire images to be processed; the images to be processed are images in various live broadcast scenarios; An image classification module, used for inputting the image to be processed into a pre-trained image classification model and outputting the image category of the image to be processed; The image classification model is used to: identify the scene contained in the image to be processed to obtain a scene recognition result; identify the image category of the image to be processed under multiple preset scenes to obtain an initial classification result corresponding to each preset scene; obtain the image category of the image to be processed based on the initial classification result corresponding to each preset scene and the scene recognition result; The scene recognition result includes the probability that the image to be processed belongs to each of the preset scenes; the initial classification result includes the probability that the image to be processed belongs to each of the preset categories in a plurality of preset categories; The step of obtaining the image category of the image to be processed based on the initial classification result corresponding to each of the preset scenes and the scene recognition result comprises: Determining a weighted weight of an initial classification result corresponding to each of the preset scenes based on a probability that the image to be processed in the scene recognition result belongs to each of the preset scenes; Based on the weighted weights and the initial classification results corresponding to each of the preset scenes, the final probability that the image to be processed belongs to each of the preset categories is determined, and the preset category corresponding to the largest final probability is determined as the image category of the image to be processed.

8. A training device for an image classification model, It is characterized in that The training device comprises: A sample set acquisition module, used to acquire a sample set; the sample set includes a plurality of samples, each sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; wherein the scene category label is used to indicate one of a plurality of preset scenes; the sample image is a collected live broadcast image; A model training module, used to perform machine learning training on the initial model of the image classification model based on the sample set to obtain the image classification model; The initial model of the image classification model includes a feature extraction network, a scene classification network, image classification networks corresponding to multiple preset scenes, and a classification output network; The model training module is also used for: Training the feature extraction network and the scene classification network based on the sample set to obtain a trained feature extraction network and a trained scene classification network; For each image classification network corresponding to a preset scene, the image classification network corresponding to the current preset scene is trained by using samples in the sample set whose scene category label is the current preset scene to obtain a trained image classification network; Based on the sample set, the trained feature extraction network, the trained scene classification network, a plurality of the trained image classification networks and the classification output network are trained to obtain a final image classification model; The step of training the trained feature extraction network, the trained scene classification network, a plurality of trained image classification networks and the classification output network based on the sample set to obtain a final image classification model includes: Determining a training sample based on the sample set; the training sample includes a sample image, and a scene category label and an image category label corresponding to the sample image; Inputting the sample image into the trained feature extraction network to output a first result, and inputting the first result into the trained scene classification network to output a second result; Inputting the first result into the plurality of trained image classification networks respectively to obtain third results corresponding to a plurality of preset scenes; inputting the second result and the third result corresponding to each preset scene into the classification output network to obtain an output result; Based on the output result and the image category label, determine the loss amount; based on the loss amount, update the weight parameters of the trained feature extraction network, the trained scene classification network, multiple trained image classification networks and the classification output network; continue to perform the step of determining training samples based on the sample set until each network converges to obtain an image classification model.

9. An electronic device, It is characterized in that The electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image classification method described in any one of claims 1 to 3 or the training method of the image classification model described in any one of claims 4 to 6.

10. A machine-readable storage medium, It is characterized in that The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the image classification method described in any one of claims 1 to 3 or the training method of the image classification model described in any one of claims 4 to 6.

Citation Information

Patent Citations

  • An image classification method and device

    CN106228193A

  • Image recognition method and device

    CN107690660A

  • Image classification method and device

    CN110163301A

  • Image classification method, device and system based on scene and storage medium

    CN110490225A

  • Video classification method and device and electronic equipment

    CN110766096A