Image classification method, device, equipment, readable storage medium and product

Abstract

The embodiment of the present application discloses an image classification method, device, equipment, readable storage medium and product, wherein the method includes: obtaining an initial training set, using the initial training set to train an initial prediction model to obtain an intermediate prediction model; using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain a first image prediction result; the first image prediction result is used to indicate the sample category of each sample image; based on the first image prediction result, a first sample image and a second sample image are determined from the initial training set; image annotation is performed on the first sample image to obtain a first sample annotated image, and the intermediate prediction model is iteratively trained using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the image category of the image. The embodiment of the present application can improve the model learning effect and the accuracy of image classification.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method, device, equipment, readable storage medium and product. Background Art

[0002] Image classification tasks are widely used in various scenarios. In order to improve the efficiency of image classification, classification models are generally trained and the trained classification models are sampled for image classification. However, in the process of training classification models, the selection of training samples is a major problem. The selected training samples are prone to problems such as uneven distribution of training samples, insufficient data volume of training samples, and low quality of training samples. Therefore, in the process of training classification models, the classification model learning effect is poor, which in turn reduces the accuracy of image classification. Summary of the invention

[0003] The embodiments of the present application provide an image classification method, apparatus, device, readable storage medium and product, which can improve the model learning effect and thus improve the accuracy of image classification.

[0004] In a first aspect, the present application provides an image classification method, comprising:

[0005] Obtaining an initial training set, using the initial training set to train an initial prediction model, and obtaining an intermediate prediction model;

[0006] Using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain a first image prediction result of each sample image; the first image prediction result is used to indicate a sample category of each sample image;

[0007] Determining a first sample image and a second sample image from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories;

[0008] The first sample image is annotated to obtain a first sample annotated image, and the intermediate prediction model is iteratively trained using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the image category of the image.

[0009] In a second aspect, the present application provides an image classification device, the device comprising:

[0010] An initial training unit, used to obtain an initial training set, and use the initial training set to train an initial prediction model to obtain an intermediate prediction model;

[0011] An image prediction unit, used to perform image prediction processing on each sample image in the initial training set using the intermediate prediction model to obtain a first image prediction result for each sample image; the first image prediction result is used to indicate a sample category of each sample image;

[0012] An image determination unit, configured to determine a first sample image and a second sample image from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories;

[0013] The image annotation unit is used to annotate the first sample image to obtain a first sample annotated image, and iteratively train the intermediate prediction model using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the image category of the image.

[0014] In a third aspect, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0015] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store computer program codes, and the processor is used to call the computer program codes so that a computer device including the processor executes the image classification method.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the above-mentioned image classification method.

[0017] In a fifth aspect, the present application provides a computer program product or a computer program, characterized in that the computer program product or the computer program includes computer instructions, which, when executed by a processor, implement the image classification method provided in the various optional methods in the first aspect of the present application.

[0018] In the embodiment of the present application, the initial prediction model is trained by acquiring the initial training set to obtain the intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result of each sample image, thereby determining the first sample image and the second sample image from the initial training set based on the first image prediction result, and annotating the first sample image to obtain the first sample image, and iteratively training the intermediate prediction model using the first sample annotated image and the second sample annotated image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. By training the target prediction model in an iterative training manner, the sample images in the initial training set are divided into the first sample image and the second sample image, and the model is trained after the image annotation of the first sample image, the number of first sample images in the initial training set can be gradually reduced. In the process of training the target prediction model, by annotating the first sample image and combining the second sample image training model, semi-supervised training of the model can be achieved, the model learning effect can be improved, and then the model stability can be improved. By using the trained target prediction model to predict the image category of the image, the accuracy of category prediction can be improved, thereby improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a network architecture diagram of an image classification system provided in an embodiment of the present application;

[0021] Figure 2 It is a flowchart of an image classification method provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a scene for dividing image categories provided in an embodiment of the present application;

[0023] Figure 4 It is a flowchart of another image classification method provided in an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the composition structure of an image classification device provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The technical solution of the present application can be applied to image prediction processing to determine the classification result of the image, so as to push the image based on the image classification result. For example, it can be applied to the content sharing platform, by accurately classifying the image, so that high-quality content can be distributed according to the classification result of the image, and the high-quality content distribution can improve the user experience, and has a great contribution to the scenarios such as user account retention or content fragmentation of the platform. Among them, the user account may refer to the account used by the user who logs in to the content sharing platform. The technical solution of the embodiment of the present application can also be applied to any scenario where image classification is required, for example, it may include but is not limited to image classification systems, image search systems, image recognition systems, etc., and the embodiment of the present application does not limit this. The collection, use and processing of data such as sample images, images to be processed, and pushed images involved in the embodiment of the present application require the consent of the corresponding personal information subject, and need to comply with the provisions of relevant laws and regulations.

[0028] Please refer to Figure 1 , Figure 1 is a network architecture diagram of an image classification system provided in an embodiment of the present application, such as Figure 1 As shown, the computer device can exchange data with the terminal device, and the number of the terminal devices can be one or at least two. For example, when the number of the terminal devices is multiple, the terminal device may include Figure 1The terminal device 101a, the terminal device 101b and the terminal device 101c in the computer device 102 are as follows. Taking the terminal device 101a as an example, the computer device 102 can obtain the initial training set to train and obtain the intermediate prediction model, and use the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result. Further, the computer device 102 can determine the first sample image and the second sample image from the initial training set based on the first image prediction result, and perform image annotation on the first sample image to obtain the first sample annotated image, and iteratively train the intermediate prediction model using the first sample annotated image and the second sample image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. Optionally, the computer device 102 can also obtain the image to be processed from the terminal device 101a, and predict the image category of the image to be processed based on the target prediction model, so that the image category of the image to be processed can be sent to the terminal device 101a. Alternatively, the computer device 102 can also determine the image to be pushed that needs to be pushed based on the image category of the image to be processed, and send the image to be pushed to the terminal device 101a.

[0029] The computer device mentioned in the embodiments of the present application may refer to a server or a terminal device, or may also be a system composed of a server and a terminal device, which is not limited in the embodiments of the present application. The terminal device may be an electronic device, including but not limited to a mobile phone, a tablet computer, a laptop computer, a vehicle-mounted device, etc. The server mentioned above may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. When the server is an independent physical server, the server can perform image classification independently. When the server is multiple physical servers, multiple physical servers can collaborate to perform image classification.

[0030] For further information, see Figure 2 , Figure 2 is a flow chart of an image classification method provided in an embodiment of the present application; Figure 2 As shown, the image classification method can be applied to a computer device, and the image classification method includes but is not limited to the following steps:

[0031] S101, obtaining an initial training set, using the initial training set to train an initial prediction model, and obtaining an intermediate prediction model.

[0032] In some content sharing platforms, high-quality content distribution plays an important role in improving user experience. The premise of high-quality content distribution is accurate image classification. Only by classifying images and determining the classification results of images can we select appropriate content according to the classification results of images and distribute high-quality content. Therefore, how to improve the accuracy of image classification is very important.

[0033] In view of this, an image classification method is provided in an embodiment of the present application, which trains an initial prediction model, uses the trained initial prediction model to perform image prediction processing on each sample image in the initial training set, and obtains an image prediction result, thereby selecting the first sample image and the second sample image in the initial training set according to the image prediction result. The first sample image is then annotated, and the initial prediction model is iteratively trained using the first sample image and the second sample image after image annotating. During the iterative training process, the target prediction model is continuously learned, so the model learning effect can be improved, and then a better classification effect can be achieved in subsequent image classification tasks.

[0034] In an embodiment of the present application, the initial training set may include multiple sample images, and the sample images may include any type of images, such as but not limited to images captured by a camera device, images drawn by an image drawing device, images captured by a drawn image, images edited by an image, images composed of one or more images, or images composed of images and text, or images composed of text, etc., which are not limited in the embodiment of the present application. In an optional implementation, the sample images may be images obtained from a content sharing platform, for example, each registered account on the content sharing platform can upload any image, and all accounts on the content sharing platform can view the uploaded images and perform operations on the images such as liking, commenting, sharing, and collecting.

[0035] Among them, using the initial training set to train the initial prediction model to obtain the intermediate prediction model may refer to: using all or part of the sample images in the initial training set to train the initial prediction model so that the initial prediction model has the ability to classify images, and when the initial prediction model converges, the initial prediction model at this time is determined as the intermediate prediction model. Model convergence may, for example, refer to the loss function of the initial prediction model being less than a loss threshold. When the loss function of the initial prediction model is greater than or equal to the loss threshold, the initial prediction model does not converge, and it is necessary to continue to adjust the model parameters in the initial prediction model to reduce the loss function of the initial prediction model so that the loss function of the initial prediction model is less than the loss threshold.

[0036] In a possible implementation, when the initial training set is obtained, each sample image in the initial training set can be annotated to obtain a sample image label for each sample image. The sample image label can be used to indicate the true image category of the sample image. In the process of training the initial prediction model, by inputting the sample image, the initial prediction model can obtain the sample image category of the sample image by performing image prediction processing on the sample image. The sample image category can refer to the model prediction category obtained by the sample image through the initial prediction model. The loss function of the initial prediction model can be determined by the difference between the sample image category and the sample image label, so that it can be determined whether the initial prediction model converges by comparing the size relationship between the loss function and the loss threshold. When annotating the sample image, a double-blind annotation method can be used to annotate the image, thereby improving the accuracy of the image annotation, making the sample image label of the annotated sample image more accurate, and thus improving the accuracy of subsequent model training.

[0037] Specifically, double-blind labeling may refer to labeling the same sample image by using multiple different image labeling devices. If the labeling results obtained by multiple image labeling devices for labeling the same sample image are the same, the labeling result is determined as the sample image label of the sample image. If the labeling results obtained by multiple image labeling devices for labeling the same sample image are different, a third image labeling device may be used to label the sample image to obtain a labeling result, and the labeling result with the largest number of identical labeling results among the multiple labeling results may be used as the final sample image label of the sample image. By using the double-blind labeling image labeling method to label the same sample image, different image labeling devices have different understandings of the same sample image, and therefore may focus on different image details when labeling the image, so the labeled sample image labels may be different. By comparing the labeling results of the same sample image by multiple image labeling devices and selecting the final sample image label, the accuracy of image labeling can be improved.

[0038] In one embodiment, when obtaining the initial training set, the sample images may be expanded to obtain the initial training set, increase the number of training samples, and thus improve the model learning effect. For example, an initial sample image may be obtained; at least one data enhancement algorithm may be used to perform image enhancement processing on the initial sample image to obtain at least two sample images; and an initial training set may be constructed based on the at least two sample images.

[0039] Among them, at least one data enhancement algorithm may include, for example, but is not limited to, random flip, random resize crop, random erasing, affine transformation, sharpness, random gamma, random brightness contrast, random brightness, random partial area cutout, normalization, etc. By using a variety of data enhancement algorithms to perform image enhancement processing on the initial sample images, the number of sample images can be increased, the sample images can be expanded, and the total amount of sample images in the initial training set can be expanded, thereby enhancing the model learning effect.

[0040] In a possible implementation, a data enhancement algorithm for a sample image can be automatically selected, for example, a data enhancement algorithm can be selected in combination with image attributes corresponding to the sample image for data enhancement processing. Specifically, the image attributes of the initial sample image can be obtained, and a data enhancement algorithm matching the image attributes of the initial sample image can be selected from at least one data enhancement algorithm; the initial sample image can be enhanced using the data enhancement algorithm matching the image attributes of the initial sample image to obtain at least two sample images; and an initial training set can be constructed based on the at least two sample images.

[0041] The image attributes may include but are not limited to the size of the image, the resolution of the image, the number of objects in the image, and the like. The objects in the image may include but are not limited to the total number of objects in one or more combinations of people, animals, plants, or other objects in the image. For example, if the sample image includes a person, a tree, and a cat, the number of objects in the sample image is 3. For example, when the image attribute of the initial sample image is the size of the initial sample image, if the size of the initial sample image is less than the size threshold, the data enhancement algorithm that matches the size of the initial sample image may include but is not limited to random flipping, random noise, random contrast adjustment, normalization, random brightening, and the like. If the size of the initial sample image is greater than or equal to the size threshold, the data enhancement algorithm that matches the size of the initial sample image may include but is not limited to random scaling and cropping, random partial area shearing, random erasing, affine transformation, sharpening, random quantization, random partial area shearing, and the like.

[0042] Alternatively, for example, when the image attribute of the initial sample image is the resolution of the initial sample image, if the resolution of the initial sample image is less than the resolution threshold, the data enhancement algorithm that matches the resolution of the initial sample image may include but is not limited to random flipping, random noise, random contrast adjustment, normalization, random brightening, etc. If the resolution of the initial sample image is greater than or equal to the resolution threshold, the data enhancement algorithm that matches the resolution of the initial sample image may include but is not limited to random scaling and cropping, random partial area clipping, random erasing, affine transformation, sharpening, random quantization, random partial area clipping, etc.

[0043] Alternatively, for example, when the image attribute of the initial sample image is the number of objects in the initial sample image, if the number of objects in the initial sample image is less than the number threshold, the data enhancement algorithm that matches the number of objects in the initial sample image may include but is not limited to random flipping, random noise, random contrast adjustment, etc. If the number of objects in the initial sample image is greater than or equal to the number threshold, the data enhancement algorithm that matches the number of objects in the initial sample image may include but is not limited to random flipping, random scaling and cropping, random erasing, affine transformation, sharpening, random noise, random contrast adjustment, random brightening, random partial area shearing, normalization, etc. Optionally, the corresponding data enhancement algorithm may also be selected in combination with the size of the image, the resolution of the image, the number of objects in the image, or any of more image attributes, which is not limited in the embodiments of the present application.

[0044] In an embodiment of the present application, for images whose size is smaller than a size threshold, that is, the image size is small, some data enhancement algorithms that will not make the image size smaller can be selected to process the image, thereby improving the accuracy of image enhancement. For images whose resolution is lower than a resolution threshold, data enhancement algorithms that will not reduce the image resolution can be selected to process the image, thereby improving the accuracy of image enhancement. For images whose number of objects in the sample image is less than a number threshold, data enhancement methods such as random partial area shearing are not selected to avoid the sample image after shearing not including the object or the object being incomplete, which is not convenient for subsequent model learning, thereby improving the accuracy of image enhancement. By selecting the corresponding data enhancement algorithm in combination with the image attributes of the initial sample image, targeted image enhancement processing can be achieved to improve the accuracy of image processing.

[0045] Optionally, the initial prediction model may, for example, select an efficient network model (EfficientNet series model) as a model backbone, or the initial prediction model may also include, but is not limited to, a convolutional neural network model (CNN), a recurrent neural network model (RNN), a deep neural network model (DNN), and the like.

[0046] Optionally, when selecting sample images from a content sharing platform, sample images under each image category can be selected in combination with the image category information of the sample images to enhance the diversity of sample data. The image category information can be selected when uploading the image to the content sharing platform. The image category information can be used to indicate the category of the sample image. For example, the image category information can include multi-level classifications. The multi-level category information can include, for example, three-level classifications, such as a first-level classification of food, a second-level classification of food display, a third-level classification of food display, and so on. By obtaining the image category information of the sample image when acquiring the sample image, it is possible to determine the image category to which the sample image belongs, so that when constructing the initial training set, sample images corresponding to a variety of image category information can be obtained respectively, the categories of training samples can be increased, and the distribution of training samples can be balanced to facilitate subsequent model learning.

[0047] The technical solution of the present application can be applied to any image classification scenario, and can also predict any image category. The specific image categories that need to be predicted can be determined according to the needs. When training the target prediction model, the corresponding sample images can be selected according to the specific needs, and the corresponding image categories can be predicted. In other words, the technical solution in the embodiment of the present application is universal, and prediction can be made for any image category. The embodiment of the present application is mainly described by the binary classification of the scene category and the layout category of the predicted image, but more categories can also be classified, which will not be explained in detail here.

[0048] S102, using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain a first image prediction result for each sample image.

[0049] In an embodiment of the present application, an intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the probability that each sample image is a plurality of image categories. The plurality of image categories may be a plurality of pre-set image categories. The image category corresponding to the maximum probability is selected from the probabilities of the plurality of image categories as the sample image category of the sample image. The first image prediction result of each sample image may include the sample image category of each sample image and the probability corresponding to the sample image category of each sample image. The first image prediction result of each sample image is used to indicate the sample category of each sample image. The sample category may include a first sample category and a second sample category. The sample images of the first sample category need to be processed, such as image annotation, for training the intermediate prediction model, and the sample images of the second sample category may be used to directly train the intermediate prediction model.

[0050] It can be understood that the input of the intermediate prediction model is a sample image, and the intermediate prediction model can perform image prediction processing on the sample image to obtain the probability that the sample image is a plurality of image categories. The output of the intermediate prediction model can be a plurality of image categories and the probabilities of a plurality of image categories, or the output of the intermediate prediction model can be the maximum probability among the probabilities of a plurality of image categories and the image category corresponding to the maximum probability. The embodiments of the present application do not limit this.

[0051] S103: Determine a first sample image and a second sample image from an initial training set based on the first image prediction result.

[0052] In the embodiment of the present application, the sample image category of each sample image in the initial training set can be predicted by the intermediate prediction model. Since each sample image in the initial training set is annotated to obtain the sample image label of each sample image when the initial training set is obtained, it is possible to determine whether the prediction result of the intermediate prediction model for each sample image is accurate based on the sample image label and the sample image category of each sample image. The first sample image and the second sample image can be further determined from the initial training set in combination with the first prediction result of each sample image, and the first sample image and the second sample image belong to different sample categories.

[0053] Among them, the first sample image may refer to an image that needs to be processed, such as image annotation, for training the intermediate prediction model, and the second sample image may refer to an image used to directly train the intermediate prediction model. For example, the first sample image may refer to an image with poor prediction effect during the image prediction process performed by the intermediate prediction model, for example, the probabilities of predicting that the sample image is of multiple image categories are all equal, and it is difficult to determine the final sample image category of the sample image. The second sample image may refer to an image with good prediction effect during the image prediction process performed by the intermediate prediction model, for example, the probability of predicting that the sample image is of multiple image categories has a probability of one image category that is significantly greater than the probabilities of other image categories. The first sample image may refer to a difficult sample image (data_hard), and the second sample image may refer to an easy sample image (data_easy).

[0054] In one implementation, the first sample image and the second sample image can be determined in combination with the probability that each sample image belongs to multiple image categories, and the first image prediction result includes the maximum probability of each sample image belonging to multiple image categories. Specifically, if the maximum probability of each sample image belonging to multiple image categories is less than a first threshold, it is determined that each sample image belongs to the first sample image; if the maximum probability of each sample image belonging to multiple image categories is greater than a second threshold, it is determined that each sample image belongs to the second sample image; the first threshold is less than the second threshold.

[0055] In the embodiment of the present application, since there are sample images of multiple image categories, there may be confusion between the sample images of multiple image categories, that is, the probability that a certain sample image belongs to two different image categories is relatively large. In this case, it is difficult to predict which image category the sample image belongs to. Therefore, when training the intermediate prediction model, by obtaining the maximum probability of each sample image belonging to multiple image categories, and comparing the maximum probability with two different thresholds, if the maximum probability is less than the first threshold, it means that the probability of each sample image belonging to multiple image categories is similar. That is, the intermediate prediction model is difficult to predict which image category each sample image belongs to. For example, it may mean that the sample image label of each sample image may be wrong when obtaining the initial training set, then each sample image can be determined as the first sample image, and these first sample images can be annotated again later, and the annotated images can be used to train the intermediate prediction model. If the maximum probability is greater than the second threshold, it means that the probability of each sample image belonging to a certain image category is significantly greater than the probability of belonging to other image categories, so each sample image can be determined as the second sample image, and these second sample images can be used to continue training the intermediate prediction model later.

[0056] Optionally, since the sample image category of each sample image in the initial training set can be predicted by the intermediate prediction model, whether the prediction result of the intermediate prediction model for each sample image is accurate can be determined based on the sample image label and sample image category of each sample image. If the prediction result for a certain sample image is inaccurate, that is, the sample image label and sample image category of a certain sample image are not the same, then the sample image can be determined as the first sample image. Optionally, if the maximum probability of each sample image belonging to multiple image categories is greater than or equal to a first threshold value, and less than or equal to a second threshold value, then each sample image can be determined to belong to the first sample image or the second sample image, or subsequent processing may not be performed on this type of sample image, for example, this type of sample image is not image labeled and is not used for training the intermediate prediction model, and the embodiments of the present application are not limited to this.

[0057] In a possible implementation, the image category may include a scene category and a layout category, and the sample category of the sample image may be determined in combination with the probabilities of the scene category and the layout category. The first image prediction result includes the maximum probability of each sample image being a plurality of image categories, the first image prediction result includes the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category, the first probability is the maximum probability of each sample image being a plurality of scene categories, and the second probability is the maximum probability of each sample image being a plurality of layout categories. Specifically, if the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both less than the first threshold, it is determined that each sample image belongs to the first sample image. If the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both greater than the second threshold, it is determined that each sample image belongs to the second sample image.

[0058] The scene category may include multiple levels of classification, for example, the first-level classification may include real-shot category, non-real-shot category, semi-real-shot category, etc. The first-level classification may also include second-level classification, for example, the second-level classification under the real-shot category may include but is not limited to selfies, portraits, objects, natural scenery, animals, outdoor landscapes, indoor layouts, etc. The second-level classification under the non-real-shot category may also include but is not limited to animation, screenshots, posters, movies, etc. The layout category may also include single-image category, image and text category, multi-image category, text category, etc. Each sample image may include at least one of the scene category or layout category.

[0059] Among them, real-shot images, for example, may refer to images obtained by photographing real objects, and the real objects may include but are not limited to people, objects, natural scenery, animals, and the like. Non-real-shot images, for example, may refer to images drawn by an image drawing device, or images obtained by intercepting images displayed on a terminal device, or film and television images. Semi-real-shot images, for example, may refer to images obtained by combining real-shot images and non-real-shot images. Single-image images, for example, may refer to images that include only one image. Graphics and text images, for example, may refer to images that include images and text, such as images obtained by editing text on an image. Multi-image images, for example, may refer to images obtained by combining multiple images by splicing, cutting, rotating, etc. Text images, for example, may refer to images that include only text and no graphics.

[0060] like Figure 3 As shown, Figure 3 It is a scene schematic diagram for dividing image categories provided by an embodiment of the present application, wherein the image category may include a scene category and a layout category, and the scene category and the layout category may be two leaf node categories, and the categories under the two leaf nodes of the scene category and the layout category may be in a coexistence relationship, that is, the layout category and the scene category are in a coexistence relationship, but the leaf node categories under the scene category are mutually exclusive, that is, the real shot, non-real shot, and semi-real shot are mutually exclusive, and the single image, picture and text, multiple images, and text under the layout category are mutually exclusive. Therefore, two classifiers can be set in the initial prediction model, namely, a scene classifier for predicting the scene category and a layout classifier for predicting the layout category, respectively predicting the probability of the image being a scene category and a layout category. The scene classifier can predict the probability of the output image being 14 categories, namely, the probabilities of scene categories such as real shot, non-real shot, semi-real shot, selfie, portrait, object, natural scenery, animal, outdoor landscape, indoor layout, cartoon animation, screenshot, poster, film and television under the scene category. The layout classifier can be used to predict the probability of the output image being classified into four categories, namely, the probabilities of being a single image, image and text, multiple images, and text.

[0061] In the embodiment of the present application, since the image category includes the scene category and the layout category, when predicting the image category of each sample image, the probability of each sample image being a plurality of scene categories and a plurality of layout categories can be predicted respectively, thereby selecting the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category. The first sample image and the second sample image can be determined by the relationship between the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category and the first threshold and the second threshold. Since the sample category of the sample image is determined by combining the scene category and the layout category, the accuracy of the sample classification can be achieved.

[0062] Optionally, when the image category includes a scene category and a layout category, the sample image category of each sample image may include a sample scene category of each sample image and a sample layout category of each sample image, and the sample image label of each sample image may include a sample scene label of each sample image and a sample layout label of each sample image. Since the sample scene category and sample layout category of each sample image in the initial training set can be predicted by the intermediate prediction model, based on the sample scene label in the sample image label of each sample image and the sample scene category in the sample image category, as well as the sample layout label in the sample image label of each sample image and the sample layout category in the sample image category, it can be determined whether the prediction result of the intermediate prediction model for each sample image is accurate. If the prediction result for a certain sample image is inaccurate, that is, the sample scene label and the sample scene category of a certain sample image are different, or the sample layout label and the sample layout category of the sample image are different, then the sample image can be determined as the first sample image. Optionally, if the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both greater than or equal to a first threshold and less than or equal to a second threshold, it can be determined that each sample image belongs to the first sample image or the second sample image, or subsequent processing may not be performed on this type of sample image, for example, image labeling or training of an intermediate prediction model is not performed on this type of sample image, and the embodiments of the present application are not limited to this.

[0063] It is understandable that the image categories may also include more categories besides the scene categories and layout categories. Then, the probabilities of more image categories may be combined to determine which sample type each sample image in the initial training set belongs to. This embodiment of the present application will not elaborate on this.

[0064] In a possible implementation, the initial training set can be divided into a training set and a test set, and the number of images in the training set and the test set can be set according to demand. Then, each sample image in the training set in the initial training set can be used to train the initial prediction model to obtain an intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the test set in the initial training set to obtain the first image prediction result of each sample image. By dividing the initial training set into a training set and a test set, using the sample images in the training set to train the initial prediction model to obtain an intermediate prediction model, and using the intermediate prediction model to perform image prediction processing on the sample images in the test set, the probability of data interference can be reduced and the model learning effect can be improved.

[0065] S104, annotating the first sample image to obtain a first sample annotated image, and iteratively training an intermediate prediction model using the first sample annotated image and the second sample image to obtain a target prediction model.

[0066] In an embodiment of the present application, since the first sample image is a difficult sample image, the first sample annotated image is obtained by annotating the first sample image, and the first sample annotated image and the second sample image can be used to iteratively train the intermediate prediction model to obtain a target prediction model. Among them, the target prediction model can be used to predict the image category of the image. Image annotation can refer to labeling the sample image to determine the sample image label of the sample image. When the sample image is subsequently predicted by the intermediate prediction model, the sample image category of the sample image can be obtained, so that based on the sample image category and the sample image label of the sample image, it can be determined whether the image classification result of the intermediate prediction model for the sample image is accurate, and then it can be determined whether the model parameters in the intermediate prediction model are adjusted.

[0067] In one embodiment, a double-blind labeling method can be used to annotate the first sample image to obtain a first sample annotated image, thereby improving the accuracy of image labeling. Specifically, at least two image labeling devices can be used to annotate the first sample image to obtain at least two image labeling results; one image labeling device corresponds to one image labeling result; if at least two image labeling results are the same, any one of the at least two image labeling results is used as the first sample annotated image. If at least two image labeling results are not the same, the first sample image is annotated based on a target image labeling device to obtain a target image labeling result, and the first sample annotated image is determined based on the target image labeling result and at least two image labeling results; the target image labeling device is a device other than the at least two image labeling devices.

[0068] Among them, double-blind labeling can refer to two or two image labeling devices labeling the same sample image, that is, two or two image labeling devices label the same sample image, and each image labeling device may focus on different image details for the same sample image. Therefore, by using multiple image labeling devices to label the same sample image, the accuracy of image labeling can be improved. Take two image labeling devices performing double-blind labeling as an example for explanation. The first image labeling device labels the sample image to obtain a first image labeling result such as a first label, and the second image labeling device labels the sample image to obtain a second image labeling result such as a second label. If the first label and the second label are the same, the first label can be determined as the sample image label of the sample image. If the first label and the second label are different, the target image labeling result such as a third label obtained by the target image labeling device for labeling the sample image can be obtained. If the target image labeling result is the same as the first image labeling result, the first image labeling result is determined as the sample image label of the sample image. If the target image labeling result and the second image labeling result are the same, the second image labeling result is determined as the sample image label of the sample image.

[0069] Optionally, the target image annotation result of the target image annotation device can be obtained by annotation by the target image annotation device, or can be obtained by input by a management personnel on the target image annotation device. It can be understood that the image annotation method is not limited in the embodiment of the present application, that is, the image annotation method can also be a method other than double-blind annotation, as long as the purpose of image annotation can be achieved. Optionally, the image annotation device used in the double-blind annotation to annotate the first sample image at this time can be different from the image annotation device used to annotate the sample image when obtaining the initial training set, thereby improving the accuracy of image annotation.

[0070] In a possible implementation, if double-blind labeling is performed by labeling a sample image through three or more image labeling devices, three or more image labeling results can be obtained, and the image labeling results with the same labeling results accounting for a larger number of the total image labeling results among the three or more image labeling results are determined as the sample image labels of the sample image. By using multiple image labeling devices to label the sample image, and determining the image labeling results with the same labeling results accounting for a larger number of the total image labeling results as the sample image labels of the sample image, the accuracy of image labeling can be improved.

[0071] In one embodiment, the intermediate prediction model is iteratively trained using the first sample annotated image and the second sample image to obtain a target prediction model. The iterative training process may be as follows:

[0072] The intermediate prediction model is used to perform image prediction processing on the last first sample annotated image and the last second sample image to obtain a second image prediction result, and the first sample image and the second sample image are determined from the last first sample annotated image and the last second sample image based on the second image prediction result, until the number of first sample images is less than the number threshold, and the iterative training is stopped to obtain the target prediction model. The second image prediction result is used to indicate the sample category of the last first sample annotated image and the sample category of the last second sample image.

[0073] In an embodiment of the present application, the first sample image (difficult sample image) can be determined from the initial training set through the iterative training process, and a new first sample image can be determined from the last first sample image through continuous iterative training. This process can continuously reduce the number of first sample images. After image annotation is performed on each determined first sample image, prediction is continued. The last first sample image is divided into a first sample image and a second sample image according to the prediction result, and image annotation and image prediction are performed on the newly divided first sample image in turn, which can gradually reduce the number of first sample images. When the number of first sample images is less than the quantity threshold, it means that the intermediate prediction model has learned more content and can be put into use for image classification. Therefore, the iterative training can be stopped, and the intermediate prediction model at this time is determined as the target prediction model, which can achieve efficient iteration of semi-supervised self-learning of the model.

[0074] In one embodiment, the image category may include at least one of a scene category and a layout category. The image may be predicted to determine at least one of the scene category and the layout category of the image, so as to push the image in a targeted manner based on the scene category and the layout category of the image, thereby improving the user experience. Specifically, the image to be processed may be obtained; the image to be processed may be processed using a target prediction model to perform image prediction processing on the image to be processed to obtain a third image prediction result, and at least one of the scene category and the layout category of the image to be processed may be obtained based on the third image prediction result; at least one image to be pushed for image push may be obtained from the recommendation system; at least one image to be pushed may be screened based on at least one of the scene category and the layout category of the image to be processed, and the screened image to be pushed may be pushed to the target terminal.

[0075] The image to be processed is any image that needs to be classified. The image to be processed can be obtained by uploading it through a target terminal, or by detecting a click operation on the current page. The target terminal can be, for example, Figure 1 One or more of the terminal devices 101a, 101b or 101c may also be other terminal devices, which is not limited in the embodiments of the present application. The third image prediction result may include the probability that the image to be processed is a plurality of scene categories and / or the probability of a plurality of layout categories. Obtaining at least one of the scene category and layout category of the image to be processed based on the third image prediction result may refer to: determining the scene category corresponding to the maximum probability among the probabilities that the image to be processed is a plurality of scene categories as the scene category of the image to be processed, and determining the layout category corresponding to the maximum probability among the probabilities that the image to be processed is a plurality of layout categories as the layout category of the image to be processed. That is to say, in the embodiments of the present application, image push may be performed based on any one of the scene category and layout category of the image to be processed or a combination of the two.

[0076] The image push method in the embodiment of the present application can be applied to two different scenarios, such as content fragmentation scenario and image search scenario. Different image push methods can be used for different scenarios, as follows:

[0077] In the content fragmentation scenario, at least one image to be pushed for image push can be obtained from the recommendation system; based on at least one of the scene category and layout category of the image to be processed, at least one image to be pushed is subjected to image screening processing, and images whose scene category and / or layout category in at least one image to be pushed are the same or similar to the scene category and / or layout category of the image to be processed are screened out, and the remaining images are determined as the screened images to be pushed, and the screened images to be pushed are pushed to the target terminal. By screening the images in the recommendation system that are the same or similar to the scene category and / or layout category of the image to be processed, it is possible to break up similar images in the recommendation system, avoid continuously pushing similar images to the target terminal, and reduce the user experience. By pushing more images with different image categories from the image to be processed to the target terminal, the recent preferences of the user account can be determined in combination with the operation of the target terminal, thereby facilitating targeted push and improving the user account retention rate.

[0078] In the scenario of image search for images, at least one image to be pushed for image push can be obtained from the recommendation system; at least one image to be pushed is subjected to image screening based on at least one of the scene category and layout category of the image to be processed, and images whose scene category and / or layout category in at least one image to be pushed are different from or dissimilar to the scene category and / or layout category of the image to be processed are screened out, and the remaining images are determined as screened images to be pushed, and the screened images to be pushed are pushed to the target terminal. By recommending images with image categories similar to those of the image to be processed, image search can be achieved, thereby improving user experience.

[0079] Optionally, the current scene can be determined as a content fragmentation scene or an image search scene in combination with the operation of the target terminal on the image to be processed. For example, when a search instruction for the image to be processed is obtained from the target terminal, it can be determined as an image search scene. For another example, when the target terminal performs a target operation such as an "not interested" operation on the image to be processed, or the target terminal stays on the image to be processed for less than a duration threshold, it can be determined as a content fragmentation scene. By judging the current scene, the corresponding image push method can be selected in combination with the corresponding scene to improve the user experience.

[0080] In the embodiment of the present application, an efficient network model is designed as the backbone network of the model for the problem of image scene classification. Two classification prediction heads are set when predicting scene categories and layout categories, which can achieve simultaneous prediction of scene categories and layout categories. In the sample image expansion stage, by using an incremental learning algorithm, that is, using a variety of data enhancement algorithms, the amount of data of the training samples can be increased, avoiding the problem of small training data scale, thereby enhancing the robustness of the model. Through a target prediction model, both the scene category (14 categories) of the output image and the layout category (4 categories) of the output image can be predicted. There is no need to use multiple models to predict the scene category and layout category respectively, which can save multiple model deployment resource problems. In addition, since the training data can be sampled based on the image category information when selecting the training sample, that is, the sample images under various image category information are selected, the quality of the training data can be improved. In addition, by combining the model with the difficult sample images whose maximum probability of image category is less than the first threshold for double-blind labeling, it is possible to obtain semi-supervised training data from easy to difficult, thereby efficiently solving the bottleneck problems of uneven distribution of training samples and the scale of training data, achieving rapid accumulation of effective training data, and spiral growth of data and models, thereby improving model training efficiency.

[0081] In the embodiment of the present application, the initial prediction model is trained by acquiring the initial training set to obtain the intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result of each sample image, thereby determining the first sample image and the second sample image from the initial training set based on the first image prediction result, and annotating the first sample image to obtain the first sample image, and iteratively training the intermediate prediction model using the first sample annotated image and the second sample annotated image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. By training the target prediction model in an iterative training manner, the sample images in the initial training set are divided into the first sample image and the second sample image, and the model is trained after the image annotation of the first sample image, the number of first sample images in the initial training set can be gradually reduced. In the process of training the target prediction model, by annotating the first sample image and combining the second sample image training model, semi-supervised training of the model can be achieved, the model learning effect can be improved, and then the model stability can be improved. By using the trained target prediction model to predict the image category of the image, the accuracy of category prediction can be improved, thereby improving the accuracy of image classification.

[0082] For further information, see Figure 4 , Figure 4 is a flow chart of another image classification method provided in an embodiment of the present application; Figure 4As shown, the image classification method can be applied to a computer device, and the image classification method includes but is not limited to the following steps:

[0083] S201, obtaining an initial sample image.

[0084] Here, the sample images include images on the content sharing platform, such as images uploaded by a registered account on the content sharing platform, or images in a video. The images in the video may include, for example, the cover image, the first image, or the key frame image of the video, or any image in the video, which is not limited in the embodiments of the present application.

[0085] S202, performing image enhancement processing on the initial sample image using a data enhancement algorithm to obtain at least two sample images, and constructing an initial training set based on the at least two sample images.

[0086] In the embodiment of the present application, the specific implementation of steps S201 to S202 can refer to the implementation of the above-mentioned step S101, which will not be repeated here.

[0087] S203, annotating each sample image in the initial training set to obtain a sample scene label and a sample layout label of each sample image.

[0088] In an embodiment of the present application, each sample image in the initial training set may be annotated to obtain a sample image label for each sample image. Taking the example that the predicted image category includes a scene category and a layout category, the sample image category may include a sample scene category and a sample layout category, and the sample image label may include a sample scene label and a sample layout label. Here, the specific image annotation method can refer to the image annotation method in the above step S104, which will not be repeated here.

[0089] S204, using the initial prediction model to perform image prediction processing on each sample image in the initial training set to obtain a sample scene category and a sample layout category of each sample image.

[0090] Here, a scene prediction head may be used to predict the sample scene category of each sample image, and a layout prediction head may be used to predict the sample layout category of each sample image. Optionally, for example, an initial prediction model may be first trained using an initial training set, so that the initial prediction model has a certain image prediction capability. For example, when the initial prediction model converges, the initial prediction model at this time is saved as an intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain a sample scene category and a sample layout category of each sample image.

[0091] S205, determining a first loss based on a sample scene label and a sample scene category of each sample image, determining a second loss based on a sample layout label and a sample layout category of each sample image, training an initial prediction model based on the first loss and the second loss, and obtaining a target prediction model.

[0092] In an embodiment of the present application, a first loss of the initial prediction model for a scene category can be determined based on a sample scene label and a sample scene category of each sample image, a second loss of the initial prediction model for a layout category can be determined based on a sample layout label and a sample layout category of each sample image, the initial prediction model is trained based on the first loss and the second loss to obtain an intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain a first image prediction result for each sample image; a first sample image and a second sample image are determined from the initial training set based on the first image prediction result; annotate the first sample image to obtain a first sample annotated image, and iteratively train the intermediate prediction model using the first sample annotated image and the second sample image to obtain a target prediction model.

[0093] Optionally, the first loss may be used to indicate the difference between the model prediction value (i.e., the sample scene category) and the sample true value (i.e., the sample scene label), and the second loss may be used to indicate the difference between the model prediction value (i.e., the sample layout category) and the sample true value (i.e., the sample layout label). The first loss may, for example, include but is not limited to a 14-classification cross entropy loss (CEloss), and the second loss may include but is not limited to a 4-classification cross entropy loss.

[0094] Optionally, during the model training phase, the sample images in the initial training set can be combined with the image category information as weak supervision information. When selecting sample images to construct the initial training set, the corresponding sample images can be sampled under specific image category information. Compared with random sampling under various image category information, the probability of predicting that the sample image is of the image category information is higher, thereby ensuring the balance of samples in the initial training set. Since categories are easily confused, each person has different understandings of the same sample image, and the initial training set (data_version0) can be double-blind labeled. By sending the double-blind labeled sample images to the initial prediction model for learning and training, and waiting for the initial prediction model to converge, an intermediate prediction model (model1) can be obtained. Since the sampling is performed on sample images under multiple image categories, batch prediction can be performed on sample images under different image categories to improve the model prediction efficiency. For the sample images in the high threshold part, that is, the maximum probability output by the two prediction heads is greater than the second threshold (that is, the binary classification score is greater than the second threshold, such as 0.7), the sample images in the high threshold part can be directly used as the images in the initial training set, and the second sample images, i.e., easy sample images (data_easy), are added. For the maximum probability output by the two prediction heads is less than the first threshold (that is, the binary classification score is less than the first threshold, such as 0.4), the sample images can be uniformly sampled according to the image category information, that is, a part of the sample images are selected under each image category for double-blind labeling to obtain new difficult sample images (data_hard). And the labeled difficult sample images and easy sample images are used to train the intermediate prediction model.

[0095] Furthermore, when training the intermediate prediction model, an iterative training method is adopted. The easy sample images obtained by model self-supervision and the difficult sample images after double-blind annotation can be used to train and update the intermediate prediction model each time. The intermediate prediction model can then generate new easy sample data and difficult sample data, forming a semi-supervised data and model spiral iterative optimization method, which can greatly reduce the high cost of confusing data labeling, quickly accumulate effective data, and realize efficient iteration of model semi-supervised self-learning.

[0096] In addition, when the accuracy of the target prediction model in the two classification tasks of scene category and layout category is greater than the accuracy threshold, that is, when the accuracy of the target prediction model meets the business needs, the prediction model can be put into use in a large number of downstream applications, thereby improving the image classification accuracy of downstream applications. The image annotation method in the embodiment of the present application can reduce the annotation cost and improve the model iteration efficiency.

[0097] In the embodiment of the present application, the specific implementation methods of training the initial prediction model, using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain the first sample image and the second sample image, and how to iteratively train based on the first sample image and the second sample image to obtain the target prediction model can refer to the implementation methods of the above-mentioned steps S102 to S104, which will not be repeated here.

[0098] In the embodiment of the present application, the initial prediction model is trained by acquiring the initial training set to obtain the intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result of each sample image, thereby determining the first sample image and the second sample image from the initial training set based on the first image prediction result, and annotating the first sample image to obtain the first sample image, and iteratively training the intermediate prediction model using the first sample annotated image and the second sample annotated image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. By training the target prediction model in an iterative training manner, the sample images in the initial training set are divided into the first sample image and the second sample image, and the model is trained after the image annotation of the first sample image, the number of first sample images in the initial training set can be gradually reduced. In the process of training the target prediction model, by annotating the first sample image and combining the second sample image training model, semi-supervised training of the model can be achieved, the model learning effect can be improved, and then the model stability can be improved. By using the trained target prediction model to predict the image category of the image, the accuracy of category prediction can be improved, thereby improving the accuracy of image classification.

[0099] The method of the embodiment of the present application is introduced above, and the device of the embodiment of the present application is introduced below.

[0100] See also Figure 5 , Figure 5 : is a schematic diagram of the composition structure of an image classification device provided in an embodiment of the present application. The above-mentioned image classification device can be deployed on a computer device; the image classification device can be used to execute the corresponding steps in the image classification method provided in an embodiment of the present application. The image classification device 50 includes:

[0101] An initial training unit 501 is used to obtain an initial training set, and use the initial training set to train an initial prediction model to obtain an intermediate prediction model;

[0102] An image prediction unit 502 is used to perform image prediction processing on each sample image in the initial training set using the intermediate prediction model to obtain a first image prediction result for each sample image; the first image prediction result is used to indicate a sample category of each sample image;

[0103] An image determining unit 503 is used to determine a first sample image and a second sample image from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories;

[0104] The image annotation unit 504 is used to annotate the first sample image to obtain a first sample annotated image, and iteratively train the intermediate prediction model using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the image category of the image.

[0105] Optionally, the first image prediction result includes the maximum probability of each sample image being a plurality of image categories; the image determination unit 503 is specifically configured to:

[0106] If the maximum probability among the probabilities that each sample image belongs to multiple image categories is less than a first threshold, then determining that each sample image belongs to the first sample image;

[0107] If the maximum probability among the probabilities that each sample image belongs to multiple image categories is greater than a second threshold, it is determined that each sample image belongs to a second sample image; and the first threshold is less than the second threshold.

[0108] Optionally, the first image prediction result includes the maximum probability of each sample image being a plurality of image categories, the image categories include scene categories and layout categories, the first image prediction result includes a first probability corresponding to each sample image being a scene category and a second probability corresponding to each sample image being a layout category, the first probability is the maximum probability of each sample image being a plurality of scene categories, and the second probability is the maximum probability of each sample image being a plurality of layout categories; the image determination unit 503 is specifically used to:

[0109] If the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both less than a first threshold, then determining that each sample image belongs to the first sample image;

[0110] If the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both greater than a second threshold, then each sample image is determined to belong to the second sample image; and the first threshold is less than the second threshold.

[0111] Optionally, the image annotation unit 504 is specifically configured to:

[0112] Using at least two image annotation devices to perform image annotation on the first sample image to obtain at least two image annotation results; one image annotation device corresponds to one image annotation result;

[0113] If the at least two image annotation results are the same, any one of the at least two image annotation results is used as the first sample annotated image;

[0114] If the at least two image annotation results are different, a target image annotation result is obtained based on image annotation of the first sample image, and the first sample annotated image is determined based on the target image annotation result and the at least two image annotation results; the target image annotation device is a device other than the at least two image annotation devices.

[0115] Optionally, the iterative training process is as follows:

[0116] Using the intermediate prediction model to perform image prediction processing on the last first sample annotated image and the last second sample image to obtain a second image prediction result, and determining the first sample image and the second sample image from the last first sample annotated image and the last second sample image based on the second image prediction result, until the number of first sample images is less than a number threshold, stopping iterative training to obtain a target prediction model;

[0117] The second image prediction result is used to indicate the sample category of the previous first sample annotated image and the sample category of the previous second sample image.

[0118] Optionally, the initial training unit 501 is specifically used for:

[0119] Get an initial sample image;

[0120] Performing image enhancement processing on the initial sample image using at least one data enhancement algorithm to obtain at least two sample images;

[0121] An initial training set is constructed based on the at least two sample images.

[0122] Optionally, the image category includes at least one of a scene category and a layout category; the image classification device 50 further includes: an image pushing unit 505, the image pushing unit 505 is used to:

[0123] Get the image to be processed;

[0124] Using the target prediction model to perform image prediction processing on the image to be processed to obtain a third image prediction result, and obtaining at least one of a scene category and a layout category of the image to be processed based on the third image prediction result;

[0125] Obtaining at least one image to be pushed for image pushing from the recommendation system;

[0126] Based on at least one of the scene category and the layout category of the image to be processed, the at least one image to be pushed is subjected to image screening processing, and the screened image to be pushed is pushed to the target terminal.

[0127] It should be noted that Figure 5 For the contents not mentioned in the corresponding embodiments, please refer to the description of the method embodiments, which will not be repeated here.

[0128] In the embodiment of the present application, the initial prediction model is trained by acquiring the initial training set to obtain the intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result of each sample image, thereby determining the first sample image and the second sample image from the initial training set based on the first image prediction result, and annotating the first sample image to obtain the first sample image, and iteratively training the intermediate prediction model using the first sample annotated image and the second sample annotated image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. By training the target prediction model in an iterative training manner, the sample images in the initial training set are divided into the first sample image and the second sample image, and the model is trained after the image annotation of the first sample image, the number of first sample images in the initial training set can be gradually reduced. In the process of training the target prediction model, by annotating the first sample image and combining the second sample image training model, semi-supervised training of the model can be achieved, the model learning effect can be improved, and then the model stability can be improved. By using the trained target prediction model to predict the image category of the image, the accuracy of category prediction can be improved, thereby improving the accuracy of image classification.

[0129] See also Figure 6 , Figure 6 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 6 As shown, the above-mentioned computer device 60 may include: a processor 601, a network interface 604 and a memory 605. In addition, the above-mentioned computer device 60 may also include: a user interface 603, and at least one communication bus 602. Among them, the communication bus 602 is used to realize the connection and communication between these components. Among them, the user interface 603 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 603 may also include a standard wired interface and a wireless interface. The network interface 604 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 605 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 605 may optionally also be at least one storage device located away from the aforementioned processor 601. As Figure 6 As shown, the memory 605 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0130] exist Figure 6 In the computer device 60 shown, the network interface 604 can provide a network communication function; the user interface 603 is mainly used to provide an input interface for the user; and the processor 601 can be used to call the device control application stored in the memory 605 to achieve:

[0131] Obtaining an initial training set, using the initial training set to train an initial prediction model, and obtaining an intermediate prediction model;

[0132] Using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set to obtain a first image prediction result of each sample image; the first image prediction result is used to indicate a sample category of each sample image;

[0133] Determining a first sample image and a second sample image from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories;

[0134] The first sample image is annotated to obtain a first sample annotated image, and the intermediate prediction model is iteratively trained using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the image category of the image.

[0135] It should be understood that the computer device 60 described in the embodiment of the present application can execute the above Figure 2 and Figure 4 The description of the above-mentioned image classification method in the corresponding embodiment can also be performed as described above. Figure 5 The description of the above-mentioned image classification device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0136] In the embodiment of the present application, the initial prediction model is trained by acquiring the initial training set to obtain the intermediate prediction model, and the intermediate prediction model is used to perform image prediction processing on each sample image in the initial training set to obtain the first image prediction result of each sample image, thereby determining the first sample image and the second sample image from the initial training set based on the first image prediction result, and annotating the first sample image to obtain the first sample image, and iteratively training the intermediate prediction model using the first sample annotated image and the second sample annotated image to obtain the target prediction model, so that the image category of the image can be predicted using the target prediction model. By training the target prediction model in an iterative training manner, the sample images in the initial training set are divided into the first sample image and the second sample image, and the model is trained after the image annotation of the first sample image, the number of first sample images in the initial training set can be gradually reduced. In the process of training the target prediction model, by annotating the first sample image and combining the second sample image training model, semi-supervised training of the model can be achieved, the model learning effect can be improved, and then the model stability can be improved. By using the trained target prediction model to predict the image category of the image, the accuracy of category prediction can be improved, thereby improving the accuracy of image classification.

[0137] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, which includes program instructions, which, when executed by a computer, causes the computer to execute the method of the aforementioned embodiment, and the computer may be part of the computer device mentioned above. For example, it is the above-mentioned processor 601. As an example, the program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or, executed on multiple computer devices distributed at multiple locations and interconnected by a communication network, and multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0138] Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0139] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image classification method, characterized in that: The method comprises: Acquire an initial training set, and use the initial training set to train an initial prediction model to obtain an intermediate prediction model; Using the intermediate prediction model to perform image prediction processing on each sample image in the initial training set, obtaining a first image prediction result of each sample image; the first image prediction result is used to indicate the sample category of each sample image, the first image prediction result includes the maximum probability of each sample image being a plurality of image categories, the plurality of image categories include a plurality of scene categories and a plurality of layout categories, the first image prediction result includes the maximum probability of each sample image being a plurality of scene categories and the maximum probability of the plurality of layout categories; A first sample image and a second sample image are determined from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories; the first sample image is an image whose maximum probability among the probabilities of the multiple scene categories and the maximum probability among the probabilities of the multiple layout categories are both less than a first threshold, and the second sample image is an image whose maximum probability among the probabilities of the multiple scene categories and the maximum probability among the probabilities of the multiple layout categories are both greater than a second threshold, and the first threshold is less than the second threshold; The first sample image is annotated to obtain a first sample annotated image, and the intermediate prediction model is iteratively trained using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the scene category and layout category of the image, and the scene category and layout category of the image are used to perform image screening processing on at least one image to be pushed before image pushing.

2. The method according to claim 1, characterized in that The determining a first sample image and a second sample image from the initial training set based on the first image prediction result comprises: If the maximum probability among the probabilities that each sample image belongs to multiple image categories is less than a first threshold, determining that each sample image belongs to the first sample image; If the maximum probability among the probabilities that each sample image belongs to multiple image categories is greater than a second threshold, it is determined that each sample image belongs to the second sample image.

3. The method according to claim 1, characterized in that The image categories include scene categories and layout categories, the first image prediction result includes a first probability corresponding to each sample image being a scene category and a second probability corresponding to each sample image being a layout category, the first probability is the maximum probability among the probabilities that each sample image is a plurality of scene categories, and the second probability is the maximum probability among the probabilities that each sample image is a plurality of layout categories; The determining a first sample image and a second sample image from the initial training set based on the first image prediction result comprises: If the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both less than a first threshold, then determining that each sample image belongs to the first sample image; If the first probability corresponding to each sample image being a scene category and the second probability corresponding to each sample image being a layout category are both greater than a second threshold, it is determined that each sample image belongs to the second sample image.

4. The method according to claim 1, characterized in that: The step of performing image annotation on the first sample image to obtain a first sample annotated image includes: Using at least two image annotation devices to perform image annotation on the first sample image to obtain at least two image annotation results; one image annotation device corresponds to one image annotation result; If the at least two image annotation results are the same, taking any one of the at least two image annotation results as the first sample annotated image; If the at least two image annotation results are different, image annotation is performed on the first sample image based on a target image annotation device to obtain a target image annotation result, and the first sample annotated image is determined based on the target image annotation result and the at least two image annotation results; the target image annotation device is a device other than the at least two image annotation devices.

5. The method according to claim 1, characterized in that The iterative training process is as follows: Using the intermediate prediction model to perform image prediction processing on the last first sample annotated image and the last second sample image to obtain a second image prediction result, and determining the first sample image and the second sample image from the last first sample annotated image and the last second sample image based on the second image prediction result, until the number of first sample images is less than a number threshold, stopping iterative training to obtain a target prediction model; The second image prediction result is used to indicate the sample category of the previous first sample annotated image and the sample category of the previous second sample image.

6. The method according to claim 1, characterized in that The obtaining of the initial training set comprises: Get an initial sample image; Performing image enhancement processing on the initial sample image using at least one data enhancement algorithm to obtain at least two sample images; An initial training set is constructed based on the at least two sample images.

7. The method according to any one of claims 1 to 6, characterized in that: The image category includes at least one of a scene category and a layout category; the method further includes: Get the image to be processed; Using the target prediction model to perform image prediction processing on the image to be processed to obtain a third image prediction result, and obtaining at least one of a scene category and a layout category of the image to be processed based on the third image prediction result; Obtaining at least one image to be pushed for image pushing from the recommendation system; The at least one image to be pushed is subjected to image screening processing based on at least one of the scene category and the layout category of the image to be processed, and the screened image to be pushed is pushed to the target terminal.

8. An image classification device, characterized in that: The device comprises: An initial training unit, used to obtain an initial training set, and use the initial training set to train an initial prediction model to obtain an intermediate prediction model; an image prediction unit, configured to perform image prediction processing on each sample image in the initial training set using the intermediate prediction model to obtain a first image prediction result for each sample image; the first image prediction result is used to indicate a sample category of each sample image, the first image prediction result includes a maximum probability of each sample image being a plurality of image categories, the plurality of image categories include a plurality of scene categories and a plurality of layout categories, the first image prediction result includes a maximum probability of each sample image being a plurality of scene categories and a maximum probability of each sample image being a plurality of layout categories; An image determination unit is used to determine a first sample image and a second sample image from the initial training set based on the first image prediction result; the first sample image and the second sample image belong to different sample categories; the first sample image is an image whose maximum probability among the probabilities of the multiple scene categories and the maximum probability among the probabilities of the multiple layout categories are both less than a first threshold, and the second sample image is an image whose maximum probability among the probabilities of the multiple scene categories and the maximum probability among the probabilities of the multiple layout categories are both greater than a second threshold, and the first threshold is less than the second threshold; An image annotation unit is used to annotate the first sample image to obtain a first sample annotated image, and iteratively train the intermediate prediction model using the first sample annotated image and the second sample image to obtain a target prediction model; the target prediction model is used to predict the scene category and layout category of the image, and the scene category and layout category of the image are used to perform image screening processing on at least one image to be pushed before image push.

9. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images