Image processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides an image processing method, device, equipment and storage medium. The method of the embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the face score of the material image according to the face area in the material image through a face scoring model, and the face scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the face score of the material image, so as to realize automatic annotation of the face score of the material image, reduce a lot of labor costs, and greatly improve the annotation efficiency of the face score of the material image.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the development of Internet technology, online shopping has greatly facilitated people's lives, and e-commerce platforms have emerged one after another. For major e-commerce online stores, there are many application scenarios for images. For example, when users comment on purchased products, they can upload pictures of the products they use, and e-commerce platforms can show users pictures of models and products.

[0003] In order to provide a better display effect for products, e-commerce platforms hope to give priority to showing users pictures of people with good looks and products. E-commerce operators manually mark the pictures of products and people to be displayed in the material library according to the looks of the people in the pictures, and display the pictures with higher looks in the material library on the e-commerce platform.

[0004] However, as the business volume of e-commerce platforms increases, the number of images in the material library will grow linearly or even exponentially, and the demand for image annotation will also grow linearly or even exponentially. The labor cost is too high, which is a waste of human resources and very inefficient. Summary of the invention

[0005] The embodiments of the present invention provide an image processing method, apparatus, device and storage medium to solve the problems in the prior art that manual appearance score labeling is highly subjective, the labeled appearance score is inaccurate, and as the business volume of the e-commerce platform increases, the number of pictures in the material library will grow linearly or even exponentially, the demand for picture labeling will also grow linearly or even exponentially, the labor cost is too high, and human resources are wasted.

[0006] One aspect of an embodiment of the present invention is to provide an image processing method, comprising:

[0007] Obtaining the material image to be processed;

[0008] Detecting whether the material image includes a face area;

[0009] If the material image includes a face region, determining the face region in the material image;

[0010] Calculating the appearance score of the material image by using an appearance scoring model according to the face area in the material image, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set;

[0011] The material image is annotated according to the appearance score of the material image.

[0012] Another aspect of an embodiment of the present invention is to provide an image processing device, comprising:

[0013] A preprocessing module, used for acquiring material images to be processed;

[0014] A face detection module, used to detect whether the material image includes a face area;

[0015] The face detection module is further used to determine the face area in the material image if the material image includes the face area;

[0016] An appearance scoring module, used to calculate the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set;

[0017] The annotation processing module is used to perform annotation processing on the material image according to the appearance score of the material image.

[0018] Another aspect of an embodiment of the present invention is to provide an image processing device, including:

[0019] a memory, a processor, and a computer program stored on the memory and executable on the processor,

[0020] When the processor runs the computer program, the above method is implemented.

[0021] Another aspect of the embodiments of the present invention is to provide a computer-readable storage medium storing a computer program.

[0022] When the computer program is executed by a processor, the above-mentioned method is implemented.

[0023] The image processing method, apparatus, device and storage medium provided by the embodiments of the present invention obtain a material image to be processed; detect whether the material image includes a face area; if the material image includes a face area, determine the face area in the material image; calculate the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotate the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs and greatly improving the annotation efficiency of the appearance score of the material image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of an image processing method provided in Embodiment 1 of the present invention;

[0025] Figure 2A flow chart of an image processing method provided in Embodiment 2 of the present invention;

[0026] Figure 3 A flowchart of an image processing method provided in Embodiment 3 of the present invention;

[0027] Figure 4 A schematic diagram of the structure of an image processing device provided in Embodiment 4 of the present invention;

[0028] Figure 5 A schematic diagram of the structure of an image processing device provided in Embodiment 5 of the present invention;

[0029] Figure 6 A schematic diagram of the structure of an image processing device provided in Embodiment 4 of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of an image processing device provided in Embodiment 7 of the present invention.

[0031] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the embodiments of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0033] First, the terms involved in the embodiments of the present invention are explained:

[0034] Image processing: Technology that uses computers to analyze images to achieve desired results, such as image classification.

[0035] Deep learning: Deep learning is currently the most booming branch of machine learning and the most widely used technology in the entire field of artificial intelligence. It is a representation learning method developed based on artificial neural networks.

[0036] Alex network: Alex is the name of the inventor. The entire network consists of 8 layers of neurons. This structure increases the complexity of the network while adding measures to prevent overfitting.

[0037] Convolutional neural network: It is a feedforward neural network whose artificial neurons can respond to surrounding units within a part of the coverage area and perform well in image processing.

[0038] Pooling: Pooling is to divide the image into non-overlapping regions according to the window size, and then aggregate the elements in each region. Aggregation methods include: maximum pooling and average pooling.

[0039] Rectified Linear Unit (Relu), also known as the rectified linear unit, is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0040] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0041] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0042] Embodiment 1

[0043] Figure 1 This is a flow chart of the image processing method provided in the first embodiment of the present invention. The embodiment of the present invention provides an image processing method to address the problems that manual beauty score labeling in the prior art is highly subjective, the labeled beauty scores are inaccurate, and as the business volume of the e-commerce platform increases, the number of pictures in the material library will grow linearly or even exponentially, the demand for picture labeling will also grow linearly or even exponentially, the labor cost is too high, and human resources are wasted. The method in this embodiment is applied to the image processing device of the e-commerce platform. The image processing device can be a server of the e-commerce platform, a computer device, or a cluster of multiple computer devices. In other embodiments, the method can also be applied to other devices. This embodiment uses the server as an example for schematic description. Figure 1 As shown, the specific steps of this method are as follows:

[0044] Step S101: Obtain a material image to be processed.

[0045] The material image may be a picture of a product and a person, for example, a picture of a model showing the product, or a picture uploaded by a user showing the product, etc.

[0046] In this embodiment, the material image to be processed can be stored in one or more specified original material directories, and the original material directory can be set by the technical staff according to actual needs. The information of the material image can be stored in multiple data tables. The material master table is used to record the relevant information of the material image, such as the identification information, storage path, publisher information, release time, etc. of the material image. The material interaction table is used to record the likes data, comment data and other interactive data of the material image. The material label table is used to record the annotation data corresponding to the material image, where the annotation data can at least include the appearance score of the material image, and whether the material image includes facial area and other annotation information. The material commodity table is used to record the information of the commodity corresponding to the material image, for example, the identification, type, price and other commodity information of the commodity corresponding to the material image.

[0047] Specifically, the server may obtain original data of the material image from multiple data tables of the material image information; pre-process the original data and extract key information of the material image; and obtain the material image to be processed according to the key information of the material image.

[0048] Optionally, preprocessing the raw data of the material image may include: filtering out the data of the material image whose value of the preset field is a missing value or an abnormal value according to the raw data of the material image, and normalizing the data of each material image. The preset field may be a field in any one of a plurality of data tables for storing material image information, and the preset field may include one or more fields. The specific fields included in the preset field may be set by a technician according to actual needs, and this embodiment does not make specific limitations here.

[0049] For example, if certain key information in a material image is missing or abnormal, the data of the material image is filtered out, and there is no need to calculate the appearance score of the material image.

[0050] In addition, the key information of the material image extracted from the data of the material image may include the identification of the material image, the storage path, etc.; the extracted key information may be set by the technician according to actual needs, and this embodiment does not make any specific limitation here.

[0051] In this embodiment, the material image to be processed may include one or more images.

[0052] Step S102: Detect whether the material image includes a face area.

[0053] After the material images to be processed are acquired, for each material image, a face detection method is used to detect whether the material image includes a face area.

[0054] If the material image includes a face area, continue to execute steps S103-S105 to mark the material image with a beauty score.

[0055] If the material image does not include a face area, there is no need to execute subsequent steps S103-S105, and step S106 can be directly executed to set the appearance score of the material image to a preset minimum appearance score.

[0056] Optionally, if the material image does not include a face area, the material image may be marked as an unusable image.

[0057] Step S103: If the material image includes a face area, determine the face area in the material image.

[0058] If the material image includes a face region, the position of the face region in the material image is determined.

[0059] Optionally, after determining the face region in the material image, the face region in the material image may be marked, for example, the face region in the material image may be circled with a rectangular frame or a circular frame.

[0060] Step S104: Calculate the appearance score of the material image according to the face area in the material image through the appearance scoring model, where the appearance scoring model is obtained by training a convolutional neural network using a training set.

[0061] In this embodiment, before calculating the appearance score of the material image through the appearance scoring model, a training set is obtained in advance, and the convolutional neural network is trained using the training set to obtain the appearance scoring model. The training set includes multiple training data, each of which includes a sample image and the appearance score of the sample image marked in advance.

[0062] The appearance scoring model is used to classify the appearance of material images, divide the material images into categories with different appearance scores, and use the appearance score corresponding to the category as the appearance score of the material image, thereby obtaining the appearance score of the material image.

[0063] Step S105: label the material image according to its appearance score.

[0064] After calculating the appearance score of the material image, the appearance score of the material image is labeled.

[0065] Specifically, the beauty score of the material image can be automatically stored in correspondence with the material image. For example, the beauty score of the material image can be recorded in the data of the material image in the data table.

[0066] Optionally, the appearance score of each material image can be stored in a two-dimensional matrix using a one-hot encoding method according to the appearance scores of the pre-divided multiple levels, and the value of the position corresponding to the level of the appearance score of the material image is set to 1, and the remaining positions are set to 0.

[0067] In addition, in this embodiment, after calculating the appearance score of the material image, other methods can be used to mark the appearance score of the material image, which is not specifically limited in this embodiment.

[0068] Step S106: If the material image does not include a face area, the appearance score of the material image is set to a preset minimum appearance score.

[0069] The embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs, and greatly improving the annotation efficiency of the appearance score of the material image.

[0070] Embodiment 2

[0071] Figure 2 This is a flow chart of the image processing method provided in the second embodiment of the present invention. Based on the above-mentioned first embodiment, in this embodiment, before calculating the appearance score of the material image through the appearance scoring model according to the face area in the material image, it also includes: obtaining a training set, the training set includes a plurality of sample images with annotated appearance scores; using the training set to train the convolutional neural network to obtain the appearance scoring model. Figure 2 As shown, the specific steps of this method are as follows:

[0072] Step S201: Obtain a training set, where the training set includes a plurality of sample images with labeled appearance scores.

[0073] The sample image refers to a material image that has been annotated with a beauty score. The training set includes multiple training data, each of which includes a sample image and a pre-annotated beauty score of the sample image.

[0074] In this embodiment, the sample image can be stored in one or more designated sample material directories, and the sample material directory can be set by the technician according to actual needs. The storage of the sample image information can be stored in the same manner as the information of the material image to be processed. Specifically, the information of the sample image can be stored in multiple data tables. The sample master table is used to record the relevant information of the sample image, such as the identification information, storage path, publisher information, release time, etc. of the sample image. The sample interaction table is used to record the like data, comment data and other interactive data of the sample image. The sample label table is used to record the annotation data corresponding to the sample image, wherein the annotation data may at least include the appearance score of the sample image, and whether the material image includes a face area and other annotation information. The sample product table is used to record the information of the product corresponding to the sample image, for example, the identification, type, price and other product information annotation information of the product corresponding to the sample image.

[0075] In addition, the annotation information of the sample image can be obtained by professional technicians by annotating the sample image according to factors such as the image clarity, image completeness, and appearance of the portrait in the image, based on a preset annotation specification document.

[0076] Optionally, before manually annotating the sample images, in order to facilitate manual annotation of the sample images, the server can obtain the original data of the sample images, pre-process the original data of the sample images, filter out sample images with missing or abnormal information, retain available sample images with complete information, normalize the data of each available sample image, and extract key information of the available sample images. The server selects multiple available sample images based on the extracted key information of the available sample images and displays them for the technicians to view and manually annotate the sample images' beauty scores and whether they include face areas.

[0077] In addition, the server preprocesses the raw data of the sample image, including: filtering out the data of the sample image whose value of the preset field is a missing value or an abnormal value according to the raw data of the sample image, and normalizing the data of each sample image. The preset field can be a field in any one of a plurality of data tables for storing sample image information, and the preset field can include one or more fields. The specific fields included in the preset field can be set by the technician according to actual needs, and this embodiment does not make specific limitations here.

[0078] For example, if certain key information in a sample image is missing or abnormal, the data of the sample image is filtered out, and there is no need to calculate the appearance score of the sample image.

[0079] In addition, the key information of the sample image extracted from the data of the sample image may include the identification of the sample image, the storage path, etc.; the specific information included in the key information can be set by the technicians according to actual needs, and this embodiment does not make specific limitations here.

[0080] In this embodiment, after obtaining the annotation information of the sample images, the sample images with different appearance scores may be stored in different directories respectively.

[0081] Specifically, the storage directories corresponding to the appearance scores of each level may be pre-set, and different appearance scores correspond to different storage directories; according to the appearance score of the sample image, the sample image is stored in the storage directory corresponding to the appearance score of the sample image.

[0082] In this step, when obtaining the training set, the number of sample images of each level of appearance score in the training set can be determined according to the proportion of sample images of different appearance scores in the training set that are preset, and sample images can be obtained from the storage directory corresponding to each level of appearance score based on the number of sample images of each level of appearance score in the training set, and the annotation information of the sample images can be obtained.

[0083] For example, the appearance score may be pre-set to include 10 levels, and the corresponding appearance scores from low to high are 1 point, 2 points, 3 points, 4 points, 5 points, 6 points, 7 points, 8 points, 9 points and 10 points respectively.

[0084] Step S202: Use a training set to train the convolutional neural network to obtain an appearance scoring model.

[0085] In this embodiment, the convolutional neural network adopts the Alex network. In this embodiment, the convolutional neural network architecture consists of 8 layers of neurons, of which the first 5 layers are convolutional layers for extracting image features, and the last 3 layers are fully connected layers for calculating the appearance score.

[0086] Convolution is a commonly used linear filtering method. Using convolution can achieve image noise reduction, sharpening and other effects. Simply put, for each pixel in the picture, the product of its domain pixel and the corresponding position element of the filter matrix is ​​calculated, and then all the products are accumulated as the output value of the pixel position. In terms of network structure, there are three main differences between the convolution layer node and the fully connected layer node: one is local perception, the second is weight sharing, and the third is multiple convolution kernels. Local perception means that for each computing unit, only the input near its pixel position needs to be considered, and it does not need to be connected to all nodes in the previous layer. This is in line with the user's understanding of the image, that is, the image is relatively continuous, and the combination of local information can form various line shapes. Weight sharing means that when a picture is convolved, the convolution kernel will slide over each pixel of the picture one by one, and the parameters for processing each pixel are the same. Each convolution kernel is a feature extractor. If there is only one convolution kernel, only one feature can be extracted, which is obviously not enough. In this embodiment, the convolution layer uses multiple convolution kernels to ensure full feature extraction in a multi-core convolution manner.

[0087] In order to prevent overfitting, the following two methods are adopted in the convolutional neural network model of this embodiment: one is to set dropout to randomly discard some neurons; the other is to perform batch normilization processing.

[0088] The beauty scoring model uses Relu as the activation function of the convolution layer, and the calculation of the beauty score of the material image is better than using the Sigmoid function as the activation function of the convolution layer. Relu is a restricted activation function, which introduces a lot of sparsity to the network during use. At least half of the neurons will not be activated, thus strengthening the extraction of strong features and the collapse of weak features, enhancing the learning effect.

[0089] For the Relu function, it is used as an activation function for regression, and the output result is approximately normal distribution. In this embodiment, a random matrix with a mean of 0 and a certain standard deviation can be generated by a random normal distribution as an initialization parameter, and the standard deviation can be gradually increased during the training process, so that the weight of the convolutional neural network can obtain an elastic increase.

[0090] In addition, the appearance scoring model uses the Alex network to create a competition mechanism for the activities of local neurons, making the values ​​with relatively large responses become relatively larger and inhibiting other neurons with smaller feedback, thereby enhancing the generalization ability of the appearance scoring model.

[0091] During the training of the convolutional neural network, parameters such as the input image size of the convolutional neural network, the number of convolution kernels of the convolutional layer, and the sliding step size of the convolutional layer can be debugged to optimize the convolutional neural network to obtain a better appearance scoring model.

[0092] Optionally, this embodiment provides a set of parameter settings for the appearance scoring model. Specifically, the size of the input image of the appearance scoring model can be 128*128 pixels; the first layer of the appearance scoring model uses a 7*7 convolution kernel, a sliding step size of 3 pixels, outputs 96 feature maps, and performs maximum pooling; the second layer of the appearance scoring model uses a 5*5 convolution kernel, the convolution generates 256 feature maps, and performs maximum pooling; the third layer of the appearance scoring model uses a 3*3 convolution kernel, and outputs 384 feature maps; the fourth layer of the appearance scoring model uses a 3*3 convolution kernel, and outputs 384 feature maps; the fifth layer of the appearance scoring model uses a 3*3 convolution kernel, outputs 256 feature maps, and performs pooling; the sixth and seventh layers of the appearance scoring model are fully connected layers, each containing 4096 hidden layers; the eighth layer of the appearance scoring model is a softMax layer, which calculates the appearance score of the material image.

[0093] In addition, during the training of the convolutional neural network, one or more training parameters such as the learning rate, number of iterations, batch size, attenuation step size, and the proportion of dropout retained neurons can be debugged to train a better appearance scoring model.

[0094] Optionally, this embodiment provides a set of better training parameters. Specifically, the learning rate can be 0.001; the number of iterations can be set to 3000; the training batch size can be set to 10; the decay step size can be set to 3; the proportion of dropout retained neurons can be set to 0.75, indicating that 25% of the neurons are randomly discarded to prevent overfitting.

[0095] For example, different learning rates will have a great impact on the convergence of the appearance scoring model. The same appearance scoring model will show very different effects when using different learning rates. In this embodiment, the learning rate is first set to 0.0001, and the learning rate is gradually increased. Combined with the evaluation indicators of the appearance scoring model, after multiple rounds of testing and tuning, the appearance scoring model has the best effect when the learning rate is adjusted to 0.001 on the adopted training set, and the learning rate can be set to 0.001.

[0096] The convolutional neural network can be tuned to adjust parameters such as the input image size, the number of convolution kernels in the convolutional layer, and the sliding step size of the convolutional layer to optimize the convolutional neural network and obtain a better appearance scoring model.

[0097] Optionally, after the convolutional neural network training is completed, the trained model can be tested using the test data in the test set. In addition, the beauty scores of the images obtained by testing the model can be stored in the form of a linked list, and the display device can be controlled to display the beauty scores of the images.

[0098] For example, there are 500 images in the test set. The model obtained after training is used to calculate the beauty scores of the specified 10 images, and the beauty scores of the 10 images are saved in a linked list. The beauty scores of the 10 images are displayed in the following form: [4 3 4 4 3 4 3 5 4 4], which means that the beauty score of the first image is 4, the beauty score of the second image is 3, the beauty score of the third image is 4, and so on. The technicians can determine the effect of the trained model based on the beauty scores of these 10 images. If the effect is not good, the model can be further trained; if the effect is good, there is no need to continue training.

[0099] In this embodiment, the storage format of the material pictures and the pictures in the training set and the test set is not specifically limited. For example, the pictures can be stored in jpg format.

[0100] Step S203: Obtain the material image to be processed.

[0101] This step is similar to the above step S101 and will not be described in detail in this embodiment.

[0102] Step S204: Detect whether the material image includes a face area.

[0103] Step S205: If the material image includes a face region, determine the face region in the material image.

[0104] The above steps S204-S205 are the process of performing face detection on the material image. After obtaining the material image to be processed, for each material image, a face detection method is used to detect whether the material image includes a face area. If the material image includes a face area, the location of the face area is determined, and the subsequent steps S206-S207 are continued to be executed to mark the material image with a beauty score.

[0105] If the material image does not include a face area, the subsequent step S208 is executed to set the appearance score of the material image to a preset minimum appearance score.

[0106] Optionally, if the material image does not include a face area, the material image may be marked as an unusable image.

[0107] In this embodiment, face detection is performed on the material image to determine whether the material image includes a face region and to determine the position of the face region in the material image, which can be achieved by using a face detection model.

[0108] Optionally, in this embodiment, a Haar cascade classifier may be used as a face detection model to implement face detection.

[0109] Before using the face detection model to perform face detection on the material image, the face detection model is trained in advance using a training set for face detection so that the detection accuracy of the face detection model meets the preset conditions. The preset conditions can be set by the technician according to actual needs, and this embodiment does not specifically limit them.

[0110] Optionally, the training set used to train the face detection model can be the same training set as the training set used to train the appearance scoring model. The training data in the training set used to train the face detection model includes sample images, and pre-labeled sample images of whether they include a face area and the appearance score of the sample images.

[0111] In addition, the training set used to train the face detection model may not be the same as the training set used to train the appearance scoring model. The training data in the training set used to train the face detection model includes sample images and pre-labeled sample images of whether they include a face area.

[0112] Optionally, a down-sampling ratio coefficient in a training set used for training a face detection model may be preset.

[0113] Optionally, the material image is usually a 3-channel image. In this embodiment, before performing face detection on the material image, the material image can be converted into a grayscale image, and face detection is performed on the grayscale image converted from the material image through a face detection model to determine the position of the face area in the material image.

[0114] Optionally, after determining the face region in the material image, the face region in the material image may be marked, for example, the face region in the material image may be circled with a rectangular frame or a circular frame.

[0115] Step S206: Calculate the appearance score of the material image according to the face area in the material image through the appearance scoring model, where the appearance scoring model is obtained by training the convolutional neural network using the training set.

[0116] In this embodiment, the appearance score of the material image is calculated by the appearance scoring model according to the face area in the material image, which can be specifically implemented in the following manner:

[0117] According to the preset size of the image input into the appearance scoring model, the material image is cropped into an image of the preset size including the face area; the cropped image is input into the appearance scoring model to calculate the appearance score of the cropped image; and the appearance score of the cropped image is used as the appearance score of the material image.

[0118] Optionally, in this embodiment, the material images to be processed can be cropped in batches, and all the material images to be processed can be cropped to a preset size. Specifically, the directory where the material images to be processed are located can be imported, and the material images in the directory where the material images to be processed are located can be read and cropped respectively, and the cropped material images of the preset size can be stored in a pre-set new directory to improve the cropping efficiency of the material images.

[0119] Furthermore, after the cropped material images of the preset size are stored in a preset new directory, the images in the new directory may be converted in batches to the format of the input images preset by the appearance scoring model. For example, the images in the new directory may be uniformly converted to the Tensorflow-specific format or the TFRecord format.

[0120] Optionally, after the material images are annotated according to their appearance scores, the material images with different appearance scores may be stored in different directories.

[0121] Specifically, the storage directories corresponding to the appearance scores of each level may be pre-set, and different appearance scores correspond to different storage directories; according to the appearance score of the material image, the material image is stored in the storage directory corresponding to the appearance score of the material image.

[0122] Step S207: label the material image according to its appearance score.

[0123] After calculating the appearance score of the material image, the appearance score of the material image is labeled.

[0124] Specifically, the beauty score of the material image can be automatically stored in correspondence with the material image. For example, the beauty score of the material image can be recorded in the data of the material image in the data table.

[0125] Optionally, the appearance score of each material image can be stored in a two-dimensional matrix using a one-hot encoding method according to the appearance scores of the pre-divided multiple levels, and the value of the position corresponding to the level of the appearance score of the material image is set to 1, and the remaining positions are set to 0.

[0126] In addition, in this embodiment, after calculating the appearance score of the material image, other methods can be used to mark the appearance score of the material image, which is not specifically limited in this embodiment.

[0127] Step S208: If the material image does not include a face area, the appearance score of the material image is set to a preset minimum appearance score.

[0128] Among them, the preset minimum appearance score can be set by technical personnel according to actual needs.

[0129] The embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs, and greatly improving the annotation efficiency of the appearance score of the material image.

[0130] Embodiment 3

[0131] Figure 3 A flow chart of the image processing method provided for the third embodiment of the present invention. Based on the above-mentioned first or second embodiment, in this embodiment, after the material image is annotated according to the appearance score of the material image, it also includes: upon receiving a request to view the material image of a product, obtaining all the material images corresponding to the product; sorting all the material images corresponding to the product according to the appearance score of all the material images corresponding to the product; and controlling the display device to display the material image corresponding to the product according to the sorting of all the material images corresponding to the product. Figure 3 As shown, after the material image is annotated according to the appearance score of the material image, the method further includes the following specific steps:

[0132] Step S301: upon receiving a request to view a product material image, all material images corresponding to the product are acquired.

[0133] When a user browses a product list or views a product, or visits other pages including material images of products, and sends an access request to the server, the server determines that it has received the request to view the material images of the products and obtains all material images corresponding to the products.

[0134] Step S302: sort all the material images corresponding to the products according to their appearance scores.

[0135] In this embodiment, for each request to view the commodity material images, the order of the material images is refreshed.

[0136] In addition, when sorting material images, the user preference score and material quality score of the material images can also be referred to. The user preference score can be calculated based on the matching degree between the user purchasing power and the price of the product corresponding to the material image by estimating the user's purchasing power through the user spending model. The material quality score can be calculated based on the material release time, material interaction data, etc.

[0137] For example, the material quality score of a material image can be calculated by calculating the image appearance score, material release time, number of material likes, number of material comments, number of material views, and number of material shares of the material image.

[0138] Optionally, all the material images corresponding to the products are sorted according to their appearance scores, which can be specifically implemented in the following ways:

[0139] Obtain user preference scores and material quality scores of all material images corresponding to the product; calculate the comprehensive ranking score of all material images corresponding to the product based on the appearance scores, user preference scores and material quality scores of all material images corresponding to the product; and sort all material images corresponding to the product based on the comprehensive ranking score of all material images corresponding to the product.

[0140] Step S303: According to the order of all the material images corresponding to the commodities, the display device is controlled to display the material images corresponding to the commodities.

[0141] The embodiment of the present invention obtains all material images corresponding to a product when a request for viewing material images of the product is received; calculates a comprehensive ranking score of all material images corresponding to the product according to the appearance scores, user preference scores and material quality scores of all material images corresponding to the product, and sorts all material images corresponding to the product according to the comprehensive ranking score; and controls a display device to display the material images corresponding to the product according to the sorting of all material images corresponding to the product, thereby showing the most valuable material images to users and improving the click-through conversion rate of the product.

[0142] Embodiment 4

[0143] Figure 4 Schematic diagram of the structure of the image processing device provided by the fourth embodiment of the present invention. The image processing device provided by the embodiment of the present invention can execute the processing flow provided by the embodiment of the image processing method. Figure 4 As shown, the image processing device 40 includes: a preprocessing module 401, a face detection module 402, an appearance scoring module 403 and a labeling processing module 404.

[0144] Specifically, the preprocessing module 401 is used to obtain a material image to be processed.

[0145] The face detection module 402 is used to detect whether the material image includes a face area.

[0146] The face detection module 402 is further configured to determine the face region in the material image if the material image includes the face region.

[0147] The appearance scoring module 403 is used to calculate the appearance score of the material image according to the face area in the material image through the appearance scoring model, and the appearance scoring model is obtained by training the convolutional neural network using the training set.

[0148] The annotation processing module 404 is used to perform annotation processing on the material image according to the appearance score of the material image.

[0149] The device provided in the embodiment of the present invention can be specifically used to execute the method embodiment provided in the above-mentioned embodiment 1, and the specific functions will not be repeated here.

[0150] The embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs, and greatly improving the annotation efficiency of the appearance score of the material image.

[0151] Embodiment 5

[0152] Figure 5 FIG. 5 is a schematic diagram of the structure of an image processing device provided in Embodiment 5 of the present invention. Based on the above Embodiment 4, in this embodiment, Figure 5 As shown, the image processing device 40 also includes: a model training module 405.

[0153] The model training module 405 is used to:

[0154] A training set is obtained, which includes multiple sample images with labeled appearance scores; a convolutional neural network is trained using the training set to obtain an appearance scoring model.

[0155] Optionally, the annotation processing module is also used to:

[0156] If the material image does not include a face area, the appearance score of the material image is set to a preset minimum appearance score.

[0157] Optionally, the appearance scoring module is also used to:

[0158] According to the preset size of the image input into the appearance scoring model, the material image is cropped into an image of the preset size including the face area; the cropped image is input into the appearance scoring model to calculate the appearance score of the cropped image; and the appearance score of the cropped image is used as the appearance score of the material image.

[0159] Optionally, the convolutional neural network is an Alex network.

[0160] The device provided in the embodiment of the present invention can be specifically used to execute the method embodiment provided in the above-mentioned embodiment 2, and the specific functions will not be repeated here.

[0161] The embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs, and greatly improving the annotation efficiency of the appearance score of the material image.

[0162] Embodiment 6

[0163] Figure 6 FIG. 1 is a schematic diagram of the structure of an image processing device provided in Embodiment 4 of the present invention. Based on the above Embodiment 4 or Embodiment 5, in this embodiment, Figure 6 As shown, the image processing device 40 further includes: a sorting and displaying module 406 .

[0164] The sorting display module 406 is also used for:

[0165] When a request to view a material image of a product is received, all material images corresponding to the product are obtained; all material images corresponding to the product are sorted according to the appearance scores of all material images corresponding to the product; and according to the sorting of all material images corresponding to the product, a display device is controlled to display the material images corresponding to the product.

[0166] Optionally, the sorting display module 406 is further used for:

[0167] Obtain user preference scores and material quality scores of all material images corresponding to the product; calculate the comprehensive ranking score of all material images corresponding to the product based on the appearance scores, user preference scores and material quality scores of all material images corresponding to the product; and sort all material images corresponding to the product based on the comprehensive ranking score of all material images corresponding to the product.

[0168] The device provided in the embodiment of the present invention can be specifically used to execute the method embodiment provided in the above-mentioned embodiment 3, and the specific functions will not be repeated here.

[0169] The embodiment of the present invention obtains all material images corresponding to a product when a request for viewing material images of the product is received; calculates a comprehensive ranking score of all material images corresponding to the product according to the appearance scores, user preference scores and material quality scores of all material images corresponding to the product, and sorts all material images corresponding to the product according to the comprehensive ranking score; and controls a display device to display the material images corresponding to the product according to the sorting of all material images corresponding to the product, thereby showing the most valuable material images to users and improving the click-through conversion rate of the product.

[0170] Embodiment 7

[0171] Figure 7 This is a schematic diagram of the structure of an image processing device provided by Embodiment 7 of the present invention. Figure 7 As shown, the device 70 includes: a processor 701 , a memory 702 , and a computer program stored in the memory 702 and executable by the processor 701 .

[0172] The processor 701 implements the image processing method provided by any of the above method embodiments when executing the computer program stored in the memory 702 .

[0173] The embodiment of the present invention obtains a material image to be processed; detects whether the material image includes a face area; if the material image includes a face area, determines the face area in the material image; calculates the appearance score of the material image according to the face area in the material image through an appearance scoring model, wherein the appearance scoring model is obtained by training a convolutional neural network using a training set; and annotates the material image according to the appearance score of the material image, thereby realizing automatic annotation of the appearance score of the material image, reducing a large amount of labor costs, and greatly improving the annotation efficiency of the appearance score of the material image.

[0174] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the image processing method provided by any of the above method embodiments is implemented.

[0175] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0178] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0179] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0181] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a material image to be processed; the material image includes a picture of a product and a portrait; Detecting whether the material image includes a face area; If the material image includes a face region, determining the face region in the material image; Calculating the appearance score of the material image using an appearance scoring model according to the face area in the material image; According to the beauty score of the material image, marking the material image; When receiving a request to view a product material image, obtaining all material images corresponding to the product; Obtaining user preference scores and material quality scores of all material images corresponding to the product; wherein the user preference score is calculated based on the matching degree between the user purchasing power and the price of the product corresponding to the material image, by estimating the user purchasing power through the user spending model; and the material quality score is calculated based on the material release time and the material interaction data; Calculate the comprehensive ranking score of all the material images corresponding to the product according to the appearance score, user preference score and material quality score of all the material images corresponding to the product; sorting all the material images corresponding to the product according to the comprehensive ranking scores of all the material images corresponding to the product; According to the order of all the material images corresponding to the product, control the display device to display the material images corresponding to the product; Before calculating the appearance score of the material image by using the appearance scoring model according to the face area in the material image, the method further includes: Obtain a training set, the training set comprising a plurality of sample images with annotated beauty scores; the beauty scores annotated on the sample images are obtained by annotating the sample images according to image clarity, image completeness, and beauty of a person in the image, in accordance with a preset annotation specification document; The training set is used to train a convolutional neural network to obtain the appearance scoring model, and the convolutional neural network is an Alex network whose activation function of the convolutional layer is Relu.

2. The method according to claim 1, characterized in that After detecting whether the material image includes a face area, the method further includes: If the material image does not include a face area, the appearance score of the material image is set to a preset minimum appearance score.

3. The method according to claim 1 or 2, characterized in that: The step of calculating the appearance score of the material image by using an appearance scoring model according to the face area in the material image includes: According to the preset size of the image input to the appearance scoring model, the material image is cropped into an image of the preset size including a face area; Input the cropped image into the appearance scoring model to calculate the appearance score of the cropped image; The beauty score of the cropped image is used as the beauty score of the material image.

4. An image processing device, characterized in that: include: A preprocessing module, used to obtain material images to be processed; the material images include pictures of products and portraits; A face detection module, used to detect whether the material image includes a face area; The face detection module is further used to determine the face area in the material image if the material image includes the face area; An appearance scoring module, used to calculate the appearance score of the material image through an appearance scoring model according to the face area in the material image; A labeling processing module, used for labeling the material image according to the beauty score of the material image; A sorting and displaying module is used to obtain all material images corresponding to the product when receiving a request to view the product material image; Obtaining user preference scores and material quality scores of all material images corresponding to the product; wherein the user preference score is calculated based on the matching degree between the user purchasing power and the price of the product corresponding to the material image, by estimating the user purchasing power through the user spending model; and the material quality score is calculated based on the material release time and the material interaction data; Calculate the comprehensive ranking score of all the material images corresponding to the product according to the appearance score, user preference score and material quality score of all the material images corresponding to the product; sorting all the material images corresponding to the product according to the comprehensive ranking scores of all the material images corresponding to the product; According to the order of all the material images corresponding to the product, control the display device to display the material images corresponding to the product; The model training module is used to obtain a training set, which includes a plurality of sample images with annotated appearance scores; the appearance scores annotated with the sample images are obtained by annotating the sample images according to the image clarity, image completeness and appearance of the portraits in the sample images, according to a preset annotation specification document; the convolutional neural network is trained using the training set to obtain the appearance scoring model, and the convolutional neural network is an Alex network whose activation function of the convolutional layer is Relu.

5. The device according to claim 4, characterized in that The annotation processing module is also used for: If the material image does not include a face area, the appearance score of the material image is set to a preset minimum appearance score.

6. The device according to claim 4 or 5, characterized in that The appearance scoring module is also used for: According to the preset size of the image input to the appearance scoring model, the material image is cropped into an image of the preset size including a face area; Input the cropped image into the appearance scoring model to calculate the appearance score of the cropped image; The beauty score of the cropped image is used as the beauty score of the material image.

7. An image processing device, characterized in that: include: a memory, a processor, and a computer program stored on the memory and executable on the processor, When the processor runs the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: A computer program is stored. When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images