Image processing method and device, storage medium, electronic equipment and chip

Abstract

The invention provides an image processing method and device, a storage medium, electronic equipment and a chip, relates to the technical field of image processing, and aims to provide evaluation and comprehensive evaluation of an image in multiple dimensions by combining the category and parameter information of the image, increase the interpretability of image evaluation, improve the accuracy and reduce the problem of low discrimination of image evaluation information. The image processing method comprises the following steps: acquiring a to-be-processed image; inputting the to-be-processed image into a trained image processing model to obtain image evaluation information corresponding to the to-be-processed image, the image processing model being used for determining an image category corresponding to the to-be-processed image and image parameter information of the to-be-processed image, and outputting target image evaluation information, first image evaluation information of the to-be-processed image under each image category and second image evaluation information corresponding to each piece of image parameter information at least according to the image category and the image parameter information.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and chip. Background Art

[0002] With the development of artificial intelligence technology, research and application in the field of image aesthetic evaluation has also gradually developed. Image aesthetic evaluation has evolved from the initial classification of an image's aesthetic category to giving a discrete score for each image or a distribution histogram for the score. As the granularity of aesthetic evaluation gradually decreases, people hope to be able to describe the beauty of the image more finely.

[0003] However, the image aesthetic evaluation method in the related art has the problem of insufficient interpretability, which results in insufficient differentiation of the final aesthetic evaluation scores. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, storage medium, electronic device and chip.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Get the image to be processed;

[0007] The image to be processed is input into a trained image processing model to obtain image evaluation information corresponding to the image to be processed, wherein the image processing model is used to determine an image category result corresponding to the image to be processed and image parameter information of the image to be processed, and output a target image evaluation information and a first image evaluation information of the image to be processed under each image category and a second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

[0008] Optionally, the step of inputting the image to be processed into a trained image processing model to obtain image evaluation information corresponding to the image to be processed includes:

[0009] Input the image to be processed into the trained image processing model, determine at least one of the following image category results corresponding to the image to be processed: semantic category result, composition category result, emotion category result and expressiveness category result, and determine at least one of the following image parameter information corresponding to the image to be processed: image quality information, depth of field information, color information, contrast and saturation;

[0010] Image evaluation information corresponding to the image to be processed is determined based at least on the at least one image category result and the at least one image parameter information.

[0011] Optionally, determining the image evaluation information corresponding to the image to be processed at least according to the at least one image category result and the at least one image parameter information includes:

[0012] Performing feature extraction on each of the image category results and each of the image parameter information to obtain a plurality of feature vectors, and performing vector concatenation on the plurality of feature vectors to obtain an attribute feature vector;

[0013] Determining the attribute weight value corresponding to each of the image category results and each of the image parameter information through the attribute weight network in the image processing model;

[0014] The first image evaluation information, the second image evaluation information and the target image evaluation information are determined at least according to the attribute feature vector and the attribute weight value.

[0015] Optionally, the image processing model is further used to extract features from the image to be processed through a pre-trained feature extraction network to obtain an image feature matrix, and the first image evaluation information, the second image evaluation information and the target image evaluation information are determined at least based on the attribute feature vector and the attribute weight value, including:

[0016] Performing convolution calculation on the attribute feature vector and the attribute weight value to obtain a target attribute feature vector;

[0017] The target attribute feature vector and the image feature matrix are input into the evaluation prediction network in the image processing model to obtain the first image evaluation information, the second image evaluation information and the target image evaluation information.

[0018] Optionally, the image processing model includes an image category network, an attribute weight network and a prediction network, and the image processing model is trained in the following manner:

[0019] Pre-training the image category network to obtain an initial image processing model;

[0020] Acquire a first sample image, wherein the first sample image is annotated with an image evaluation label, wherein the image evaluation label includes a target image evaluation label for the first sample image, a first image evaluation label for the first sample image under different image categories, and a second image evaluation label corresponding to image parameter information of the first sample image; input the first sample image into the initial image processing model to obtain a target prediction evaluation result corresponding to the first sample image, a first prediction evaluation result for the first sample image under each of the image categories, and a second prediction evaluation result corresponding to each of the image parameter information;

[0021] The parameters of the image processing model are adjusted according to the image evaluation label, the target prediction evaluation result, the first prediction evaluation result and the second prediction evaluation result.

[0022] Optionally, the image category network includes an emotion classification network and / or an expressiveness classification network, and the pre-training of the image category network includes:

[0023] Acquire a second sample image and / or a third sample image, wherein the second sample image is annotated with an emotion category label, and the third sample image is annotated with an expressiveness category label;

[0024] Inputting the second sample image into the emotion classification network to obtain a predicted emotion category result corresponding to the second sample image, and adjusting the parameters of the emotion classification network according to the predicted emotion category result and the emotion category label; and / or,

[0025] The third sample image is input into the expressiveness classification network to obtain a predicted expressiveness category result corresponding to the third sample image, and the parameters of the expressiveness classification network are adjusted according to the predicted expressiveness category result and the expressiveness category label.

[0026] Optionally, the image evaluation label, the emotion category label and the expressiveness category label are obtained in the following manner:

[0027] Obtaining evaluation scores of the first sample image annotated by multiple first users, and performing weighted average calculation on the evaluation scores according to a first preset weight value to obtain the target image evaluation label of the first sample image;

[0028] Acquire emotion category values ​​annotated by multiple second users on the second sample image, and perform weighted average calculation on the emotion category values ​​according to a second preset weight value to obtain the emotion category label of the second sample image;

[0029] The expressiveness category values ​​annotated by multiple third users on the third sample image are obtained, and the expressiveness category values ​​are weighted averaged according to a third preset weight value to obtain the expressiveness category label of the third sample image.

[0030] Optionally, the method is applied to a terminal device, and obtaining the image to be processed includes:

[0031] When the terminal device is in a shooting mode, a shooting preview image captured by the terminal device is obtained as an image to be processed.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0033] An acquisition module, used for acquiring an image to be processed;

[0034] A processing module is used to input the image to be processed into a trained image processing model to obtain image evaluation information corresponding to the image to be processed, wherein the image processing model is used to determine the image category result corresponding to the image to be processed and the image parameter information of the image to be processed, and output a target image evaluation information and the first image evaluation information of the image to be processed under each image category and the second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

[0035] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided in the first aspect of the present disclosure are implemented.

[0036] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0037] A processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the steps of the image processing method provided in the first aspect of the present disclosure when executed.

[0038] According to a fifth aspect of an embodiment of the present disclosure, a chip is provided, including:

[0039] A processor and an interface; the processor is used to read instructions and, when executed, implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0040] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: 1) by combining the image category results and the image parameter information, image evaluation information is obtained, which increases the analysis dimension and thus improves the accuracy of image evaluation; 2) the image processing model can output the first image evaluation information under each image category, the second image evaluation information corresponding to each image parameter information and the target image evaluation information, which increases the interpretability of the image evaluation information and reduces the situation where the image evaluation information has low discrimination.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] Figure 1is a flowchart of an image processing method according to an exemplary embodiment;

[0044] Figure 2 is a schematic diagram of a process of an image processing method according to an exemplary embodiment;

[0045] Figure 3 is a flowchart of an image processing model training according to an exemplary embodiment;

[0046] Figure 4 is a block diagram of an image processing device according to an exemplary embodiment;

[0047] Figure 5 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0050] With the development of artificial intelligence technology, research and application in the field of image aesthetic evaluation has also gradually developed. Image aesthetic evaluation has evolved from the initial classification of an image's aesthetic category, to giving a discrete score for each image, to giving a distribution histogram for the score. As the granularity of aesthetic evaluation gradually decreases, people hope to be able to describe the beauty of the image more finely.

[0051] In related technologies, usually only the information of the image itself is considered, and the global image features are learned through convolutional neural networks. With the subsequent development of deep learning, network structures and training methods that are more suitable for this task were introduced to improve performance. In addition, related technologies can give a score of 100 to an image, or recommend photos to users when setting wallpapers. However, these related technologies have the problem of insufficient interpretability, which leads to insufficient discrimination of the final image evaluation scores.

[0052] In view of this, the present disclosure provides an image processing method, device, storage medium, electronic device and chip to combine image category results and image parameter information to obtain image evaluation information, increase the analysis dimension, thereby improving the accuracy of image evaluation, increasing the interpretability of image evaluation information, and reducing the situation where the image evaluation information has low discrimination.

[0053] First, possible application scenarios of the embodiments of the present disclosure are described.

[0054] 1) When the electronic device is in shooting mode, assist the user to take better quality photos. For example, before taking a photo, open the preview screen, and as the camera moves, the image processing method provided by the embodiment of the present disclosure provides image evaluation information in real time, and determines the capture moment based on the image evaluation information, and automatically assists in pressing the shutter, or provides shooting suggestions, such as how to move the camera to get a better composition.

[0055] 2) After pressing the shutter of the electronic device's camera or in the electronic device's photo album, automatically edit or crop the photo. For example, set a sliding window to evaluate the photo, and when sliding to the photo area where the image evaluation information meets the preset conditions, automatically select the cropping ratio and crop the photo area. For another example, the image evaluation information is given by the image processing method provided by the embodiment of the present disclosure, and the image evaluation information is given, and according to the image evaluation information, the rotation strategy is used to assist in the photo straightening or enhance the photo.

[0056] 3) Screening photos or managing albums in the electronic device. For example, for multiple similar photos in the album, the image processing method provided by the embodiment of the present disclosure provides image evaluation information, and recommends a target photo to the user based on the image evaluation information.

[0057] 4) Summarize or extract the cover of the video highlights in the album of the electronic device. For example, the image evaluation information is given by the image processing method provided by the embodiment of the present disclosure, and a frame of the video is found as the cover according to the image evaluation information.

[0058] It should be understood that the above application scenarios are only possible application scenarios of the image processing method provided by the present disclosure. In actual applications, the image processing method provided by the present disclosure can also be applied to other scenarios as needed, and the embodiments of the present disclosure are not limited to this.

[0059] The image processing method provided by the present disclosure is described below.

[0060] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method comprises the following steps:

[0061] In step S101, an image to be processed is obtained.

[0062] In step S102, the image to be processed is input into the trained image processing model to obtain image evaluation information corresponding to the image to be processed.

[0063] In step S102, the image processing model can be used to determine the image category result corresponding to the image to be processed and the image parameter information of the image to be processed, and output the target image evaluation information and the first image evaluation information of the image to be processed under each image category and the second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

[0064] By combining the image category results and image parameter information, as well as the correlation between each image attribute and different image categories, image evaluation information is obtained, which increases the analysis dimension, thereby improving the accuracy of image evaluation. In addition, the first image evaluation information under each image category, the second image evaluation information corresponding to each image parameter information, and the target image evaluation information can be output, which increases the interpretability of the image evaluation information and reduces the situation where the image evaluation information has low differentiation.

[0065] In order to enable those skilled in the art to better understand the image processing method provided by the present disclosure, the above steps are described in detail with examples below.

[0066] In one possible manner, the image processing method provided by the embodiment of the present disclosure may be applied to a terminal device. Accordingly, in step S101, when the terminal device is in a shooting mode, a shooting preview image captured by the terminal device may be obtained as an image to be processed.

[0067] In other words, the image processing method provided by the embodiment of the present disclosure can assist users in taking photos of better quality.

[0068] In one possible manner, step S102 may include: inputting the image to be processed into a trained image processing model, determining at least one of the following image category results corresponding to the image to be processed: semantic category results, composition category results, emotion category results, and expressiveness category results, and determining at least one of the following image parameter information corresponding to the image to be processed: image quality information, depth of field information, color information, contrast, and saturation; determining image evaluation information corresponding to the image to be processed based on at least at least one image category result and at least one image parameter information.

[0069] It should be understood that image semantic categories may include animals, urban scenery, humans, natural pastoral scenery, and still life. Composition categories may include rule of thirds composition, vertical composition, horizontal composition, diagonal composition, curve composition, triangle composition, center composition, mirror composition, and stamp composition. Emotion categories may include joy, melancholy, and neutrality. Expressiveness categories may include whether it can inspire imagination or trigger deep thought. In the specific implementation process, the types and subcategories of the above categories may also be expanded.

[0070] It should be understood that image quality information may include noise, clarity, and exposure. Among them, noise can be calculated by visual noise, and the higher the value, the more noise. Clarity can be calculated by calculating the variance using the Laplace operator, and the higher the value, the clearer it is. Exposure can be calculated by histogram statistics, and too much low light is underexposure, and too much highlight is overexposure. Depth of field information may include far depth of field, medium depth of field, and near depth of field, which can be calculated in combination with blur detection results and other image processing methods.

[0071] It should be understood that color information can include color harmony information, i.e., the same color, complementary color, and adjacent color, by obtaining the image color RGB (red green blue) value and the existence of the same color, complementary color, or adjacent color in the regional range as a harmony reference. Among them, the same color refers to the combination of different brightness, such as dark red and light red, dark blue and light blue, dark green and light green, etc. Complementary colors refer to colors such as red and cyan, green and magenta, blue and yellow, etc., which can achieve a harmonious effect when properly matched. Adjacent colors refer to adjacent colors in spectral order, such as red and orange, orange and yellow, yellow and green, etc. Contrast can include high contrast, medium contrast, and low contrast, which can be calculated using existing image processing methods. Saturation can include high saturation, medium saturation, and low saturation, which can be calculated using existing image processing methods.

[0072] For example, refer to Figure 2 After the image is input into the image processing model, the image quality, depth of field, color harmony, contrast, saturation, composition category results, semantic category results, emotion category results and expression category results can be determined, and then the image evaluation information can be output based on these image category results and image parameter information. The image evaluation information may include the target image evaluation information and the first image evaluation information of the image under each image category, and the second image evaluation information corresponding to each image parameter information.

[0073] Through the above method, multi-dimensional information analysis and evaluation of images are performed through different image categories and image parameter information, which improves the accuracy of image evaluation, increases the interpretability of image evaluation information, and reduces the situation where the image evaluation information has low discrimination.

[0074] In one possible manner, step S102 may include: performing feature extraction on each image category result and each image parameter information to obtain multiple feature vectors, and performing vector concatenation on the multiple feature vectors to obtain an attribute feature vector; determining the attribute weight value corresponding to each image category result and each image parameter information through an attribute weight network in an image processing model; and determining the first image evaluation information, the second image evaluation information, and the target image evaluation information based at least on the attribute feature vector and the attribute weight value.

[0075] It should be understood that in the multiple feature vectors for feature extraction of each image category result and each image parameter information, the category and parameter information can be a one-to-many, one-to-one mapping relationship with the elements of the feature vector. For example, if the semantic category includes five categories, five elements can correspond to it in the feature vector, and natural numbers are used to represent different categories. Emotional categories, composition categories, and expressive categories can be similar to the mapping relationship between the categories of semantic categories and the elements in the feature vector. In addition, if the image quality information includes noise, clarity and exposure, these three sub-parameter information can correspond to an element of the feature vector respectively, and their values ​​are unnormalized values ​​that characterize the parameter information. The correspondence between the sub-parameter information of color information, depth of field information, contrast and saturation and the elements of the feature vector can also be similar to the image quality information. The preliminary integration of image category results and parameter information can be achieved by extracting and splicing features, which is the basis for realizing multi-dimensional evaluation of images.

[0076] In addition, it should be understood that the processing of feature vectors in the attribute weight network can be a linear transformation or a nonlinear transformation, and the attribute weight network can consider the impact of a single image category result or image parameter information on the final image evaluation information. The attribute weight network can further integrate and process the image category results and image parameter information, thereby preliminarily weighing the impact of the image category results and image parameter information on the image evaluation information, so as to quantify the importance of the image category results and image parameter information to the image evaluation, which is a further measure to achieve multi-dimensional evaluation of images.

[0077] In the disclosed embodiment, the attribute weight network in the image processing model can be used to determine the attribute weight value corresponding to each image category result and each image parameter information, so that the first image evaluation information, the second image evaluation information, and the target image evaluation information can be determined at least based on the attribute feature vector and the attribute weight value. For example, the first image evaluation information, the second image evaluation information, and the target image evaluation information can be determined based on the attribute feature vector and the attribute weight value. For another example, in order to reduce the feature dimension and improve the image processing efficiency, the image to be processed can be first feature extracted to obtain an image feature matrix, and then the first image evaluation information, the second image evaluation information, and the target image evaluation information can be determined based on the image feature matrix, the attribute feature vector, and the attribute weight value.

[0078] That is to say, in one possible way, the image processing model can also be used to extract features from the image to be processed through a pre-trained feature extraction network to obtain an image feature matrix. Accordingly, the first image evaluation information, the second image evaluation information, and the target image evaluation information are determined at least according to the attribute feature vector and the attribute weight value, including: performing convolution calculation on the attribute feature vector and the attribute weight value to obtain the target attribute feature vector; inputting the target attribute feature vector and the image feature matrix into the evaluation prediction network in the image processing model to obtain the first image evaluation information, the second image evaluation information, and the target image evaluation information.

[0079] It should be understood that the function of the feature extraction network can be to extract features and reduce the dimension of the image. The attribute feature vector and the attribute weight value are convolved to obtain the target attribute feature vector in the weight network, and the convolution calculation can be multiple convolution calculations. The feature extraction network can be a convolutional neural network, which can be pre-trained on the AVA (A Large-Scale Database for Aesthetic Visual Analysis, a large dataset for aesthetic visual assessment) dataset, or other datasets can be added to enrich the training set during the specific implementation process.

[0080] It should be understood that the evaluation prediction network can be a multi-layer convolutional neural network, which performs multiple convolution operations on the target attribute feature vector corresponding to the image category result and the image parameter information and the feature matrix obtained by the feature extraction network. On the one hand, the evaluation prediction network can quantify the performance of a single information of the image (such as a single image category result or a single image parameter information) and the mutual influence between the single information of the quantified image. On the other hand, the evaluation prediction network can quantify the comprehensive performance of multiple information of the image. Among them, the performance of the single information of the quantified image and the mutual influence between the information of the image can obtain the first image evaluation information and the second image evaluation information, and the comprehensive performance of the multiple information of the quantified image can obtain the target image evaluation information. Therefore, the embodiment of the present disclosure can obtain a target evaluation information for the image to be processed and the first image evaluation information of the image to be processed under each image category and the second image evaluation information corresponding to each image parameter information, thereby improving the interpretability of the image evaluation.

[0081] For example, refer to Figure 2 The feature extraction network is a convolutional neural network, which extracts features from each image category result and each image parameter information corresponding to the input image to obtain multiple feature vectors, and then concatenates multiple feature vectors to obtain an attribute feature vector. After that, the attribute feature vector is input into the attribute weight network. The attribute weight value corresponding to each image category result and each image parameter information is determined by the attribute weight network, and the target attribute feature vector is output according to the attribute feature vector and the attribute weight value. Then, the target attribute feature vector is input into the evaluation prediction network. At the same time, the input image is feature extracted by the pre-trained convolutional neural network to obtain an image feature matrix, and the image feature matrix is ​​input into the evaluation prediction network. Finally, the first image evaluation information, the second image evaluation information and the target image evaluation information are determined by the evaluation prediction network based on the image feature matrix and the target attribute feature vector.

[0082] Through the above method, the multi-dimensional image category results and image parameter information can be combined to obtain image evaluation information, which increases the analysis dimension, thereby improving the accuracy of image evaluation, increasing the interpretability of image evaluation information, and reducing the situation where the image evaluation information has low discrimination.

[0083] In one possible approach, the image processing model includes an image category network, an attribute weight network, and a prediction network, referring to Figure 3 , the image processing model can be trained through the following steps:

[0084] In step S301, the image category network is pre-trained to obtain an initial image processing model.

[0085] In step S302, a first sample image is obtained, wherein the first sample image is annotated with an image evaluation label, and the image evaluation label includes a target image evaluation label for the first sample image, a first image evaluation label of the first sample image under different image categories, and a second image evaluation label corresponding to the image parameter information of the first sample image.

[0086] In step S303, the first sample image is input into the initial image processing model to obtain a target prediction evaluation result corresponding to the first sample image, a first prediction evaluation result of the first sample image under each image category, and a second prediction evaluation result corresponding to each image parameter information.

[0087] In step S304, the parameters of the image processing model are adjusted according to the image evaluation label, the target prediction evaluation result, the first prediction evaluation result and the second prediction evaluation result.

[0088] It should be understood that the image category network may include a semantic classification network, a composition classification network, an emotion classification network, and an expressiveness network. Among them, the image category network is pre-trained, and the category labels need to be manually labeled. After the training is completed, the fixed parameters are placed in the image processing model. In other words, after the image category network is pre-trained, the training of the image processing model mainly trains the attribute weight network and the evaluation prediction network. During the training process, the EMDloss (EarthMover's Distanceloss) loss function can be used for back propagation, which is not limited to the embodiments of the present disclosure.

[0089] In one possible manner, the image category network includes an emotion classification network and / or an expressiveness classification network, and pre-training the image category network includes: obtaining a second sample image and / or a third sample image, wherein the second sample image is annotated with an emotion category label and the third sample image is annotated with an expressiveness category label; inputting the second sample image into the emotion classification network to obtain a predicted emotion category result corresponding to the second sample image, and adjusting parameters of the emotion classification network based on the predicted emotion category result and the emotion category label; and / or inputting the third sample image into the expressiveness classification network to obtain a predicted expressiveness category result corresponding to the third sample image, and adjusting parameters of the expressiveness classification network based on the predicted expressiveness category result and the expressiveness category label.

[0090] It should be understood that the emotion classification network and the expressiveness classification network can be pre-trained using a training set with manually annotated category labels. After the network is trained, the fixed parameters are placed in the image processing model.

[0091] In one possible manner, the image evaluation label, the emotion category label, and the expressiveness category label are obtained in the following manner: obtaining evaluation scores annotated by multiple first users for a first sample image, and performing weighted average calculation on the evaluation scores according to a first preset weight value to obtain a target image evaluation label for the first sample image; obtaining emotion category values ​​annotated by multiple second users for a second sample image, and performing weighted average calculation on the emotion category values ​​according to a second preset weight value to obtain an emotion category label for the second sample image; obtaining expressiveness category values ​​annotated by multiple third users for a third sample image, and performing weighted average calculation on the expressiveness category values ​​according to a third preset weight value to obtain an expressiveness category label for the third sample image.

[0092] It should be understood that users can be divided according to their photography level, such as ordinary users, entry-level photography users, and professional photography users. Weights can be assigned to the annotation scores given by different groups of people to take the weighted average as a label. The labels obtained by the above annotation method take into account the evaluation of the image by both the public aesthetics and professional photography. Among them, the first preset weight value corresponding to the first user, the second preset weight value corresponding to the second user, and the third preset weight value corresponding to the third user can be set according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0093] In addition, it should be understood that the emotion category value is used to represent the emotion category corresponding to the second sample image. For example, if the emotion categories include joy, depression, and neutrality, then the emotion category value of 2 can be used to represent the corresponding emotion category of joy, the emotion category value of 0 can be used to represent the corresponding emotion category of depression, and the emotion category value of 1 can be used to represent the corresponding emotion category of neutrality. The disclosed embodiment does not limit this. Based on this, after calculating the weighted average of the emotion category values ​​of different users, the emotion category label corresponding to the second sample image can be determined according to the emotion category that is closer to the weighted average.

[0094] Similarly, the expressiveness category value is used to represent the expressiveness category corresponding to the third sample image. For example, the expressiveness category may include two categories: whether it can stimulate imagination or trigger deep thinking. Then, the expressiveness category value of 1 may be used to represent that the corresponding expressiveness category is able to stimulate imagination or trigger deep thinking, and the expressiveness category value of 0 may be used to represent that the corresponding expressiveness category is unable to stimulate imagination or trigger deep thinking. The disclosed embodiment does not limit this. Based on this, after performing a weighted average calculation on the expressiveness category values ​​of different users, the emotion category label corresponding to the second sample image may be determined based on the expressiveness category that is closer to the weighted average.

[0095] By using any of the above image processing methods, the image category results and image parameter information can be combined to obtain image evaluation information, which increases the analysis dimension and thus improves the accuracy of image evaluation. In addition, the image processing model can output the first image evaluation information under each image category, the second image evaluation information corresponding to each image parameter information, and the target image evaluation information, which increases the interpretability of the image evaluation information and reduces the situation where the image evaluation information has low discrimination.

[0096] Figure 4 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Figure 4 The device includes an acquisition module 400 and a processing module 401.

[0097] The acquisition module 400 is configured to acquire an image to be processed;

[0098] The processing module 401 is configured to input the image to be processed into a trained image processing model to obtain image evaluation information corresponding to the image to be processed, wherein the image processing model is used to determine the image category result corresponding to the image to be processed and the image parameter information of the image to be processed, and output a target image evaluation information and the first image evaluation information of the image to be processed under each image category and the second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

[0099] Optionally, the processing module 401 is configured to:

[0100] Input the image to be processed into the trained image processing model, determine at least one of the following image categories corresponding to the image to be processed: semantic category, composition category, emotion category and expressiveness category, and determine at least one of the following image parameter information corresponding to the image to be processed: image quality information, depth of field information, color information, contrast and saturation;

[0101] Image evaluation information corresponding to the image to be processed is determined based on at least one image category and at least one image parameter information.

[0102] Optionally, the processing module 401 is configured to:

[0103] Perform feature extraction on each image category and each image parameter information to obtain multiple feature vectors, and perform vector concatenation on the multiple feature vectors to obtain an attribute feature vector;

[0104] Determine the attribute weight value corresponding to each image category and each image parameter information through the attribute weight network in the image processing model;

[0105] The first image evaluation information, the second image evaluation information and the target image evaluation information are determined at least according to the attribute feature vector and the attribute weight value.

[0106] Optionally, the image processing model is further used to extract features from the image to be processed through a pre-trained feature extraction network to obtain an image feature matrix, and the processing module 401 is configured as follows:

[0107] Perform convolution calculation on the attribute feature vector and the attribute weight value to obtain the target attribute feature vector;

[0108] The target attribute feature vector and the image feature matrix are input into the evaluation prediction network in the image processing model to obtain the first image evaluation information, the second image evaluation information and the target image evaluation information.

[0109] Optionally, the image processing model includes an image category network, an attribute weight network and a prediction network, and the image processing device further includes a training module, and the training module is configured to:

[0110] Pre-train the image category network to obtain the initial image processing model;

[0111] Acquire a first sample image, where the first sample image is annotated with an image evaluation label, wherein the image evaluation label includes a target image evaluation label for the first sample image, first image evaluation labels for the first sample image under different image categories, and a second image evaluation label corresponding to image parameter information of the first sample image;

[0112] Inputting the first sample image into the initial image processing model, obtaining a target prediction evaluation result corresponding to the first sample image, a first prediction evaluation result of the first sample image under each image category, and a second prediction evaluation result corresponding to each image parameter information;

[0113] Adjust the parameters of the image processing model according to the image evaluation label, the target prediction evaluation result, the first prediction evaluation result and the second prediction evaluation result.

[0114] Optionally, the image category network includes an emotion classification network and / or an expressiveness classification network, and the training module is configured to:

[0115] Acquire a second sample image and / or a third sample image, wherein the second sample image is annotated with an emotion category label, and the third sample image is annotated with an expressiveness category label;

[0116] Inputting the second sample image into the emotion classification network to obtain a predicted emotion category corresponding to the second sample image, and adjusting parameters of the emotion classification network according to the predicted emotion category and the emotion category label; and / or,

[0117] The third sample image is input into the expressiveness classification network to obtain a predicted expressiveness category corresponding to the third sample image, and the parameters of the expressiveness classification network are adjusted according to the predicted expressiveness category and the expressiveness category label.

[0118] Optionally, the image processing device further includes a label processing module, and the label processing module is configured to:

[0119] Obtaining evaluation scores of the first sample image annotated by multiple first users, and performing weighted average calculation on the evaluation scores according to a first preset weight value to obtain a target image evaluation label of the first sample image;

[0120] Obtaining emotion category values ​​annotated by multiple second users on the second sample image, and performing weighted average calculation on the emotion category values ​​according to a second preset weight value to obtain an emotion category label for the second sample image;

[0121] The expressiveness category values ​​annotated by multiple third users on the third sample image are obtained, and the expressiveness category values ​​are weighted averaged according to the third preset weight value to obtain the expressiveness category label of the third sample image.

[0122] Optionally, the image processing device is applied to an electronic device, and the acquisition module is configured to:

[0123] When the electronic device is in a shooting mode, a shooting preview image captured by the electronic device is obtained as an image to be processed.

[0124] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0125] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided by the present disclosure are implemented.

[0126] Figure 5 1 is a block diagram of an apparatus 500 for image processing according to an exemplary embodiment. For example, the apparatus 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0127] Reference Figure 5 , the device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output interface 512 , a sensor component 514 , and a communication component 516 .

[0128] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned image processing method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0129] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0130] The power supply component 506 provides power to the various components of the device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 500.

[0131] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0132] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the device 500 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0133] The input / output interface 512 provides an interface between the processing component 502 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0134] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500, the sensor assembly 514 can also detect the position change of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and the temperature change of the device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0135] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0136] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned image processing method.

[0137] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of the apparatus 500 to perform the above-mentioned image processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0138] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned image processing method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instruction may be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned image processing method may be implemented; alternatively, the integrated circuit or chip may receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned image processing method.

[0139] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned image processing method when executed by the programmable device.

[0140] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0141] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is input into a trained image processing model to obtain image evaluation information corresponding to the image to be processed, wherein the image processing model is used to determine an image category result corresponding to the image to be processed and image parameter information of the image to be processed, and output a target image evaluation information and a first image evaluation information of the image to be processed under each image category and a second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

2. The method according to claim 1, characterized in that The step of inputting the image to be processed into the trained image processing model to obtain image evaluation information corresponding to the image to be processed includes: Input the image to be processed into the trained image processing model, determine at least one of the following image category results corresponding to the image to be processed: semantic category result, composition category result, emotion category result and expressiveness category result, and determine at least one of the following image parameter information corresponding to the image to be processed: image quality information, depth of field information, color information, contrast and saturation; Image evaluation information corresponding to the image to be processed is determined based at least on the at least one image category result and the at least one image parameter information.

3. The method according to claim 2, characterized in that The determining, at least based on the at least one image category result and the at least one image parameter information, image evaluation information corresponding to the image to be processed includes: Performing feature extraction on each of the image category results and each of the image parameter information to obtain a plurality of feature vectors, and performing vector concatenation on the plurality of feature vectors to obtain an attribute feature vector; Determining the attribute weight value corresponding to each of the image category results and each of the image parameter information through the attribute weight network in the image processing model; The first image evaluation information, the second image evaluation information and the target image evaluation information are determined at least according to the attribute feature vector and the attribute weight value.

4. The method according to claim 3, characterized in that The image processing model is further used to extract features from the image to be processed through a pre-trained feature extraction network to obtain an image feature matrix, and the first image evaluation information, the second image evaluation information and the target image evaluation information are determined at least according to the attribute feature vector and the attribute weight value, including: Performing convolution calculation on the attribute feature vector and the attribute weight value to obtain a target attribute feature vector; The target attribute feature vector and the image feature matrix are input into the evaluation prediction network in the image processing model to obtain the first image evaluation information, the second image evaluation information and the target image evaluation information.

5. The method according to claim 1, characterized in that The image processing model includes an image category network, an attribute weight network and a prediction network, and the image processing model is trained in the following manner: Pre-training the image category network to obtain an initial image processing model; Acquire a first sample image, where the first sample image is annotated with an image evaluation label, wherein the image evaluation label includes a target image evaluation label for the first sample image, first image evaluation labels for the first sample image under different image categories, and a second image evaluation label corresponding to image parameter information of the first sample image; Inputting the first sample image into the initial image processing model, obtaining a target prediction evaluation result corresponding to the first sample image, a first prediction evaluation result of the first sample image under each of the image categories, and a second prediction evaluation result corresponding to each of the image parameter information; The parameters of the image processing model are adjusted according to the image evaluation label, the target prediction evaluation result, the first prediction evaluation result and the second prediction evaluation result.

6. The method according to claim 5, characterized in that The image classification network includes an emotion classification network and / or an expressiveness classification network, and the pre-training of the image classification network includes: Acquire a second sample image and / or a third sample image, wherein the second sample image is annotated with an emotion category label, and the third sample image is annotated with an expressiveness category label; Inputting the second sample image into the emotion classification network to obtain a predicted emotion category result corresponding to the second sample image, and adjusting the parameters of the emotion classification network according to the predicted emotion category result and the emotion category label; and / or, The third sample image is input into the expressiveness classification network to obtain a predicted expressiveness category result corresponding to the third sample image, and the parameters of the expressiveness classification network are adjusted according to the predicted expressiveness category result and the expressiveness category label.

7. The method according to claim 6, characterized in that The image evaluation label, the emotion category label and the expressiveness category label are obtained in the following manner: Obtaining evaluation scores of the first sample image annotated by multiple first users, and performing weighted average calculation on the evaluation scores according to a first preset weight value to obtain the target image evaluation label of the first sample image; Acquire emotion category values ​​annotated by multiple second users on the second sample image, and perform weighted average calculation on the emotion category values ​​according to a second preset weight value to obtain the emotion category label of the second sample image; The expressiveness category values ​​annotated by multiple third users on the third sample image are obtained, and the expressiveness category values ​​are weighted averaged according to a third preset weight value to obtain the expressiveness category label of the third sample image.

8. The method according to any one of claims 1 to 7, characterized in that: Applied to electronic equipment, the step of obtaining an image to be processed includes: When the electronic device is in a shooting mode, a shooting preview image captured by the electronic device is obtained as an image to be processed.

9. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; A processing module is used to input the image to be processed into a trained image processing model to obtain image evaluation information corresponding to the image to be processed, wherein the image processing model is used to determine the image category result corresponding to the image to be processed and the image parameter information of the image to be processed, and output a target image evaluation information and the first image evaluation information of the image to be processed under each image category and the second image evaluation information corresponding to each image parameter information at least based on the image category result and the image parameter information.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: The electronic device comprises: A processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method described in any one of claims 1-8 when executed.

12. A chip, characterized in that: The method comprises a processor and an interface; the processor is used to read instructions to execute the method according to any one of claims 1 to 8.

Images