Image processing method and device, storage medium and chip

Abstract

The invention relates to an image processing method and device, a storage medium and a chip, and relates to the technical field of aesthetic evaluation. The method includes: acquiring a to-be-processed picture; based on a pre-trained aesthetics evaluation model, determining an aesthetics score corresponding to the to-be-processed picture; the aesthetic evaluation model is obtained by training a training sample through a machine learning method, and the training sample comprises an image sample, a text description sample of the image sample and an aesthetic score sample corresponding to the image sample. By using the image processing method provided by the invention, the aesthetic score for evaluating the aesthetic degree of the to-be-processed picture can be obtained, so that quantitative analysis of the aesthetic degree of the to-be-processed picture is realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of aesthetic evaluation, and in particular to an image processing method, device, storage medium and chip. Background Art

[0002] Currently, in the field of aesthetic evaluation, aesthetic evaluation models can only determine whether the acquired picture is beautiful, but cannot perform a quantitative aesthetic evaluation of the picture.

[0003] For example, when a user asks whether a picture is beautiful, the aesthetic evaluation model can only output an answer of whether the picture is beautiful or not, but cannot output the specific degree of beauty of the picture. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, storage medium and chip.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Get the screen to be processed;

[0007] Based on a pre-trained aesthetic evaluation model, an aesthetic score corresponding to the image to be processed is determined; the aesthetic evaluation model is trained using a machine learning method using training samples, and the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

[0008] Optionally, determining the aesthetic score corresponding to the picture to be processed based on a pre-trained aesthetic evaluation model includes:

[0009] Based on the aesthetic evaluation model, a first aesthetic score corresponding to the picture to be processed is determined, wherein the first aesthetic score is a score used to reflect the clarity of the picture subject of the picture to be processed; the text description sample includes a first text description sample, and the first text description sample is used to describe the picture content in the image sample; the aesthetic score sample includes a first aesthetic score sample, and the first aesthetic score sample is a score used to reflect the clarity of the picture subject in the image sample.

[0010] Optionally, determining the aesthetic score corresponding to the picture to be processed based on a pre-trained aesthetic evaluation model includes:

[0011] Based on the aesthetic evaluation model, a second aesthetic score corresponding to the picture to be processed is determined, and the second aesthetic score is a score used to reflect the visual effect of the picture to be processed; the text description sample includes a second text description sample, and the second text description sample is used to describe the visual effect of the image sample; the aesthetic score sample includes a second aesthetic score sample, and the second aesthetic score sample is a score used to reflect the visual effect of the image sample.

[0012] Optionally, determining the aesthetic score corresponding to the picture to be processed based on a pre-trained aesthetic evaluation model includes:

[0013] Based on the aesthetic evaluation model, a third aesthetic score corresponding to the picture to be processed is determined, and the third aesthetic score is a score used to reflect the degree of expression of the photographic elements of the picture to be processed; the text description sample includes a third text description sample, and the third text description sample is used to describe the photographic elements of the image sample; the aesthetic score sample includes a third aesthetic score sample, and the third aesthetic score sample is a score used to reflect the degree of expression of the photographic elements.

[0014] Optionally, determining the aesthetic score corresponding to the picture to be processed based on a pre-trained aesthetic evaluation model includes:

[0015] Based on the aesthetic evaluation model, determining a first aesthetic score, a second aesthetic score, and a third aesthetic score corresponding to the picture to be processed;

[0016] The aesthetic score is obtained according to the first aesthetic score, the second aesthetic score and the third aesthetic score.

[0017] Optionally, obtaining the aesthetic score according to the first aesthetic score, the second aesthetic score and the third aesthetic score includes:

[0018] The aesthetic score is obtained according to the first aesthetic score, the first weight corresponding to the first aesthetic score, the second aesthetic score, the first weight corresponding to the second aesthetic score, the third aesthetic score and the weight of the third aesthetic score.

[0019] Optionally, the method further comprises:

[0020] When the aesthetic score reaches a threshold, the picture to be processed displayed in the preview picture of the terminal is photographed to obtain a target image.

[0021] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0022] An acquisition module is configured to acquire a picture to be processed;

[0023] The score determination module is configured to determine the aesthetic score corresponding to the image to be processed based on a pre-trained aesthetic evaluation model; wherein the aesthetic evaluation model is trained using a machine learning method using training samples, and the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

[0024] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0025] processor;

[0026] a memory for storing processor-executable instructions;

[0027] Wherein, the processor is configured to:

[0028] Execute the steps of the image processing method provided in the first aspect of the present disclosure.

[0029] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided in the first aspect of the present disclosure are implemented.

[0030] According to a fifth aspect of an embodiment of the present disclosure, a chip is provided, including a processor and an interface; the processor is used to read instructions to execute the steps of the image processing method provided in the first aspect of the present disclosure.

[0031] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0032] The aesthetic score of the image to be processed can be obtained through a pre-trained aesthetic evaluation model, so as to quantitatively evaluate the aesthetic expression of the image to be processed based on the aesthetic score; and the aesthetic evaluation model is trained using image samples, text description samples and aesthetic score samples as training samples, which can help the aesthetic evaluation model understand the intrinsic connection between the aesthetic score and the quality of the image sample.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0036] Figure 2 It is a schematic diagram showing a screen to be processed in a preview interface of a terminal according to an exemplary embodiment.

[0037] Figure 3 is a schematic diagram of an image sample according to an exemplary embodiment.

[0038] Figure 4 is a schematic diagram showing an image to be processed according to an exemplary embodiment.

[0039] Figure 5 is a schematic diagram of an image sample according to an exemplary embodiment.

[0040] Figure 6 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0041] Figure 7 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0042] Figure 8 is a block diagram of an image processing apparatus according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0044] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0045] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.

[0046] In step S11, a picture to be processed is obtained.

[0047] The picture to be processed may be a picture of an image input after being photographed by a user, or may be a picture that appears in a preview interface of a terminal when a user uses the terminal to take a picture.

[0048] For example, when a user uses a question-and-answer chat software (chatgpt) on the Internet to ask whether the picture is beautiful, the user can upload the image to the chat box of the question-and-answer chat software and send it to the question-and-answer chat software to obtain the picture to be processed.

[0049] For example, see Figure 2 As shown, when a user is taking a photo with a mobile phone, the picture captured by the terminal camera will appear in the preview interface of the mobile phone, and the picture in the preview interface is the picture to be processed.

[0050] In step S12, based on a pre-trained aesthetic evaluation model, the aesthetic score corresponding to the image to be processed is determined; the aesthetic evaluation model is trained using a machine learning method using training samples, and the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

[0051] The aesthetic evaluation model can be a large language model (LLM). A convolutional neural network can be used to extract features from the input image to be processed to obtain image features; the feature vector of the image is input into the aesthetic evaluation model to generate a text description related to the image content. In this process, the aesthetic evaluation model can achieve alignment and interaction between vision (image) and language (text description), better understand and process the relationship between vision and language, and improve the understanding of images.

[0052] The aesthetic evaluation model includes a pre-training stage and a fine-tuning stage.

[0053] The pre-training stage refers to the aesthetic evaluation model learning the features of image samples and text description samples through self-supervision on a large scale of image samples and text description samples. During the pre-training stage, the aesthetic evaluation model will observe a large number of text description samples and try to predict missing words or sentences in the text description samples, or understand the words in the sentences through the context, thereby helping the aesthetic evaluation model learn the semantics, grammar and context understanding capabilities of the text description samples, so that the aesthetic evaluation model can generate coherent text and understand human language.

[0054] The fine-tuning stage refers to further optimizing the aesthetic evaluation model after pre-training so that the aesthetic evaluation model can adapt to the aesthetic score calculation task. In the fine-tuning stage, the image samples, the text description samples corresponding to the image samples, and the aesthetic score samples corresponding to the image samples will be used to train the aesthetic evaluation model so that some network parameters such as weights and thresholds in the aesthetic evaluation model can be adjusted to better adapt to the aesthetic score calculation task. The focus of the present disclosure is on fine-tuning some network parameters of the aesthetic evaluation model through image samples, the text description samples of the image samples, and the aesthetic score samples corresponding to the image samples during the fine-tuning stage of the aesthetic evaluation model so that the adjusted aesthetic evaluation model can be more suitable for the aesthetic score calculation task of the present disclosure. The aesthetic score calculation task is used to calculate the aesthetic score of the image to be processed.

[0055] For example, image samples can be used as input parameters of the aesthetic evaluation model, and text description samples and aesthetic score samples can be used as labels of the aesthetic evaluation model. The aesthetic evaluation model outputs text descriptions and aesthetic scores. A first error between the text description output by the aesthetic evaluation model and the text description samples used as labels, as well as a second error between the aesthetic score output by the aesthetic evaluation model and the aesthetic score samples used as labels are determined. The above steps are repeatedly performed. When the first error is less than a first preset value and the second error is less than a second preset value, it means that the error between the text description and aesthetic score output by the aesthetic evaluation model and the accurate label is small, and the text description and aesthetic score output by the aesthetic evaluation model are accurate. At this time, it can be determined that the fine-tuning stage of the aesthetic evaluation model is over.

[0056] Image samples refer to images used to train the aesthetic evaluation model; text description samples are true and accurate text descriptions corresponding to manually annotated image samples. Text description samples are used to describe the clarity, visual effects, and photographic elements of the image subject, thereby reflecting the semantic content of the image sample and the ideas that the image sample wants to reflect; aesthetic score samples are true and accurate aesthetic scores corresponding to manually annotated image samples. The aesthetic score samples reflect the aesthetic degree of the image sample. For example, the higher the aesthetic score sample, the more beautiful the image sample is, and the clarity, visual effects, and photographic elements of the image subject in the image sample are better.

[0057] Of course, in addition to outputting the aesthetic score, the aesthetic evaluation model can also output the reason for obtaining the aesthetic score, so that the user can know the basis on which the aesthetic evaluation model obtains the aesthetic score. Specifically, in addition to outputting the aesthetic score of the image to be processed, the aesthetic evaluation model can also explain the reason for obtaining the aesthetic score in terms of the clarity of the subject of the image to be processed, the visual effects, and the degree of expression of photographic elements.

[0058] For example, the score range of the aesthetic evaluation model is [0, 10], and the aesthetic evaluation model gives the picture to be processed 8 points. And from the clarity of the main body of the picture, it is shown that the main body of the picture to be processed is clear, and it gets 2 points; from the visual effect of the picture, it is shown that the visual effect of the picture to be processed is very good, and it gets 4 points; from the expression level of the photographic elements of the picture, it is shown that the expression level of the photographic elements of the picture to be processed is high, and it gets 2 points. Finally, the aesthetic score of the picture to be processed is 8 points. The score range of the aesthetic evaluation model can be [0, 10] or [0, 100], and the present disclosure does not limit this.

[0059] It is understandable that the aesthetic score samples can be independent of the text description samples. The aesthetic score samples and the text description samples can be used as two training labels to train the aesthetic evaluation model respectively. The aesthetic score samples can also be embedded in the text description samples as a training label to train the aesthetic evaluation model. After the aesthetic evaluation model is trained with the latter, the text description output by the aesthetic evaluation model is mixed with the aesthetic score. For example, the main body of the picture is clear and can get 2 points.

[0060] The related technology cannot evaluate the aesthetic scores of different images to be processed. Users can only get the answer of whether each image to be processed is beautiful or not. When the user inputs the image A to be processed into the question-and-answer chat software, the question-and-answer chat software can only answer that the image A to be processed is very beautiful; when the user inputs the image B to be processed into the question-and-answer chat software, the question-and-answer chat software can only answer that the image B to be processed is also beautiful. However, the user cannot compare which image A to be processed is more beautiful than the image B to be processed based on the reply of the question-and-answer chat software.

[0061] In order to quantitatively obtain the aesthetic score of the image to be processed, the image samples can be labeled with scores based on a data-driven deep learning algorithm. After the image to be processed is input into the deep learning algorithm, the aesthetic score can be predicted. However, the data quality of the image samples used to train the deep learning algorithm is not high, and its generalization is poor.

[0062] Therefore, the present disclosure first proposes to use a large language aesthetic evaluation model to evaluate the aesthetic scores of the processed images, which utilizes the extensive world knowledge of the aesthetic evaluation model and is no longer limited to the above-mentioned image samples with low data quality, thereby solving the problems of low data quality and poor generalization.

[0063] Of course, the aesthetic score can also be obtained through the aesthetic evaluation model in a question-and-answer manner, but the aesthetic evaluation model can only output the aesthetic score, and it cannot understand the intrinsic relationship between the aesthetic score and the actual quality of the image to be processed.

[0064] Therefore, the present disclosure proposes to use three types of samples, namely, image samples, text description samples and aesthetic score samples, to train the aesthetic evaluation model. It can not only train the aesthetic evaluation model to obtain aesthetic scores, but also train the aesthetic evaluation model to learn the language description of the quality of the image sample when obtaining the aesthetic score. The combination of text description samples and aesthetic score samples can train the aesthetic evaluation model to learn the intrinsic connection between the quality of image samples and the aesthetic scores.

[0065] For example, the aesthetic score sample can be set to 8 points, and the text description sample corresponding to the image sample is "The picture depicts a little boy holding a watermelon. He is looking at the camera with a smile on his face. He should be sitting at a table full of fruits. Behind him are two little girls who are feeding each other. The whole environment seems to be in a farmyard. The sun shines on people. Everything is beautiful and harmonious" to train the aesthetic evaluation model. Therefore, the aesthetic evaluation model can learn that the clearer the main body of the picture, the better the visual effect, and the higher the degree of expression of photographic elements in the picture to be processed, the higher the aesthetic score sample will be, thereby understanding the intrinsic connection between the quality of the image sample and the aesthetic score sample.

[0066] It can be understood that the text description samples serve as a bridge between the image samples and the aesthetic score samples, thereby establishing an intrinsic connection between the image samples and the aesthetic score samples.

[0067] Through the above technical scheme, the aesthetic score of the image to be processed can be obtained through a pre-trained aesthetic evaluation model, so that the aesthetic expression degree of the image to be processed can be quantitatively evaluated. For example, it can be known that the aesthetic score of the image to be processed A is 8 points, and the aesthetic score of the image to be processed B is 6 points, so that the user can know that the image to be processed B is more beautiful than the image to be processed A; and training the aesthetic evaluation model with image samples, text description samples and aesthetic score samples as training samples can help the aesthetic evaluation model understand the intrinsic connection between the aesthetic score and the quality of the image sample.

[0068] The following describes a specific embodiment involved in the above step S12, wherein the aesthetic score sample involved in step S12 includes a first aesthetic score sample, and the text description sample includes a first text description sample. This embodiment is used to explain that the aesthetic evaluation model can obtain the first aesthetic score according to the clarity of the main body of the picture in the to-be-processed picture.

[0069] Based on the aesthetic evaluation model, a first aesthetic score corresponding to the picture to be processed is determined, where the first aesthetic score is a score used to reflect the clarity of the main body of the picture to be processed.

[0070] Since the aesthetic evaluation model can obtain the first aesthetic score of the image to be processed according to the clarity of the main body of the image, when training the aesthetic evaluation model, it is necessary to use the first text description sample and the first aesthetic score sample as training labels so that the aesthetic evaluation model has the ability to obtain the first aesthetic score according to the clarity of the main body of the image.

[0071] The first text description sample is used to describe the picture content in the image sample, and the picture content includes at least one of the picture subject, secondary objects, foreground, background, positional relationship between the picture subject and secondary objects, positional relationship between the foreground and the background, action and expression of the picture subject, action and expression of the secondary objects, etc. in the image sample.

[0072] For example, see Figure 3 The image samples shown, the aesthetic assessment model can identify Figure 3 The first text description sample corresponding to the objects in the box, such as the little boy, watermelon, little girl, and fruit on the table, is: The picture depicts a little boy holding a watermelon (the main body and foreground of the picture), looking at the camera with a smile on his face (the action and expression of the main body of the picture). He should be sitting at the table (the position relationship between the main body of the picture and the secondary object), and the table is full of fruits. There are two little girls behind him (the relationship between the foreground and the background), feeding each other (the action and expression of the secondary object).

[0073] The first aesthetic score sample is a score used to reflect the clarity of the subject of the picture in the image sample. The maximum score of the first aesthetic score sample can be 2 points. Whether the subject of the picture is prominent or clear can be used as a sign to evaluate the clarity of the subject of the picture. For a clear subject, 2 points can be given, for a less clear subject, 1 point can be given, and for an unclear subject, 0 points can be given. Of course, 1.5 points and 0.5 points can be given depending on the situation, but this is not shown in the present disclosure.

[0074] For example, see Figure 3 As shown, Figure 3 The little boy holding a watermelon is the main subject of the picture, and the main subject of the picture is clear, so Figure 3 The corresponding first aesthetic score should be 2 points.

[0075] Of course, since the first text description sample is also used as a training label to train the aesthetic evaluation model, the aesthetic evaluation model can output the reason for obtaining the first aesthetic score while outputting the first aesthetic score. Figure 4 After processing the image shown, the aesthetic assessment model can identify Figure 4The first aesthetic score of the image to be processed is 0, and it is explained that the main body of the image to be processed is not prominent and the main body of the image is not clear. The reason why the main body of the image is not prominent is that Figure 4 The picture to be processed shown does not have a subject that catches the viewer's eye, and the primary and secondary objects are not clear; the reason why the subject of the picture is not clear is that without the subject of the picture, there is no question of whether the subject of the picture is clear.

[0076] Through the above technical scheme, the first text description sample describing the picture content of the image sample and the first aesthetic score sample reflecting the clarity of the picture subject in the image sample can be used as labels to train the aesthetic evaluation model. In this way, the trained aesthetic evaluation model can obtain the first aesthetic score describing the clarity of the picture subject of the picture to be processed after acquiring the image to be processed; and training the aesthetic evaluation model with the first text description sample and the first aesthetic score sample can enable the aesthetic evaluation model to learn the intrinsic connection between the first text description sample and the first aesthetic score sample, thereby obtaining a more accurate aesthetic score.

[0077] For example Figure 3 As shown, the first text description sample "The picture depicts a little boy holding a watermelon. He is looking at the camera with a smile on his face. He should be sitting at a table full of fruits. Behind him are two little girls who are feeding each other. The whole environment seems to be in a farmyard. The sun shines on people. Everything is beautiful and harmonious" will correspond to a higher first aesthetic score sample, that is, the aesthetic evaluation model will learn that the intrinsic connection between the quality of the image sample and the first aesthetic score sample is that the image sample has a higher degree of picture theme clarity. The better the image sample, the higher the first aesthetic score sample. When the aesthetic evaluation model receives a picture to be processed that is similar to the picture described by the first text description sample again, it will output a higher aesthetic score and the reason for the higher aesthetic score.

[0078] The following describes a specific embodiment of step S12, wherein the aesthetic score sample involved in step S12 includes a second aesthetic score sample, and the text description sample includes a second text description sample. This embodiment is used to explain that the aesthetic evaluation model can obtain the second aesthetic score according to the visual effect in the picture to be processed.

[0079] Based on the aesthetic evaluation model, a second aesthetic score corresponding to the picture to be processed is determined, where the second aesthetic score is a score used to reflect the visual effect of the picture to be processed.

[0080] Since the aesthetic evaluation model can obtain the second aesthetic score of the picture to be processed according to the visual effect of the picture, when training the aesthetic evaluation model, it is necessary to use the second text description sample and the second aesthetic score sample as training labels so that the aesthetic evaluation model has the ability to obtain the second aesthetic score according to the visual effect of the picture.

[0081] The second text description sample is used to describe the visual effect of the image sample. The visual effect includes at least one of color, light and shadow, composition and image quality. The second text description sample includes an advantage description of the visual effect and a disadvantage description of the visual effect. The advantage description is used to describe at least one of color, light and shadow, composition and image quality, and the disadvantage description is also used to describe at least one of color, light and shadow, composition and image quality.

[0082] For example, see Figure 3 As shown, Figure 3 The corresponding second text description sample includes a description of the advantages of visual effects as follows.

[0083] Color: The picture is rich in colors and high in saturation. The little boy is wearing blue clothes, which form complementary colors with the red / yellow of the fruit. The combination of cool and warm colors is harmonious and full of life.

[0084] Light and shadow: It can be seen that the photo was taken on a sunny day. The boy's face is exposed to the sun, while the fruit is in the shadow. The little girl behind is also in the sun, while the environment inside the house is in the shadow. The use of light and shadow makes the main body of the picture stand out and the layers are clear.

[0085] Composition: The little boy in the foreground and the little girl in the background form a diagonal composition, which echoes each other, making the picture concise yet full of life.

[0086] It is understandable that if color, light, shadow and composition appear in the description of the advantages of the second text description sample, they cannot continue to appear in the description of the disadvantages of the second text description sample. Similarly, if color, light, shadow and composition appear in the description of the disadvantages, they cannot continue to appear in the description of the advantages.

[0087] For example, if the second text description sample describes the advantages of color, then it cannot continue to describe the disadvantages of color; similarly, if the second text description sample describes the advantages of light and shadow, then it cannot continue to describe the disadvantages of color.

[0088] The second aesthetic score sample is a score used to reflect the visual effect of the image sample. The second aesthetic score sample is the sum of the color score, the light and shadow score, the composition score and the image quality score. The maximum score of the second aesthetic score sample can be 4 points. For color, if the color appears in the description of the advantages, the color score is 1 point, and if the color appears in the description of the disadvantages, the color score is 0 point. For the light and shadow score, if the light and shadow appears in the description of the advantages, the light and shadow score is 1 point, and if the light and shadow appears in the description of the disadvantages, the light and shadow score is 0 point. For the composition score, if the composition appears in the description of the advantages, the composition score is 1 point, and if the composition appears in the description of the disadvantages, the composition score is 0 point. In addition, among the color score, the light and shadow score and the composition score, if the second text description sample does not describe the advantages and disadvantages of the color, light and shadow or composition, the three items are scored 0.5 points respectively. For the image quality score, if the image quality is normal, it is scored 1 point, if the image quality is slightly worse, it is scored 0.5 points, and if the image quality is very bad, it is scored 0 points.

[0089] For example, see Figure 3 As shown, Figure 3 The colors are rich, the light and shadow are beautiful, and the diagonal composition is Figure 3 The picture quality is clear, that is, the color, light and shadow, composition and picture quality have all reached the best, so the second aesthetic score sample of this image sample is 4 points.

[0090] Of course, since the second text description sample is used as a training label to train the aesthetic evaluation model, the aesthetic evaluation model can output the reason for obtaining the second aesthetic score while obtaining the second aesthetic score. Figure 4 After the image to be processed is shown, the aesthetic evaluation model can obtain a second aesthetic score of 1 for the image to be processed, and explains that since the image to be processed has poor colors, ugly light and shadow, no composition, and clear picture quality, only the picture quality got 1 point, and the other colors, light and shadow, and composition had no scores.

[0091] Through the above technical scheme, the second text description sample describing the visual effect of the image sample and the second aesthetic score sample reflecting the clarity of the main body of the picture in the image sample can be used as labels to train the aesthetic evaluation model. In this way, the trained aesthetic evaluation model can obtain the second aesthetic score that describes the visual effect of the picture to be processed after acquiring the image to be processed; and training the aesthetic evaluation model with the second text description sample and the second aesthetic score can enable the aesthetic evaluation model to learn the intrinsic connection between the second text description sample and the second aesthetic score sample, thereby obtaining a more accurate aesthetic score.

[0092] For example Figure 3As shown, the second text description sample "The picture depicts a little boy holding a watermelon. He is looking at the camera with a smile on his face. He should be sitting at a table full of fruits. There are two little girls behind him, feeding each other. The whole environment seems to be in a farmyard. The sun shines on people. Everything is beautiful and harmonious" will correspond to a higher second aesthetic score sample, that is, the aesthetic evaluation model will learn the intrinsic connection between the quality of the image sample and the second aesthetic score sample, that is, the better the visual effect of the image sample, the better the image sample, and the higher the second aesthetic score sample. When the aesthetic evaluation model receives a to-be-processed picture similar to the picture described by the second text description sample again, it will output a higher aesthetic score and the reason for the higher aesthetic score.

[0093] The following describes a specific embodiment involved in the above step S12, wherein the aesthetic score sample involved in step S12 includes a third aesthetic score sample, and the text description sample includes a third text description sample. This embodiment is used to explain that the aesthetic evaluation model can obtain the third aesthetic score according to the degree of expression of the photographic elements of the image to be processed.

[0094] Based on the aesthetic evaluation model, a third aesthetic score corresponding to the picture to be processed is determined, and the third aesthetic score is a score used to reflect the degree of expression of the photographic elements of the picture to be processed.

[0095] Since the aesthetic evaluation model can obtain the third aesthetic score of the picture to be processed according to the degree of expression of the photographic elements of the picture to be processed, when training the aesthetic evaluation model, it is necessary to use the third text description samples and the third aesthetic score samples as training labels so that the aesthetic evaluation model has the ability to obtain the third aesthetic score according to the degree of expression of the photographic elements of the picture to be processed.

[0096] The third text description sample is used to describe the photographic elements of the image sample. The photographic elements include: at least one of the difficulty coefficient of photographing the image sample, the story of the image sample, the connotation of the image sample, the expressiveness of the image sample, the aesthetic taste of the image sample, and the photographic skills of the image sample. The shooting difficulty coefficient refers to the time cost spent by the photographer to shoot the image sample. The higher the time cost, the greater the shooting difficulty coefficient. The story of the image sample refers to whether the image sample is imaginative. For example, please refer to Figure 5 As shown, the aesthetic recognition model can identify Figure 5The person in the box of the sample image shown in the picture will make people wonder whether the protagonist in the image is waiting for someone. The connotation of the sample image refers to whether the image will make people think deeply. For example, a sample image shows a messy kitchen behind a takeaway, which will make people think deeply about the benefits of the fast-moving consumer goods industry. The expressiveness of the sample image refers to the information, emotions and opinions that the sample image can express. For example, a sample image shows the interaction between a cat and a person, which shows the intimacy between the cat and the person.

[0097] For example, see Figure 3 The image sample shown, its corresponding third text description can be: the whole environment seems to be in a farmyard, the sun shines on people, everything is beautiful and harmonious, then the image sample reflects the expressiveness of the image sample, expressing the beautiful and harmonious information of farm life.

[0098] The third aesthetic score sample is a score used to reflect the degree of expression of the photographic element. The maximum score of the third aesthetic score can be 4 points. For each photographic element in the photographic elements, if the photographic element is presented, 0.5 points can be given, and if the photographic element is not presented, 0 points can be given.

[0099] For example, see Figure 3 As shown, it expresses the two photographic elements of the image sample, namely, expressiveness and photographic skills, so the third aesthetic score sample of the image sample is 1 point.

[0100] Of course, since the third text description sample is used as a training label to train the aesthetic evaluation model, the aesthetic evaluation model can output the reason for obtaining the third aesthetic score while obtaining the third aesthetic score. Figure 4 After the image to be processed is shown, the aesthetic evaluation model can obtain the third aesthetic score of the image to be processed is 0.5 points, and explains that it is because the image shows a noisy living room environment and has the photographic element of the expressiveness of the image sample, but does not reflect the other photographic elements, so it has 0.5 points.

[0101] Through the above technical scheme, the third text description sample describing the photographic elements of the image sample and the third aesthetic score sample reflecting the degree of expression of the photographic elements of the image sample can be used as labels to train the aesthetic evaluation model. In this way, the trained aesthetic evaluation model can obtain the third aesthetic score describing the degree of expression of the photographic elements of the image to be processed after acquiring the image to be processed; and training the aesthetic evaluation model with the third text description sample and the third aesthetic score sample can enable the aesthetic evaluation model to learn the intrinsic connection between the third text description sample and the third aesthetic score sample, thereby obtaining a more accurate aesthetic score.

[0102] For example Figure 5 As described above, the third text description sample "The picture depicts a little boy holding a watermelon. He is looking at the camera with a smile on his face. He should be sitting at a table full of fruits. Behind him are two little girls who are feeding each other. The whole environment seems to be in a farmyard. The sun shines on people. Everything is beautiful and harmonious" will correspond to a higher third aesthetic score sample, that is, the aesthetic evaluation model will learn that the intrinsic connection between the quality of the image sample and the third aesthetic score sample is that the higher the degree of expression of the photographic elements of the image sample, the better the image sample, and the higher the third aesthetic score sample. When the aesthetic evaluation model receives a to-be-processed picture similar to the picture described by the third text description sample again, it will output a higher aesthetic score and the reason for the higher aesthetic score.

[0103] After obtaining the first aesthetic score sample, the second aesthetic score sample and the third aesthetic score sample, the annotator may use the sum of the first aesthetic score sample, the second aesthetic score sample and the third aesthetic score sample as the aesthetic score sample corresponding to the image sample.

[0104] Since different annotators have different aesthetic tastes, the following two annotation methods can be used to obtain aesthetic score samples.

[0105] Labeling method 1: Multiple different staff members label the aesthetic score samples of the same image sample to obtain multiple aesthetic scores, and then perform weighted averaging on the multiple aesthetic scores and the weights corresponding to the multiple aesthetic scores to obtain the final aesthetic score of the image sample.

[0106] The corresponding weights of multiple aesthetic scores depend on the aesthetic level of the annotator who annotates the aesthetic score. Therefore, the higher the aesthetic level of the annotator, the higher the corresponding weight of the aesthetic score annotated by the annotator.

[0107] The aesthetic level of the annotator depends on the annotator's education level, personality, photography experience, etc. Generally speaking, the higher the education level and photography experience of a person, the higher the aesthetic level of the annotator, and naturally the higher the weight of the aesthetic score annotated by the annotator.

[0108] Labeling method 2: One labeler labels the aesthetic score of an image sample, and finally an labeler with a higher aesthetic level will correct all the aesthetic scores.

[0109] The following describes a specific embodiment of the above step S12. After obtaining the first aesthetic score, the second aesthetic score and the third aesthetic score, the aesthetic score can be obtained according to the first aesthetic score, the second aesthetic score and the third aesthetic score, including the following two methods:

[0110] Method 1: The sum of the first aesthetic score, the second aesthetic score and the third aesthetic score is taken as the final aesthetic score.

[0111] For example, Figure 4 The first aesthetic score is 0, the second aesthetic score is 1, and the third aesthetic score is 0.5. Then the final aesthetic score of the image to be processed is 1.5.

[0112] Method 2: Obtain the aesthetic score according to the first aesthetic score, the first weight corresponding to the first aesthetic score, the second aesthetic score, the first weight corresponding to the second aesthetic score, the third aesthetic score and the weight of the third aesthetic score.

[0113] Multiply the first aesthetic score by the first weight to obtain a first value; multiply the second aesthetic score by the second weight to obtain a second value; multiply the third aesthetic score by the third weight to obtain a third value; finally, the sum of the first value, the second value and the third value is taken as the aesthetic score.

[0114] For example, Figure 4 The first aesthetic score is 0, the first weight is 0.4, the second aesthetic score is 1, the second weight is 0.4, the third aesthetic score is 0.5, and the third weight is 0.2, then the final aesthetic score is 0.5.

[0115] Of course, the relationship between the first weight, the second weight and the third weight can be set according to actual conditions. For example, if the user pays more attention to the first aesthetic score, the first weight can be set larger; if the user pays more attention to the second aesthetic score, the second weight can be set larger. The present disclosure does not impose any restrictions on this.

[0116] Through the above technical scheme, the aesthetic score output by the final aesthetic evaluation model can be obtained by combining the first aesthetic score, the second aesthetic score and the third aesthetic score. It can also be understood that the aesthetic score of the image to be processed is obtained by combining factors such as the clarity of the main body of the image to be processed, the visual effect of the image, and the degree of expression of the photographic elements of the image. Therefore, the aesthetic score obtained is more accurate.

[0117] The following introduces an optional embodiment involved in the above-mentioned step S11 and step S12, which is used to combine the aesthetic score with the shooting scene.

[0118] When the aesthetic score reaches a threshold, the picture to be processed displayed in the preview picture of the terminal is photographed to obtain a target image.

[0119] See also Figure 2As shown, the terminal can obtain the aesthetic score of the picture to be processed in the preview screen through the aesthetic recognition model. When the aesthetic score reaches the threshold, it means that the aesthetic level of the picture to be processed is high. At this time, it can respond to the user's photo-taking instruction to shoot the picture to be processed and obtain the target image. In this way, the main body of the target image is clearer, the visual effect is better, and the photographic elements it expresses are richer, thereby enhancing the user's experience of using the terminal camera.

[0120] Figure 6 is a block diagram of an image processing device according to an exemplary embodiment. Figure 6 The image processing device 600 includes: an acquisition module 620 and a score determination module 630.

[0121] The acquisition module 620 is configured to acquire the picture to be processed;

[0122] The score determination module 630 is configured to determine the aesthetic score corresponding to the image to be processed based on a pre-trained aesthetic evaluation model; wherein the aesthetic evaluation model is trained using a machine learning method using training samples, and the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

[0123] Optionally, the score determination module 630 includes:

[0124] The first aesthetic score determination submodule is configured to determine a first aesthetic score corresponding to the picture to be processed based on the aesthetic evaluation model, the first aesthetic score being a score used to reflect the clarity of the picture subject of the picture to be processed; the text description sample includes a first text description sample, the first text description sample is used to describe the picture content in the image sample; the aesthetic score sample includes a first aesthetic score sample, the first aesthetic score sample is a score used to reflect the clarity of the picture subject in the image sample.

[0125] Optionally, the score determination module 630 includes:

[0126] The second aesthetic score determination submodule is configured to determine a second aesthetic score corresponding to the picture to be processed based on the aesthetic evaluation model, wherein the second aesthetic score is a score used to reflect the visual effect of the picture to be processed; the text description sample includes a second text description sample, and the second text description sample is used to describe the visual effect of the image sample; the aesthetic score sample includes a second aesthetic score sample, and the second aesthetic score sample is a score used to reflect the visual effect of the image sample.

[0127] Optionally, the score determination module 630 includes:

[0128] The second aesthetic score determination submodule is configured to determine a third aesthetic score corresponding to the image to be processed based on the aesthetic evaluation model, wherein the third aesthetic score is a score used to reflect the degree of expression of the photographic elements of the image to be processed; the text description sample includes a third text description sample, and the third text description sample is used to describe the photographic elements of the image sample; the aesthetic score sample includes a third aesthetic score sample, and the third aesthetic score sample is a score used to reflect the degree of expression of the photographic elements.

[0129] Optionally, the score determination module 630 includes:

[0130] A prediction submodule, configured to determine a first aesthetic score, a second aesthetic score, and a third aesthetic score corresponding to the picture to be processed based on the aesthetic evaluation model;

[0131] The first calculation submodule is configured to obtain the aesthetic score according to the first aesthetic score, the second aesthetic score and the third aesthetic score.

[0132] Optionally, the first calculation submodule includes:

[0133] The second calculation submodule is configured to obtain the aesthetic score according to the first aesthetic score, the first weight corresponding to the first aesthetic score, the second aesthetic score, the first weight corresponding to the second aesthetic score, the third aesthetic score and the weight of the third aesthetic score.

[0134] Optionally, the image processing device 600 includes:

[0135] The judgment module is configured to shoot the to-be-processed picture displayed in the preview picture of the terminal to obtain a target image when the aesthetic score reaches a threshold.

[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image processing method provided by the present disclosure are implemented.

[0138] Figure 7 is a block diagram of an apparatus 700 for image processing according to an exemplary embodiment. For example, the apparatus 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0139] Reference Figure 7 , the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output interface 712 , a sensor component 714 , and a communication component 716 .

[0140] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned image processing method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0141] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0143] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0145] The input / output interface 712 provides an interface between the processing component 702 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0146] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned image processing method.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to complete the above-mentioned image processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0150] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned image processing method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instruction may be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned image processing method may be implemented; alternatively, the integrated circuit or chip may receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned image processing method.

[0151] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned image processing method when executed by the programmable device.

[0152] Figure 8 8 is a block diagram of an apparatus 800 for image processing according to an exemplary embodiment. For example, the apparatus 800 may be provided as a server. Figure 8 The apparatus 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions executable by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned image processing method.

[0153] The device 800 may also include a power supply component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output interface 858. The device 800 may operate based on an operating system stored in the memory 832.

[0154] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Get the screen to be processed; Determining an aesthetic score corresponding to the image to be processed based on a pre-trained aesthetic evaluation model; The aesthetic evaluation model is trained by a machine learning method using training samples, wherein the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

2. The method according to claim 1, characterized in that The step of determining the aesthetic score corresponding to the picture to be processed based on the pre-trained aesthetic evaluation model includes: Based on the aesthetic evaluation model, a first aesthetic score corresponding to the picture to be processed is determined, wherein the first aesthetic score is a score used to reflect the clarity of the picture subject of the picture to be processed; the text description sample includes a first text description sample, and the first text description sample is used to describe the picture content in the image sample; the aesthetic score sample includes a first aesthetic score sample, and the first aesthetic score sample is a score used to reflect the clarity of the picture subject in the image sample.

3. The method according to claim 1, characterized in that The step of determining the aesthetic score corresponding to the picture to be processed based on the pre-trained aesthetic evaluation model includes: Based on the aesthetic evaluation model, a second aesthetic score corresponding to the picture to be processed is determined, and the second aesthetic score is a score used to reflect the visual effect of the picture to be processed; the text description sample includes a second text description sample, and the second text description sample is used to describe the visual effect of the image sample; the aesthetic score sample includes a second aesthetic score sample, and the second aesthetic score sample is a score used to reflect the visual effect of the image sample.

4. The method according to claim 1, characterized in that The step of determining the aesthetic score corresponding to the picture to be processed based on the pre-trained aesthetic evaluation model includes: Based on the aesthetic evaluation model, a third aesthetic score corresponding to the picture to be processed is determined, and the third aesthetic score is a score used to reflect the degree of expression of the photographic elements of the picture to be processed; the text description sample includes a third text description sample, and the third text description sample is used to describe the photographic elements of the image sample; the aesthetic score sample includes a third aesthetic score sample, and the third aesthetic score sample is a score used to reflect the degree of expression of the photographic elements.

5. The method according to claim 1, characterized in that The step of determining the aesthetic score corresponding to the picture to be processed based on the pre-trained aesthetic evaluation model includes: Determining a first aesthetic score, a second aesthetic score, and a third aesthetic score corresponding to the picture to be processed based on the aesthetic evaluation model; The aesthetic score is obtained according to the first aesthetic score, the second aesthetic score and the third aesthetic score.

6. The method according to claim 5, characterized in that The step of obtaining the aesthetic score according to the first aesthetic score, the second aesthetic score and the third aesthetic score comprises: The aesthetic score is obtained according to the first aesthetic score, the first weight corresponding to the first aesthetic score, the second aesthetic score, the first weight corresponding to the second aesthetic score, the third aesthetic score and the weight of the third aesthetic score.

7. The method according to claim 1, characterized in that The method further comprises: When the aesthetic score reaches a threshold, the picture to be processed displayed in the preview picture of the terminal is photographed to obtain a target image.

8. An image processing device, characterized in that: include: An acquisition module is configured to acquire a picture to be processed; The score determination module is configured to determine the aesthetic score corresponding to the image to be processed based on a pre-trained aesthetic evaluation model; wherein the aesthetic evaluation model is trained using a machine learning method using training samples, and the training samples include image samples, text description samples of the image samples, and aesthetic score samples corresponding to the image samples.

9. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Execute the steps of the method described in any one of claims 1 to 7 of the present disclosure.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

11. A chip, characterized in that: The method comprises a processor and an interface; the processor is used to read instructions to execute the method according to any one of claims 1 to 7.

Images