Quality optimization method and device for smart photo album

Abstract

The present disclosure is about a quality optimization method and device for a smart album, which involves image processing. The method includes: obtaining the category information of the images in the smart album; filling the category information of the images into the prompt template to obtain the first prompt text; encoding the first prompt text to obtain the text features of the first prompt text; encoding the image to obtain the visual features of the image; predicting the first mask and the second mask according to the text features and the visual features to obtain the preference information and quality information of the image; optimizing the quality of the target task of the smart album according to the preference information and quality information of the image. The present disclosure predicts the preference information and quality information of the image according to the text features of the first prompt text and the visual features of the image. Then, the quality of the target task of the smart album is optimized to provide users with higher quality smart album services and enhance the user's perceived experience of the smart album.

Description

Technical Field

[0001] The present disclosure relates to image processing, and more particularly to a method and device for optimizing the quality of a smart photo album. Background Art

[0002] With the development of mobile terminal camera technology, more and more users use mobile terminals to take photos and upload them to cloud-based smart albums for storage, backup, creation and sharing. In related technologies, most smart albums do not consider the album quality experience after the intelligent classification of tags is completed. After obtaining the image classification labels, the display order of the pictures in the label set is randomly arranged or arranged only by the time dimension, and the picture cover of each label set is randomly selected or selected only based on the pixel quality of the picture. Although the current smart album can aggregate pictures of the same category in the user's album through the classification model, it is convenient for users to manage and browse. However, for users, the random arrangement of picture sets and the random selection of covers bring a general viewing experience. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a quality optimization method and device for a smart photo album.

[0004] According to a first aspect of an embodiment of the present disclosure, a quality optimization method for a smart photo album is provided, comprising:

[0005] Obtaining category information of pictures in the smart album;

[0006] Filling the category information of the picture into a prompt template to obtain a first prompt text; wherein the prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture;

[0007] Performing encoding processing on the first prompt text to obtain text features of the first prompt text;

[0008] Performing encoding processing on the image to obtain visual features of the image;

[0009] Predicting the first mask and the second mask according to the text feature and the visual feature to obtain preference information and quality information of the image;

[0010] The quality of the target task of the smart album is optimized according to the preference information and quality information of the pictures.

[0011] According to a second aspect of an embodiment of the present disclosure, a method for training a picture evaluation model is provided, comprising:

[0012] Obtain the category label information, preference label information and quality label information of the image to be evaluated;

[0013] Fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain a prompt text;

[0014] Encoding the prompt text to obtain text features of the prompt text;

[0015] Performing encoding processing on the image to be evaluated to obtain visual features of the image to be evaluated;

[0016] The image evaluation model is trained according to the text features, the visual features, the category label information, the preference label information and the quality label information.

[0017] According to a third aspect of an embodiment of the present disclosure, a quality optimization device for a smart photo album is provided, including:

[0018] A first acquisition module, used to acquire category information of pictures in the smart album;

[0019] A second acquisition module is used to fill the category information of the picture into the prompt template to obtain a first prompt text; wherein the prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture;

[0020] A third acquisition module is used to encode the first prompt text to obtain text features of the first prompt text;

[0021] A fourth acquisition module, used to encode the image to obtain visual features of the image;

[0022] a fifth acquisition module, configured to predict the first mask and the second mask according to the text feature and the visual feature, and obtain preference information and quality information of the image;

[0023] The optimization module is used to optimize the quality of the target task of the smart album according to the preference information and quality information of the picture.

[0024] According to a fourth aspect of an embodiment of the present disclosure, a training device for an image evaluation model is provided, comprising:

[0025] A first acquisition module is used to acquire category label information, preference label information and quality label information of the image to be evaluated;

[0026] A second acquisition module is used to fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain a prompt text;

[0027] A third acquisition module is used to encode the prompt text to obtain text features of the prompt text;

[0028] A fourth acquisition module, used to perform encoding processing on the image to be evaluated to obtain visual features of the image to be evaluated;

[0029] A training module is used to train the image evaluation model according to the text features, the visual features, the category label information, the preference label information and the quality label information.

[0030] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect or the method described in the second aspect is implemented.

[0031] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect or the method described in the second aspect is implemented.

[0032] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: according to the text features of the first prompt text and the visual features of the picture, the preference information and quality information of the picture are predicted. Based on the predicted preference information and quality information of the picture, the quality of the target task of the smart album is optimized, so that the management of the pictures by the smart album is more in line with the user's preferences, and considering the picture quality, a higher quality smart album service is provided to the user, thereby improving the user's perceptual experience of the smart album.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] Figure 1 is a flow chart of a quality optimization method for a smart photo album according to an exemplary embodiment;

[0036] Figure 2 is a flow chart showing another quality optimization method of a smart photo album according to an exemplary embodiment;

[0037] Figure 3 is a flow chart showing another method for optimizing the quality of a smart photo album according to an exemplary embodiment;

[0038] Figure 4 is a flowchart of searching for cover images of a category picture collection according to an exemplary embodiment;

[0039] Figure 5 is a flow chart of a method for determining a thumbnail making strategy according to an exemplary embodiment;

[0040] Figure 6 is a flowchart of a method for training a picture evaluation model according to an exemplary embodiment;

[0041] Figure 7 is a schematic diagram of a picture evaluation model according to an exemplary embodiment;

[0042] Figure 8 is a structural block diagram of a quality optimization device for a smart photo album according to an exemplary embodiment;

[0043] Fig. 9 is a structural block diagram of a training device for a picture evaluation model according to an exemplary embodiment;

[0044] Fig.10 is a block diagram of an electronic device 1000 according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0046] In the technical solution of the present disclosure, the acquisition, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0047] Figure 1 is a flow chart of a quality optimization method for a smart photo album according to an exemplary embodiment. It should be noted that the quality optimization method for a smart photo album in the embodiment of the present disclosure can be used in the quality optimization device for a smart photo album in the embodiment of the present disclosure, and the quality optimization device for a smart photo album in the embodiment of the present disclosure can be configured in an electronic device. Figure 1As shown, the quality optimization method of the smart photo album may include but is not limited to the following steps.

[0048] Step 101, obtaining category information of pictures in the smart album.

[0049] The smart album may be a cloud-based smart album. It should be noted that the category information of the image can be directly obtained from the classification result of the smart album, and the value is limited and determined.

[0050] Step 102: Fill the category information of the picture into the prompt template to obtain the first prompt text. The prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture.

[0051] As an example, the prompt template may be in the form of "a picture of {category information}, the user preference is {MASK1}, and the subjective score is {MASK2}". Assuming that the category information of a certain picture is "person", the "person" category information is filled into the prompt template to obtain the first prompt text "a picture of {person}, the user preference is {MASK1}, and the subjective score is {MASK2}".

[0052] Step 103: Encode the first prompt text to obtain text features of the first prompt text.

[0053] Step 104: encode the image to obtain visual features of the image.

[0054] Step 105 , predicting the first mask and the second mask according to the text features and the visual features to obtain preference information and quality information of the image.

[0055] In some embodiments of the present disclosure, the text features and the visual features may be fused to obtain fused features, and the first mask MASK1 and the second mask MASK2 are predicted based on the fused features, thereby obtaining preference information and quality information of the image.

[0056] Optionally, the first mask and the second mask may be predicted based on the fusion feature and a prediction module in a pre-trained image evaluation model to obtain preference information and quality information of the image. The prediction module in the image evaluation model has learned the ability to predict preference information and quality information of the image based on the fusion feature.

[0057] Step 106, optimizing the quality of the target task of the smart album according to the preference information and quality information of the pictures.

[0058] It should be noted that the target task may include multiple tasks, such as setting a category cover image, making thumbnails, and arranging pictures for display. As an example, for the category cover image setting task, the pictures in the smart album may be classified according to category information. In each category of pictures, the picture with the highest preference information and quality information is searched, and the picture is used as the cover image of the picture collection of that category.

[0059] As another example, for the picture display arrangement task, the pictures in the smart album can be arranged in descending order according to the preference information and quality information to optimize the quality of the smart album.

[0060] By implementing the disclosed embodiment, the preference information and quality information of the picture are predicted based on the text features of the first prompt text and the visual features of the picture. The quality of the target task of the smart album is optimized based on the predicted preference information and quality information of the picture, so that the management of the picture by the smart album is more in line with the user's preference, and considering the picture quality, a higher quality smart album service is provided to the user, which can solve the quality optimization problem of the smart album in the case of a huge number of pictures and uneven shooting levels to a certain extent, and enhance the user's perception experience of the smart album.

[0061] Figure 2 FIG. 1 is a flow chart showing another method for optimizing the quality of a smart photo album according to an exemplary embodiment. Figure 2 As shown, the quality optimization method of the smart photo album may include but is not limited to the following steps.

[0062] Step 201, obtaining category information of pictures in the smart album.

[0063] Step 202: Fill the category information of the picture into the prompt template to obtain the first prompt text.

[0064] The prompt template includes a first mask and a second mask, the first mask is used to represent the to-be-predicted preference information of the picture, and the second mask is used to represent the to-be-predicted quality information of the picture.

[0065] Step 203: Encode the first prompt text to obtain text features of the first prompt text.

[0066] Step 204, divide the picture into a plurality of picture blocks, and determine the main area of ​​the picture.

[0067] In one implementation, the subject and subject area in the picture may be determined based on the classification result of the smart album.

[0068] Step 205 , position coding is performed on the picture blocks including the main body area among the multiple picture blocks to obtain the position coding of the picture.

[0069] As an example, assume that it is determined that there is a subject b in picture I, determine the area of ​​subject b, and divide the picture into multiple picture blocks I1, I2, ..., I5 of fixed size. The area containing subject b in picture block I2 and picture block I3 is determined based on the segmentation coordinate calculation. Considering that significant content plays a significant role in user perception of picture quality, and the significant area is usually the foreground content of the picture, and the pure background content has little effect on the perceived quality, only the picture blocks containing the main area are positionally encoded, and the position encoding of the remaining picture blocks that do not contain the main image area is 0. The position encoding PE of picture I can refer to the following formula.

[0070]

[0071] Step 206 , according to the position coding of the picture, a plurality of picture blocks are encoded based on a visual encoder in a pre-trained picture evaluation model to obtain visual features of the picture.

[0072] By encoding the position of the image blocks containing the main area, the focus on the salient content features of the image can be enhanced, and the calculation amount of the position encoding can be reduced. As an example, the image can be linearly mapped to the image blocks and then added to the position encoding of the image, and then input into the visual encoder to obtain the visual features of the image.

[0073] Step 207, calculating the similarity between the text features and the visual features to obtain a similarity matrix.

[0074] In some embodiments of the present disclosure, the similarity of text features and visual features can be calculated by the following formula to obtain a similarity matrix P:

[0075]

[0076] Among them, F is the visual feature and G is the text feature.

[0077] Step 208: According to the similarity matrix, the text feature with the highest similarity is connected to the visual feature channel dimension to obtain a fused feature.

[0078] Step 209 , predicting the first mask and the second mask according to the fusion feature to obtain preference information and quality information of the picture.

[0079] Step 210, optimizing the quality of the target task of the smart album according to the preference information and quality information of the pictures.

[0080] In the embodiments of the present disclosure, steps 201 to 203 and steps 209 to 210 may be implemented in any manner in the embodiments of the present disclosure, and the present disclosure does not specifically limit this and will not elaborate on it.

[0081] By implementing the disclosed embodiment, the visual features of the image are obtained based on the position coding of the subject, which enhances the focus on the significant content features of the image and reduces the amount of calculation. Then, based on the text features of the first prompt text and the visual features of the image, the preference information and quality information of the image are predicted to improve the prediction accuracy of the preference information and quality information. Based on the predicted preference information and quality information of the image, the quality of the target task of the smart album is optimized, so that the management of the image by the smart album is more in line with the user's preferences, and considering the quality of the image, a higher quality smart album service is provided to the user, further enhancing the user's perceptual experience of the smart album.

[0082] Figure 3 FIG. 1 is a flow chart showing another method for optimizing the quality of a smart photo album according to an exemplary embodiment. Figure 3 As shown, the quality optimization method of the smart photo album may include but is not limited to the following steps.

[0083] Step 301, obtaining category information and attribute information of pictures in the smart album.

[0084] In some embodiments of the present disclosure, the attribute information of a picture may include picture size, picture resolution, picture source, picture content, etc.

[0085] Step 302: Fill the category information of the picture into the prompt template to obtain the first prompt text. The prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture.

[0086] Step 303: Encode the first prompt text to obtain text features of the first prompt text.

[0087] Step 304: encode the image to obtain visual features of the image.

[0088] Step 305 , predicting the first mask and the second mask according to the text features and the visual features to obtain preference information and quality information of the image.

[0089] Step 306: Divide the pictures in the smart album into multiple candidate picture sets according to the attribute information.

[0090] In one implementation, pictures in a smart album may be divided into two alternative picture sets according to the source of the pictures, namely, pictures from shooting sources and pictures from other sources (eg, transferred, downloaded, screenshots, etc.).

[0091] Alternatively, the images may be divided into two alternative image sets according to the image sizes, namely, normal-size images and extra-long images.

[0092] Alternatively, the pictures in the smart album can be divided according to a variety of attribute information. As an example, all the pictures from the shooting sources in the smart album can be used as the first candidate picture set, and the pictures from other sources can be used as the second candidate picture set. The size of the pictures in the second candidate picture set is judged, and the maximum aspect ratio max_ratio, the minimum aspect ratio min_ratio, and the minimum short side length min_length are set. When the aspect ratio of the picture is less than the minimum value min_ration or greater than the maximum value max_ratio, the picture is moved from the second candidate picture set to the third candidate set. When the short side length of the picture is less than the minimum value min_length, it is moved from the second candidate picture set to the third candidate set. The next step is to judge the picture content of the second candidate picture set after the size screening. When the picture is a black and white picture, has no content such as a solid color picture, or has only simple content such as a pure text picture, it is moved from the second candidate picture set to the fourth candidate set. Among them, black and white images can be judged by the color level of the image, such as calculating the variance mean of the differences between the three color channels of the image, and judging it as a black and white image when the mean is less than a certain threshold threshold_bw. Pure color images can be judged by image edge detection and other methods, such as extracting the image contour after binarization, and judging it as a pure image when the contour is less than a certain threshold threshold_pure.

[0093] Step 307 , based on the preference information and quality information of the pictures, the quality of the target task of the smart album is optimized in combination with a plurality of candidate picture sets.

[0094] It should be noted that when images are divided into multiple candidate image sets according to their attribute information, the priority of candidate image sets with different attribute information can be considered when optimizing the quality of the target task of the smart photo album. For example, for the shooting source candidate image set and other source candidate image set divided according to the image source, in the target task of setting the category cover image, the corresponding category cover image with high preference information and quality information can be searched in the shooting source candidate image set according to the preference information and quality information of the image.

[0095] In some embodiments of the present disclosure, for the task of setting a category cover image, assuming that the pictures in the smart album are divided into a first candidate picture set, a second candidate picture set, a third candidate picture set, and a fourth candidate picture set according to the various attribute information in step 306, the multiple candidate picture sets can be divided into candidate picture subsets corresponding to the multiple categories according to the category information of the pictures. As an example, for the person category, the pictures with the category information of "person" in the first candidate picture set are taken as the first candidate picture subset corresponding to the person category, and the second candidate picture subset, the third candidate picture subset, and the fourth candidate picture subset can be obtained in the same way.

[0096] According to the candidate image subsets corresponding to each of the multiple categories, a category image set corresponding to each category is generated. Taking the aforementioned person category as an example, the category image set corresponding to the person category includes the first candidate image subset, the second candidate image subset, the third candidate image subset and the fourth candidate image subset corresponding to the person category, that is, including all the images of the person category in the smart album. Determine the priority search order of the multiple candidate image subsets corresponding to the current category, search for the first image in the multiple candidate image subsets corresponding to the current category according to the preference information and quality information of the image in the priority search order, and use the first image as the cover image of the category image set corresponding to the current category.

[0097] In one implementation, the priority search order of multiple candidate image subsets can refer to Figure 4 As shown in the flow chart. For the current category, first set a minimum quality score threshold such as 85, and search for the picture with the highest quality information in the first candidate picture subset corresponding to the current category. If its quality information is greater than the set threshold, no subsequent candidate picture subset search will be performed. Otherwise, the second candidate picture subset search will be performed, and so on to the third candidate picture subset. If there is no picture that meets the conditions in the three candidate picture subsets, take the one with the highest quality information in the three candidate picture subsets. When there is the same quality information, priority is given to preference information. For example, the preference information priority is set to 1>2>3>5>7>6>8>4. When the preference information is still consistent, it can be randomly selected from the first candidate picture subset, the second candidate picture subset and the third candidate picture subset.

[0098] In some embodiments of the present disclosure, for the thumbnail making task, assuming that the pictures in the smart album are divided into a first candidate picture set, a second candidate picture set, a third candidate picture set, and a fourth candidate picture set according to the various attribute information in step 306, the thumbnail making strategy corresponding to each picture in the multiple candidate picture sets can be determined according to the preference information and quality information of the pictures. The corresponding thumbnail making strategy is adopted for each picture in each candidate picture set to obtain a thumbnail of each picture.

[0099] In one implementation, the thumbnail creation strategy corresponding to the image can be determined by referring to Figure 5As shown in the flowchart. First, determine whether the image belongs to the third candidate image set. If the image belongs to the third candidate image set and the aspect ratio is less than 1 / 3 or greater than 3, the image is an overlong image, so the image intelligent cropping strategy is used to reconstruct the image. The intelligent cropping strategy may include subject integrity cropping guided by subject coordinates, or aesthetic enhancement intelligent cropping based on the depth model, etc., and finally obtain a secondary reconstructed image with an aspect ratio within a certain range. If the image belongs to the third candidate image set and the aspect ratio is greater than or equal to 1 / 3 and less than or equal to 3, the image is a small image and does not belong to an overlong image. The small image will subsequently adopt a thumbnail compression production strategy based on resolution. If the image does not belong to the third candidate image set, determine whether the image belongs to the fourth candidate image set. If the image belongs to the fourth candidate image set, it means that the image is a pure image (pure color image or image containing only simple content), and the image belonging to the fourth candidate image set directly adopts a thumbnail compression production strategy based on resolution. If the image is not the third or fourth candidate image set, the image enhancement strategy is executed according to the preference information and quality information of the image. When the quality information is greater than or equal to the minimum picture quality score threshold (such as 60), a thumbnail compression production strategy based on resolution is adopted. When the quality information is less than the minimum picture quality score threshold, a picture enhancement strategy based on preference information is adopted. In some embodiments of the present disclosure, the preference information may include whether to like and the corresponding reason with reference to the following table.

[0100] Preference Information Preference reason 1 like Color Harmony 2 like Perfect composition 3 like Interesting content 4 dislike Blurry pictures 5 dislike Low contrast and brightness 6 dislike Content is messy 7 dislike Lack of color 8 dislike The picture is meaningless

[0101] When the preference information of a picture is 4, it means that the picture is blurred, and the deblurring enhancement strategy can be used. When the user preference classification is 8, semantic enhancement can be implemented based on the subject coordinates or semantics to highlight the key points.

[0102] In addition, a resolution-based thumbnail compression production strategy is adopted for intelligent cropped images, quality enhanced images and the fourth alternative set images. For example, four different resolutions of 128*128|480*480|800*800|1080*1080 and two different compression rates of 90|70 can be set. In actual use, the selection needs to be based on the storage capacity of the cloud smart album system and the resolution of the user's terminal.

[0103] In some embodiments of the present disclosure, for the picture display arrangement task, assuming that the pictures in the smart album are divided into a first alternative picture set, a second alternative picture set, a third alternative picture set and a fourth alternative picture set according to multiple attribute information in step 306, the pictures in the multiple alternative picture sets can be sorted according to the preference information and quality information of the pictures to determine the picture display arrangement of the smart album.

[0104] In one implementation, the pictures in the first candidate picture set, the second candidate picture set, and the third candidate picture set are divided into a first category (pictures that the user likes, such as preference information 1, 2, 3) and a second category (pictures that the user does not like, such as preference information 4, 5, 6, 7, 8) according to preference information. According to quality information, the first category, the second category, and the fourth candidate picture set are sorted in order from high quality information to low quality information. Thus, the priority of picture arrangement is determined as first category high quality -> first category low quality -> second category high quality -> second category low quality -> fourth candidate set high quality -> fourth candidate set low quality, and the picture display arrangement of the smart album is determined according to the priority.

[0105] By implementing the disclosed embodiment, the preference information and quality information of the picture are accurately predicted based on the text features of the first prompt text and the visual features of the picture. The picture is divided into multiple candidate picture sets according to the attribute information of the picture, and then based on the predicted preference information and quality information of the picture, the quality of the target task of the smart album is optimized in combination with multiple candidate picture sets, so that the management of the picture by the smart album is more in line with the user's preference, and considering the picture quality, a higher quality smart album service is provided to the user, further improving the user's perception experience of the smart album.

[0106] The present disclosure also proposes a method for training a picture evaluation model, wherein the model is used to predict preference information and quality information of a picture. Figure 6 FIG. 1 is a flowchart of a method for training a picture evaluation model according to an exemplary embodiment. Figure 6 As shown, the training method of the image evaluation model may include but is not limited to the following steps.

[0107] Step 601: Obtain category label information, preference label information, and quality label information of the image to be evaluated.

[0108] Among them, the category label information of the image to be evaluated can be directly obtained from the classification results of the smart album, and the value is limited and determined. The quality label information mos can be obtained through comprehensive scoring by multiple testers based on multiple comprehensive dimensions (including noise, clarity, brightness, contrast, color richness, artistic beauty, etc.). The preference label information can also be obtained through the subjective preference scores and corresponding reasons given by multiple testers, and the preference score with the largest proportion is used as the preference label information of the image to be evaluated.

[0109] Step 602: fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain the prompt text.

[0110] As an example, the prompt text (text prompt) can be "a picture of {person}, user preference is {2}, and subjective score is {4}", "a picture of {building}, user preference is {8}, and subjective score is {1}", "a picture of {flower}, user preference is {1}, and subjective score is {5}", etc.

[0111] Step 603: Encode the prompt text to obtain text features of the prompt text.

[0112] In the disclosed embodiment, the prompt text can be encoded based on the text encoder in the picture evaluation model to obtain the text features of the prompt text. Optionally, the encoder can be a transformer-type text encoding model, such as the ChatGPT series pre-trained model.

[0113] Step 604: Encode the image to be evaluated to obtain visual features of the image to be evaluated.

[0114] In the disclosed embodiment, the image to be evaluated may be encoded based on the visual encoder in the image evaluation model to obtain text features of the prompt text.

[0115] In one implementation, Figure 7 As shown, the image to be evaluated can be divided into multiple image blocks of fixed size, such as a size of 224*224. The main area (or main position) of the image to be evaluated is determined, and the image blocks containing the main area in the multiple image blocks are position-encoded based on the position encoder in the image evaluation model to obtain the position encoding of the image to be evaluated. According to the position encoding of the image to be evaluated, the multiple image blocks are encoded based on the visual encoder to obtain the visual features of the image to be evaluated. Optionally, the encoder can adopt a pre-trained visual encoding model such as ViT / B-32. Among them, the position encoding PEˊ of the image to be evaluated can refer to the following formula.

[0116]

[0117] Among them, r represents a random initial value, which is a learning parameter and can be dynamically adjusted during the iteration process. The encodings of other positions are set to 0 and are not updated.

[0118] Step 605 , training the image evaluation model based on the text features, visual features, category label information, preference label information, and quality label information.

[0119] In some embodiments of the present disclosure, text features and visual features may be fused to obtain fused features. The fused features are input into a prediction module in an image evaluation model to obtain predicted category information, predicted preference information, and predicted quality information output by the prediction module. A loss value of the image evaluation model is generated based on the category label information, preference label information, quality label information, predicted category information, predicted preference information, and predicted quality information. The image evaluation model is trained based on the loss value.

[0120] In one implementation, the similarity of the text features and the visual features can be calculated to obtain a similarity matrix. As an example, assume that the text feature is G, with a dimension of V×D, where V is the number of possible text prompts. The visual feature is F, with a dimension of N×D, where N is the number of image blocks. Perform cosine similarity calculation on the text feature G and the visual feature F to obtain a similarity matrix P. Among them, P(i,j) represents the similarity between the i-th image block and the j-th text prompt feature. According to the similarity matrix, the text features with the highest similarity are connected to the visual feature channel dimension to obtain a fused feature. That is, determine the column position of the maximum value in each row of the similarity matrix P, and the column position of the maximum value in the m-th row of the similarity matrix P is recorded as u m For each image block, extract the uth m Row vector G um , and use it as the text similarity feature to be fused, with a dimension of 1×D. Thus, for N image blocks, the text similarity feature to be fused with a dimension of N×D can be obtained. The text similarity feature to be fused is connected with the visual feature F to obtain the fusion feature Pˊ, with a dimension of (N+N)×D.

[0121] Optionally, in order to improve the accuracy of model prediction, in some embodiments of the present disclosure, a contrast loss value may be introduced to train the image evaluation model. In one implementation, the contrast loss value may be determined according to the similarity matrix with reference to the following formula.

[0122]

[0123]

[0124] Among them, logit(F,G) v , represents the vth value in logit(F,G), which is the similarity calculated between the image to be evaluated and the prompt feature of the text that actually matches the image. τ is a smoothing control parameter, which can be an empirical value. This contrast loss value is used to evaluate the match between visual features and text features.

[0125] like Figure 7As shown, the prediction module can include multiple Head structures to perform different tasks, including label classification tasks, preference classification tasks, and quality regression tasks. Among them, the labels to be identified in the label classification task are the picture labels within the scope defined by the smart album, which can be implemented using a visual recognition classification structure, such as MLP+softmax. The label classification loss is the cross entropy loss, as shown in the following formula:

[0126]

[0127] Among them, t i is the category label information of the i-th image to be evaluated, p i is the predicted category information of the i-th image to be evaluated, and k is the number of categories.

[0128] In the embodiment of the present disclosure, the user preference prediction model is modeled as a multi-classification task, which can also be implemented using a visual recognition classification structure, such as MLP+softmax, and the label classification loss is the cross entropy loss l 偏好 , the format is consistent with the label classification.

[0129] The quality regression task completes the image quality score prediction through a regression model. For example, one possible implementation method is MLP+Linear regression. The quality regression loss function uses MSE loss, as shown in the following formula:

[0130]

[0131] Among them, mos i is the quality label information of the i-th image to be evaluated, p i is the predicted quality information of the i-th picture to be evaluated, and M is the number of pictures to be evaluated.

[0132] This generates the loss value of the image evaluation model as shown in the following formula:

[0133] L=λ1l 对比 +λ2l 标签 +λ3l 偏好 +λ4l 质量

[0134] Among them, λ1, λ2, λ3, and λ4 are balance coefficients, which can be selected based on experience and dynamically adjusted, such as setting different size coefficient values ​​according to the importance of different modules in the iteration stage.

[0135] The image evaluation model is trained according to the loss value L. After the model converges, the preference information and quality information of the image can be predicted based on the prediction module.

[0136] By implementing the disclosed embodiments, the image evaluation model is trained to accurately predict the preference information and quality information of the image, so that the quality of the smart album can be optimized based on the preference information and quality information predicted by the model, and the album data can be processed at low cost and high precision, thereby improving the user's perception experience of the smart album.

[0137] Figure 8 FIG. 1 is a structural block diagram of a quality optimization device for a smart photo album according to an exemplary embodiment. Figure 8 As shown, the quality optimization device of the smart photo album includes: a first acquisition module 801, a second acquisition module 802, a third acquisition module 803, a fourth acquisition module 804, a fifth acquisition module 805 and an optimization module 806.

[0138] Among them, the first acquisition module 801 is used to obtain category information of pictures in the smart album.

[0139] The second acquisition module 802 is used to fill the category information of the picture into the prompt template to obtain the first prompt text. The prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture.

[0140] The third acquisition module 803 is used to encode the first prompt text to obtain text features of the first prompt text.

[0141] The fourth acquisition module 804 is used to encode the picture to obtain the visual features of the picture.

[0142] The fifth acquisition module 805 is used to predict the first mask and the second mask according to the text features and the visual features to obtain the preference information and quality information of the picture.

[0143] The optimization module 806 is used to optimize the quality of the target task of the smart album according to the preference information and quality information of the pictures.

[0144] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0145] Fig. 9 FIG. 1 is a structural block diagram of a training device for a picture evaluation model according to an exemplary embodiment. Fig. 9 As shown, the training device of the picture evaluation model includes: a first acquisition module 901, a second acquisition module 902, a third acquisition module 903, a fourth acquisition module 904 and a training module 905.

[0146] The first acquisition module 901 is used to acquire category label information, preference label information and quality label information of the image to be evaluated.

[0147] The second acquisition module 902 is used to fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain the prompt text.

[0148] The third acquisition module 903 is used to encode the prompt text to obtain text features of the prompt text.

[0149] The fourth acquisition module 904 is used to perform encoding processing on the picture to be evaluated to obtain visual features of the picture to be evaluated.

[0150] The training module 905 is used to train the image evaluation model according to text features, visual features, category label information, preference label information and quality label information.

[0151] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0152] Fig.10 is a block diagram of an electronic device 1000 according to an exemplary embodiment.

[0153] like Fig.10 As shown, the electronic device 1000 includes:

[0154] The memory 1001 and the processor 1002, a bus 1003 connecting different components (including the memory 1001 and the processor 1002), the memory 1001 stores a computer program, and when the processor 1002 executes the program, the quality optimization method of the smart album or the training method of the picture evaluation model of the embodiment of the present disclosure is implemented.

[0155] Bus 1003 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0156] The electronic device 1000 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 1000, including volatile and non-volatile media, removable and non-removable media.

[0157] The memory 1001 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1004 and / or cache memory 1005. The electronic device 1000 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1006 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 1003 via one or more data medium interfaces. The memory 1001 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0158] A program / utility 1008 having a set (at least one) of program modules 1007 may be stored, for example, in the memory 1001, such program modules 1007 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 1007 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0159] The electronic device 1000 may also communicate with one or more external devices 1009 (e.g., keyboard, pointing device, display 1011, etc.), one or more devices that enable a user to interact with the electronic device 1000, and / or any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 1012. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1013. Fig.10 As shown, the network adapter 1013 communicates with other modules of the electronic device 1000 via the bus 1003. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0160] The processor 1002 executes various functional applications and data processing by running the programs stored in the memory 1001 .

[0161] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the quality optimization method of the smart photo album or the training method of the picture evaluation model of the embodiment of the present disclosure, and will not be repeated here.

[0162] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium.

[0163] When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the quality optimization method of the smart album or the training method of the picture evaluation model as described above. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0164] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0165] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A quality optimization method for smart photo album, It is characterized in that include: Obtaining category information of pictures in the smart album; Filling the category information of the picture into a prompt template to obtain a first prompt text; wherein the prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture; Performing encoding processing on the first prompt text to obtain text features of the first prompt text; Performing encoding processing on the image to obtain visual features of the image; Predicting the first mask and the second mask according to the text feature and the visual feature to obtain preference information and quality information of the image; Optimizing the quality of the target task of the smart album according to the preference information and quality information of the picture; The predicting the first mask and the second mask according to the text feature and the visual feature to obtain the preference information and quality information of the picture includes: Calculating similarity between the text feature and the visual feature to obtain a similarity matrix; According to the similarity matrix, the text features with the highest similarity are connected to the visual feature channel dimension to obtain a fusion feature; Predicting the first mask and the second mask according to the fusion feature to obtain preference information and quality information of the picture; The predicting the first mask and the second mask according to the fusion feature to obtain the preference information and quality information of the picture includes: According to the fusion feature, based on a prediction module in a pre-trained picture evaluation model, the first mask position and the second mask position are predicted to obtain preference information and quality information of the picture.

2. The method according to claim 1, It is characterized in that The encoding process of the picture to obtain the visual features of the picture includes: Dividing the picture into a plurality of picture blocks; The plurality of picture blocks are encoded based on a visual encoder in a pre-trained picture evaluation model to obtain visual features of the picture.

3. The method according to claim 2, It is characterized in that The visual encoder in the pre-trained picture evaluation model encodes the multiple picture blocks to obtain visual features of the pictures, including: Determine the main area of ​​the picture; Performing position coding on the picture blocks including the main area in the multiple picture blocks to obtain the position coding of the picture; According to the position coding of the picture, the multiple picture blocks are encoded based on the visual encoder to obtain the visual features of the picture.

4. The method according to claim 1, It is characterized in that Also includes: Obtaining attribute information of pictures in the smart album; Dividing the pictures in the smart album into a plurality of candidate picture sets according to the attribute information; The step of optimizing the quality of the target task of the smart album according to the preference information and quality information of the picture includes: According to the preference information and quality information of the pictures, the quality of the target task of the smart album is optimized in combination with the multiple candidate picture sets.

5. The method according to claim 4, It is characterized in that The target task includes setting a category cover image; optimizing the quality of the target task of the smart photo album in combination with the plurality of candidate image sets according to the preference information and quality information of the image, including: According to the category information of the pictures, the multiple candidate picture sets are divided into candidate picture subsets corresponding to the multiple categories respectively; Generating a category picture set corresponding to each category according to the candidate picture subsets corresponding to each of the multiple categories; Get the priority search order of multiple candidate image subsets corresponding to the current category; According to the preference information and quality information of the picture, the first picture is searched in the multiple candidate picture subsets corresponding to the current category in the priority search order, and the first picture is used as the cover picture of the category picture set corresponding to the current category.

6. The method according to claim 4, It is characterized in that The target task includes thumbnail production; optimizing the quality of the target task of the smart photo album in combination with the plurality of candidate picture sets according to the preference information and quality information of the picture, including: Determining thumbnail creation strategies corresponding to each picture in the plurality of candidate picture sets according to the preference information and quality information of the pictures; A corresponding thumbnail making strategy is adopted for each picture in each candidate picture set to obtain thumbnails of the pictures.

7. The method according to claim 4, It is characterized in that The target task includes picture display arrangement; the quality optimization of the target task of the smart photo album based on the preference information and quality information of the pictures and in combination with the multiple candidate picture sets includes: According to the preference information and quality information of the pictures, the pictures in the multiple candidate picture sets are sorted to determine the picture display arrangement of the smart photo album.

8. A training method for an image evaluation model, It is characterized in that include: Obtain the category label information, preference label information and quality label information of the image to be evaluated; Fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain a prompt text; Encoding the prompt text to obtain text features of the prompt text; Performing encoding processing on the image to be evaluated to obtain visual features of the image to be evaluated; Training the image evaluation model according to the text features, the visual features, the category label information, the preference label information, and the quality label information; The training of the image evaluation model according to the text features, the visual features, the category label information, the preference label information and the quality label information comprises: Fusing the text feature with the visual feature to obtain a fused feature; Inputting the fusion feature into a prediction module in the image evaluation model to obtain prediction category information, prediction preference information and prediction quality information output by the prediction module; generating a loss value of the image evaluation model according to the category label information, the preference label information, the quality label information, the predicted category information, the predicted preference information, and the predicted quality information; Training the image evaluation model according to the loss value; The step of generating a loss value of the image evaluation model according to the category label information, the preference label information, the quality label information, the predicted category information, the predicted preference information, and the predicted quality information includes: Calculating similarity between the text feature and the visual feature to obtain a similarity matrix; Determine a contrast loss value according to the similarity matrix, wherein the contrast loss value is used to evaluate the matching degree between the visual feature and the text feature; A loss value of the image evaluation model is generated according to the category label information, the preference label information, the quality label information, the predicted category information, the predicted preference information, the predicted quality information and the contrast loss value.

9. A quality optimization device for smart photo album, It is characterized in that include: A first acquisition module, used to acquire category information of pictures in the smart album; A second acquisition module is used to fill the category information of the picture into the prompt template to obtain a first prompt text; wherein the prompt template includes a first mask and a second mask, the first mask is used to represent the preference information to be predicted of the picture, and the second mask is used to represent the quality information to be predicted of the picture; A third acquisition module is used to encode the first prompt text to obtain text features of the first prompt text; A fourth acquisition module, used to encode the image to obtain visual features of the image; a fifth acquisition module, configured to predict the first mask and the second mask according to the text feature and the visual feature, and obtain preference information and quality information of the image; An optimization module, used to optimize the quality of the target task of the smart album according to the preference information and quality information of the picture; The fifth acquisition module is specifically used for: Calculating similarity between the text feature and the visual feature to obtain a similarity matrix; According to the similarity matrix, the text features with the highest similarity are connected to the visual feature channel dimension to obtain a fusion feature; Predicting the first mask and the second mask according to the fusion feature to obtain preference information and quality information of the picture; The fifth acquisition module is further specifically used for: According to the fusion feature, based on a prediction module in a pre-trained picture evaluation model, the first mask position and the second mask position are predicted to obtain preference information and quality information of the picture.

10. A training device for an image evaluation model, It is characterized in that include: A first acquisition module is used to acquire category label information, preference label information and quality label information of the image to be evaluated; A second acquisition module is used to fill the category label information, preference label information and quality label information of the image to be evaluated into the prompt template to obtain a prompt text; A third acquisition module is used to encode the prompt text to obtain text features of the prompt text; A fourth acquisition module, used to perform encoding processing on the image to be evaluated to obtain visual features of the image to be evaluated; A training module, used for training the image evaluation model according to the text features, the visual features, the category label information, the preference label information and the quality label information; The training module is specifically used for: Fusing the text feature with the visual feature to obtain a fused feature; Inputting the fusion feature into a prediction module in the image evaluation model to obtain prediction category information, prediction preference information and prediction quality information output by the prediction module; generating a loss value of the image evaluation model according to the category label information, the preference label information, the quality label information, the predicted category information, the predicted preference information, and the predicted quality information; Training the image evaluation model according to the loss value; The training module is also specifically used for: Calculating similarity between the text feature and the visual feature to obtain a similarity matrix; Determine a contrast loss value according to the similarity matrix, wherein the contrast loss value is used to evaluate the matching degree between the visual feature and the text feature; A loss value of the image evaluation model is generated according to the category label information, the preference label information, the quality label information, the predicted category information, the predicted preference information, the predicted quality information and the contrast loss value.

11. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images