Material processing method and device, equipment and storage medium
By marking and optimizing the basic attributes, facial features and picture quality scores for material pictures, the AI photo image generation problem caused by poor quality of material pictures is solved, and the quality of facial feature extraction and generation is improved.
Patent Information
- Application Number
- CN202410931377.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-08
AI Technical Summary
During the AI portrait photo process, poor quality of the material picture leads to incorrect or incomplete digital feature extraction of face images, affecting the quality of the generated photo picture.
By obtaining original material pictures, marking information about basic attributes and status dimensions, facial features quality scoring dimensions and picture quality scoring dimensions, filtering out the standard material pictures, and optimizing them, including rotation, beautification and super-resolution processing, generating optimization Material pictures to improve the quality of generated AI photo pictures.
The quality of material pictures is improved, the accuracy and completeness of facial feature extraction is ensured, thereby improving the generation quality of AI photo pictures.
Smart Images

Figure CN118918042A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a material processing method, device, equipment and storage medium. Background Art
[0002] AI portrait photography is a technology that uses artificial intelligence technology to generate portrait images. In AI portrait photography, AI technology can be used to generate portrait images of a specific style based on material images. In the process of AI portrait photography, it is necessary to extract the digital features of the facial image based on the material image, and generate the portrait image based on the digital features of the facial image. The quality of the material images varies, and there may be facial occlusion, closed eyes, insufficient lighting, and inverted images. If the digital features of the facial image are extracted directly from the material image without proper processing, there may be errors in the extraction of digital features of the facial image, incomplete features, etc., which will affect the quality of the subsequently generated portrait images. Summary of the invention
[0003] The embodiments of the present application provide a material image processing method, device, equipment and storage medium for improving the quality of material images.
[0004] In a first aspect, an embodiment of the present application provides a material image processing method, the method comprising:
[0005] Get at least one original material picture;
[0006] Annotating the original material image with dimensional information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension;
[0007] Based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture;
[0008] The qualified material picture is subjected to image optimization processing to obtain at least one optimized material picture for generating a portrait picture.
[0009] In a second aspect, an embodiment of the present application provides a material image processing device, the device comprising:
[0010] An acquisition module, used to acquire at least one original material picture;
[0011] A labeling module, used to label the original material image with dimension information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension;
[0012] A screening module, configured to screen the at least one original material picture based on the dimension information of each dimension of the multiple dimensions annotated by the original material picture, to obtain at least one qualified material picture that can be used to generate a photo picture;
[0013] The optimization module is used to perform image optimization processing on the qualified material image to obtain at least one optimized material image for generating a photo image.
[0014] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:
[0015] processor;
[0016] a memory for storing instructions executable by the processor;
[0017] Among them, the processor is configured to execute the instructions to implement the material image processing method as described in the first aspect of the embodiment of the present application.
[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the material image processing method as described in the first aspect of the embodiment of the present application.
[0019] The technical solution provided in the embodiment of the present application obtains original material pictures, and annotates the original material pictures with dimensional information of each of multiple predefined dimensions, including basic attribute and status dimensions, facial feature quality scoring dimensions, and image quality scoring dimensions; based on the dimensional information of each of the multiple dimensions, the original material pictures are screened, and image optimization processing is performed on the qualified material pictures obtained after screening, which can improve the quality of the material pictures and facilitate the extraction of facial features based on high-quality material pictures to generate AI portrait pictures, thereby improving the generation quality of AI portrait pictures. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:
[0021] Figure 1 A schematic diagram of an application scenario of a material image processing method provided by an embodiment of the present application;
[0022] Figure 2 A schematic diagram of a process flow of a material image processing method provided for an embodiment of the present application;
[0023] Figure 3A schematic diagram of a dimension information labeling process provided for an embodiment of the present application.
[0024] Figure 4 A schematic diagram of a screening process of an original material picture provided for an embodiment of the present application;
[0025] Figure 5 A schematic diagram of an optimization process of a qualified material picture provided by an embodiment of the present application;
[0026] Figure 6 A schematic diagram of a module of a material image processing device provided by an embodiment of the present application;
[0027] Figure 7 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.
[0030] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0031] Please refer to Figure 1 , is a schematic diagram of an application scenario of a material picture processing method provided by an embodiment of the present application. In this application scenario, it includes: user 101, user terminal 102, and AI photo application 103. User 101 can upload original material pictures in AI photo application 103 through user terminal 102, and AI photo application can process the original material pictures based on the material picture processing method provided in the embodiment of the present application to obtain optimized material pictures, generate AI photo pictures based on the optimized material pictures, and output them to the user.
[0032] The AI photo application can be a client application, a web application, a mini-program application, etc. Now, in combination with the above application scenarios, the material image processing method provided in the embodiment of the present application is described in detail.
[0033] Please refer to Figure 2 , is a flow chart of a material image processing method provided by an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0034] S202, obtaining at least one original material picture.
[0035] The original material pictures are uploaded by users. The original material pictures are pictures uploaded by users to generate portrait pictures, which are usually portrait pictures.
[0036] S204: labeling the original material image with dimension information of each of the predefined multiple dimensions.
[0037] The predefined multiple dimensions are used to mark the original material images from multiple dimensions, so that the original material images can be subsequently filtered according to the dimensional information of the multiple dimensions.
[0038] The predefined dimensions include: basic attribute and status dimension, facial features quality rating dimension, and image quality rating dimension.
[0039] The basic attribute and state dimension is used to identify the basic attributes and state of the portrait, which may include dimensions such as number of people, gender, age, expression, glasses, hair, posture, mask state, facial occlusion state, skin color state, eye state, mouth state, etc. The dimension information of the number of people dimension is, for example, single person or multiple people, the dimension information of the gender dimension is, for example, male or female, the dimension information of the age dimension is a specific age, the dimension information of the expression dimension is, for example, smiling, serious, sad or excited, the dimension information of the glasses dimension is, for example, with glasses or without glasses, the dimension information of the hair dimension is, for example, long hair or short hair, the dimension information of the posture dimension is, for example, standing or sitting, the dimension information of the mask state is, for example, with a mask or without a mask, the dimension information of the facial occlusion state is, for example, with occlusion or without occlusion, the dimension information of the skin color state dimension is, for example, normal skin color or abnormal skin color, the dimension information of the eye state dimension is, for example, normal eyes, closed eyes, or squinting eyes, and the dimension information of the mouth state dimension is, for example, normal or crooked mouth.
[0040] The facial features quality score dimension is used to quantitatively evaluate the coordination and beauty of the facial features of a face image, and may include: eyebrow score, eye score, nose score, mouth score, ear score, etc. The dimensional information of the eyebrow score, eye score, nose score, mouth score, ear score, etc. is a specific score, such as 60, 64, 76, 82, 91, etc.
[0041] The image quality score dimension is used to quantitatively evaluate the quality of the image, which may include: face image proportion score, image resolution score, face occlusion score, etc. The dimension information of the face image proportion score, image resolution score, face occlusion score, etc. is a specific score, such as 67, 78, 57, etc.
[0042] In a specific implementation, the dimension information of each dimension in the predefined multiple dimensions of the original material picture may be determined in an appropriate manner, and the dimension information of each dimension in the predefined multiple dimensions may be annotated for the original material picture.
[0043] Figure 3 A schematic diagram of a dimension information labeling process provided for an embodiment of the present application. Figure 3 As shown, step S204, marking the dimension information of each dimension in the predefined multiple dimensions for the original material image, may specifically include the following steps:
[0044] S302: Input the original material picture into a face detection model to detect whether the original material picture contains a face image.
[0045] S304: When the original material picture includes a face image, the original material picture is input into an information annotation model to annotate the original material picture with dimension information of each dimension in the predefined multiple dimensions.
[0046] The information annotation model is a pre-trained annotation model, which is used to determine the dimensional information of each dimension of the predefined multiple dimensions of the original material image, and annotate the dimensional information of each dimension of the predefined multiple dimensions for the original material image.
[0047] In a specific implementation, a general information annotation model can be trained based on the annotated sample data set to obtain an information annotation model. The annotated samples in the annotated sample data set can be material pictures annotated with the dimension information of each dimension in the predefined multiple dimensions. The general information annotation model can be trained based on the annotated sample data set to obtain the information annotation model. The original material picture is input into the information annotation model, and the dimension information of each dimension in the predefined multiple dimensions can be machine-annotated for the original material picture.
[0048] For original material pictures of different portrait styles, the evaluation criteria for the dimensional information of each dimension in the predefined multiple dimensions may be different. In one implementation, in order to improve the accuracy of dimensional information determination and annotation, multiple information annotation models can be pre-trained, each of which corresponds to a portrait style and is used to annotate the dimensional information of each dimension in the predefined multiple dimensions for the original material image of its corresponding portrait style. When the original material picture contains a face image, the original material picture can be input into a style recognition model to identify the portrait style of the original material picture, and then, according to the portrait style of the original material picture, the original material picture is input into the information annotation model corresponding to the portrait style of the original material picture, and the information annotation model corresponding to the portrait style of the original material picture annotates the dimensional information of each dimension in the predefined multiple dimensions for the original material picture.
[0049] For example, assuming that the original material picture is in a real-life realistic style, the original material picture can be input into the information annotation model corresponding to the real-life realistic style, and the information annotation model corresponding to the real-life realistic style annotates the dimensional information of each dimension of the predefined multiple dimensions for the original material picture.
[0050] For another example, assuming that the original material picture is in anime style, the original material picture can be input into an information annotation model corresponding to the anime style, and the information annotation model corresponding to the anime style annotates the original material picture with dimensional information of each dimension in a plurality of predefined dimensions.
[0051] S306, when the original material picture does not contain a face image, output the original material picture to a reviewer, so that the reviewer can manually annotate the original material picture with dimensional information of each dimension of the predefined multiple dimensions when manually reviewing that the original material picture contains a face image.
[0052] When the face image detection model detects that the original material picture contains a face image, the original material picture is input into the information annotation model for machine annotation; when the face image detection model detects that the original material picture does not contain a face image, the original material picture is output to the reviewer, who performs manual annotation after manual review. This can combine machine annotation with manual annotation to avoid missing screening and missing annotation of the original material picture, thereby improving the integrity of the material picture based on which the photo picture is generated and improving the generation quality of the photo picture.
[0053] In order to improve the annotation accuracy of the information annotation model, in one implementation, further, after manually annotating the dimension information of each dimension in the predefined multiple dimensions for the original material picture, the manually annotated original material picture can be stored in the annotated sample data set to update the annotated sample data set; then, at set time intervals, the information annotation model is trained based on the updated annotated sample data set to update the information annotation model. Among them, the manually annotated original material picture is the original material picture annotated with the dimension information of each dimension in the predefined multiple dimensions obtained after manual annotation. In this way, annotation samples can be continuously collected during the application process, the information annotation model can be continuously iterated and updated, and the annotation accuracy of the information annotation model can be improved.
[0054] S206: Based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture.
[0055] As mentioned above, the multiple dimensions include: basic attribute and state dimension, facial features quality score dimension and picture quality score dimension. In the embodiment of the present application, the at least one original material picture is screened based on the dimensional information of each dimension in the basic attribute and state dimension, the facial features quality score dimension and the picture quality score dimension to obtain at least one qualified material picture that can be used to generate a portrait picture.
[0056] See also Figure 4 , is a schematic diagram of a screening process of an original material picture provided by an embodiment of the present application. Figure 4 As shown, step S206, based on the dimension information of each dimension in the multiple dimensions annotated by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture, which may specifically include the following steps:
[0057] S402: Screen the at least one original material picture based on the basic attributes annotated by the original material picture and the dimension information of the state dimension to obtain at least one qualified candidate material picture.
[0058] Specifically, the basic attributes and status dimensions include: number of people, eye status, mouth status, facial occlusion status, skin color status and other dimensions. The dimensional information of the number of people status dimension includes: single person or multiple people. The dimensional information of the eye status dimension includes: eyes open or eyes closed, and the dimensional information of the mouth status dimension includes: normal or crooked. The dimensional information of the facial occlusion status dimension includes: face occlusion or face unocclusion. The dimensional information of the skin color status dimension includes: normal or abnormal skin color. According to the dimensional information of the number of people status, eye status, mouth status, facial occlusion status, skin color status and other dimensions, the at least one original material picture can be screened, and original material pictures with multiple people, closed eyes, crooked mouth, occluded face, and abnormal skin color can be screened out, so as to obtain at least one qualified candidate material picture.
[0059] In the specific implementation, a pre-configured feature screening strategy can be obtained, and the feature screening strategy is used to screen out material pictures that do not meet the standards, for example: screen out material pictures annotated with any of the following dimensional information: multiple people, closed eyes, crooked mouth, face occlusion, abnormal skin color. Afterwards, based on the feature screening strategy and the dimensional information of the basic attributes and state dimensions annotated on the original material picture, the at least one original material picture can be screened to obtain the qualified candidate material picture.
[0060] By obtaining a preconfigured feature screening strategy, and based on the preconfigured feature screening strategy and the dimensional information of the basic attributes and status dimensions annotated on the original material image, screening the at least one original material image to obtain the qualified candidate material image, the configurability of the feature screening strategy can be provided, which is convenient for improving the flexibility of feature screening.
[0061] S404, based on the dimension information of the facial features quality score dimension and the dimension information of the picture quality score dimension annotated on the qualified candidate material picture, the at least one qualified candidate material picture is screened to obtain at least one qualified material picture.
[0062] Specifically, the facial features quality scoring dimensions include: eyebrow scoring, eye scoring, nose scoring, mouth scoring, and ear scoring. The dimensional information of the eyebrow scoring, eye scoring, nose scoring, mouth scoring, and ear scoring dimensions is a specific score. The picture quality scoring dimensions include: face image proportion scoring, picture resolution scoring, face occlusion scoring, and other dimensions. The dimensional information of the face image proportion scoring, picture resolution scoring, and face occlusion scoring dimensions is a specific score. For each dimension in the facial features quality scoring dimension and the picture quality scoring dimension, the qualified scores of each dimension in the pre-configured facial features quality scoring dimension and the picture quality scoring dimension can be obtained; when the dimensional information of each dimension in the facial features quality scoring dimension and the picture quality scoring dimension of the qualified candidate material picture is greater than the qualified score of the corresponding dimension, the qualified candidate material picture is determined as the qualified material picture.
[0063] By obtaining the qualified scores of each dimension in the pre-configured facial feature quality scoring dimension and the picture quality scoring dimension; when the dimensional information of each dimension in the facial feature quality scoring dimension and the picture quality scoring dimension of the qualified candidate material picture is greater than the qualified score of the corresponding dimension, the qualified candidate material picture is determined as the qualified material picture, which can provide configurability of the qualified score and further improve the flexibility of material picture screening.
[0064] In specific implementation, the qualified scores of each dimension in the pre-configured facial quality scoring dimension and the picture quality scoring dimension can be obtained, and then, for each qualified candidate material picture, the dimension information of each dimension in the facial quality scoring dimension and the picture quality scoring dimension of the qualified candidate material picture is compared with the qualified scores of the corresponding dimensions respectively. If the dimension information of each dimension is greater than the qualified scores of the corresponding dimensions, the qualified candidate material picture can be determined as the qualified material picture. Among them, the dimension information of each dimension in the facial quality scoring dimension and the picture quality scoring dimension is a specific score.
[0065] For example, assuming that the qualified candidate material picture is picture 1, the dimension information of its facial features quality score dimension is: eyebrow score: 65, eye score: 70, nose score 80, mouth score 75, ear score 80, and the dimension information of the picture quality score dimension is: face image proportion score: 70, picture resolution score 80, face occlusion score 90. The qualified scores of each dimension in the eyebrow score, eye score, nose score, mouth score, ear score, face image proportion score, picture resolution score, and face occlusion score are all 60. The scores of each dimension in the facial features quality score dimension and the picture quality score dimension of picture 1 exceed their corresponding qualified scores. At this time, picture 1 can be determined as a qualified material picture.
[0066] For another example, assuming that the qualified candidate material picture is picture 2, the dimension information of its facial features quality score dimension is: eyebrow score: 65, eye score: 50, nose score 80, mouth score 75, ear score 80, and the dimension information of the picture quality score dimension is: face image proportion score: 70, picture resolution score 80, face occlusion score 90. The qualified scores of each dimension in the eyebrow score, eye score, nose score, mouth score, ear score, face image proportion score, picture resolution score, and face occlusion score are all 60. In the facial features quality score dimension and picture quality score dimension of picture 2, the scores of other dimensions except the eye dimension all exceed their corresponding qualified scores, but the score of the eye score dimension does not exceed the qualified score of the eye dimension. At this time, it is determined that picture 2 is not a qualified material picture.
[0067] S208, performing image optimization processing on the qualified material image to obtain at least one optimized material image for generating a portrait image.
[0068] Optimizing the source images is the basis for generating portrait images. The AI portrait application then extracts facial image feature information based on the optimized source images and generates portrait images based on the facial image feature information.
[0069] Image optimization processing is used to further improve the quality of material images, which may include: portrait positive image processing, portrait beautification processing, image super-resolution processing, etc. Figure 5 Schematic diagram of an optimization process of a qualified material picture provided by an embodiment of the present application. Figure 5 As shown, step S208, performing image optimization processing on the at least one qualified material image to obtain at least one optimized material image for generating a portrait image, may specifically include the following steps:
[0070] S502: Screen the at least one qualified material picture based on the similarity between each qualified material picture in the at least one qualified material picture to obtain at least one optimized candidate material picture.
[0071] Specifically, for the at least one qualified material picture, the similarity between the qualified material picture and other qualified material pictures can be calculated, and then, based on the similarity between the qualified material picture and other qualified material pictures, the qualified material pictures with lower similarity to other qualified material pictures can be screened out, so as to obtain optimized candidate material pictures.
[0072] For example, assuming that the qualified material pictures are Picture 1, Picture 2, Picture 3, and Picture 4, the similarity between Picture 1, Picture 2, and Picture 3 is high, and the similarity between Picture 4 and Picture 1, Picture 2, and Picture 3 is low, then Picture 4 can be screened out, and the remaining Picture 1, Picture 2, and Picture 3 can be determined as optimization candidate material pictures.
[0073] By screening at least one qualified material picture based on the similarity between each qualified material picture in the at least one qualified material picture, material pictures whose portraits are inconsistent with other pictures, or whose portrait styles are inconsistent with other pictures can be screened out, so as to avoid these material pictures interfering with facial image feature extraction and other processing when generating AI photo pictures, thereby improving the quality of AI photo picture generation.
[0074] S504: Detect whether the optimization candidate material picture is a normal material picture.
[0075] Specifically, it is possible to detect whether the direction of the face image object in the optimization candidate material picture is positive, if so, it is determined that the optimization candidate material picture is a positive material picture, if not, it is determined that the optimization candidate material picture is not a positive material picture. Wherein, positive means that the face image directly faces the camera or the observer, without significant rotation or tilt, and the main facial features such as the eyes, nose and mouth are directly facing the camera, in the most direct and frontal viewing direction.
[0076] S505: When the optimization candidate material picture is not a normal material picture, the optimization candidate material picture is rotated to become a normal material picture.
[0077] When the preset direction is positive, the non-preset direction is non-positive, such as the face image is inverted, the face image is tilted, etc. When the face image object is non-positive, the optimized candidate material image can be rotated to an angle where the portrait object is positive, thereby obtaining a positive material image;
[0078] S506, when the optimization candidate material picture is a normal material picture, a face image region is captured for the optimization candidate material picture to obtain a face image material picture.
[0079] Specifically, according to the position of the face image in the positive image material picture, the face image area in the positive image material picture can be cut out in proportion, so as to obtain the face image material picture.
[0080] S508, beautify the face image material picture to obtain a beautified material picture.
[0081] Specifically, the face image material picture may be input into the skin beautification model, and the face image material picture may be subjected to skin beautification processing, thereby obtaining a beautified material picture.
[0082] S510, performing image super-resolution processing on the beautified material picture to obtain an optimized material picture.
[0083] Specifically, the beautified material picture may be input into an image super-resolution processing model, and image super-resolution processing may be performed on the beautified material picture to improve the resolution of the beautified material picture, thereby obtaining an optimized material picture.
[0084] After obtaining at least one optimized material picture, the at least one optimized material picture can be saved so that the AI photo application can subsequently generate a photo picture based on the at least one optimized material picture.
[0085] In one implementation, to avoid too many optimized material images, optimized material images with total scores ranked before a preset order can be selected from at least one optimized material image according to the total scores of the facial features quality score dimension of the optimized material images, thereby obtaining a preset number of optimized material images. Afterwards, the AI photo application can generate photo images based on the preset number of optimized material images.
[0086] For example, assuming that the optimized material pictures obtained are Picture 1, Picture 2, Picture 3, and Picture 4, and the number is 4, and the preset number of optimized material pictures based on which the AI photo application generates photo pictures is 3, assuming that the total score of the facial features quality scoring dimension of Picture 1 is 800, and the total score of the facial features quality scoring dimension of Picture 2 is 900, assuming that the total score of the facial features quality scoring dimension of Picture 3 is 700, and the total score of the facial features quality scoring dimension of Picture 4 is 750, after sorting Picture 1, Picture 2, Picture 3, and Picture 4 from high to low according to the total score of the facial features quality scoring dimension, the order is: Picture 2, Picture 1, Picture 4, Picture 3, then the top 3 pictures, i.e., Picture 2, Picture 1, and Picture 4, can be determined as the optimized material pictures for generating photo pictures.
[0087] The material picture processing method of the embodiment of the present application obtains the original material picture, and annotates the original material picture with dimensional information of each of multiple predefined dimensions, including basic attribute and status dimension, facial feature quality score dimension, and picture quality score dimension; based on the dimensional information of each of the multiple dimensions, the original material picture is screened, and the qualified material picture obtained after screening is optimized, which can improve the quality of the material picture and facilitate the extraction of facial features based on high-quality material pictures to generate AI portrait pictures, thereby improving the generation quality of AI portrait pictures.
[0088] In addition, Figure 2 Corresponding to the material picture processing method shown, the embodiment of the present application also provides a material picture processing device. Figure 6 FIG. 6 is a schematic diagram of a module of a material image processing device 600 provided by an embodiment of the present application. Figure 6 As shown, the material picture processing device 600 includes:
[0089] The acquisition module 601 is used to acquire at least one original material picture;
[0090] The labeling module 602 is used to label the original material image with dimension information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension;
[0091] A screening module 603 is used to screen the at least one original material picture based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, so as to obtain at least one qualified material picture that can be used to generate a photo picture;
[0092] The optimization module 604 is used to perform image optimization processing on the qualified material image to obtain at least one optimized material image for generating a photo image.
[0093] The material picture processing device of the embodiment of the present application obtains the original material picture, and annotates the original material picture with dimensional information of each of the predefined multiple dimensions, including the basic attribute and state dimension, the facial feature quality scoring dimension, and the picture quality scoring dimension; based on the dimensional information of each of the multiple dimensions, the original material picture is screened, and the qualified material picture obtained after the screening is optimized, which can improve the quality of the material picture and facilitate the extraction of facial features based on the high-quality material picture to generate AI photo pictures, thereby improving the generation quality of AI photo pictures.
[0094] Specifically, the annotation module includes:
[0095] A face recognition unit, used to input the original material picture into a face detection model to detect whether the original material picture contains a face image;
[0096] a machine annotation unit, configured to input the original material picture into an information annotation model to annotate the original material picture with dimension information of each of the predefined multiple dimensions when the original material picture contains a face image;
[0097] The manual review unit is used to output the original material picture to a reviewer when the original material picture does not contain a face image, so that the reviewer can manually mark the dimensional information of each dimension of the predefined multiple dimensions for the original material picture when manually reviewing the original material picture to find out that the original material picture contains a face image.
[0098] Optionally, the machine labeling unit is also used to input the original material picture into a portrait style recognition model to identify the portrait style of the face image when the original material picture includes a face image; and according to the portrait style, input the original material picture into an information labeling model corresponding to the portrait style to label the original material picture with dimensional information of each dimension of the predefined multiple dimensions.
[0099] Optionally, the device further comprises:
[0100] A storage module, configured to store the manually annotated original material picture into an annotated sample data set after the dimension information of each dimension in the predefined multiple dimensions is manually annotated for the original material picture, so as to update the annotated sample data set;
[0101] The training module is used to train the information annotation model based on the updated annotated sample data set at every set time period to update the information annotation model.
[0102] Specifically, the screening module includes:
[0103] a qualified candidate screening unit, configured to screen the at least one original material picture based on the basic attributes and the dimension information of the state dimension annotated on the original material picture, to obtain at least one qualified candidate material picture;
[0104] The qualified screening unit screens the at least one qualified candidate material picture based on the dimension information of the facial features quality score dimension and the dimension information of the image quality score dimension annotated on the qualified candidate material picture to obtain at least one qualified material picture.
[0105] Specifically, the qualified candidate screening unit includes:
[0106] The screening strategy acquisition subunit is used to acquire a pre-configured feature screening strategy; the feature screening strategy includes: screening out material images marked with any of the following dimensional information: multiple people, closed eyes, crooked mouth, face obstruction, abnormal skin color;
[0107] a qualified candidate screening subunit, configured to screen the at least one original material picture based on the feature screening strategy and the basic attributes and state dimension information annotated on the original material picture to obtain the qualified candidate material picture;
[0108] The standard screening unit comprises:
[0109] A subunit for obtaining the standard score, used to obtain the standard score of each dimension in the pre-configured facial features quality scoring dimension and the image quality scoring dimension;
[0110] The qualified screening subunit is used to determine the qualified candidate material picture as a qualified material picture when the dimensional information of each dimension in the facial feature quality scoring dimension and the image quality scoring dimension annotated on the qualified candidate material picture is greater than the qualified score of the corresponding dimension.
[0111] Specifically, the optimization module includes:
[0112] an optimization candidate screening unit, configured to screen the at least one qualified material picture based on the similarity between the qualified material pictures in the at least one qualified material picture, to obtain at least one optimization candidate material picture;
[0113] A positive image detection unit, used to detect whether the optimization candidate material picture is a positive image material picture;
[0114] A rotation unit, used for rotating the optimization candidate material picture to a positive image material picture when the optimization candidate material picture is not a positive image material picture;
[0115] A capture unit, configured to capture a face image region of the optimization candidate material picture to obtain a face image material picture when the optimization candidate material picture is a normal image material picture;
[0116] The beautification unit is used to beautify the face image material picture to obtain a beautified material picture.
[0117] Obviously, the data processing device in the embodiment of the present application can be used as the above Figure 2 The execution subject of the material picture processing method shown in FIG. Figure 2 The functions realized are as follows. Since the principles are the same, they will not be described here.
[0118] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.
[0119] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0120] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0121] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0122] Get at least one original material picture;
[0123] Annotating the original material image with dimensional information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension;
[0124] Based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture;
[0125] The qualified material picture is subjected to image optimization processing to obtain at least one optimized material picture for generating a portrait picture.
[0126] Specifically, the marking of the dimension information of each dimension in the predefined multiple dimensions for the original material image includes:
[0127] Inputting the original material picture into a face detection model to detect whether the original material picture contains a face image;
[0128] When the original material picture includes a face image, the original material picture is input into the information annotation model to annotate the original material picture with dimension information of each dimension of the predefined multiple dimensions;
[0129] When the original material picture does not contain a face image, the original material picture is output to a reviewer, so that the reviewer can manually mark the dimensional information of each dimension of the predefined multiple dimensions for the original material picture when manually reviewing that the original material picture contains a face image.
[0130] Optionally, when the original material picture includes a face image, the original material picture is input into an information annotation model, and dimension information of each dimension of the predefined multiple dimensions is annotated for the original material picture, including:
[0131] When the original material picture includes a face image, the original material picture is input into a portrait style recognition model to recognize the portrait style of the face image;
[0132] According to the portrait style, the original material picture is input into the information annotation model corresponding to the portrait style, and the dimension information of each dimension in the predefined multiple dimensions is annotated for the original material picture.
[0133] Optionally, after manually labeling the dimension information of each dimension of the predefined multiple dimensions for the original material image, the manually labeled original material image is stored in a labeled sample data set to update the labeled sample data set;
[0134] At set time intervals, the information annotation model is trained based on the updated annotated sample data set to update the information annotation model.
[0135] Specifically, the screening of the at least one original material picture based on the dimension information of each dimension of the multiple dimensions annotated by the original material picture to obtain at least one qualified material picture that can be used to generate a photo picture includes:
[0136] Based on the basic attributes and the dimension information of the state dimension annotated on the original material picture, the at least one original material picture is screened to obtain at least one qualified candidate material picture;
[0137] Based on the dimensional information of the facial features quality scoring dimension and the dimensional information of the picture quality scoring dimension annotated on the qualified candidate material picture, the at least one qualified candidate material picture is screened to obtain at least one qualified material picture.
[0138] Specifically, the screening of the at least one original material picture based on the basic attributes and the dimension information of the state dimension annotated on the original material picture to obtain at least one qualified candidate material picture includes:
[0139] Obtain a preconfigured feature screening strategy; the feature screening strategy includes: screening out material images annotated with any of the following dimensional information: multiple people, closed eyes, crooked mouth, face obstruction, abnormal skin color;
[0140] Based on the feature screening strategy and the dimension information of the basic attributes and status dimensions annotated on the original material picture, the at least one original material picture is screened to obtain the qualified candidate material picture;
[0141] The step of screening the at least one qualified candidate material picture based on the dimension information of the facial features quality score dimension and the dimension information of the image quality score dimension annotated on the qualified candidate material picture to obtain at least one qualified material picture includes:
[0142] Obtain the compliance scores of each dimension in the pre-configured facial features quality scoring dimension and image quality scoring dimension;
[0143] When the dimension information of each dimension in the facial feature quality score dimension and the image quality score dimension annotated on the qualified candidate material picture is greater than the qualified score of the corresponding dimension, the qualified candidate material picture is determined as the qualified material picture.
[0144] Specifically, the performing image optimization processing on the at least one qualified material image to obtain at least one optimized material image for generating a portrait image includes:
[0145] Based on the similarity between each qualified material picture in the at least one qualified material picture, the at least one qualified material picture is screened to obtain at least one optimized candidate material picture;
[0146] Detecting whether the optimization candidate material picture is a positive material picture;
[0147] When the optimization candidate material picture is not a normal material picture, rotating the optimization candidate material picture to become a normal material picture;
[0148] When the optimization candidate material picture is a positive material picture, performing face image region interception on the optimization candidate material picture to obtain a face image material picture;
[0149] The face image material picture is beautified to obtain a beautified material picture.
[0150] The above application Figure 2 The method performed by the material picture processing device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of this specification can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0151] It should be understood that the electronic device of the embodiment of the present application can implement the material picture processing device in Figure 2 Functions of the embodiments shown. Since the principles are the same, the embodiments of the present application will not be described in detail here.
[0152] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0153] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 2 The method of the embodiment shown is specifically used to perform the following operations:
[0154] Get at least one original material picture;
[0155] Annotating the original material image with dimensional information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension;
[0156] Based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture;
[0157] The qualified material picture is subjected to image optimization processing to obtain at least one optimized material picture for generating a portrait picture.
[0158] Specifically, the marking of the dimension information of each dimension in the predefined multiple dimensions for the original material image includes:
[0159] Inputting the original material picture into a face detection model to detect whether the original material picture contains a face image;
[0160] When the original material picture includes a face image, the original material picture is input into the information annotation model to annotate the original material picture with dimension information of each dimension of the predefined multiple dimensions;
[0161] When the original material picture does not contain a face image, the original material picture is output to a reviewer, so that the reviewer can manually mark the dimensional information of each dimension of the predefined multiple dimensions for the original material picture when manually reviewing that the original material picture contains a face image.
[0162] Optionally, when the original material picture includes a face image, the original material picture is input into an information annotation model, and dimension information of each dimension of the predefined multiple dimensions is annotated for the original material picture, including:
[0163] When the original material picture includes a face image, the original material picture is input into a portrait style recognition model to recognize the portrait style of the face image;
[0164] According to the portrait style, the original material picture is input into the information annotation model corresponding to the portrait style, and the dimension information of each dimension in the predefined multiple dimensions is annotated for the original material picture.
[0165] Optionally, after manually labeling the dimension information of each dimension of the predefined multiple dimensions for the original material image, the manually labeled original material image is stored in a labeled sample data set to update the labeled sample data set;
[0166] At set time intervals, the information annotation model is trained based on the updated annotated sample data set to update the information annotation model.
[0167] Specifically, the screening of the at least one original material picture based on the dimension information of each dimension of the multiple dimensions annotated by the original material picture to obtain at least one qualified material picture that can be used to generate a photo picture includes:
[0168] Based on the basic attributes and the dimension information of the state dimension annotated on the original material picture, the at least one original material picture is screened to obtain at least one qualified candidate material picture;
[0169] Based on the dimensional information of the facial features quality scoring dimension and the dimensional information of the picture quality scoring dimension annotated on the qualified candidate material picture, the at least one qualified candidate material picture is screened to obtain at least one qualified material picture.
[0170] Specifically, the screening of the at least one original material picture based on the basic attributes and the dimension information of the state dimension annotated on the original material picture to obtain at least one qualified candidate material picture includes:
[0171] Obtain a preconfigured feature screening strategy; the feature screening strategy includes: screening out material images annotated with any of the following dimensional information: multiple people, closed eyes, crooked mouth, face obstruction, abnormal skin color;
[0172] Based on the feature screening strategy and the dimension information of the basic attributes and status dimensions annotated on the original material picture, the at least one original material picture is screened to obtain the qualified candidate material picture;
[0173] The step of screening the at least one qualified candidate material picture based on the dimension information of the facial features quality score dimension and the dimension information of the image quality score dimension annotated on the qualified candidate material picture to obtain at least one qualified material picture includes:
[0174] Obtain the compliance scores of each dimension in the pre-configured facial features quality scoring dimension and image quality scoring dimension;
[0175] When the dimension information of each dimension in the facial feature quality score dimension and the image quality score dimension annotated on the qualified candidate material picture is greater than the qualified score of the corresponding dimension, the qualified candidate material picture is determined as the qualified material picture.
[0176] Specifically, the performing image optimization processing on the at least one qualified material image to obtain at least one optimized material image for generating a portrait image includes:
[0177] Based on the similarity between each qualified material picture in the at least one qualified material picture, the at least one qualified material picture is screened to obtain at least one optimized candidate material picture;
[0178] Detecting whether the optimization candidate material picture is a positive material picture;
[0179] When the optimization candidate material picture is not a normal material picture, rotating the optimization candidate material picture to become a normal material picture;
[0180] When the optimization candidate material picture is a positive material picture, performing face image region interception on the optimization candidate material picture to obtain a face image material picture;
[0181] The face image material picture is beautified to obtain a beautified material picture.
[0182] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0183] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the protection scope of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the protection scope of this specification.
[0184] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0185] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0186] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. A material picture processing method, the method comprising: Get at least one original material picture; Annotating the original material image with dimensional information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension; Based on the dimension information of each dimension in the multiple dimensions marked by the original material picture, the at least one original material picture is screened to obtain at least one qualified material picture that can be used to generate a photo picture; The qualified material picture is subjected to image optimization processing to obtain at least one optimized material picture for generating a portrait picture.
2. According to the method of claim 1, the step of labeling the original material image with dimension information of each of the predefined multiple dimensions comprises: Inputting the original material picture into a face detection model to detect whether the original material picture contains a face image; When the original material picture includes a face image, the original material picture is input into the information annotation model to annotate the original material picture with dimension information of each dimension of the predefined multiple dimensions; When the original material picture does not contain a face image, the original material picture is output to a reviewer, so that the reviewer can manually mark the dimensional information of each dimension of the predefined multiple dimensions for the original material picture when manually reviewing that the original material picture contains a face image.
3. The method according to claim 2, wherein when the original material image includes a face image, the original material image is input into an information annotation model, and dimension information of each dimension of the predefined multiple dimensions is annotated for the original material image, comprising: When the original material picture includes a face image, the original material picture is input into a portrait style recognition model to recognize the portrait style of the face image; According to the portrait style, the original material picture is input into the information annotation model corresponding to the portrait style, and the dimension information of each dimension in the predefined multiple dimensions is annotated for the original material picture.
4. The method according to claim 2, further comprising: After manually labeling the dimension information of each dimension in the predefined multiple dimensions for the original material image, storing the manually labeled original material image in the labeled sample data set to update the labeled sample data set; At set time intervals, the information annotation model is trained based on the updated annotated sample data set to update the information annotation model.
5. The method according to claim 1, The step of screening the at least one original material picture based on the dimension information of each dimension in the multiple dimensions annotated by the original material picture to obtain at least one qualified material picture that can be used to generate a photo picture includes: Based on the basic attributes and the dimension information of the state dimension annotated on the original material picture, the at least one original material picture is screened to obtain at least one qualified candidate material picture; Based on the dimensional information of the facial features quality scoring dimension and the dimensional information of the picture quality scoring dimension annotated on the qualified candidate material picture, the at least one qualified candidate material picture is screened to obtain at least one qualified material picture.
6. The method according to claim 5, The step of screening the at least one original material picture based on the basic attributes and the dimension information of the state dimension annotated on the original material picture to obtain at least one qualified candidate material picture includes: Get preconfigured feature filtering strategies; The feature screening strategy includes: screening out material images with any of the following dimension information: multiple people, closed eyes, crooked mouth, face obstruction, abnormal skin color; Based on the feature screening strategy and the dimension information of the basic attributes and status dimensions annotated on the original material picture, the at least one original material picture is screened to obtain the qualified candidate material picture; The step of screening the at least one qualified candidate material picture based on the dimension information of the facial features quality score dimension and the dimension information of the image quality score dimension annotated on the qualified candidate material picture to obtain at least one qualified material picture includes: Obtain the compliance scores of each dimension in the pre-configured facial features quality scoring dimension and image quality scoring dimension; When the dimension information of each dimension in the facial feature quality score dimension and the image quality score dimension annotated on the qualified candidate material picture is greater than the qualified score of the corresponding dimension, the qualified candidate material picture is determined as the qualified material picture.
7. According to the method of claim 1, the step of performing image optimization processing on the at least one qualified material image to obtain at least one optimized material image for generating a portrait image comprises: Based on the similarity between each qualified material picture in the at least one qualified material picture, the at least one qualified material picture is screened to obtain at least one optimized candidate material picture; Detecting whether the optimization candidate material picture is a positive material picture; When the optimization candidate material picture is not a normal material picture, rotating the optimization candidate material picture to become a normal material picture; When the optimization candidate material picture is a positive material picture, performing face image region interception on the optimization candidate material picture to obtain a face image material picture; The face image material picture is beautified to obtain a beautified material picture.
8. A material picture processing device, the device comprising: An acquisition module, used to acquire at least one original material picture; A labeling module, used to label the original material image with dimension information of each of the predefined multiple dimensions; the multiple dimensions include: basic attribute and state dimension, facial features quality rating dimension, and image quality rating dimension; A screening module, configured to screen the at least one original material picture based on the dimension information of each dimension of the multiple dimensions annotated by the original material picture, to obtain at least one qualified material picture that can be used to generate a photo picture; The optimization module is used to perform image optimization processing on the qualified material image to obtain at least one optimized material image for generating a photo image.
9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.
10. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
Face image quality evaluation method based on multidirectional evaluation standard and system thereof
CN108960087A
Automatic face auditing method, system, device and readable storage medium
CN111222433A
Image processing method and device, medium and computing equipment
CN115115599A
Generating synthetic photo-realistic images
US20200320341A1
Styled image generation method, model training method, apparatus, device, and medium
US20230401682A1
