Image processing method and device, model training method and device, electronic equipment and medium

Abstract

The present disclosure provides an image processing method and device, a model training method and device, an electronic device and a medium, relates to the field of image processing, and in particular to the technical field of computer vision and deep learning. The specific implementation scheme is as follows: detecting an eye state of a target image displayed in an image display interface; in response to the target image including a target object whose eye state does not meet a set condition, displaying an eye adjustment control in the image display interface; in response to detecting an interaction operation on the eye adjustment control, extracting a visual feature of the target object, the visual feature at least including a facial local feature; and based on the visual feature, adjusting the eye of the target object to obtain an adjusted target image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to the fields of computer vision and deep learning, and particularly to an image processing method, a model training method, an apparatus, an electronic device, and a medium. Background Technology

[0002] During image capture, unexpected factors such as blinking and lighting interference often lead to abnormal eye appearances in photos, including closed eyes, inconsistent gaze direction, and dull eyes. While advancements in image processing technology have enabled general facial retouching, such as blemish removal and expression adjustment, the effectiveness of eye retouching still needs improvement. Summary of the Invention

[0003] This disclosure provides an image processing method, a model training method, an apparatus, an electronic device, and a medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising:

[0005] Perform eye state detection on the target image displayed in the image display interface;

[0006] In response to the presence of a target object in the target image whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface;

[0007] In response to detecting an interaction with the eye adjustment control, visual features of the target object are extracted, the visual features including at least local facial features;

[0008] Based on the visual features, eye adjustments are made to the target object to obtain the adjusted target image.

[0009] According to another aspect of this disclosure, a model training method is provided, comprising:

[0010] Obtain the model to be trained and sample images, wherein the sample images include sample objects whose eye state does not meet the set conditions, and the sample images correspond to eye annotation data;

[0011] The sample image is input into the model to be trained to extract the visual features of the sample object, and based on the visual features, the eye adjustment of the sample object is performed to obtain the adjusted sample image output by the model to be trained; wherein, the visual features include at least local facial features;

[0012] The model to be trained is trained based on the eye data and eye annotation data of the sample object in the adjusted sample image.

[0013] According to another aspect of this disclosure, an image processing apparatus is provided, comprising:

[0014] The detection module is used to detect the eye state of the target image displayed in the image display interface;

[0015] The control display module is used to display eye adjustment controls on the image display interface in response to a target object in the target image whose eye state does not meet the set conditions.

[0016] A feature extraction module is used to extract visual features of the target object in response to detecting an interactive operation on the eye adjustment control, the visual features including at least local facial features;

[0017] The adjustment module is used to perform eye adjustments on the target object based on the visual features to obtain an adjusted target image.

[0018] According to another aspect of this disclosure, a model training apparatus is provided, comprising:

[0019] The acquisition module is used to acquire the model to be trained and sample images, wherein the sample images include sample objects whose eye state does not meet the set conditions, and the sample images correspond to eye annotation data;

[0020] An input module is used to input the sample image into the model to be trained to extract the visual features of the sample object, and to perform eye adjustments on the sample object based on the visual features to obtain the adjusted sample image output by the model to be trained; wherein, the visual features include at least local facial features;

[0021] The training module is used to train the model to be trained based on the eye data and eye annotation data of the sample object in the adjusted sample image.

[0022] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image processing method proposed in one aspect of this disclosure or the model training method proposed in another aspect of this disclosure.

[0026] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing the computer to perform the image processing method proposed in one aspect of this disclosure or the model training method proposed in another aspect of this disclosure.

[0027] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method proposed in one aspect of this disclosure or the model training method proposed in another aspect of this disclosure.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0030] Figure 1 This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure;

[0031] Figure 2 This is a schematic flowchart of an image processing method provided according to another embodiment of the present disclosure;

[0032] Figure 3 This is a schematic flowchart of an image processing method provided according to another embodiment of the present disclosure;

[0033] Figure 4 This is a schematic flowchart of a model training method provided according to another embodiment of the present disclosure;

[0034] Figure 5 This is a schematic flowchart of a model training method provided according to another embodiment of the present disclosure;

[0035] Figure 6 A schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure;

[0036] Figure 7 A schematic diagram of the structure of a model training device according to an embodiment of the present disclosure;

[0037] Figure 8 This is a block diagram of an electronic device used to implement the image processing method or model training method of the embodiments of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] In related technologies, the repair function relies on the user to actively trigger it, such as manually selecting the "closed eye repair" tool to perform closed eye repair, which results in a lengthy user operation chain.

[0040] Therefore, to address at least one of the aforementioned problems, this disclosure proposes an image processing method, a model training method, an apparatus, an electronic device, and a medium. This disclosure can be applied to cloud storage, image processing tools, etc.

[0041] The image processing method, model training method, apparatus, electronic device, and medium of the present disclosure are described below with reference to the accompanying drawings.

[0042] Figure 1 This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure.

[0043] This disclosure illustrates an example where the image processing method is configured in an image processing apparatus, which can be applied to any electronic device to enable the electronic device to perform text processing functions.

[0044] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0045] like Figure 1 As shown, the image processing method may include the following steps S101 to S104:

[0046] Step S101: Perform eye state detection on the target image displayed in the image display interface.

[0047] The image display interface includes an image preview interface, an image editing interface, and other interfaces that can display images. The image display interface may include interactive controls such as image cropping controls, image deletion controls, and image sharing controls. The target image refers to at least one image displayed in the image display interface. For example, the object in the target image includes people and / or animals. Eye state detection includes at least one of the following: closed eye detection, inconsistent gaze direction detection, and dull gaze detection.

[0048] As an example, eye state detection can be performed on a target image using a trained eye state detection model.

[0049] In the case of eye state detection being closed-eye detection, as an example, the EAR (Eye Aspect Ratio) corresponding to each object in the target image can be obtained; closed-eye detection can then be performed on the target image based on the EAR.

[0050] It should be noted that face detection can be performed on the target image first to obtain face detection boxes, and then eye state detection can be performed based on each face detection box.

[0051] In step S102, in response to the target image including a target object whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface.

[0052] The target object can be at least one object in the target image that has closed eyes, inconsistent gaze direction, or a dull gaze; the eye adjustment control is used to trigger the eye adjustment function, and the size, color, and display position of the eye adjustment control can be preset. For example, the eye adjustment control can be an eye adjustment button.

[0053] Therefore, when a target image is detected to contain a target object whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface. This transforms the "user finds function" mode into a "function reaches user" mode, lowering the operational threshold and improving the user experience.

[0054] Step S103: In response to detecting an interaction with the eye adjustment control, extract the visual features of the target object, including at least local facial features.

[0055] Interactive operations can include click operations, double-click operations, and touch operations; visual features can include facial features, hair color, and skin color; facial features can include eye features, nose features, and the position of facial features.

[0056] As an example, visual features of a target object can be extracted using an encoder.

[0057] Step S104: Based on visual features, perform eye adjustments on the target object to obtain the adjusted target image.

[0058] As an example, based on the correspondence between visual features and open-eye images, an open-eye image matching the visual features of the target object can be queried in an open-eye image database; based on the queried open-eye image, the eyes of the target object can be adjusted to obtain the adjusted target image.

[0059] As another example, visual features can be input into a trained decoder to obtain an adjusted target image output by the decoder; wherein the decoder is used to indicate the mapping relationship between the visual features and the adjusted target image.

[0060] Therefore, the decoder can accurately map the adjusted target image based on visual features, and efficiently complete the eye adjustment.

[0061] The image processing method of this disclosure performs eye state detection on the target image displayed in the image display interface. In response to the target image including a target object whose eye state does not meet the set conditions, an eye adjustment control is displayed in the image display interface. This can transform the "user finds function" mode into the "function reaches user" mode, reducing the operation threshold. Based on the visual characteristics of the target object, the eye adjustment of the target object can realize automated and personalized eye repair and adjustment, improving the user experience.

[0062] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0063] Visual features include eyebrow features, facial light and shadow distribution features, and eye closure features. To clearly illustrate how the aforementioned features are extracted in any embodiment of this disclosure, this disclosure also proposes an image processing method.

[0064] Figure 2 This is a schematic flowchart of an image processing method provided according to another embodiment of the present disclosure.

[0065] like Figure 2 As shown, the image processing method may include the following steps S201 to S206:

[0066] Step S201: Perform eye state detection on the target image displayed in the image display interface.

[0067] In step S202, in response to the target image including a target object whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface.

[0068] The explanations of steps S201 and S202 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0069] Step S203: In response to detecting an interactive operation on the eye adjustment control, based on the target image, obtain the first region image corresponding to the eyebrow region of the target object; based on the first region image, obtain the eyebrow texture features of the target object; based on the eyebrow texture features, obtain the eyebrow features.

[0070] The first region image can be obtained based on the eyebrow detection box; eyebrow texture features are used to indicate the direction, thickness, density, etc. of eyebrow texture.

[0071] In any embodiment of this disclosure, the first region image is converted to grayscale to obtain a corresponding first grayscale image; based on the first grayscale image, a grayscale co-occurrence matrix is ​​obtained; and based on the grayscale co-occurrence matrix, eyebrow texture features are determined.

[0072] Among them, statistical quantities such as contrast, energy, and entropy can be calculated based on the gray-level co-occurrence matrix, and eyebrow texture features can be determined based on the statistical quantities.

[0073] The grayscale co-occurrence matrix can reflect the spatial distribution pattern of grayscale levels in eyebrows and accurately reflect their texture characteristics. Therefore, obtaining eyebrow texture features based on the grayscale co-occurrence matrix helps to match the original texture of eyebrows when making eye adjustments, so that the adjusted and repaired eyes and eyebrows can be naturally connected, avoiding a stiff feeling and improving the realism and naturalness of eye adjustment and repair.

[0074] In any embodiment of this disclosure, a grayscale conversion is performed on a first region image to obtain a corresponding first grayscale image; a first LBP value corresponding to any pixel in the first grayscale image is obtained based on a set local binary mode (LBP) operator; and eyebrow texture features are obtained based on the first LBP value corresponding to any pixel.

[0075] The LBP operator can include an n*n LBP operator and a circular LBP operator. Here, n is a positive integer.

[0076] As an example, an LBP feature map can be obtained based on the first LBP value corresponding to any pixel in the first region image; the LBP feature map can then be used as the eyebrow texture feature.

[0077] As another example, the occurrence count of the first LBP value can be counted; based on the occurrence count of each first LBP value, the eyebrow texture features can be obtained.

[0078] The LBP operator takes a neighborhood as the center of a pixel, compares the gray values ​​of the neighboring pixels with those of the center pixel, and generates a binary code. It can perceive subtle differences in gray values ​​between pixels and thus accurately capture the gray values ​​and texture changes of the eyebrows. Therefore, based on the first LBP value, the texture features of the eyebrows are determined, which helps to achieve a natural connection between the texture of the eyes and eyebrows, making the adjusted target image more realistic and natural.

[0079] One possible approach is to use eyebrow texture features as eyebrow features.

[0080] As another possible implementation, image features are extracted from the first region image to obtain image features; based on the image features and eyebrow texture features, eyebrow features are obtained.

[0081] Specifically, an image feature extraction network can be used to extract image features from the first region image; the image features and eyebrow texture features can be spliced ​​together to obtain eyebrow features.

[0082] Image features can reflect global information such as the overall shape and color distribution of eyebrows, while eyebrow texture features can reflect detailed information such as the local texture direction and thickness of eyebrows. Therefore, combining image features and eyebrow texture features for eye adjustments can make the adjusted eye and eyebrow shape and texture more compatible, and the repair effect more realistic.

[0083] Step S204: Based on the target image, obtain the second region image corresponding to the facial region of the target object; perform grayscale conversion on the second region image to obtain a second grayscale image; based on the second grayscale image, obtain a grayscale histogram to obtain the facial light and shadow distribution features.

[0084] The facial region of the target object can refer to the entire facial region of the target object, or it can refer to a local facial region including the eye area of ​​the target object.

[0085] Among them, the grayscale histogram can reflect the distribution of pixels with different grayscale values ​​in the image, and thus indirectly reflect the overall distribution of light and shadow.

[0086] There may be shadows around the eyes, such as the shadow cast by a hat brim on the face. The characteristics of facial light and shadow distribution can reflect the distribution of light and shadow around the eyes. Therefore, adjusting the eyes based on the characteristics of facial light and shadow distribution can avoid the inconsistency between the light and shadow of the eyes and the light and shadow of the surrounding area, that is, the abnormal brightness of the eyes after adjustment, and ensure a natural transition of light and shadow around the eyes.

[0087] Step S205: Based on the target image, obtain the key points of the target object's eyes; based on the key points of the eyes, determine the eyelid closure rate, and determine the degree of eye closure feature based on the eyelid closure rate.

[0088] Key points for the eyes can include key points at the inner corner of the eye, key points at the outer corner of the eye, and key points on the upper and lower eyelids.

[0089] As an example, the eyelid closure rate of the target object's two eyes can be obtained; the eyelid closure rate of the target object's two eyes can be used as a feature of the degree of eye closure.

[0090] By combining the characteristics of the degree of eye closure for eye adjustments, the condition of the eyes can be accurately located, and then the shape of the eyelids and skin texture can be adjusted in a targeted manner, making the eye adjustment effect more realistic and natural.

[0091] It should be noted that the order of steps 203-205 is not intended to limit the order in which the various features in the visual features are acquired. That is, this disclosure does not limit the order in which the eyebrow features, facial light and shadow distribution features, and eye closure features are acquired.

[0092] In any embodiment of this disclosure, in response to detecting an interactive operation on the eye adjustment control, an eye adjustment interface is displayed, and the eye adjustment progress is displayed on the eye adjustment interface.

[0093] The eye adjustment interface does not include the interactive controls displayed in the image display interface. In other words, in response to the detection of an interactive operation on the eye adjustment controls, the user cannot interact with the target image.

[0094] For example, the progress of eye adjustments can be displayed on the eye adjustment interface via a toast notification.

[0095] It should be noted that in response to the detection of interactive operations on the eye adjustment controls, the eye adjustment interface can be displayed and visual features can be extracted simultaneously.

[0096] Therefore, in response to the detection of interactive operations on the eye adjustment controls, the eye adjustment interface is displayed, enabling the eye adjustment process to be started quickly; displaying the adjustment progress allows users to understand the progress of the eye adjustment more intuitively.

[0097] In any embodiment of this disclosure, in response to detecting a closing operation on the eye adjustment interface, the eye adjustment interface is exited and the target image before eye adjustment is displayed on the image display interface; or, in response to detecting a saving operation on the adjusted target image displayed in the eye adjustment interface, the adjusted target image is stored.

[0098] For example, the eye adjustment interface includes page control controls, save controls, and exit / save controls. The page control controls are enabled during the display of the eye adjustment interface.

[0099] The closing operation for the eye adjustment interface can refer to closing the page control controls, or the interactive operation for the exit save control.

[0100] As a result, users can quickly return from the eye adjustment interface to the target image displayed on the image display interface, or quickly and conveniently save the adjusted target image on the eye adjustment interface, meeting different user needs and thus improving the user experience.

[0101] Step S206: Based on visual features, perform eye adjustments on the target object to obtain the adjusted target image.

[0102] The explanation of step S206 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0103] The image processing method of this disclosure combines eyebrow texture features to obtain eyebrow features, which helps to achieve a natural connection between the eye and eyebrow textures, making the adjusted target image more realistic and natural. Combining facial light and shadow distribution features for eye adjustment can avoid inconsistencies between the light and shadow of the eyes and the light and shadow of the surrounding area, ensuring a natural transition of light and shadow in the eyes. Combining eye closure degree features for eye adjustment can accurately locate the eye state, and then make targeted adjustments to eyelid shape, skin texture, etc., making the eye adjustment effect more realistic and natural.

[0104] Visual features also include the texture features of clothing collars and the geometric features of headwear. To clearly illustrate how the aforementioned features are obtained in any embodiment of this disclosure, this disclosure also proposes an image processing method.

[0105] Figure 3 This is a schematic flowchart of an image processing method provided according to another embodiment of the present disclosure.

[0106] like Figure 3 As shown, the image processing method may include the following steps S301 to S305:

[0107] Step S301: Perform eye state detection on the target image displayed in the image display interface.

[0108] In step S302, in response to the target image including a target object whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface.

[0109] The explanation of steps S301 and S302 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0110] Step S303: In response to detecting an interactive operation on the eye adjustment control, based on the target image, obtain a third region image corresponding to the neckline area of ​​the clothing of the target object; obtain the second LBP value corresponding to any pixel in the third region image; and determine the texture features of the clothing neckline based on the number of occurrences of any second LBP value.

[0111] The third region image may include the neckline area of ​​clothing, and may also include a partial facial region of the target object, such as the chin area. For example, the neckline of clothing may include the neckline of an academic gown.

[0112] The LBP operator for obtaining the second LBP value or the LBP operator for obtaining the first LBP value can be the same or different.

[0113] Adjusting the eyes by combining them with the texture features of the neckline area can not only match the overall texture and style of the garment after adjustment, but also preserve the edges of the garment, avoiding blurry edges after eye adjustment.

[0114] In any embodiment of this disclosure, the texture feature of the clothing collar includes at least one of the following: the average number of occurrences of the second LBP value; the variance of the number of occurrences of the second LBP value; and the energy of the number of occurrences of the second LBP value.

[0115] The mean reflects the overall distribution of the texture, the variance reflects the degree of dispersion of the distribution, and the energy represents the complexity of the texture. Combining these values ​​can comprehensively reflect the texture characteristics of the clothing collar.

[0116] Step S304: Perform edge detection on the headwear of the target object, and obtain the contour information of the headwear based on the edge detection results; obtain the geometric features of the headwear based on the contour information.

[0117] Among them, headwear includes headbands, hats, and other clothing / decorations worn on the head.

[0118] The geometric features of headwear can reflect the outline and shape of headwear, providing reference information for eye adjustments. Therefore, adjusting the eyes based on the geometric features of headwear can make the adjusted eyes and headwear more visually harmonious and natural.

[0119] In any embodiment of this disclosure, the headwear includes a hat, and the outline information includes at least one of the following: the edge length of the brim; the curvature of the brim; the edge length of the crown; and the curvature of the crown. Combining the foregoing data allows for a more accurate characterization of the hat's outline.

[0120] For example, hats may include academic caps, baseball caps, etc.

[0121] Based on the geometric features of the hat, the relationship between the headwear and the face can be accurately located. Therefore, when adjusting the eyes in combination with the geometric features of the hat, on the one hand, the outline of the headwear can be avoided, and on the other hand, the adjusted eyes can be more harmonious and natural with the overall headwear.

[0122] It should be noted that the order of steps 303-304 is not intended to limit the order in which the features in the visual features are acquired. That is, this disclosure does not limit the order in which the texture features of the clothing collar and the geometric features of the headwear are acquired.

[0123] Step S305: Based on visual features, perform eye adjustments on the target object to obtain the adjusted target image.

[0124] The explanation of step S305 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0125] The image processing method of this disclosure combines the texture features of the clothing collar area for eye adjustment, which not only achieves matching between the adjusted eyes and the overall texture style of the clothing, but also preserves the edges of the clothing, avoiding blurring of the clothing edges after eye adjustment. The geometric features of the headwear reflect the outline and shape of the headwear, providing reference information for eye adjustment. Combining the geometric features of the headwear for eye adjustment makes the adjusted eyes and headwear more visually harmonious and natural, improving the quality of the adjusted image.

[0126] This disclosure also proposes a model training method. Figure 4 This is a schematic flowchart of a model training method provided according to another embodiment of the present disclosure.

[0127] This disclosure illustrates the example of the model training method being configured in a model training device, which can be applied to any electronic device to enable the electronic device to perform model training functions.

[0128] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0129] like Figure 4 As shown, the model training method may include the following steps S401 to S403:

[0130] Step S401: Obtain the model to be trained and sample images, wherein the sample images include sample objects whose eye state does not meet the set conditions, and the sample images have corresponding eye annotation data.

[0131] The model to be trained may include an encoder and a decoder to be trained; the number of sample images is at least one, and the sample images may be images of various scenes and under various lighting conditions; the number of sample objects whose eye states do not meet the set conditions included in the sample images is at least one; the eye annotation data may include reference eye images whose eye states corresponding to the sample objects meet the set conditions.

[0132] For example, the scene may include graduation season scene, travel scene, daily life scene, artistic photography scene, etc.; the lighting conditions may include outdoor strong light, indoor lighting, etc.

[0133] Step S402: Input the sample image into the model to be trained to extract the visual features of the sample object, and perform eye adjustments on the sample object based on the visual features to obtain the adjusted sample image output by the model to be trained; wherein, the visual features include at least local facial features.

[0134] The encoder to be trained is used to extract the visual features of the sample object; the decoder to be trained is used to perform eye adjustments on the sample object based on the visual features and output the adjusted sample image.

[0135] The encoder to be trained includes a first branch and a second branch. When the visual features include eyebrow features, facial light and shadow distribution features, eye closure features, clothing collar texture features, and head and clothing geometric features, the first branch can be used to extract eyebrow features and eye closure features, while the second branch can be used to extract facial light and shadow distribution features, clothing collar texture features, and head and clothing geometric features.

[0136] Step S403: Train the model to be trained based on the eye data and eye annotation data of the sample objects in the adjusted sample images.

[0137] The eye data may include the eye images of the sample objects in the adjusted sample images.

[0138] As an example, a first loss can be calculated based on the difference between the eye data and eye annotation data of the sample object in the adjusted sample image; the model to be trained can then be trained based on the first loss.

[0139] For example, a first loss can be calculated based on the image difference between the eye image of the sample object and the reference eye image in the adjusted sample image.

[0140] The model training method of this disclosure involves inputting a sample image into a model to be trained to extract visual features of the sample object, and then adjusting the eyes of the sample object based on the visual features to obtain an adjusted sample image output by the model to be trained. The visual features include at least local facial features. The model to be trained is then trained based on the eye data and eye annotation data of the sample object in the adjusted sample image. This enables the model to learn the extraction of visual features and the correlation between visual features and eye adjustments, thereby achieving automated and personalized eye repair and adjustment, and improving user experience.

[0141] The sample images also correspond to clothing labeling data and eye occlusion labeling data. In order to clearly illustrate how the model to be trained is based on the eye data and eye labeling data of the sample object in the adjusted sample image in any embodiment of this disclosure, this disclosure also proposes a model training method.

[0142] Figure 5 This is a schematic flowchart of a model training method provided according to another embodiment of the present disclosure.

[0143] like Figure 5 As shown, the model training method may include the following steps S501 to S504:

[0144] Step S501: Determine the first loss based on the difference between the eye data and eye annotation data of the sample object in the adjusted sample image.

[0145] In any embodiment of this disclosure, eye data includes eye key point data, and correspondingly, eye annotation data includes eye key point annotation data.

[0146] For example, the eye key point data includes the inner corner of the eye key point, the outer corner of the eye key point, and the key points of the upper and lower eyelids.

[0147] Eye key point data can accurately locate the structure and shape of the eye. Therefore, by training the model based on the eye key point data and the eye key point annotation data, the model can learn the rules of eye features in depth. When repairing and adjusting, it can better fit the real eye shape, avoid repair distortion, make the repaired eye natural and vivid, and improve the repair effect and the overall image quality.

[0148] It should be noted that eye data can also include data used to assist in eye adjustments, such as key eyebrow point data.

[0149] Step S502: Determine the second loss based on the difference between the clothing data and clothing annotation data of the sample objects in the adjusted sample image.

[0150] In any embodiment of this disclosure, the clothing data includes at least one of the following: clothing neckline outline data, hat brim edge data, and shoulder line position data; correspondingly, the clothing annotation data includes at least one of the following: clothing neckline outline annotation data, hat brim edge annotation data, and shoulder line position annotation data.

[0151] As an example, edge detection can be performed on the adjusted sample image, and based on the edge detection results, at least one of the following can be obtained: clothing neckline outline data, hat brim edge data, and shoulder line position data.

[0152] By combining data on clothing neckline contours, hat brim edges, shoulder line positions, and corresponding annotations for model training, the model can not only consider more related information when making eye adjustments, but also avoid problems such as misalignment and blurring of clothing edges (such as clothing neckline contours, hat brim edges, and shoulder lines).

[0153] Step S503: Based on the difference between the eye occlusion data and the eye occlusion annotation data of the sample object in the adjusted sample image, determine the third loss.

[0154] Among them, eye occlusion data can be used to indicate whether there is an obstruction in the eye, and can also be used to indicate the location, shape, outline and other information of the obstruction.

[0155] In any embodiment of this disclosure, eye occlusion data includes hair occlusion data and / or glasses occlusion data; correspondingly, eye occlusion annotation data includes hair occlusion annotation data and / or glasses occlusion annotation data.

[0156] For example, hair occlusion data includes data such as the outline and color of the hair obscuring the eyes; eyeglass occlusion data includes data such as the outline, color, and position of the eyeglasses.

[0157] By combining hair occlusion data, glasses occlusion data, and corresponding labeled data for model training, the model can fully learn the eye features under occlusion conditions, avoiding problems such as hair distortion and glasses disappearance after eye adjustments.

[0158] Step S504: Train the model to be trained based on at least one of the first loss, the second loss, and the third loss.

[0159] As an example, the model to be trained can be trained based on any one or any two of the first loss, second loss, and third loss.

[0160] As another example, the first loss, the second loss, and the third loss can be weighted and summed to obtain the target loss; the model to be trained is then trained based on the target loss.

[0161] The model training method of this disclosure determines a first loss based on the difference between the eye data and eye annotation data of the sample object in the adjusted sample image. This first loss can accurately measure the adjustment error of eye repair, guide the model to optimize the eye repair effect, and improve the repair accuracy. It also determines a second loss based on the difference between the clothing data and clothing annotation data of the sample object in the adjusted sample image. This second loss enables the model to learn clothing-related features, assisting in eye repair and enhancing the coordination between eye repair and the overall image. Furthermore, it determines a third loss based on the difference between the eye occlusion data and eye occlusion annotation data of the sample object in the adjusted sample image. This third loss allows the model to adapt to eye occlusion situations, enabling accurate eye repair even with occlusion, thus improving repair robustness. Training the model based on at least one of the first, second, and third losses comprehensively optimizes the model from multiple factors, improving the quality of the model's eye repair.

[0162] To clearly illustrate how this disclosure trains the model and how it is used for eye adjustments, the following example demonstrates how to restore closed eyes in graduation season scene images:

[0163] Model training process:

[0164] (1) Obtain a training dataset containing graduation season portraits. The sample images in the training dataset cover different lighting conditions (strong outdoor light, indoor ceremony light), different styles of graduation gowns (including different colors such as black / red / blue, high collar / low collar, different fabrics such as draped fabrics), the angle of wearing the graduation cap (straight, slightly tilted), and hairstyles (short hair for boys, shoulder-length hair for girls, updo), etc. Among them, the sample images correspond to eye key point annotation data, clothing collar outline annotation data, cap brim edge annotation data, shoulder line position annotation data, hair occlusion annotation data, and glasses occlusion annotation data.

[0165] (2) Input the sample images in the training dataset into the model to be trained; wherein the model to be trained includes an encoder and a decoder, and the encoder includes a first branch and a second branch.

[0166] (3) Extract the eyebrow features and eye closure features of the sample object in the sample image through the first branch, and extract the facial light and shadow distribution features, clothing collar texture features and head clothing geometric features of the sample object in the sample image through the second branch.

[0167] (4) The features extracted from the first branch and the second branch are fused to obtain fused features; the fused features are input into the decoder for eye adjustment to obtain the adjusted sample image output by the decoder.

[0168] (5) Obtain eye key point data, clothing neckline outline data, hat brim edge data, shoulder line position data, hair occlusion data and glasses occlusion data from the adjusted sample image, and combine the differences between the aforementioned data and the corresponding labeled data to train the model to be trained, and obtain the trained target model.

[0169] Model application process:

[0170] Upon detecting an interactive operation with the eye adjustment controls displayed in the image display interface, the target image displayed in the image display interface is input into the target model. The first branch of the encoder in the target model extracts the eyebrow features and eye closure features of the target object in the target image. The second branch of the encoder extracts the facial light and shadow distribution features, clothing collar texture features, and headwear geometric features of the target object. The decoder decodes the fusion features extracted by the first and second branches to adjust the eyes of the target object, and finally obtains the adjusted target image.

[0171] This disclosure optimizes features such as hairstyle and clothing (e.g., graduation gown collar and academic cap) for graduation portraits, ensuring that the eye lighting and shadow are consistent with the academic cap shadow after the eyes are closed, and that the skin tone of the eyes matches the collar area of ​​the graduation gown. This avoids problems such as blurred clothing edges, abnormal eye brightness caused by conflicts between lighting and the hat structure, and distorted hairstyle texture.

[0172] With the above Figures 1 to 3 Corresponding to the image processing method provided in the embodiments, this disclosure also provides an image processing apparatus. Since the image processing apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 3 The image processing method provided in the embodiments corresponds to the image processing apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0173] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided according to an embodiment of the present disclosure.

[0174] like Figure 6 As shown, the image processing device 600 may include: a detection module 610, a control display module 620, a feature extraction module 630, and an adjustment module 640.

[0175] Among them, the detection module 610 is used to detect the eye state of the target image displayed in the image display interface;

[0176] The control display module 620 is used to display eye adjustment controls on the image display interface in response to a target object in the target image whose eye state does not meet the set conditions.

[0177] The feature extraction module 630 is used to extract visual features of the target object in response to the detection of an interaction operation with the eye adjustment control. The visual features include at least local facial features.

[0178] The adjustment module 640 is used to perform eye adjustments on the target object based on visual features to obtain the adjusted target image.

[0179] In one possible implementation of this disclosure, the visual features include eyebrow features, and the feature extraction module 630 is specifically used for:

[0180] Based on the target image, obtain the first region image corresponding to the eyebrow region of the target object;

[0181] Based on the first region image, obtain the eyebrow texture features of the target object;

[0182] Eyebrow features are obtained based on eyebrow texture features.

[0183] In one possible implementation of this disclosure, the feature extraction module 630 is specifically used for:

[0184] Image features are extracted from the first region image to obtain the image features;

[0185] Eyebrow features are obtained based on image features and eyebrow texture features.

[0186] In one possible implementation of this disclosure, the feature extraction module 630 is specifically used for:

[0187] Perform grayscale conversion on the first region image to obtain the corresponding first grayscale image;

[0188] Based on the first grayscale image, obtain the grayscale co-occurrence matrix;

[0189] Based on the gray-level co-occurrence matrix, the texture features of eyebrows are determined.

[0190] In one possible implementation of this disclosure, the feature extraction module 630 is specifically used for:

[0191] Perform grayscale conversion on the first region image to obtain the corresponding first grayscale image;

[0192] Based on the set local binary mode LBP operator, the first LBP value corresponding to any pixel in the first grayscale image is obtained;

[0193] The eyebrow texture features are obtained based on the first LBP value corresponding to any pixel.

[0194] In one possible implementation of this disclosure, the visual features include facial light and shadow distribution features, and the feature extraction module 630 is specifically used for:

[0195] Based on the target image, obtain the second region image corresponding to the facial region of the target object;

[0196] Perform grayscale conversion on the second region image to obtain a second grayscale image;

[0197] Based on the second grayscale image, a grayscale histogram is obtained to obtain the facial light and shadow distribution features.

[0198] In one possible implementation of this disclosure, the visual features include eye closure degree features, and the feature extraction module 630 is specifically used for:

[0199] Based on the target image, obtain the key points of the target object's eyes;

[0200] Based on key points of the eye, the eyelid closure rate is determined, and the degree of eye closure characteristics are determined based on the eyelid closure rate.

[0201] In one possible implementation of this disclosure, the visual features include the texture features of the clothing collar, and the feature extraction module 630 is specifically used for:

[0202] Based on the target image, obtain the third region image corresponding to the neckline area of ​​the clothing of the target object;

[0203] Obtain the second LBP value corresponding to any pixel in the third region image;

[0204] The texture features of the clothing collar are determined based on the number of occurrences of any second LBP value.

[0205] In one possible implementation of this disclosure, the texture feature of the clothing collar includes at least one of the following:

[0206] The average number of times the second LBP value occurs;

[0207] The variance of the number of occurrences of the second LBP value;

[0208] The energy of the number of times the second LBP value appears.

[0209] In one possible implementation of this disclosure, the visual features include geometric features of the headwear, and the feature extraction module 630 is specifically used for:

[0210] Edge detection is performed on the headwear of the target object, and the contour information of the headwear is obtained based on the edge detection results;

[0211] Based on the contour information, the geometric features of the headwear are obtained.

[0212] In one possible implementation of this disclosure, the headwear includes a hat, and the outline information includes at least one of the following:

[0213] The length of the brim of the hat;

[0214] The curvature of the brim;

[0215] Length of the brim of the hat crown;

[0216] The curvature of the hat crown.

[0217] In one possible implementation of this disclosure, the adjustment module 640 is specifically used for:

[0218] The visual features are input into the decoder to obtain the adjusted target image output by the decoder; the decoder is used to indicate the mapping relationship between the visual features and the adjusted target image.

[0219] In one possible implementation of this disclosure, the device further includes an interface display module, used for:

[0220] In response to the detection of an interaction with the eye adjustment controls, the eye adjustment interface is displayed, and the progress of the eye adjustment is shown on the eye adjustment interface.

[0221] In one possible implementation of this disclosure, the device further includes an operation module for:

[0222] In response to detecting a closing operation on the eye adjustment interface, exit the eye adjustment interface and display the target image before eye adjustment on the image display interface;

[0223] Alternatively, in response to detecting a save operation for the adjusted target image displayed in the eye adjustment interface, the adjusted target image is stored.

[0224] The image processing apparatus of this disclosure performs eye state detection on the target image displayed in the image display interface. In response to the target image including a target object whose eye state does not meet the set conditions, it displays eye adjustment controls on the image display interface. This can transform the "user finds function" mode into the "function reaches user" mode, reducing the operation threshold. Based on the visual characteristics of the target object, it performs eye adjustment on the target object, which can realize automated and personalized repair and adjustment of the eyes, improving the user experience.

[0225] With the above Figures 4 to 5 Corresponding to the model training method provided in the embodiments, this disclosure also provides a model training apparatus. Since the model training apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 5The model training method provided in the embodiments corresponds to the model training device provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0226] Figure 7 This is a schematic diagram of the structure of a model training device provided according to an embodiment of the present disclosure.

[0227] like Figure 7 As shown, the model training device 700 may include: an acquisition module 710, an input module 720, and a training module 730.

[0228] The acquisition module 710 is used to acquire the model to be trained and sample images. The sample images include sample objects whose eye states do not meet the set conditions, and the sample images have corresponding eye annotation data.

[0229] The input module 720 is used to input sample images into the model to be trained in order to extract visual features of the sample objects, and to perform eye adjustments on the sample objects based on the visual features to obtain the adjusted sample images output by the model to be trained; wherein, the visual features include at least local facial features.

[0230] Training module 730 is used to train the model to be trained based on the eye data and eye annotation data of the sample objects in the adjusted sample images.

[0231] In one possible implementation of this disclosure, the sample image also corresponds to clothing labeling data and eye occlusion labeling data, and the training module 730 is specifically used for:

[0232] The first loss is determined based on the difference between the eye data and eye annotation data of the sample objects in the adjusted sample images;

[0233] The second loss is determined based on the difference between the clothing data and clothing annotation data of the sample objects in the adjusted sample images;

[0234] The third loss is determined based on the difference between the eye occlusion data and the eye occlusion annotation data of the sample objects in the adjusted sample images;

[0235] The model to be trained is trained based on at least one of the first loss, the second loss, and the third loss.

[0236] In one possible implementation of this disclosure, the eye data includes key eye point data;

[0237] Clothing data includes at least one of the following: clothing neckline outline data, hat brim data, and shoulder line position data;

[0238] Eye occlusion data includes hair occlusion data and / or glasses occlusion data.

[0239] The model training apparatus of this disclosure inputs a sample image into the model to be trained to extract visual features of the sample object, and performs eye adjustments on the sample object based on the visual features to obtain an adjusted sample image output by the model to be trained; wherein, the visual features include at least local facial features; the model to be trained is trained based on the eye data and eye annotation data of the sample object in the adjusted sample image; the model to be trained can learn the extraction of visual features and the correlation between visual features and eye adjustments, thereby realizing automated and personalized eye repair and adjustment, and improving user experience.

[0240] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0241] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0242] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0243] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0244] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods and / or model training methods. For example, in some embodiments, the image processing methods and / or model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing methods and / or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform image processing methods and / or model training methods by any other suitable means (e.g., by means of firmware).

[0245] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0246] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0247] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0248] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0249] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0250] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0251] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0252] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0253] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: Perform eye state detection on the target image displayed in the image display interface; In response to the presence of a target object in the target image whose eye state does not meet the set conditions, an eye adjustment control is displayed on the image display interface; In response to detecting an interaction with the eye adjustment control, visual features of the target object are extracted, the visual features including at least eyebrow features and / or clothing collar texture features; Based on the visual features, eye adjustments are made to the target object to obtain the adjusted target image.

2. The method according to claim 1, wherein, The visual features include eyebrow features, and the extraction of visual features of the target object includes: Based on the target image, obtain the first region image corresponding to the eyebrow region of the target object; Based on the first region image, obtain the eyebrow texture features of the target object; The eyebrow features are obtained based on the eyebrow texture features.

3. The method according to claim 2, wherein, The process of obtaining the eyebrow features based on the eyebrow texture features includes: Image features are extracted from the first region image to obtain image features; The eyebrow features are obtained based on the image features and the eyebrow texture features.

4. The method according to claim 2, wherein, The step of obtaining the eyebrow texture features of the target object based on the first region image includes: The first region image is converted to grayscale to obtain the corresponding first grayscale image; Based on the first grayscale image, obtain the grayscale co-occurrence matrix; The eyebrow texture features are determined based on the gray-level co-occurrence matrix.

5. The method according to claim 2, wherein, The step of obtaining the eyebrow texture features of the target object based on the first region image includes: The first region image is converted to grayscale to obtain the corresponding first grayscale image; Based on the set local binary mode LBP operator, the first LBP value corresponding to any pixel in the first grayscale image is obtained; The eyebrow texture feature is obtained based on the first LBP value corresponding to any pixel.

6. The method according to claim 1, wherein, The visual features include facial light and shadow distribution features, and the extraction of visual features of the target object includes: Based on the target image, obtain a second region image corresponding to the facial region of the target object; The second region image is converted to grayscale to obtain a second grayscale image; Based on the second grayscale image, a grayscale histogram is obtained to obtain the facial light and shadow distribution features.

7. The method according to claim 1, wherein, The visual features include eye closure degree features, and the extraction of visual features of the target object includes: Based on the target image, obtain the key points of the eye of the target object; Based on the key eye points, the eyelid closure rate is determined, and the degree of eye closure is determined based on the eyelid closure rate.

8. The method according to claim 1, wherein, The visual features include the texture features of the clothing collar, and the extraction of the visual features of the target object includes: Based on the target image, obtain a third region image corresponding to the neckline area of ​​the clothing of the target object; Obtain the second LBP value corresponding to any pixel in the third region image; The texture features of the clothing collar are determined based on the number of occurrences of any of the second LBP values.

9. The method according to claim 8, wherein, The texture feature of the clothing collar includes at least one of the following: The average number of occurrences of the second LBP value; The variance of the number of occurrences of the second LBP value; The energy of the number of times the second LBP value appears.

10. The method according to claim 1, wherein, The visual features include the geometric features of the head and clothing, and the extraction of the visual features of the target object includes: Edge detection is performed on the headwear of the target object, and the contour information of the headwear is obtained based on the edge detection results; Based on the contour information, the geometric features of the headwear are obtained.

11. The method according to claim 10, wherein, The headwear includes a hat, and the outline information includes at least one of the following: The length of the brim of the hat; The curvature of the brim; Length of the brim of the hat crown; The curvature of the hat crown.

12. The method according to any one of claims 1-11, wherein, The step of adjusting the target object based on the visual features to obtain the adjusted target image includes: The visual features are input into the decoder to obtain the adjusted target image output by the decoder; wherein the decoder is used to indicate the mapping relationship between the visual features and the adjusted target image.

13. The method according to any one of claims 1-11, wherein, The method further includes: In response to detecting an interaction with the eye adjustment control, an eye adjustment interface is displayed, and the progress of the eye adjustment is shown on the eye adjustment interface.

14. The method according to claim 13, wherein, The method further includes: In response to detecting a closing operation on the eye adjustment interface, the eye adjustment interface is exited, and the target image before eye adjustment is displayed on the image display interface; Alternatively, in response to detecting a save operation for the adjusted target image displayed in the eye adjustment interface, the adjusted target image is stored.

15. A model training method, comprising: Obtain the model to be trained and sample images, wherein the sample images include sample objects whose eye state does not meet the set conditions, and the sample images correspond to eye annotation data; The sample image is input into the model to be trained to extract the visual features of the sample object, and based on the visual features, the eye features of the sample object are adjusted to obtain the adjusted sample image output by the model to be trained; wherein, the visual features include at least eyebrow features and / or clothing collar texture features; The model to be trained is trained based on the eye data and eye annotation data of the sample object in the adjusted sample image.

16. The method according to claim 15, wherein, The sample images also correspond to clothing labeling data and eye occlusion labeling data. The step of training the model to be trained based on the eye data and eye labeling data of the sample object in the adjusted sample images includes: Based on the difference between the eye data and the eye annotation data of the sample object in the adjusted sample image, a first loss is determined; Based on the difference between the clothing data of the sample object in the adjusted sample image and the clothing annotation data, a second loss is determined; The third loss is determined based on the difference between the eye occlusion data and the eye occlusion annotation data of the sample object in the adjusted sample image; The model to be trained is trained based on at least one of the first loss, the second loss, and the third loss.

17. The method according to claim 16, wherein, The method includes at least one of the following: The eye data includes key eye point data; The clothing data includes at least one of the following: clothing neckline outline data, hat brim edge data, and shoulder line position data; The eye occlusion data includes hair occlusion data and / or glasses occlusion data.

18. An image processing apparatus, comprising: The detection module is used to detect the eye state of the target image displayed in the image display interface; The control display module is used to display eye adjustment controls on the image display interface in response to a target object in the target image whose eye state does not meet the set conditions. A feature extraction module is used to extract visual features of the target object in response to detecting an interactive operation on the eye adjustment control, wherein the visual features include at least eyebrow features and / or clothing collar texture features; The adjustment module is used to perform eye adjustments on the target object based on the visual features to obtain an adjusted target image.

19. A model training device, comprising: The acquisition module is used to acquire the model to be trained and sample images, wherein the sample images include sample objects whose eye state does not meet the set conditions, and the sample images correspond to eye annotation data; An input module is used to input the sample image into the model to be trained to extract the visual features of the sample object, and to perform eye adjustments on the sample object based on the visual features to obtain the adjusted sample image output by the model to be trained; wherein, the visual features include at least eyebrow features and / or clothing collar texture features; The training module is used to train the model to be trained based on the eye data and eye annotation data of the sample object in the adjusted sample image.

20. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-14 or 15-17.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14 or 15-17.

22. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-14 or 15-17.

Images