Image Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

Embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium. The method includes: in response to a special effect trigger operation, collecting a to-be-processed image including a target object; processing the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; and displaying the target special effect image on a target display interface. The technical solution provided by the embodiments of the present disclosure realizes fine adjustment of the pupil based on the pupil attribute condition label, adds a special effect to the eye, makes the generated special effect image consistent with the condition label, improves the accuracy of adding the eye special effect, and further makes the displayed image more realistic when the special effect image is displayed on the display interface, achieving the technical effect of meeting the user experience requirements.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of technology, more and more application software has entered users' lives, gradually enriching users' spare time, such as short video application software. Users can record their lives in the form of videos, photos, etc. Optionally, corresponding videos can be shot based on short video software. To further improve the richness and interest of video content, special effects processing is usually performed on video frames in the video.

[0003] When adding special effects to an image, the existing method usually uses software such as PS (Photoshop, image processing software) to process. It can also be to perform special effects processing on the image using a pre-trained adversarial model to obtain a special effect image.

[0004] When determining the special effect image by the above method, there are problems of large computational complexity and poor authenticity of the special effect image, which leads to a poor user experience. Summary of the Invention

[0005] Embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium to achieve the technical effect of adjusting pupil features in multiple aspects, thereby improving the authenticity of special effect images.

[0006] In a first aspect, embodiments of the present disclosure provide an image processing method, which includes:

[0007] In response to a special effect trigger operation, collect a to-be-processed image including a target object;

[0008] Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label;

[0009] Display the target special effect image on a target display interface.

[0010] In a second aspect, embodiments of the present disclosure further provide an image processing apparatus, which includes:

[0011] A to-be-processed image collection module, configured to collect a to-be-processed image including a target object in response to a special effect trigger operation;

[0012] A target special effect image acquisition module, configured to process the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil feature adjustment model, so as to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label;

[0013] An image display module, configured to display the target special effect image on a target display interface.

[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the device includes:

[0015] One or more processors;

[0016] A storage device, configured to store one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method according to any one of the embodiments of the present disclosure.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image processing method according to any one of the embodiments of the present disclosure is implemented.

[0019] The technical solution of the embodiment of the present disclosure, by responding to a special effect trigger operation, acquires an image to be processed including a target object, and then processes the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil feature adjustment model to obtain a target special effect image, and displays the target special effect image on a target display interface, realizes multi-faceted adjustment of pupil features, thereby improving the richness and vividness of image content. Description of the Drawings

[0020] Combined with the drawings and referring to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original components and elements are not necessarily drawn to scale.

[0021] Figure 1 It is a schematic flowchart of an image processing method provided by Embodiment 1 of the present disclosure;

[0022] Figure 2 It is a schematic flowchart of an image processing method provided by Embodiment 2 of the present disclosure;

[0023] Figure 3 It is a schematic flowchart of an image processing method provided by Embodiment 3 of the present disclosure;

[0024] Figure 4a Schematic flowchart of determining the first training sample provided in the third embodiment of the present disclosure;

[0025] Figure 4b Schematic diagram of the first theoretical image provided in the third embodiment of the present disclosure;

[0026] Figure 5 Schematic flowchart of an image processing method provided in the fourth embodiment of the present disclosure;

[0027] Figure 6a Schematic diagram of the eye structure segmentation image provided in the fourth embodiment of the present disclosure;

[0028] Figure 6b Schematic diagram of the eye reconstruction image with different pupil positions provided in the fourth embodiment of the present disclosure;

[0029] Figure 7 Schematic flowchart of an image processing method provided in the fifth embodiment of the present disclosure;

[0030] Figure 8 Schematic diagram of the eye image with pupil size change provided in the fifth embodiment of the present disclosure;

[0031] Figure 9 Schematic flowchart of an image processing method provided in the sixth embodiment of the present disclosure;

[0032] Figure 10a Schematic diagram of the eye reconstruction image provided in the sixth embodiment of the present disclosure;

[0033] Figure 10b Schematic diagram of the image to be used provided in the sixth embodiment of the present disclosure;

[0034] Figure 10c Schematic diagram of the image to be used provided in the sixth embodiment of the present disclosure;

[0035] Figure 10d Schematic diagram of the image to be used provided in the sixth embodiment of the present disclosure;

[0036] Figure 11 Block diagram of the structure of an image processing device provided in the seventh embodiment of the present disclosure;

[0037] Figure 12 Schematic diagram of the structure of an electronic device provided in the eighth embodiment of the present disclosure. Detailed implementation manners

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0039] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0040] The term "including" and its variations used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0041] It should be noted that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless clearly specified otherwise in the context, it should be understood as "one or more".

[0042] Before introducing the technical solution, an exemplary description of the application scenario can be given first. The technical solution of the present disclosure can be applied to any picture that requires special effect display. For example, during video shooting, special effects processing can be performed on the object being shot to obtain the target special effect picture to be displayed; it can also be applied during static image shooting. For example, after an image is taken by the built-in camera of the terminal device, the taken image is processed into a special effect image for special effect display. In this embodiment, the added special effects can be eye special effects of various objects. For example, the eye special effects can be special effects of pupil dilation, constriction, and relative offset of the pupil center point from the eye center point. In this embodiment, the target object can be a user or various pets captured.

[0043] Embodiment 1

[0044] Figure 1The flowchart of an image processing method provided by Embodiment 1 of the present disclosure is applicable to any image display scenario supported by the Internet for the situation of processing the eye image of a target object into a special effect image and displaying it. This method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC, or a server, etc. Any image display scenario can be executed by the server, the client, or the cooperation between the client and the server.

[0045] As Figure 1 , the method of this embodiment includes:

[0046] S110. In response to a special effect trigger operation, collect a to-be-processed image including the target object.

[0047] It should be noted that the above has briefly described various applicable scenarios and will not be elaborated specifically here. Among them, the device for executing the image processing method provided by the embodiments of the present disclosure can be integrated in an application software that supports image processing functions, and this software can be installed in an electronic device. Optionally, the electronic device can be a mobile terminal or a PC, etc. The application software can be a type of software for image / video processing, and its specific application software will not be elaborated one by one here as long as it can implement image / video processing. It can also be a specially developed application program to implement software for adding special effects and displaying special effects, or integrated in a corresponding page, and users can implement special effect addition processing through the page integrated in the PC.

[0048] Among them, the to-be-processed image can be understood as an image that needs to be processed. This image can be an image collected based on the application software or an image pre-stored by the application software from the storage space. In actual applications, an image including the target object can be captured in real time or periodically based on the application software, and at this time, special effects can be directly added to the object. It can also be that after detecting that the user triggers a special effect addition control, the image is sent to the server, and the server adds special effects to the target object in the collected to-be-processed image. Correspondingly, the to-be-processed image can include the target object, and the target subject can be a user or a pet. It should be noted that the video frames corresponding to the captured video can also be processed. For example, a target object corresponding to the captured video can be preset. When it is detected that the target object exists in the image corresponding to the video frame, the image corresponding to the video frame can be used as the to-be-processed image, so that subsequent special effect processing can be performed on the target object in each video frame image in the video.

[0049] It should be noted that the number of target objects in the same shooting scene can be one or more. Whether it is one or more, the technical solution provided by the present disclosure can be used to determine the special effect display image.

[0050] Specifically, when it is detected that a special effect needs to be added to the target object in the image to be processed, the image to be processed including the target object can be collected to add a special effect to the target object in the image to be processed, so as to obtain a target special effect image corresponding to the image to be processed.

[0051] In this embodiment, the special effect trigger operation includes at least one of the following: detecting that the target object is included in the field of view; detecting that the target object triggers a target special effect action; detecting that a special effect generation control is triggered.

[0052] Among them, the special effect generation control can be a button displayed on the display interface of the application software. The touch of this button indicates that the image to be processed needs to be collected and the special effect processing of the image to be processed is required. In practical applications, if the user triggers this button, it can be considered that the image function of the special effect display is to be triggered, that is, a corresponding special effect needs to be added to the target object. Among them, what the added special effect is specifically can be consistent with the special effect triggered by the user. The target special effect action can be a preset body movement. The preset body movement can be matched with the added special effect, or it can be understood that different special effects correspond to different body movements. The target special effect action in this technical solution can be a head-turning action or a blinking action. This method further improves the intelligence of special effect recognition and addition. For example, according to the shooting field of view range of the mobile terminal, it can be determined whether the body movement of the target object within the field of view is consistent with the preset body movement. If so, it means that the special effect addition operation is triggered. For example, the preset body movement is a "blinking" action. If the body movement of the target subject triggers a blinking action, it means that the special effect trigger operation is triggered. Another implementation method can be to judge whether the field of view range contains the target object according to the shooting field of view range of the mobile terminal. If so, it means that the special effect addition operation is triggered. For example, a pet cat can be preset as the target object in advance. When it is detected that the pet cat is included in the field of view area, it means that the special effect trigger operation is triggered.

[0053] S120. Process the image to be processed and at least one target condition label corresponding to the image to be processed based on the target pupil feature adjustment model to obtain a target special effect image.

[0054] Among them, the target pupil feature adjustment model is a model for adjusting pupil attributes. The target conditional label can be understood as a pupil attribute label. The target conditional label can be a pupil position label and / or a pupil size label. The target special effect image is an image that matches the image to be processed and at least one target conditional label. Exemplarily, when the target conditional label is label 1 with a pupil radius of 5 mm, after processing the image to be processed and label 1 based on the target pupil feature adjustment model, a target special effect image with a pupil size of 5 mm can be obtained.

[0055] It should be noted that the target conditional label identical to the pupil feature of the target object can be set according to actual needs; the target conditional label can also be set according to the ultimately required effect. In practical applications, to improve the interaction efficiency with the user, the target conditional label can be built into the program so that when processing the image to be processed, the corresponding target conditional label can be retrieved.

[0056] Specifically, after receiving the image to be processed, the server can input the image to be processed and at least one pre-set target conditional label into the target pupil adjustment model to obtain a target special effect image that matches the target conditional label.

[0057] S130. Display the target special effect image on the target display interface.

[0058] Specifically, after obtaining the target special effect image, in order to enable the user to understand the final special effect image, the target special effect image can be displayed on the target display interface. Among them, the target display interface is the display interface of the terminal device corresponding to when uploading the image to be processed.

[0059] The technical solution of the embodiment of the present disclosure, by responding to the special effect trigger operation, collecting the image to be processed including the target object, and then processing the image to be processed and at least one target conditional label corresponding to the image to be processed based on the target pupil feature adjustment model to obtain the target special effect image, and displaying the target special effect image on the target display interface, realizes the multi-faceted adjustment of the pupil feature, thereby improving the richness and vividness of the image content.

[0060] Embodiment Two

[0061] Figure 2 It is a schematic flowchart of an image processing method provided by the second embodiment of the present disclosure. On the basis of the foregoing embodiment, S120 is further refined. Among them, the same or corresponding technical terms as those in the above embodiment will not be described in detail here.

[0062] As Figure 2 shown, the method specifically includes the following steps:

[0063] S210. In response to a special effect trigger operation, collect a to-be-processed image including a target object.

[0064] S220. Determine at least one target condition label corresponding to the to-be-processed image.

[0065] In this embodiment, determining at least one target condition label includes: obtaining at least one pre-set target condition label; or, determining at least one target condition label corresponding to the target object according to the pose information of the target object in the to-be-processed image.

[0066] It can be understood that: the target condition labels can be pre-stored in a database. When it is necessary to generate what kind of special effects, the target condition labels matching the special effects can be directly retrieved. After receiving the to-be-processed image, the target condition labels and the to-be-processed image can be used as the inputs of a target pupil adjustment model to obtain a target special effect image.

[0067] In order to improve the adaptability between the target object in the to-be-processed image and the added special effects, condition labels adapted to the morphological information can be determined according to the morphological information corresponding to the target object in the to-be-processed image as the target condition labels. For example, if the target object tilts its head and does not fully expose its eyes, the eye state of the target object can be analyzed through an algorithm to determine what size the eyes should be processed into to achieve the best special effect. For example, if the best pupil size for the special effect is 5mm, then the pupil size of 5mm can be used as the target condition label. It can also be to determine the target condition labels while improving the adaptability between the target object and the added special effects and meeting the user's special effect requirements. For example, the special effect requirements can be to add a questioning special effect, a happy special effect, an angry special effect, etc., and determine the target condition labels in such a situation.

[0068] Among them, the pose information can be the head deflection angle, or the eye state, or the facial expression.

[0069] It can be understood that the target condition labels are determined according to the pose information of the target object in the to-be-processed image. For example, an algorithm can be used to analyze the pose information of the target object, and then according to the actual requirement of making a special effect, it can be determined what kind of eye special effect with condition labels should be added to the target object for the best effect. For example, when wanting to make a questioning expression for a cat, an algorithm can be used to analyze what size the cat's eyes should be adjusted to and what position the pupils should be adjusted to for the best effect, and the pupil size and pupil position at this time can be used as the target condition labels.

[0070] It should also be noted that before determining the target condition label, the position of the eye can be divided and calibrated to determine the condition label corresponding to different positions of the pupil in the eye. Optionally, the label corresponding to the coincidence of the center point of the pupil and the center point of the eye is recorded as 0.5. If the center point of the pupil is left-offset relative to the center point of the eye, the label can be decreased sequentially; if the center point of the pupil is right-shifted relative to the center point of the eye, the label can be increased sequentially. The leftmost label is 0 and the rightmost label is 1. The pupil size label can be defined by the radius of the pupil. Optionally, for pupil radii of 2 mm, 5 mm, 8 mm, the pupil size labels can be 2, 5, 8, etc. The target condition label can be determined from multiple pre-set pupil size labels and pupil position labels.

[0071] In this embodiment, the at least one target condition label includes a target size label corresponding to the target display size of the pupil, and / or a target position label corresponding to the target relative display position of the pupil in the eye.

[0072] It can be understood that: the target size label is used to represent the label of the pupil display radius. The target position label is used to represent the label of the offset information of the pupil center point in the eye. The target size label and the target position label can be used as the target condition label. In this embodiment, the number of labels in the target condition label has a certain relationship with the trained target pupil adjustment model. The specific way can be: if the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is changed, the at least one target condition label includes the target size label; if the target pupil feature adjustment model is trained under the condition that the size label is fixed and the position label is changed, the at least one target condition label includes the target position label; if the target pupil feature adjustment model is trained under the condition that the size label and the position label are changed, the at least one target condition label includes the target position label and the target size label.

[0073] It can be understood that the target pupil feature adjustment model can be pre-trained by using the condition label. For example, the model can be trained under the condition that the condition label remains unchanged, the condition label changes partially, or the condition label combination changes.

[0074] Exemplarily, if the transformed conditional label is a position label, the size label can be fixed. A p2p neural network is trained with different position labels and corresponding images to be trained, so as to obtain a target pupil feature adjustment model. The input of the pupil feature adjustment model can be a position label and a corresponding image to be processed, and the pupil position in the obtained target special effect map is consistent with the position label, and the pupil size is consistent with the fixed size label. If the transformed conditional label is a size label, the position label can be fixed. A p2p neural network is trained with different size labels and corresponding images to be trained, so as to obtain a target pupil feature adjustment model. The input of the pupil feature adjustment model can be a size label and a corresponding image to be processed, and the pupil size in the obtained target special effect map is consistent with the size label, and the pupil position is consistent with the fixed position label. If the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is transformed, the position label and the size label can be randomly combined, and the combined label and the original image are used as the input of the model to be trained, and the corresponding theoretical image is used as the output of the model, so as to train the target pupil feature adjustment model.

[0075] S230. Use the at least one target conditional label and the image to be processed as the input of the target pupil adjustment model to obtain the target special effect map.

[0076] In this embodiment, the image to be processed and the at least one selected target conditional label can be input into the target pupil adjustment model. The model can process the image to be processed and output a target special effect map under the target conditional label. For example, if the target conditional label is a pupil size of 8 mm and a pupil position of 0.5, a special effect map with a pupil radius change of 8 mm and the pupil moved to the middle position of the eye can be generated.

[0077] In this embodiment, the specific manner of processing the image to be processed based on the target pupil feature adjustment model can be: if the at least one target conditional label includes a target size label and a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the image to be processed to match the target size label and the target position label to obtain the target special effect map; if the at least one target conditional label includes a target size label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the image to be processed to match the target size label and the position label used when training the target pupil feature adjustment model; if the at least one target conditional label includes a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the image to be processed to match the target position label and the size label used when training the target pupil feature adjustment model.

[0078] S240. Display the target special effect image on the target display interface.

[0079] In the technical solution of the embodiments of the present disclosure, by obtaining a target condition label including a target size label and a target position label corresponding to the image to be processed, and then using at least one target condition label and the image to be processed as the input of the target pupil adjustment model to obtain a target special effect map, it realizes fine adjustment of the pupil based on the pupil size and position condition labels, improves the accuracy and richness of adding eye special effects, and further improves the technical effect of the user experience.

[0080] Embodiment III

[0081] Figure 3 FIG. is a schematic flowchart of an image processing method provided in Embodiment III of the present disclosure. On the basis of the foregoing embodiments, a target pupil feature adjustment model can be pre-trained to determine a target special effect map based on the target pupil feature adjustment model. The specific implementation manner can refer to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above embodiments will not be described in detail here.

[0082] As Figure 3 shown, the method specifically includes the following steps:

[0083] S310. Obtain a first training sample set.

[0084] To improve the accuracy of the model, as many and rich training samples as possible can be obtained. The first training sample set includes a plurality of first training samples, and each first training sample includes a first original input image and a first theoretical image. The first original input image is an image captured by a photographing device, or an image reconstructed by an image reconstruction model, or an image pre-stored in a storage space. At this time, no corresponding special effects are added to the first original input image, that is, the pupil features in the image are not modified. At the same time, the first original input image must include the target part of the target object. The target part can be the eye part. The first theoretical image is generated based on a pre-trained target pupil position adjustment model and a target pupil size adjustment model. At this time, the first theoretical image is the image that the pupil feature adjustment model to be trained is expected to output.

[0085] Among them, the target pupil position adjustment model is used to adjust the pupil position of the object in the first original input image. The target pupil size adjustment model is used to adjust the pupil size of the object in the first original input image. The original input image can be processed successively based on the target pupil position adjustment model and the target pupil size adjustment model to obtain the first theoretical image for training the target pupil feature adjustment model.

[0086] In this embodiment, the determination of the first theoretical image for each training sample may be as follows: obtain the first original input image; input the first original input image and the to-be-displayed pupil position label into the target pupil position adjustment model to obtain the to-be-used image; input the to-be-used image and the to-be-displayed pupil size label into the target pupil size adjustment model to obtain the first theoretical image.

[0087] Among them, the to-be-displayed pupil position label refers to the label of the position information of the pupil in the eye after the target pupil position model processes the image. For example, if it is expected that the pupil deviates from the center point of the eye by 20%, the to-be-displayed pupil position label may be 0.2; if it is expected to deviate from the center point of the eye by 90%, the to-be-displayed pupil position label may be 0.9. The to-be-displayed pupil position label 0.9 and the first original input image can be input into the target pupil position adjustment model at the same time to obtain the to-be-used image with the pupil deviating from the center point of the eye by 90%. The to-be-displayed pupil size label is the label of the pupil size after the target pupil size model processes the image. For example, if it is expected that the radius of the pupil in the eye is 5mm, the to-be-displayed pupil size label may be 5; if it is expected that the radius of the pupil in the eye is 2mm, the to-be-displayed pupil size label is 2. The to-be-displayed pupil size label 5 and the image with the pupil deviating from the center point by 90% can be used as the input of the target pupil size adjustment model to obtain the first theoretical image with the pupil deviating from the center point by 90% and the pupil size of 5mm at the same time.

[0088] By using the above method, the first theoretical image corresponding to each first original input image can be obtained, and thus the first training sample can be determined based on the first original input image and the corresponding first theoretical image. Based on each first training sample, the first training sample set is determined.

[0089] The process of determining the first training sample described above may be: as Figure 4a shown, input the first original input image (such as Figure 4b A) and the to-be-displayed pupil position label into the target pupil position adjustment model to obtain the to-be-used image with the pupil moved to the specified position. Input the to-be-used image and the to-be-displayed pupil size label into the target pupil size adjustment model to obtain the special effect image with the pupil size changed to a fixed size. At this time, the obtained special effect image is the first theoretical image with the pupil position moved and the size changed, see Figure 4b B. It is just a schematic diagram and is not limited thereto.

[0090] It should be noted that in practical applications, it is also possible that the first original input image is first processed by the target pupil size adjustment model and then processed by the pupil size model to obtain the first theoretical image. As long as the first theoretical image can be obtained, the specific processing order of the models is not limited. On the basis of the above technical solution, there may be a target pupil adjustment model that wants to obtain a fixed pupil size or pupil position. In this case, optimization can be performed when determining the first theoretical image.

[0091] Optionally, if the pupil size label to be displayed is a fixed value, each first theoretical image is an image with the same pupil size but different pupil positions; if the pupil position label to be displayed is a fixed value, each first theoretical image is an image with the same pupil position but different pupil sizes; if the pupil size label to be displayed changes and the pupil position label to be displayed changes, each first theoretical image is an image with different pupil positions and pupil sizes.

[0092] It can be understood that if you want to obtain a first theoretical image with a fixed pupil size, after obtaining each image to be used, you can keep the pupil size label to be displayed as a fixed value. Use the image to be used and the fixed pupil size label to be displayed as the input of the target pupil size adjustment model to obtain a first theoretical image with a fixed pupil size and a certain offset in pupil position. At this time, the pupil sizes of all first theoretical images are the same. If you want to obtain a first theoretical image with a fixed pupil position, you can set the pupil size label to be displayed as a fixed value. After inputting the first original input image and the fixed pupil position label to be displayed into the target pupil position adjustment model, an image to be used with the same pupil position can be obtained. Further, input the image to be used and the pupil size label to be displayed that can change into the target pupil size adjustment model to obtain a first theoretical image with an unchanged pupil position but a changing pupil size. If you want to obtain a first theoretical image with both a fixed pupil size and a fixed pupil position, it can be stated that both the pupil size label to be displayed and the pupil position label to be displayed are fixed values. If you want to obtain a first theoretical image with a changing pupil size and pupil position, the above-described detailed method can be used to determine the first theoretical image.

[0093] S320. For each first training sample, use the first original image of the current first training sample as the input of the pupil feature adjustment model to be trained, and use the first theoretical image as the output of the pupil feature adjustment model to be trained, and adjust the model parameters of the pupil feature adjustment model to be trained.

[0094] Among them, the model parameters in the pupil feature adjustment model to be trained are default values. The model parameters in the pupil feature adjustment model to be trained are corrected through the training samples to obtain the target pupil feature adjustment model.

[0095] It should be noted that the processing method for each training sample is the same. Taking the processing of one training sample as an example for illustration.

[0096] Specifically, input the first original input image in the current training sample into the pupil feature adjustment model to be trained, and use the first theoretical image as the output of the pupil feature adjustment model to be trained, so as to adjust the model parameters in the pupil feature adjustment model to be trained, and continuously optimize the model parameters, so that when the original image is input, the pupil feature adjustment model to be trained can output the expected image.

[0097] S330. Take the convergence of the loss function of the pupil feature adjustment model to be trained as the training objective to obtain the target pupil feature adjustment model.

[0098] Among them, the convergence of the preset loss function can be taken as the training objective. When it is determined that the preset loss function of the pupil feature adjustment model to be trained converges, it means that this model can be used as the target pupil feature adjustment model.

[0099] It should be noted that since the model parameters in the pupil feature adjustment model to be trained are not corrected, then, at this time, there may also be corresponding differences between the special effect image output by the model and the first theoretical image corresponding to the current training sample. The model parameters in the pupil feature adjustment model to be trained can be continuously corrected so that the pupil feature adjustment model to be trained can output the expected first theoretical image.

[0100] S340. In response to the special effect trigger operation, collect the image to be processed including the target object.

[0101] S350. Process the image to be processed and at least one target condition label corresponding to the image to be processed based on the target pupil feature adjustment model to obtain the target special effect image.

[0102] S360. Display the target special effect image on the target display interface.

[0103] The technical solution of the embodiment of the present disclosure determines the first theoretical image in the training sample based on the target pupil position adjustment model and the target pupil size adjustment model. Train the pupil feature adjustment model to be trained based on the first original input image and the corresponding first theoretical image to obtain the target pupil feature adjustment model, so that the target pupil feature adjustment model adjusts the pupil features of the collected image, improving the technical effects of the efficiency and richness of adding special effects.

[0104] Embodiment 4

[0105] Figure 5It is a schematic flowchart of an image processing method provided in the fourth embodiment of the present disclosure. On the basis of the foregoing embodiment, a target pupil position adjustment model can be trained to generate an image of pupil position change based on the target pupil position adjustment model, and then a corresponding first theoretical image can be generated based on the image. The specific implementation manner can refer to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above embodiment will not be elaborated here.

[0106] As Figure 5 shown, the method specifically includes the following steps:

[0107] S410. Obtain a second training sample set.

[0108] In order to improve the accuracy of the model, as many and as rich training samples as possible can be obtained. The second training sample set includes a plurality of second training samples, and each second training sample includes a second input image with different pupil positions, a second theoretical output image, and a pupil position label corresponding to the second theoretical output image. The second input image and the second theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil.

[0109] At this time, the second input image is the input image of the pupil position adjustment model to be trained, and this image is a real eye image reconstructed by the target pupil model. The second theoretical output image is the image expected to be output by the pupil position adjustment model to be trained. Optionally, for the same image including the eye part, two images with different pupil positions in the eye are generated based on the pupil reconstruction model. The user can mark the sclera area and the pupil area in the eye part of the image according to actual needs, and reconstruct the image based on the target pupil reconstruction model to obtain an image with a pupil offset. Since the pupil area can be marked according to actual needs during marking, the pupil in the reconstructed image may have a certain difference from the pupil in the original image.

[0110] Exemplarily, referring to Figure 6a , a large number of images containing eyes can be collected by using a camera device, or a large number of images containing eyes can be generated based on an image generation model. The pupil area and the sclera area of the eye can be marked. Optionally, a pupil position is randomly selected in the eye and filled with black, and the area other than the black area in the eye socket is filled with white to obtain the sclera area, as shown in A in Fig. 6a in the segmentation map. The image A is input into the target pupil reconstruction model, and the reconstructed image is as shown in Figure 6a B in. However, it should be noted that the above-mentioned images are only schematic diagrams and are not limited thereto.

[0111] Based on the above method, training samples for training the to-be-trained pupil position adjustment model can be constructed based on the target pupil reconstruction model. Refer to Figure 6b , for the same image, two to-be-input images with different pupil positions are respectively marked, and the two to-be-input images are respectively processed based on the target pupil reconstruction model to obtain two theoretical images with different pupil positions, that is, two corresponding real eye images can be obtained (such as Figure 6b A and B in). However, it should be noted that the above-mentioned images are only schematic diagrams and are not limited thereto. When obtaining the above samples, if it is necessary to use A in 6b as the input of the to-be-trained pupil position adjustment model and the output of the to-be-trained pupil position adjustment model in B in 6b. Then when determining B in 6b, the to-be-displayed pupil position label corresponding to the pupil position can be determined. Based on the above method, the input image, theoretical image, and pupil position label corresponding to the theoretical image in the second training sample can be obtained.

[0112] S420. For each second training sample, use the second input image and pupil position label of the current training sample as the input of the to-be-trained pupil position adjustment model, and use the second theoretical output image as the output of the to-be-trained pupil position adjustment model to adjust the model parameters of the to-be-trained pupil position adjustment model.

[0113] Among them, the model parameters in the to-be-trained pupil position adjustment model are default values. This model can be a p2p network model.

[0114] Exemplarily, continue to refer to Figure 6b , determine the position of the pupil in the B image in 6b to obtain the pupil position label. Input the pupil position label and the A image in 6b into the to-be-trained pupil position adjustment model, use B in 6b as the output of the model, and adjust the model parameters in the model.

[0115] Specifically, input the second input image in the current training sample into the to-be-trained pupil position adjustment model, and use the second theoretical output image as the output of the to-be-trained pupil position adjustment model, so as to adjust the model parameters in the to-be-trained pupil feature adjustment model, and continuously optimize the model parameters, so that when the original image and position label are input, the to-be-trained pupil position adjustment model can output an expected image consistent with the position label.

[0116] S430. Use the convergence of the loss function of the to-be-trained pupil position adjustment model as the training goal to obtain the target pupil position adjustment model.

[0117] Among them, the preset loss function convergence can be used as the training goal. When it is determined that the preset loss function of the to-be-trained pupil position adjustment model converges, it means that this model can be used as the target pupil position adjustment model.

[0118] It should be noted that since the model parameters in the to-be-trained pupil feature adjustment model are not corrected, there may be corresponding differences between the special effect image output by the model at this time and the second theoretical output image corresponding to the current training sample. The model parameters in the to-be-trained pupil position adjustment model can be continuously corrected so that the to-be-trained pupil feature adjustment model can output the desired second theoretical output image. In this embodiment, the advantage of determining the target pupil position adjustment model is that based on the obtained first theoretical image, and then based on the first theoretical image and the corresponding first original input image, the target pupil position adjustment model can be trained.

[0119] S440. In response to the special effect trigger operation, collect the to-be-processed image including the target object.

[0120] S450. Based on the target pupil feature adjustment model, process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image to obtain the target special effect image.

[0121] S460. Display the target special effect image on the target display interface.

[0122] The technical solution of the embodiment of the present disclosure can train the target pupil position adjustment model by determining the second input image and the second theoretical output image. Furthermore, based on the target pupil position adjustment model, a theoretical image consistent with some features in the second theoretical image can be obtained, and based on this, the first training sample can be obtained, improving the accuracy and convenience of determining the first training sample.

[0123] Embodiment Five

[0124] Figure 7 FIG. is a schematic flowchart of an image processing method provided in Embodiment Five of the present disclosure. On the basis of the foregoing embodiments, a target pupil size adjustment model can be trained to generate an image with a changing pupil size based on the target pupil size adjustment model, and then a corresponding first theoretical image can be generated based on this image. The specific implementation manner can refer to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above embodiments will not be described in detail here.

[0125] As Figure 7 shown, the method specifically includes the following steps:

[0126] S510. Obtain the third training sample set.

[0127] Among them, the third training sample set includes multiple third training samples. Each third training sample includes a third input image with different pupil sizes, a third theoretical output image, and a pupil size label corresponding to the third theoretical image. The third input image refers to the input of the model. The third theoretical output image refers to the output of the model. The third input image and the third theoretical output image are determined based on a pre-trained target pupil reconstruction model, which is used to reconstruct an image consistent with the marked pupil.

[0128] Exemplarily, a large number of images containing eyes are generated based on an image generation model. The pupil area and sclera area of the eyes can be marked, and then the marked images are input into the target pupil reconstruction model to obtain images with adjusted sizes. For example, referring to Figure 8 in the original image A, the pupil size is marked, and there are differences in the marked pupil sizes. The marked images are input into the target pupil reconstruction model to obtain eye images with changed pupil sizes. One of the images can be used as the input for training the pupil size adjustment model, and the other image can be used as the output for training the pupil size adjustment model. At this time, the input also includes the pupil size label of the output image. In this way, multiple third training samples can be obtained. The set of all third training samples is the third training sample set.

[0129] S520. For each third training sample, use the third input image and the pupil size label of the current training sample as the input of the to-be-trained pupil position adjustment model, and use the third theoretical output image as the output of the to-be-trained pupil size adjustment model to adjust the model parameters of the to-be-trained pupil size adjustment model.

[0130] Among them, the model parameters in the to-be-trained pupil position adjustment model are default parameters.

[0131] Specifically, the third input image and the pupil size label in the current training sample can be input into the to-be-trained pupil size adjustment model, and the third theoretical output image is used as the output of the to-be-trained pupil size adjustment model. That is to say, the model can be trained when both the input parameters and output parameters of the to-be-trained pupil size adjustment model are fixed, so as to adjust the model parameters in the to-be-trained pupil size adjustment model and continuously optimize the model parameters, so that when the original image and the pupil size label are input, the model can output the expected third theoretical output image, and the pupil size in the third theoretical output image matches the input pupil size label.

[0132] S530. Take the convergence of the loss function of the to-be-trained pupil size adjustment model as the training goal to obtain the target pupil size adjustment model.

[0133] Among them, the convergence of the preset loss function can be used as the training objective. When it is determined that the preset loss function of the to-be-trained pupil size adjustment model converges, it indicates that the adjustment result meets the scheme requirements, and the trained model has been obtained, thereby obtaining the target pupil size adjustment model.

[0134] It should be noted that since the model parameters in the to-be-trained pupil size adjustment model are not corrected, at this time, there may be corresponding differences between the special effect image output by the model and the third theoretical output image corresponding to the current training sample. The model parameters in the to-be-trained pupil size adjustment model can be continuously corrected so that the to-be-trained pupil size adjustment model can output the expected third theoretical output image.

[0135] Specifically, when it is detected that the loss function of the to-be-trained pupil size adjustment model converges, it indicates that the training of the to-be-trained pupil size adjustment model is completed, and at this time, the iterative training can be stopped. If it is detected that the loss function does not converge currently, the training samples can be further obtained to continue training the to-be-trained pupil size adjustment model until the loss function converges. It can be considered that the to-be-trained pupil size adjustment model is trained well, so that when the to-be-processed image and the target size label are input into the trained to-be-trained pupil size adjustment model, the model can adjust the pupil of the target object in the to-be-processed image to the size corresponding to the target size label, so that a special effect image with size change can be generated. The trained to-be-trained pupil size adjustment model can be used as the target pupil size adjustment model.

[0136] S540. In response to the special effect trigger operation, collect the to-be-processed image including the target object.

[0137] S550. Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on the target pupil feature adjustment model to obtain the target special effect image.

[0138] S560. Display the target special effect image on the target display interface.

[0139] The technical solution of the embodiment of the present disclosure obtains an image consistent with the marked pupil based on the target pupil reconstruction model. Correspondingly, the third input image and the third theoretical output image are obtained as training samples to train the model, and the trained target pupil size adjustment model is obtained, so as to control the pupil size change of the target object in the image based on the target pupil size adjustment model, improving the efficiency and accuracy of adding special effects.

[0140] Embodiment Six

[0141] Figure 9The flowchart of an image processing method provided in Embodiment 6 of the present disclosure. On the basis of the foregoing embodiments, a pupil reconstruction model to be trained can be trained based on training samples to obtain a target pupil reconstruction model. For the specific implementation manner, reference can be made to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above embodiments will not be elaborated herein.

[0142] As Figure 9 shown, the method specifically includes the following steps:

[0143] S610. Determine a plurality of fourth training samples, where the fourth training samples include a preprocessed image and an image to be used including pupil markings determined according to a preset image processing algorithm for the preprocessed image.

[0144] It should be noted that before training the target pupil reconstruction model, it is necessary to first obtain training samples to train the model based on the training samples. To improve the accuracy of the model, as many and rich training samples as possible can be obtained.

[0145] Among them, the preset image processing algorithm is an algorithm for performing pupil marking on the obtained image. The preprocessed image can be an image including a target object. There are various ways to obtain the preprocessed image. For example, it can be an image captured by a camera device, an image pre-stored in a storage space, or an image generated based on an image generation model.

[0146] After obtaining the preprocessed image, the corresponding theoretical output image can be determined, and then the target pupil reconstruction model can be trained based on the preprocessed image and the corresponding theoretical output image. The image reconstructed based on the pupil marking can be used as the image to be used. That is, an image can be constructed according to the pupil marking and used as the image to be used. The image to be used is a grayscale image of the pupil marking. It can be understood that the training samples include the pupil marking image and the reconstructed image.

[0147] In this embodiment, determining each fourth training sample can be: obtaining a preprocessed image including a target part and determining the grayscale image of the target part; determining a first region and a second region in the grayscale image according to a preset image processing algorithm; where the target part is the eye part, the first region is the pupil region, and the second region is the sclera region; and obtaining an image to be used including pupil markings based on the first region and the second region.

[0148] It should be noted that a camera device can be used to photograph the eye part of the target object, and multiple images containing the eyes can be collected as preprocessed images. To improve the accuracy of dividing the eyes and determining the pupil area, an algorithm can be used to convert the preprocessed image into a grayscale image, and the grayscale value can include any value from 0 to 255, where 255 represents pure white and 0 represents pure black. Further, a preset image processing algorithm and the corresponding grayscale value can be used to determine the pupil area and the sclera area in the eye part, and mark the sclera area and the pupil area to obtain an input image input to the to-be-trained pupil reconstruction model. Exemplarily, refer to Figure 10a , after the image is collected to obtain a preprocessed image, the pupil area and the sclera area in the eye can be marked to obtain an image to be used, such as A in 10a. Perform image reconstruction processing on A in 10a to obtain B in 10a. This is only a schematic diagram and is not limited thereto.

[0149] In this embodiment, determining the preset image processing algorithm may include a grayscale mean algorithm and a grayscale threshold algorithm. Correspondingly, there are at least three methods for determining the image to be used, specifically:

[0150] The first way to determine the image to be used can be: the preset image processing algorithm includes a grayscale mean algorithm, and determining the first region and the second region in the grayscale image according to the preset image processing algorithm includes: determining the grayscale mean of the grayscale image; taking the pixel points with grayscale values lower than the grayscale mean as the first type of pixel points, and taking the pixel points with grayscale values higher than the grayscale mean as the second type of pixel points; based on the first type of pixel points and the second type of pixel points, determining the first region and the second region.

[0151] Specifically, after converting the preprocessed image into a grayscale image, the grayscale mean of the target part in the grayscale image can be calculated. The pixel points with grayscale values lower than the grayscale mean in the target part can be taken as the first type of pixel points, and the pixel points with grayscale values higher than the grayscale mean can be taken as the second type of pixel points. The region composed of the first type of pixel points can be taken as the first region, and the region composed of the second type of pixel points can be taken as the second region. The first region can be the pupil area, and the second region can be the sclera area.

[0152] The second way to determine the image to be used can be: the preset image processing algorithm includes a grayscale threshold algorithm, and determining the first region and the second region in the grayscale image according to the preset image processing algorithm includes: taking the pixel points with grayscale values less than the preset grayscale value threshold as the first type of pixel points, and taking the pixel points with grayscale values greater than or equal to the preset grayscale threshold as the second type of pixel points; based on the first type of pixel points and the second type of pixel points, determining the first region and the second region.

[0153] Specifically, before obtaining the grayscale image of the eye region, a preset grayscale value threshold is determined. Pixel points in the grayscale image with grayscale values less than the preset grayscale value threshold can be regarded as the first type of pixel points, and pixel points greater than or equal to the preset grayscale value threshold can be regarded as the second type of pixel points. Based on the first type of pixel points and the second type of pixel points, the sclera region and the pupil region can be determined.

[0154] A third way to determine the image to be used can be: determining first pixels to be determined with grayscale means greater than the preset grayscale value threshold, and determining second pixels to be determined with grayscale values less than the preset grayscale value threshold, and based on the first pixels to be determined and the second pixels to be determined, determining a first segmented image; determining third pixels to be determined with grayscale means greater than the grayscale value mean of the target part, determining fourth pixels to be determined with grayscale values less than the grayscale value mean, and based on the third pixels to be determined and the fourth pixels to be determined, determining a second segmented image; according to the first pixels to be determined and the third pixels to be determined in the first segmented image and the second segmented image, determining the pixel intersection and the pixel union; if the ratio of the pixel intersection to the pixel union is greater than the preset ratio threshold, then based on the pixel intersection, determining a first region and a second region.

[0155] To further improve the accuracy of the image to be used, the first pixels to be determined and the second pixels to be determined can be determined based on the first method. The third pixels to be determined and the fourth pixels to be determined are determined based on the second method. The first pixels to be determined and the third pixels to be determined correspond to the pupil region, and the second pixels to be determined and the fourth pixels to be determined correspond to the sclera region. The first segmented image can be the pupil region determined by the first method. The second segmented image can be the pupil region determined by the second method. For the same eye, the number of pixel intersections and the number of unions in the first segmented image and the second segmented image can be determined. Based on the number of intersections and the number of unions, the ratio threshold is determined. If the ratio threshold is greater than the preset ratio threshold, then the intersection region can be regarded as the pupil region, and the eye region outside the intersection can be regarded as the sclera region. That is, the first region and the second region can be determined.

[0156] Exemplarily, referring to 10b, a region composed of pixel points with grayscale values lower than the grayscale mean can be selected as the pupil region, and the region outside the initial pupil region in the eye part is the sclera region. B in 10b can represent the grayscale image of the cat's eye image, and A in 10b can represent the image to be used that divides the grayscale image into the pupil region and the sclera region. Refer to Figure 10c, the area composed of pixel points below the preset gray value threshold can be selected as the pupil area, and the area composed of pixel points greater than the preset gray value threshold can be used as the sclera area. B in 10c can be represented as the grayscale image of the cat eye image, and A in 10c can be represented as the image to be used after dividing the grayscale image into the pupil area and the sclera area. It is only a schematic diagram and is not limited thereto.

[0157] Furthermore, in order to ensure the quality of the training data pairs, the Intersection-over-Union (IOU) of the two eyes in the image can be calculated based on the first segmented image and the second segmented image respectively to obtain a high-quality image to be used. The pixel intersection and pixel union of the first pixel point to be determined and the third pixel point to be determined can be calculated using an algorithm. Furthermore, the ratio of the pixel intersection to the pixel union can be calculated using an algorithm. This ratio can be compared with a preset ratio threshold. If the ratio is greater than the preset ratio threshold, the pixel intersection result can be used as the image to be used. For example, referring to Figure 10d A in is the first segmented image, B in 10d is the second segmented image, and C in 10d is the pixel intersection result image. However, it should be noted that the above-mentioned images are only schematic diagrams and are not limited thereto. And the image to be used and the corresponding preprocessed image are used as the fourth training sample. The pupil reconstruction model to be trained is trained based on the fourth training sample so that the model can reconstruct a real eye image through the binary mask of the eye.

[0158] S620. For each fourth training sample, use the image to be used as the input parameter of the pupil reconstruction model to be trained, and use the preprocessed image as the output parameter of the pupil reconstruction model to be trained, and adjust the model parameters of the pupil reconstruction model to be trained.

[0159] Among them, the model parameters in the pupil reconstruction model to be trained are default parameters.

[0160] Specifically, after obtaining the fourth training sample set, the image to be used in the current training sample can be input into the pupil reconstruction model to be trained, and the corresponding preprocessed image is used as the output of the pupil reconstruction model to be trained. That is to say, when the input parameter and the output parameter of the pupil reconstruction model to be trained are both fixed, the model parameters in the pupil reconstruction model to be trained can be adjusted so that when the image to be used is input, the model can output the expected preprocessed image, and the preprocessed image is the original image of the image to be used.

[0161] S630. Use the convergence of the loss function of the pupil reconstruction model to be trained as the training goal to obtain the target pupil reconstruction model.

[0162] Among them, the convergence of the preset loss function can be used as the training objective. When it is determined that the preset loss function of the to-be-trained pupil reconstruction model converges, it indicates that the adjustment result meets the requirements of the solution, and the trained model has been obtained, thereby obtaining the target pupil reconstruction model.

[0163] It should be noted that since the model parameters in the to-be-trained pupil reconstruction model are uncorrected, at this time, there may be corresponding differences between the reconstructed image output by the model and the preprocessed image corresponding to the current training sample. The model parameters in the to-be-trained pupil reconstruction model can be continuously corrected so that the to-be-trained pupil reconstruction model can output the desired preprocessed image.

[0164] Specifically, when it is detected that the loss function of the to-be-trained pupil reconstruction model converges, it indicates that the training of the to-be-trained pupil reconstruction model is completed, and at this time, the iterative training can be stopped. If it is detected that the loss function does not converge currently, the training samples can be further obtained to continue training the to-be-trained pupil reconstruction model until the loss function converges. It can be considered that the to-be-trained pupil reconstruction model is trained well, so that when the to-be-used image is input into the trained to-be-trained pupil reconstruction model, the model can reconstruct the real eye image, that is, the original image that has not been processed, from the to-be-used image. The trained to-be-trained pupil reconstruction model can be used as the target pupil reconstruction model.

[0165] In this embodiment, after training the target pupil reconstruction model, the training samples for constructing the training target pupil position adjustment model and the target pupil size adjustment model can be based on the target pupil reconstruction model. Further, based on the target pupil position adjustment model and the target pupil size adjustment model, the training samples for training the target pupil feature adjustment model are determined, and then the target feature pupil adjustment model is trained.

[0166] S640. In response to the special effect trigger operation, collect the to-be-processed image including the target object.

[0167] S650. Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on the target pupil feature adjustment model to obtain the target special effect image.

[0168] S660. Display the target special effect image on the target display interface.

[0169] The technical solution of the embodiment of the present disclosure determines a to-be-used image including pupil markings in a to-be-processed image through a preset image processing algorithm, and then uses the to-be-used image as an input parameter of a to-be-trained pupil reconstruction model, and uses the to-be-processed image as an output parameter of the to-be-trained pupil reconstruction model, and adjusts the model parameters of the to-be-trained pupil reconstruction model to obtain a trained target pupil reconstruction model, realizing pupil marking processing on the to-be-processed image based on the preset image processing algorithm to obtain the to-be-used image, improving the quality of training data, and improving the efficiency and accuracy of model training.

[0170] Embodiment VII

[0171] Figure 11 It is a structural block diagram of an image processing device provided in Embodiment VII of the present disclosure, which can execute the image processing method provided in any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method. As Figure 11 shown, the device specifically includes: a to-be-processed image acquisition module 710, a target special effect image acquisition module 720, and an image display module 730.

[0172] Among them, the to-be-processed image acquisition module 710 is configured to acquire a to-be-processed image including a target object in response to a special effect trigger operation; the target special effect image acquisition module 720 is configured to process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label; the image display module 730 is configured to display the target special effect image on a target display interface.

[0173] Based on the above technical solutions, the to-be-processed image acquisition module 710 includes a special effect trigger operation setting unit.

[0174] The special effect trigger operation setting unit is configured to detect that the visual field area includes a target object; detect that the target object triggers a target special effect action; detect that a special effect generation control is triggered.

[0175] Based on the above technical solutions, the target special effect image acquisition module 720 includes a target condition label determination unit and a target special effect image acquisition unit.

[0176] The target condition label determination unit is configured to determine at least one target condition label corresponding to the to-be-processed image;

[0177] The target special effect image acquisition unit is configured to use the at least one target condition label and the to-be-processed image as inputs of the target pupil adjustment model to obtain the target special effect image.

[0178] Based on the above technical solutions, the target condition label determination unit includes a target condition label acquisition subunit.

[0179] The target condition label acquisition subunit is used to acquire at least one pre-set target condition label; or, determine at least one target condition label corresponding to the target object according to the pose information of the target object in the image to be processed.

[0180] Based on the above technical solutions, the target condition label determination unit further includes a target condition label setting subunit.

[0181] The target condition label setting subunit is used for the at least one target condition label to include a target size label corresponding to the target display size of the pupil, and / or a target position label corresponding to the target relative display position of the pupil in the eye.

[0182] Based on the above technical solutions, the target condition label setting subunit includes that the target size label includes a first unit, the target position label includes a first unit, and the target position and size label includes a first unit.

[0183] The target size label includes a first unit, and is used for if the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is changed, then the at least one target condition label includes the target size label;

[0184] The target position label includes a first unit, and is used for if the target pupil feature adjustment model is trained under the condition that the size label is fixed and the position label is changed, then the at least one target condition label includes the target position label;

[0185] The target position and size label includes a first unit, and is used for if the target pupil feature adjustment model is trained under the condition that the size label and the position label are changed, then the at least one target condition label includes the target position label and the target size label.

[0186] Based on the above technical solutions, the target size label includes a first unit, including a first subunit of the target size label. The target position label includes a first unit, including a first subunit of the target position label. The target position and size label includes a first unit, including a first subunit of the target position and size label.

[0187] The first subunit of the target size label is used for if the at least one target condition label includes the target size label, then based on the target pupil feature adjustment model, adjusting the pupil of the target object in the image to be processed to match the target size label and the position label used when training the target pupil feature adjustment model;

[0188] The target position label includes a first sub-unit, which is used to, if the at least one target condition label includes the target position label, adjust the model based on the target pupil feature adjustment model, and adjust the pupil of the target object in the to-be-processed image to match the target position label and the size label used when training the target pupil feature adjustment model;

[0189] The target position and size label includes a first sub-unit, which is used to, if the at least one target condition label includes the target size label and the target position label, adjust the model based on the target pupil feature adjustment model, and adjust the pupil of the target object in the to-be-processed image to match the target size label and the target position label, so as to obtain the target special effect image.

[0190] On the basis of the above technical solutions, the device further includes: a target pupil feature adjustment model acquisition module. Among them, the target pupil feature adjustment model acquisition module includes a first training sample set acquisition unit, a model parameter adjustment unit, and a target pupil feature adjustment model acquisition unit.

[0191] The first training sample set acquisition unit is used to acquire a first training sample set; among them, the first training sample set includes a plurality of first training samples, and the first training sample includes a first original input image and a first theoretical image, and the first theoretical image is generated based on a pre-trained target pupil position adjustment model and a target pupil size adjustment model;

[0192] The model parameter adjustment unit is used to, for each first training sample, use the first original image of the current first training sample as the input of the to-be-trained pupil feature adjustment model, use the first theoretical image as the output of the to-be-trained pupil feature adjustment model, and adjust the model parameters of the to-be-trained pupil feature adjustment model;

[0193] The target pupil feature adjustment model acquisition unit is used to take the convergence of the loss function of the to-be-trained pupil feature adjustment model as the training target, and obtain the target pupil feature adjustment model.

[0194] On the basis of the above technical solutions, the first training sample set acquisition unit includes a first original input image acquisition sub-unit, a to-be-used image acquisition sub-unit, and a first theoretical image acquisition sub-unit.

[0195] The first original input image acquisition sub-unit is used to acquire a first original input image;

[0196] The to-be-used image acquisition sub-unit is used to input the first original input image and the to-be-displayed pupil position label into the target pupil position adjustment model to obtain a to-be-used image;

[0197] The first theoretical image acquisition subunit is configured to input the image to be used and the pupil size label to be displayed into the target pupil size adjustment model, and obtain the first theoretical image.

[0198] Based on the above technical solutions, the first training sample set acquisition unit further includes a fixed value first subunit, a fixed value second subunit, and a label change third subunit.

[0199] The fixed value first subunit is configured to, if the pupil size label to be displayed is a fixed value, the first theoretical images are images with the same pupil size and different pupil positions.

[0200] The fixed value second subunit is configured to, if the pupil position label to be displayed is a fixed value, the first theoretical images are images with the same pupil position and different pupil sizes.

[0201] The label change third subunit is configured to, if the pupil size label to be displayed changes and the pupil position label to be displayed changes, the first theoretical images are images with different pupil positions and pupil sizes.

[0202] Based on the above technical solutions, the device further includes: a target pupil position adjustment model acquisition module. Among them, the target pupil position adjustment model acquisition module includes a second training sample set acquisition unit, a model parameter adjustment unit, and a target pupil position adjustment model acquisition unit.

[0203] The second training sample set acquisition unit is configured to acquire a second training sample set; among them, the second training sample set includes a plurality of second training samples, and the second training samples include second input images with different pupil positions, second theoretical output images, and pupil position labels corresponding to the second theoretical output images. The second input images and the second theoretical output images are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil.

[0204] The model parameter adjustment unit is configured to, for each second training sample, use the second input image and the pupil position label of the current training sample as the input of the pupil position adjustment model to be trained, and use the second theoretical output image as the output of the pupil position adjustment model to be trained, and adjust the model parameters of the pupil position adjustment model to be trained.

[0205] The target pupil position adjustment model acquisition unit is configured to use the convergence of the loss function of the pupil position adjustment model to be trained as the training target, and obtain the target pupil position adjustment model.

[0206] Based on the above technical solutions, the device further includes: a target pupil size adjustment model acquisition module. Among them, the target pupil size adjustment model acquisition module includes a third training sample set acquisition unit, a model parameter adjustment unit, and a target pupil size adjustment model acquisition unit.

[0207] The third training sample set acquisition unit is used to acquire a third training sample set; among them, the third training sample set includes a plurality of third training samples, and each third training sample includes a third input image with different pupil sizes, a third theoretical output image, and a pupil size label corresponding to the third theoretical image. The third input image and the third theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil.

[0208] The model parameter adjustment unit is used to, for each third training sample, use the third input image and the pupil size label of the current training sample as the input of the to-be-trained pupil position adjustment model, use the third theoretical output image as the output of the to-be-trained pupil size adjustment model, and adjust the model parameters of the to-be-trained pupil size adjustment model.

[0209] The target pupil size adjustment model acquisition unit is used to take the convergence of the loss function of the to-be-trained pupil size adjustment model as the training goal to obtain the target pupil size adjustment model.

[0210] Based on the above technical solutions, the device further includes: a target pupil reconstruction model acquisition module. Among them, the target pupil reconstruction model acquisition module includes a fourth training sample set acquisition unit, a model parameter adjustment unit, and a target pupil reconstruction model acquisition unit.

[0211] The fourth training sample set acquisition unit is used to determine a plurality of fourth training samples. Each fourth training sample includes a preprocessed image and an image to be used including a pupil mark determined according to a preset image processing algorithm.

[0212] The model parameter adjustment unit is used to, for each fourth training sample, use the image to be used as the input parameter of the to-be-trained pupil reconstruction model, use the preprocessed image as the output parameter of the to-be-trained pupil reconstruction model, and adjust the model parameters of the to-be-trained pupil reconstruction model.

[0213] The target pupil reconstruction model acquisition unit is used to take the convergence of the loss function of the to-be-trained pupil reconstruction model as the training goal to obtain the target pupil reconstruction model.

[0214] Based on the above technical solutions, the fourth training sample set acquisition unit includes a preprocessed image acquisition subunit, a region determination subunit, and a to-be-used image acquisition subunit.

[0215] The preprocessed image acquisition subunit is configured to acquire a preprocessed image including a target part and determine a grayscale image of the target part;

[0216] The region determination subunit is configured to determine a first region and a second region in the grayscale image according to a preset image processing algorithm; wherein, the target part is an eye part, the first region is a pupil region, and the second region is a sclera region;

[0217] The to-be-used image acquisition subunit is configured to obtain a to-be-used image including pupil markings based on the first region and the second region.

[0218] Based on the above technical solutions, the region determination subunit includes a first region determination unit, a second region determination unit, and a third region determination unit.

[0219] Wherein, the preset image processing algorithm includes a grayscale mean algorithm. The first region determination unit includes a first grayscale mean determination subunit, a first pixel point determination subunit, and a first region determination subunit.

[0220] The first grayscale mean determination subunit is configured to determine the grayscale mean of the grayscale image;

[0221] The first pixel point determination subunit is configured to use the pixel points lower than the grayscale mean as the first type of pixel points and the pixel points higher than the grayscale mean as the second type of pixel points;

[0222] The first region determination subunit is configured to determine the first region and the second region based on the first type of pixel points and the second type of pixel points.

[0223] Wherein, the preset image processing algorithm includes a grayscale threshold algorithm. The second region determination unit includes a second pixel point determination subunit and a second region determination subunit.

[0224] The second pixel point determination subunit is configured to use the pixel points with grayscale values less than a preset grayscale value threshold as the first type of pixel points and the pixel points with grayscale values greater than or equal to the preset grayscale threshold as the second type of pixel points;

[0225] The second region determination subunit is configured to determine the first region and the second region based on the first type of pixel points and the second type of pixel points.

[0226] Based on the above technical solutions, the region determination third unit includes a first segmentation image determination third subunit, a second segmentation image determination third subunit, a pixel intersection and union determination third subunit, and a region determination third subunit.

[0227] The first segmentation image determination third subunit is configured to determine first pixels to be determined with a grayscale mean greater than a preset grayscale value threshold, and determine second pixels to be determined with a grayscale value less than the preset grayscale value threshold, and based on the first pixels to be determined and the second pixels to be determined, determine a first segmentation image;

[0228] The second segmentation image determination third subunit is configured to determine third pixels to be determined with a grayscale mean greater than the grayscale mean of the target part, determine fourth pixels to be determined with a grayscale value less than the grayscale mean, and based on the third pixels to be determined and the fourth pixels to be determined, determine a second segmentation image;

[0229] The pixel intersection and union determination third subunit is configured to determine a pixel intersection and a pixel union according to the first pixels to be determined and the third pixels to be determined in the first segmentation image and the second segmentation image;

[0230] The region determination third subunit is configured to, if the ratio of the pixel intersection to the pixel union is greater than a preset ratio threshold, determine a first region and a second region based on the pixel intersection.

[0231] The technical solution of the embodiment of the present disclosure, by responding to a special effect trigger operation, collecting a to-be-processed image including a target object, processing the to-be-processed image based on a target pupil feature adjustment model, and at least one target condition label corresponding to the to-be-processed image, obtaining a target special effect image, and displaying the target special effect image on a target display interface, solves the problem in the prior art that when using image retouching technology to generate a special effect image, the authenticity of the obtained special effect image is relatively low, thus causing a poor user experience. It realizes fine adjustment of the pupil based on the pupil attribute condition label, adds a special effect to the eye, makes the generated special effect image consistent with the condition label, improves the accuracy of adding the eye special effect, and further makes the displayed image more realistic when the special effect image is displayed on the display interface, achieving the technical effect of meeting the user experience requirements.

[0232] The image processing device provided by the embodiment of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.

[0233] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present disclosure.

[0234] Embodiment 8

[0235] Figure 12 is a schematic structural diagram of an electronic device provided in Embodiment 8 of the present disclosure. The following refers to Figure 12 , which shows a schematic structural diagram of an electronic device 800 suitable for implementing the embodiments of the present disclosure (such as Figure 12 the terminal device or server in Figure 12 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0236] As Figure 12 shown, the electronic device 800 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 801, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0237] Generally, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 12 shows the electronic device 800 having various devices, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0238] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above functions defined in the method of the embodiment of the present disclosure are performed.

[0239] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0240] The electronic device provided in the embodiment of the present disclosure and the image processing method provided in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0241] Embodiment Nine

[0242] Embodiment Nine of the present disclosure provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the image processing method provided in the above embodiment is implemented.

[0243] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0244] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0245] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0246] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to:

[0247] In response to a special effect trigger operation, collect a to-be-processed image including a target object;

[0248] Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label.

[0249] Display the target special effect image on a target display interface.

[0250] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through an Internet service provider using the Internet).

[0251] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0252] The units involved in the embodiments of the present disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0253] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0254] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM or Flash memory), an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0255] According to one or more embodiments of this disclosure, [Example 1] provides an image processing method, which includes:

[0256] In response to a special effect trigger operation, collect a to-be-processed image including a target object;

[0257] Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label;

[0258] Display the target special effect image on a target display interface.

[0259] According to one or more embodiments of this disclosure, [Example 2] provides an image processing method, which further includes:

[0260] Optionally, the special effect trigger operation includes at least one of the following:

[0261] Detect that the field of view region includes a target object;

[0262] Detect that the target object triggers a target special effect action;

[0263] Detect that a special effect generation control is triggered.

[0264] According to one or more embodiments of the present disclosure, [Example Three] provides an image processing method, further comprising:

[0265] Optionally, determining at least one target condition label corresponding to the image to be processed;

[0266] Using the at least one target condition label and the image to be processed as inputs to the target pupil adjustment model to obtain the target special effect image.

[0267] According to one or more embodiments of the present disclosure, [Example Four] provides an image processing method, further comprising:

[0268] Obtaining at least one pre-set target condition label; or,

[0269] Determining at least one target condition label corresponding to the target object according to the pose information of the target object in the image to be processed.

[0270] According to one or more embodiments of the present disclosure, [Example Five] provides an image processing method, further comprising:

[0271] Optionally, the at least one target condition label includes a target size label corresponding to the target display size of the pupil, and / or a target position label corresponding to the target relative display position of the pupil in the eye.

[0272] According to one or more embodiments of the present disclosure, [Example Six] provides an image processing method, further comprising:

[0273] If the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is changed, then the at least one target condition label includes the target size label;

[0274] If the target pupil feature adjustment model is trained under the condition that the size label is fixed and the position label is changed, then the at least one target condition label includes the target position label;

[0275] If the target pupil feature adjustment model is trained under the condition that the size label and the position label are changed, then the at least one target condition label includes the target position label and the target size label.

[0276] According to one or more embodiments of the present disclosure, [Example Seven] provides an image processing method, further comprising:

[0277] If the at least one target condition label includes a target size label and a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target size label and the target position label, to obtain the target special effect image;

[0278] If the at least one target condition label includes a target size label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target size label and the position label used when training the target pupil feature adjustment model;

[0279] If the at least one target condition label includes a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target position label and the size label used when training the target pupil feature adjustment model.

[0280] According to one or more embodiments of the present disclosure, [Example VIII] provides an image processing method, further including:

[0281] Train the target pupil feature adjustment model;

[0282] The training to obtain the target pupil feature adjustment model includes:

[0283] Obtain a first training sample set; wherein, the first training sample set includes a plurality of first training samples, and each first training sample includes a first original input image and a first theoretical image, and the first theoretical image is generated based on a pre-trained target pupil position adjustment model and a target pupil size adjustment model;

[0284] For each first training sample, use the first original image of the current first training sample as the input of the to-be-trained pupil feature adjustment model, use the first theoretical image as the output of the to-be-trained pupil feature adjustment model, and adjust the model parameters of the to-be-trained pupil feature adjustment model;

[0285] Take the convergence of the loss function of the to-be-trained pupil feature adjustment model as the training objective to obtain the target pupil feature adjustment model.

[0286] According to one or more embodiments of the present disclosure, [Example IX] provides an image processing method, further including:

[0287] Obtain a first original input image;

[0288] Input the first original input image and the to-be-displayed pupil position label into the target pupil position adjustment model to obtain an image to be used;

[0289] Input the image to be used and the pupil size label to be displayed into the target pupil size adjustment model to obtain the first theoretical image.

[0290] According to one or more embodiments of the present disclosure, [Example Ten] provides an image processing method, further including:

[0291] If the pupil size label to be displayed is a fixed value, each first theoretical image is an image with the same pupil size but different pupil positions;

[0292] If the pupil position label to be displayed is a fixed value, each first theoretical image is an image with the same pupil position but different pupil sizes;

[0293] If the pupil size label to be displayed changes and the pupil position label to be displayed changes, each first theoretical image is an image with different pupil positions and pupil sizes.

[0294] According to one or more embodiments of the present disclosure, [Example Eleven] provides an image processing method, further including:

[0295] Train to obtain the target pupil position adjustment model:

[0296] The training to obtain the target pupil position adjustment model includes:

[0297] Obtain a second training sample set; wherein, the second training sample set includes a plurality of second training samples, and each second training sample includes a second input image with a different pupil position, a second theoretical output image, and a pupil position label corresponding to the second theoretical output image. The second input image and the second theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil;

[0298] For each second training sample, take the second input image and the pupil position label of the current training sample as the input of the pupil position adjustment model to be trained, and take the second theoretical output image as the output of the pupil position adjustment model to be trained, and adjust the model parameters of the pupil position adjustment model to be trained;

[0299] Take the convergence of the loss function of the pupil position adjustment model to be trained as the training objective to obtain the target pupil position adjustment model.

[0300] According to one or more embodiments of the present disclosure, [Example Twelve] provides an image processing method, further including:

[0301] Train to obtain the target pupil size adjustment model;

[0302] The training to obtain the target pupil size adjustment model includes:

[0303] Obtain a third training sample set; wherein, the third training sample set includes a plurality of third training samples, and each third training sample includes a third input image with different pupil sizes, a third theoretical output image, and a pupil size label corresponding to the third theoretical image. The third input image and the third theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil.

[0304] For each third training sample, use the third input image and the pupil size label of the current training sample as the input of the to-be-trained pupil position adjustment model, and use the third theoretical output image as the output of the to-be-trained pupil size adjustment model, and adjust the model parameters of the to-be-trained pupil size adjustment model.

[0305] Take the convergence of the loss function of the to-be-trained pupil size adjustment model as the training objective to obtain the target pupil size adjustment model.

[0306] According to one or more embodiments of the present disclosure, [Example Thirteen] provides an image processing method, which further includes:

[0307] Train to obtain a target pupil reconstruction model;

[0308] The training to obtain the target pupil reconstruction model includes:

[0309] Determine a plurality of fourth training samples. Each fourth training sample includes a preprocessed image and an image to be used including a pupil mark determined according to a preset image processing algorithm.

[0310] For each fourth training sample, use the image to be used as the input parameter of the to-be-trained pupil reconstruction model, and use the preprocessed image as the output parameter of the to-be-trained pupil reconstruction model, and adjust the model parameters of the to-be-trained pupil reconstruction model.

[0311] Take the convergence of the loss function of the to-be-trained pupil reconstruction model as the training objective to obtain the target pupil reconstruction model.

[0312] According to one or more embodiments of the present disclosure, [Example Fourteen] provides an image processing method, which further includes:

[0313] Obtain a preprocessed image including a target part and determine the grayscale image of the target part;

[0314] Determine a first region and a second region in the grayscale image according to a preset image processing algorithm; wherein, the target part is the eye part, the first region is the pupil region, and the second region is the sclera region;

[0315] Based on the first region and the second region, obtain an image to be used including pupil markings.

[0316] According to one or more embodiments of the present disclosure, [Example XV] provides an image processing method, further including:

[0317] Optionally, the preset image processing algorithm includes a grayscale mean algorithm;

[0318] Determine the grayscale mean of the grayscale image;

[0319] Take the pixel points below the grayscale mean as the first type of pixel points, and take the pixel points above the grayscale mean as the second type of pixel points;

[0320] Based on the first type of pixel points and the second type of pixel points, determine the first region and the second region.

[0321] According to one or more embodiments of the present disclosure, [Example XVI] provides an image processing method, further including:

[0322] Optionally, the preset image processing algorithm includes a grayscale threshold algorithm.

[0323] Take the pixel points with grayscale values less than the preset grayscale value threshold as the first type of pixel points, and take the pixel points with grayscale values greater than or equal to the preset grayscale threshold as the second type of pixel points;

[0324] Based on the first type of pixel points and the second type of pixel points, determine the first region and the second region.

[0325] According to one or more embodiments of the present disclosure, [Example XVII] provides an image processing method, further including:

[0326] Determine first pixels to be determined with grayscale means greater than the preset grayscale value threshold, and determine second pixels to be determined with grayscale values less than the preset grayscale value threshold, and based on the first pixels to be determined and the second pixels to be determined, determine a first segmented image;

[0327] Determine third pixels to be determined with grayscale means greater than the grayscale mean value of the target part, determine fourth pixels to be determined with grayscale values less than the grayscale mean value, and based on the third pixels to be determined and the fourth pixels to be determined, determine a second segmented image;

[0328] Determine a pixel intersection and a pixel union based on a first pixel to be determined and a third pixel to be determined in the first segmented image and the second segmented image;

[0329] If the ratio of the pixel intersection to the pixel union is greater than a preset ratio threshold, determine a first region and a second region based on the pixel intersection.

[0330] According to one or more embodiments of the present disclosure, [Example XVIII] provides an image processing apparatus, including:

[0331] A to-be-processed image acquisition module, configured to acquire a to-be-processed image including a target object in response to a special effect trigger operation;

[0332] A target special effect image acquisition module, configured to process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label;

[0333] An image display module, configured to display the target special effect image on a target display interface.

[0334] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principle. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, a technical solution formed by mutually replacing the above features with technical features having similar functions disclosed in the present disclosure.

[0335] In addition, although the operations are depicted in a specific order, this should not be construed as requiring that the operations be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0336] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that, Including: In response to a special effect triggering operation, collect a to-be-processed image including a target object; Process the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label; the at least one target condition label includes a target size label corresponding to a target display size of the pupil, and / or a target position label corresponding to a target relative display position of the pupil in the eye; Display the target special effect image on a target display interface; The method further includes: if the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is changed, the at least one target condition label includes a target size label.

2. The method according to claim 1, characterized in that, The special effect triggering operation includes at least one of the following: Detect that the target object is included in the visual field area; Detect that the target object triggers a target special effect action; Detect that a special effect generation control is triggered.

3. The method according to claim 1, wherein The process of processing the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image includes: Determine at least one target condition label corresponding to the to-be-processed image; Use the at least one target condition label and the to-be-processed image as inputs to the target pupil adjustment model to obtain the target special effect image.

4. The method according to claim 3, wherein The determination of at least one target condition label corresponding to the to-be-processed image includes: Obtain at least one pre-set target condition label; or, Determine at least one target condition label corresponding to the target object according to the pose information of the target object in the to-be-processed image.

5. The method according to claim 1, wherein Also including: If the target pupil feature adjustment model is trained under the condition that the size label is fixed and the position label is changed, the at least one target condition label includes a target position label; If the target pupil feature adjustment model is trained under the condition that the size label and the position label are changed, the at least one target condition label includes a target position label and a target size label.

6. The method according to claim 1 or 5, characterized in that The process of processing the to-be-processed image and at least one target condition label corresponding to the to-be-processed image based on a target pupil feature adjustment model to obtain a target special effect image includes: If the at least one target condition label includes a target size label and a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target size label and the target position label to obtain the target special effect image; If the at least one target condition label includes a target size label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target size label and the position label used when training the target pupil feature adjustment model; If the at least one target condition label includes a target position label, based on the target pupil feature adjustment model, adjust the pupil of the target object in the to-be-processed image to match the target position label and the size label used when training the target pupil feature adjustment model.

7. The method according to claim 1, characterized in that Further included are: Training the target pupil feature adjustment model; The training of the target pupil feature adjustment model includes: Obtaining a first training sample set; wherein, the first training sample set includes a plurality of first training samples, and each first training sample includes a first original input image and a first theoretical image, and the first theoretical image is generated based on a pre-trained target pupil position adjustment model and a target pupil size adjustment model; For each first training sample, use the first original image of the current first training sample as the input of the to-be-trained pupil feature adjustment model, use the first theoretical image as the output of the to-be-trained pupil feature adjustment model, and adjust the model parameters of the to-be-trained pupil feature adjustment model; Take the convergence of the loss function of the to-be-trained pupil feature adjustment model as the training objective to obtain the target pupil feature adjustment model.

8. The method according to claim 7, wherein Determining the first theoretical image in each first training sample in the first training sample set includes: Obtaining a first original input image; Input the first original input image and the to-be-displayed pupil position label into the target pupil position adjustment model to obtain an image to be used; Input the image to be used and the to-be-displayed pupil size label into the target pupil size adjustment model to obtain the first theoretical image.

9. The method according to claim 8, wherein Further included are: If the to-be-displayed pupil size label is a fixed value, each first theoretical image is an image with the same pupil size and different pupil positions; If the to-be-displayed pupil position label is a fixed value, each first theoretical image is an image with the same pupil position and different pupil sizes; If the to-be-displayed pupil size label changes and the to-be-displayed pupil position label changes, each first theoretical image is an image with different pupil positions and pupil sizes.

10. The method according to claim 7, characterized in that, Further included are: Training the target pupil position adjustment model; The training of the target pupil position adjustment model includes: Obtaining a second training sample set; wherein, the second training sample set includes a plurality of second training samples, and each second training sample includes a second input image with a different pupil position, a second theoretical output image, and a pupil position label corresponding to the second theoretical output image, and the second input image and the second theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil; For each second training sample, use the second input image and the pupil position label of the current training sample as the input of the to-be-trained pupil position adjustment model, use the second theoretical output image as the output of the to-be-trained pupil position adjustment model, and adjust the model parameters of the to-be-trained pupil position adjustment model; Taking the convergence of the loss function of the to-be-trained pupil position adjustment model as the training objective, the target pupil position adjustment model is obtained.

11. The method according to claim 7, wherein It further includes: Training a target pupil size adjustment model; The training to obtain the target pupil size adjustment model includes: Obtaining a third training sample set; wherein, the third training sample set includes a plurality of third training samples, and each third training sample includes a third input image with different pupil sizes, a third theoretical output image, and a pupil size label corresponding to the third theoretical image. The third input image and the third theoretical output image are determined based on a pre-trained target pupil reconstruction model; the target pupil reconstruction model is used to reconstruct an image consistent with the marked pupil. For each third training sample, taking the third input image and the pupil size label of the current training sample as the input of the to-be-trained pupil position adjustment model, taking the third theoretical output image as the output of the to-be-trained pupil size adjustment model, and adjusting the model parameters of the to-be-trained pupil size adjustment model. Taking the convergence of the loss function of the to-be-trained pupil size adjustment model as the training objective, the target pupil size adjustment model is obtained.

12. The method according to claim 10 or 11, characterized in that It further includes: Training a target pupil reconstruction model; The training to obtain the target pupil reconstruction model includes: Determining a plurality of fourth training samples, each of which includes a preprocessed image and an image to be used including a pupil mark determined according to a preset image processing algorithm in the preprocessed image. For each fourth training sample, taking the image to be used as the input parameter of the to-be-trained pupil reconstruction model, taking the preprocessed image as the output parameter of the to-be-trained pupil reconstruction model, and adjusting the model parameters of the to-be-trained pupil reconstruction model. Taking the convergence of the loss function of the to-be-trained pupil reconstruction model as the training objective, the target pupil reconstruction model is obtained.

13. The method according to claim 12, wherein Determining each fourth training sample includes: Obtaining a preprocessed image including a target part and determining the grayscale image of the target part. Determining a first region and a second region in the grayscale image according to a preset image processing algorithm; wherein, the target part is the eye part, the first region is the pupil region, and the second region is the sclera region. Based on the first region and the second region, an image to be used including a pupil mark is obtained.

14. The method according to claim 13, wherein The preset image processing algorithm includes a grayscale mean algorithm. The determining of the first region and the second region in the grayscale image according to the preset image processing algorithm includes: Determining the grayscale mean of the grayscale image. Taking the pixel points lower than the grayscale mean as the first type of pixel points and the pixel points higher than the grayscale mean as the second type of pixel points. Based on the first type of pixel points and the second type of pixel points, the first region and the second region are determined.

15. The method according to claim 13, wherein The preset image processing algorithm includes a grayscale threshold algorithm. The determining of the first region and the second region in the grayscale image according to the preset image processing algorithm includes: Taking the pixel points with grayscale values less than the preset grayscale value threshold as the first type of pixel points and the pixel points with grayscale values greater than or equal to the preset grayscale threshold as the second type of pixel points. Based on the first type of pixel points and the second type of pixel points, determine a first region and a second region.

16. The method according to claim 13, wherein Determining the first region and the second region in the grayscale image according to a preset image processing algorithm includes: Determine first to-be-determined pixel points with a grayscale mean greater than a preset grayscale value threshold, and determine second to-be-determined pixel points with a grayscale value less than the preset grayscale value threshold, and based on the first to-be-determined pixel points and the second to-be-determined pixel points, determine a first segmented image; Determine third to-be-determined pixel points with a grayscale mean greater than the grayscale value mean of the target part, determine fourth to-be-determined pixel points with a grayscale value less than the grayscale value mean, and based on the third to-be-determined pixel points and the fourth to-be-determined pixel points, determine a second segmented image; According to the first to-be-determined pixel points and the third to-be-determined pixel points in the first segmented image and the second segmented image, determine a pixel point intersection and a pixel point union; If the ratio of the pixel point intersection to the pixel point union is greater than a preset ratio threshold, then based on the pixel point intersection, determine the first region and the second region.

17. An image processing apparatus, characterized in that, Includes: An image to be processed acquisition module, configured to acquire an image to be processed including a target object in response to a special effect trigger operation; A target special effect image acquisition module, configured to process the image to be processed and at least one target condition label corresponding to the image to be processed based on a target pupil feature adjustment model to obtain a target special effect image; wherein, the pupil feature of the target object in the target special effect image matches the at least one target condition label; the at least one target condition label includes a target size label corresponding to a target display size of the pupil, and / or a target position label corresponding to a target relative display position of the pupil in the eye; An image display module, configured to display the target special effect image on a target display interface; The device further includes: a target condition label setting subunit, configured to if the target pupil feature adjustment model is trained under the condition that the position label is fixed and the size label is changed, then the at least one target condition label includes a target size label.

18. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method according to any one of claims 1-16.

19. A storage medium containing computer-executable instructions, the computer-executable instructions being used to execute the image processing method according to any one of claims 1-16 when executed by a computer processor.

Images