Model training method, image processing method, electronic equipment and storage medium

Through the combination of the illumination perception model and the generation model, lighting features are extracted and fused, the problem of inconsistent light and shadow effects in image processing is solved, and a more natural and coordinated light and shadow effect is achieved, which improves the visual quality of the image.

CN119068304APending Publication Date: 2024-12-03SHANGHAI GLORY SMART TECH DEV CO LTD

Patent Information

Application Number
CN202410982466.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When existing image processing technology enhances image brightness or denoising, it is difficult to effectively improve the light and shadow effect of the image, especially when the lighting information between the foreground portrait and the background image is mismatched, resulting in the lack of harmony in the synthetic image and Realism.

Method used

The training model method is adopted, combined with the illumination perception model and the generation model, and the light feature extraction and fusion layer processing is used to generate a more natural and coordinated light and shadow effect. The specific steps include inputting the illumination training image into the illumination perception model to extract the illumination features, fusing it with the image feature vector, and generating the adjusted image through the generation layer to improve the light and shadow effect.

Benefits of technology

It significantly improves the visual quality of the image, makes the processed image more natural and coordinated, enhances the light and shadow effect, ensures the consistency of light and shadow in the two images, thereby improving the overall visual experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068304A_ABST
    Figure CN119068304A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, and provides a training model method, an image processing method, electronic equipment and a storage medium. The training model method comprises the steps of performing illumination feature extraction on an illumination training image based on parameters of an illumination perception model to obtain a first panoramic illumination feature vector corresponding to the illumination training image; based on the parameters of the feature extraction layer, performing image feature extraction on the first training image to obtain an image feature vector corresponding to the first training image; based on the parameters of the fusion layer, fusing the image feature vector and the first panoramic illumination feature vector to obtain the fused feature vector based on the parameters of the generation layer, and obtaining a first generated image according to the fused feature vector; and adjusting parameters of the generative model and the illumination perception model according to the first generated image to obtain the trained generative model and the trained illumination perception model. The image shadow effect can be enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training model method, an image processing method, an electronic device and a storage medium. Background Art

[0002] With the development of image processing technology, the improvement and enhancement of light and shadow effects have become an important means to optimize the visual quality of images. At present, image processing technology can enhance the brightness of the image or perform denoising on the whole, but the light and shadow effects of the processed image still need to be enhanced. For example, in a scene where a foreground portrait is synthesized into a background image, the brightness of the image is enhanced or denoised on the whole. However, if there is a mismatch in lighting information between the foreground portrait and the background image, the synthesized image lacks visual harmony and realism. Therefore, how to enhance the light and shadow effects of an image has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a training model method, an image processing method, an electronic device and a storage medium, which can enhance the light and shadow effects of an image.

[0004] A first aspect provides a training model method, which is applied to an electronic device. The electronic device includes an illumination perception model and a generation model. The generation model includes a feature extraction layer, a fusion layer and a generation layer. The method includes: inputting an illumination training image into the illumination perception model, extracting illumination features from the illumination training image based on parameters of the illumination perception model, and obtaining a first panoramic illumination feature vector corresponding to the illumination training image; inputting a first training image of a first training set into the feature extraction layer, extracting image features from the first training image based on parameters of the feature extraction layer, and obtaining an image feature vector corresponding to the first training image; inputting the image feature vector and the first panoramic illumination feature vector into the fusion layer, and fusing the image feature vector and the first panoramic illumination feature vector based on parameters of the fusion layer to obtain a fused feature vector; inputting the fused feature vector into the generation layer, and obtaining a first generated image based on the fused feature vector based on the parameters of the generation layer, wherein the first generated image is an image generated based on the first training image with the first panoramic illumination feature vector as a conditional vector of the generation model; adjusting the parameters of the generation model and the illumination perception model based on the first generated image to obtain a trained generation model and illumination perception model.

[0005] In the embodiment of the present application, the illumination features of the image are extracted based on the illumination perception model to obtain the panoramic illumination feature vector used by the image to describe the panoramic illumination information, providing more comprehensive and detailed illumination information for the synthesis model. The synthesis model intelligently adjusts and migrates the light and shadow of the image based on the illumination information identified and quantified by the illumination perception model, thereby enhancing the light and shadow effects of the image.

[0006] In one possible implementation, the illumination training image includes a panoramic training image, the illumination perception model includes a first image encoder, the illumination training image is input into the illumination perception model, and based on parameters of the illumination perception model, illumination features are extracted from the illumination training image to obtain a first panoramic illumination feature vector corresponding to the illumination training image, comprising: inputting the panoramic training image into the first image encoder, and based on parameters of the first image encoder, illumination features are extracted from the panoramic training image to obtain a panoramic illumination feature vector corresponding to the panoramic training image, wherein the first panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the panoramic training image.

[0007] In an embodiment of the present application, a panoramic illumination feature vector of a panoramic training image is obtained based on a second image encoder, and the panoramic training image is encoded into a fusion layer to provide a more comprehensive and detailed illumination information for the synthesis model.

[0008] In one possible implementation, the illumination training image includes a local training image, the illumination perception model includes a second image encoder and a vector extension model, the illumination training image is input into the illumination perception model, and based on the parameters of the illumination perception model, illumination features are extracted from the illumination training image to obtain a first panoramic illumination feature vector corresponding to the illumination training image, including: inputting the local training image into the second image encoder, and based on the parameters of the second image encoder, illumination features are extracted from the local training image to obtain a local illumination feature vector corresponding to the local training image; inputting the local illumination feature vector corresponding to the local training image into the vector extension model, and based on the parameters of the vector extension model, expanding the local illumination feature vector into a panoramic illumination feature vector to obtain a panoramic illumination feature vector corresponding to the local training image, wherein the first panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the local training image or a panoramic illumination feature vector corresponding to the panoramic training image.

[0009] In an embodiment of the present application, a panoramic illumination feature vector of a panoramic training image is obtained based on a second image encoder, and the panoramic training image is encoded into a fusion layer to provide a more comprehensive and detailed illumination information for the synthesis model.

[0010] In one possible implementation, the first training set also includes a first target image corresponding to the first training image, and according to the first generated image, the parameters of the generation model and the lighting perception model are adjusted to obtain the trained generation model and the lighting perception model, including: constructing a first loss function according to the first target image and the first generated image; constructing a second loss function according to the panoramic illumination feature vector corresponding to the local training image and the panoramic illumination feature vector corresponding to the panoramic training image; according to the first loss function and the second loss function, adjusting the parameters of the generation model and the lighting perception model to obtain the trained generation model and the lighting perception model.

[0011] In one possible implementation, the electronic device also includes a synthesis model, and the method also includes: obtaining a first mask image corresponding to the real portrait image based on the real portrait image, wherein the image area where the target object is located in the first mask image is marked as a selected state; inputting the first mask image and the real lighting training image into the synthesis model, and based on the synthesis model, synthesizing the target object in the first mask image into the real lighting training image to obtain a second training image; inputting the second training image into the trained generation model and inputting the real lighting training image into the trained lighting perception model for fine-tuning.

[0012] In one possible implementation, the lighting perception model also includes a text generation model, and the method also includes: inputting the lighting training image into the text generation model, and obtaining a first text feature vector for describing the lighting features of the lighting training image based on the text generation model; inputting the image feature vector and the first panoramic lighting feature vector into a fusion layer, and based on the parameters of the fusion layer, fusing the image feature vector and the first panoramic lighting feature vector to obtain a fused feature vector, including: inputting the image feature vector, the first panoramic lighting feature vector, and the first text feature vector into the fusion layer, and based on the parameters of the fusion layer, fusing the image feature vector, the first panoramic lighting feature vector, and the text feature vector to obtain a fused feature vector.

[0013] A second aspect provides an image processing method, which is applied to an electronic device. The electronic device includes an illumination perception model and a generation model. The generation model includes a feature extraction layer, a fusion layer and a generation layer. The method includes inputting a first image into the illumination perception model, extracting illumination features of the first image based on parameters of the illumination perception model, and obtaining a target panoramic illumination feature vector corresponding to the first image; inputting a second image into the feature extraction layer, extracting image features of the second image based on parameters of the feature extraction layer, and obtaining a target image feature vector corresponding to the second image; inputting the target image feature vector and the target panoramic illumination feature vector into the fusion layer, and fusing the target image feature vector and the target panoramic illumination feature vector based on parameters of the fusion layer to obtain a fused target feature vector; inputting the fused target feature vector into the generation layer, and obtaining a target generated image based on the fused target feature vector based on parameters of the generation layer, wherein the target generated image is an image generated based on the second image with the target panoramic illumination feature vector as a conditional vector of the generation model.

[0014] In an embodiment of the present application, the illumination features of an image are extracted based on an illumination perception model to obtain a panoramic illumination feature vector of the image used to describe the illumination information of the panorama, providing a more comprehensive and detailed illumination information for the synthesis model. The synthesis model intelligently adjusts and migrates the light and shadow of the image according to the illumination information identified and quantified by the illumination perception model, thereby enhancing the light and shadow effects of the image. In particular, the synthesis model uses the panoramic illumination feature vector provided by the illumination perception model as a conditional vector to guide the generation of the image. The conditional vector can be effectively combined with the image features to guide the generation model to generate an image with specific features. When performing light and shadow effect migration, the illumination information of the image in the panorama is obtained through the illumination perception model, and the light source direction and scene atmosphere of the image are respected to ensure that the migrated light and shadow effects are coordinated and consistent with the processed image as a whole, and to ensure that the light and shadow effects are consistent in the two images, thereby improving the overall visual quality of the processed image.

[0015] In one possible implementation, the first image includes a target panoramic image, the lighting perception model includes a first image encoder, the first image is input into the lighting perception model, and based on parameters of the lighting perception model, lighting features are extracted from the first image to obtain a target panoramic lighting feature vector corresponding to the first image, including: inputting the target panoramic image into the first image encoder, and based on parameters of the first image encoder, lighting features are extracted from the target panoramic image to obtain a panoramic lighting feature vector corresponding to the target panoramic image, wherein the target panoramic lighting feature vector includes a panoramic lighting feature vector corresponding to the target panoramic image.

[0016] In one possible implementation, the first image includes a target local image, the lighting perception model includes a second image encoder and a vector extension model, the first image is input into the lighting perception model, and based on the parameters of the lighting perception model, lighting features are extracted from the first image to obtain a target panoramic lighting feature vector corresponding to the first image, which includes: inputting the target local image into the second image encoder, and based on the parameters of the second image encoder, lighting features are extracted from the target local image to obtain a local lighting feature vector corresponding to the target local image; inputting the local lighting feature vector corresponding to the target local image into the vector extension model, and based on the parameters of the vector extension model, expanding the local lighting feature vector into a panoramic lighting feature vector to obtain a panoramic lighting feature vector corresponding to the target local image, wherein the target panoramic lighting feature vector includes a panoramic lighting feature vector corresponding to the target local image or a panoramic lighting feature vector corresponding to the target panoramic image.

[0017] In one possible implementation, the lighting perception model also includes a text generation model, and the method also includes: inputting the first image into the text generation model, and obtaining a target text feature vector for describing the lighting characteristics of the first image based on the text generation model; inputting the target image feature vector and the target panoramic lighting feature vector into a fusion layer, and based on the parameters of the fusion layer, fusing the target image feature vector and the target panoramic lighting feature vector to obtain a fused target feature vector, including: inputting the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector into the fusion layer, and based on the parameters of the fusion layer, fusing the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector to obtain a fused target feature vector.

[0018] In one possible implementation, the electronic device also includes a synthesis model. Before inputting the second image into the feature extraction layer, the method also includes: obtaining a target mask image corresponding to the target portrait image based on the target portrait image, wherein an image area where the target object is located in the target mask image is marked as a selected state; inputting the target mask image and the first image into the synthesis model, and based on the synthesis model, synthesizing the target object in the target mask image into the first image to obtain the second image.

[0019] In one possible implementation, the method further includes: in response to the first operation, determining at least one target portrait image;

[0020] In the case where at least one target portrait image includes two or more objects, in response to the second operation, at least one target object in the at least one target portrait image is determined; and in response to the third operation, the first image is determined.

[0021] The third aspect provides an electronic device, which includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes any one of the training model methods or image processing methods above.

[0022] The fourth aspect provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed, the training model method or image processing method as described above is executed.

[0023] In addition, the technical effects brought about by the third aspect to the fourth aspect can be found in the description related to the methods of each design in the above method part, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0025] FIG. 1A to FIG. 1I A schematic diagram of an application scenario provided for an embodiment of the present application;

[0026] FIG. 2A to FIG. 2C A schematic diagram of another application scenario provided for an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of an image processing module provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the synthesis model provided in the embodiment of the present application;

[0029] Figure 5A A schematic diagram of a model training architecture provided in an embodiment of the present application;

[0030] Figure 5B A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0031] Figure 5C A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0032] Figure 5D A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0033] Figure 5EA schematic diagram of another model training architecture provided in an embodiment of the present application;

[0034] Fig. 6A A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0035] Figure 6B A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0036] Fig. 7A A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0037] Figure 7B A schematic diagram of another model training architecture provided in an embodiment of the present application;

[0038] Fig. 8A A schematic diagram of an image processing architecture provided in an embodiment of the present application;

[0039] Figure 8B A schematic diagram of another image processing architecture provided in an embodiment of the present application;

[0040] Fig.9A A schematic diagram of another image processing architecture provided in an embodiment of the present application;

[0041] Fig. 9B A schematic diagram of another image processing architecture provided in an embodiment of the present application;

[0042] Fig. 9C A schematic diagram of another image processing architecture provided in an embodiment of the present application;

[0043] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0046] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0047] In this article, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprising a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0048] It should also be noted that the method disclosed in the embodiments of the present application or the method shown in the flowchart includes one or more steps for implementing the method. Without departing from the scope of the claims, the execution order of multiple steps can be interchangeable with each other, and some of the steps can also be deleted.

[0049] The term "user interface (UI)" in the following embodiments of the present application refers to the medium for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can include visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, etc. displayed on the display screen of an electronic device.

[0050] Some embodiments will be described below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0051] As mentioned above, the existing image processing technology enhances the brightness of the image as a whole or performs denoising, but there is still room for improvement in the processing of light and shadow effects. For example, when synthesizing a foreground portrait and a background image, if the lighting information between the two does not match, for example, there are differences in brightness, shadows, highlights, etc., which affects the overall visual coordination of the synthesized image, making the synthesized image appear unnatural. The inventors found in the process of implementing the embodiments of the present application that the direction of the light source plays a role in forming a consistent light and shadow effect. The inconsistent light source directions of the image providing the light and shadow effect (such as image A) and the image applying the light and shadow effect (such as image B) will cause the direction of the shadow and highlight to conflict. At the same time, the different lighting intensities of image A and image B will also affect the contrast and detail performance. When the lighting intensities of image A and image B are not coordinated, it may cause a part of image B to be too prominent or dim. Lighting information also affects the color and color temperature of the image. Unmatched lighting information will cause inconsistent colors, which in turn affects the realism of image B. Light and shadow effects are crucial to helping observers perceive the spatial relationship and three-dimensional sense of objects. If the light and shadow effects of image A and image B do not match, this spatial perception will be destroyed. Traditional image processing technology cannot fully understand the image lighting information, resulting in the image B after the light and shadow effect is transferred not being refined enough in terms of light and shadow effects, highlighting the limitations of existing technology in light and shadow effect processing.

[0052] In view of this, the embodiments of the present application provide a training model method, an image processing method, an electronic device and a storage medium, which can enhance the light and shadow effects of an image, so that the processed image is more natural and coordinated visually. Specifically, in order to achieve a more refined and realistic visual effect, an illumination perception model is used to more accurately and finely identify and quantify the illumination information in the image, including the direction, intensity, color of the light source, and the shadows and reflections produced by the illumination on the object. In particular, the illumination features of the image are extracted based on the illumination perception model to obtain a panoramic illumination feature vector of the image used to describe the panoramic illumination information, so as to provide a more comprehensive and refined illumination information for the synthesis model. The synthesis model intelligently adjusts and migrates the light and shadow of the image according to the illumination information identified and quantified by the illumination perception model, thereby enhancing the light and shadow effects of the image. In particular, the synthesis model uses the panoramic illumination feature vector provided by the illumination perception model as a conditional vector to guide the generation of the image. The conditional vector can be effectively combined with the image features to guide the generation model to generate an image with specific features. When migrating light and shadow effects, the lighting information of the image in the panoramic view is obtained through the lighting perception model, and the light source direction and scene atmosphere of the image are respected to ensure that the migrated light and shadow effects are coordinated and consistent with the processed image as a whole, and to ensure that the light and shadow effects are consistent in the two images, thereby improving the overall visual quality of the processed image.

[0053] The training model method and image processing method provided in the embodiments of the present application can be applied to electronic devices, and the electronic devices may be mobile phones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPCs), netbooks, personal digital assistants (personal digital assistants, PDAs), household appliances and other devices. The present application does not impose any restrictions on the form of the electronic devices.

[0054] The electronic device provided in the embodiment of the present application is equipped with an image processing module. The image processing module is used to improve and enhance the light and shadow effects in the image, which not only enhances the artistic effect of the image, but also provides users with a richer and more realistic visual experience. The application fields of the image processing module may include but are not limited to: photography post-processing, film and video post-processing, game design, virtual reality (VR) and augmented reality (AR), etc. Specifically, it may include scenes such as image synthesis and image editing.

[0055] In an embodiment of the present application, an image processing module can be used to migrate the light and shadow effects of an image to another image. Migrating the light and shadow effects of an image to another image can be called "light and shadow effect migration" or "lighting consistency adjustment". Light and shadow effect migration is an image processing technology, the purpose of which is to copy or adjust the light and shadow effects in one image to another image, so that the processed image is more visually natural and coordinated. Light and shadow effects include visual elements such as brightness, shadows, highlights, and contrast in the image. The migration of light and shadow effects is not just a simple brightness adjustment, but to ensure that the direction, intensity, and color temperature of the light source are consistent in the two images. In image processing, migration refers to copying or adjusting specific visual features of an image (such as light and shadow effects) to another image. During the migration process, ensure that the two images are as consistent as possible in lighting information, including the direction, intensity, and color temperature of the light source.

[0056] The following uses the electronic device as a mobile phone and the image processing module integrated into the gallery application as an example to exemplarily introduce the image synthesis scenario provided by the embodiment of the present application.

[0057] like Figure 1A As shown, the mobile phone displays a home screen interface 11. The home screen interface 11 displays various types of application (App) icons, such as a clock icon, a calendar icon, a gallery icon 12 and a memo icon.

[0058] The mobile phone receives the user's input operation on the gallery icon 12 , and in response to the input operation on the gallery icon 12 , displays the gallery interface 13 as shown in FIG1B .

[0059] like Figure 1BAs shown, the gallery interface 13 displays albums, such as the all-photos album 14, the camera album, the my favorites album, and the screenshot album.

[0060] The mobile phone receives the user's input operation on the all photo album 14, and in response to the input operation on the all photo album 14, displays the all photo interface 15 as shown in 1C.

[0061] like Figure 1C As shown, all photos interface 15 displays photo thumbnails, such as photo thumbnail 16. All photos interface 15 can display photo thumbnails of multiple photos.

[0062] The mobile phone receives the user's input operation on the photo thumbnail 16, and in response to the input operation on the photo thumbnail 16, displays the photo interface 17 as shown in 1D.

[0063] like Figure 1D As shown, the photo interface 17 includes a display area for the target image (i.e., the image corresponding to the photo thumbnail 16) and a share icon, a favorite icon, an edit icon, a delete icon, and more icons 18. When the mobile phone receives an operation of the user selecting an icon, the mobile phone can perform a corresponding operation on the target image.

[0064] The mobile phone receives the user's input operation on the more icon 18, and displays the option box 19 as shown in 1E in response to the input operation on the more icon 18. The option box 19 includes an image synthesis option 20, a light and shadow enhancement option 21, a rotation option, and a rename option. Figure 1E The photo interface 17 shown also includes a display area for the target image.

[0065] When the user wants to turn on the image processing module of the mobile phone to call the image processing module to perform image synthesis, the user can input an operation on the image synthesis option 20 .

[0066] The mobile phone receives the user's input operation on the image synthesis option 20, and in response to the input operation on the image synthesis option 20, displays the synthesis interface 22 as shown in 1F.

[0067] like Figure 1F As shown, the synthesis interface 22 includes an image display area 23. The image display area 23 displays an image to be synthesized that has not yet been synthesized, that is, a target image.

[0068] In some embodiments, when the image to be synthesized is displayed in the image display area 23, the image area where the object in the image to be synthesized is located can also be identified. The object in the image to be synthesized can include but is not limited to: a face, a human body, a pet, a vehicle, etc. The object in the image to be synthesized can be an object selected by the user, such as a building, etc., and the object of this application is not specifically limited.

[0069] In some embodiments, when the image area where the object in the image to be synthesized is located is identified, the identified image area can also be displayed in the image display area 23. The identification method may include but is not limited to: drawing a border around the object, covering a layer of semi-transparent color block or pattern on the object to highlight it, using an arrow to point to the object, bolding the outline boundary line of the object, and using a highlight color or dynamic effect to identify the outline boundary line of the object.

[0070] like Figure 1F As shown, in the image display area 23, the contour boundary line of the human object is marked to indicate the image area 24 where the human object in the image to be synthesized is located.

[0071] In some embodiments, when the image to be synthesized is displayed in the image display area 23, the mobile phone can determine that the object in the target image area is the target object in response to the user's input operation on the target image area in the image to be synthesized in the image display area 23. The target object is the object in the image to be synthesized that will be synthesized into the background image.

[0072] Exemplarily, when the image to be synthesized includes multiple objects, the mobile phone can determine that the one or more objects in the image to be synthesized are target objects and the image area where the one or more objects are located is the target image area in response to the user's input operation on the image area where one or more objects in the image to be synthesized are located.

[0073] In some embodiments, when the image to be synthesized is displayed in the image display area 23, if the image to be synthesized includes only one object, the object can be directly determined as the target object. Figure 1F As shown, based on the fact that the image to be synthesized includes only one human object, the human object in the image area 24 is taken as the target object.

[0074] like Figure 1F As shown, the synthesis interface 22 also includes a background image display area 25. The background image display area 25 is used to display an image used as a background image. The image displayed in the background image display area 25 can be a panoramic image or a partial image.

[0075] The images displayed in the background image display area 25 may include pictures in the mobile phone, such as images taken by the mobile phone or images downloaded through the network. In some embodiments, the images displayed in the background image display area 25 may also include background templates provided by mobile phone manufacturers, etc. The background templates may include, but are not limited to: High-Dynamic Range Image (HDRI), panoramic images.

[0076] It can be understood that the embodiment of the present application does not specifically limit the source, type, etc. of the background image provided by the background image display area 25.

[0077] like Figure 1F As shown, the background image display area 25 displays an image thumbnail 251 and an image thumbnail 252. The image thumbnail 251 corresponds to the first background image, and the image thumbnail 252 corresponds to the second background image.

[0078] It can be understood that the background image display area 25 can display image thumbnails of multiple background images at the same time.

[0079] The mobile phone receives the user's input operation on the image thumbnail 251, and in response to the input operation on the image thumbnail 251, determines the image corresponding to the image thumbnail 251 (i.e., the first background image) as the background image to be synthesized, synthesizes the target object in the image to be synthesized into the first background image, and displays the image as shown in FIG. Figure 1G The image composite display area 26 is shown.

[0080] like Figure 1G As shown, a composite image obtained by synthesizing the target object in the image to be synthesized into the first background image is displayed in the image synthesis display area 26. The composite image includes the first background image and the target object in the image to be synthesized. The background area in the composite image is coordinated with the target object as a whole, thereby improving the overall visual quality of the processed image.

[0081] The mobile phone receives the user's input operation on the image thumbnail 251, and in response to the input operation on the image thumbnail 251, determines the image corresponding to the image thumbnail 252 (i.e., the second background image) as the background image to be synthesized, synthesizes the target object in the image to be synthesized into the second background image, and displays the image as shown in FIG. Figure 1H The image composite display area 26 is shown.

[0082] Figure 1H The image synthesis display area 26 displays a synthesized image obtained by synthesizing the target object in the image to be synthesized into the second background image, and the synthesized image includes the second background image and the target object in the image to be synthesized. The background area in the synthesized image is coordinated with the target object as a whole, thereby improving the overall visual quality of the processed image.

[0083] In some embodiments, the mobile phone may display more image thumbnails serving as background images for user selection in response to a user input operation on the background image display area 25 .

[0084] like Fig. 1IAs shown, the mobile phone can respond to the user's sliding operation on the background image display area 25, and in response to the sliding operation on the background image display area 25, display the image thumbnail 253 and the image thumbnail 254 in the background image display area 25. The image thumbnail 253 corresponds to the third background image, and the image thumbnail 254 corresponds to the fourth background image. Among them, the first background image and the second background image are partial images, and the third background image and the fourth background image are panoramic images, such as high dynamic range images.

[0085] The following uses a mobile phone as an electronic device and an image processing module integrated in a gallery application as an example to exemplarily introduce the light and shadow migration scenario provided in an embodiment of the present application.

[0086] When the user wants to start the image processing module and call the image processing module to transfer the light and shadow effects of an image to another image, the user can input an operation on the light and shadow enhancement option 21 .

[0087] As described above, in response to the input operation of the more icon 18, the mobile phone displays the following Figure 2A Option box 19 is shown.

[0088] The mobile phone receives the user's input operation on the light and shadow enhancement option 21 in the option box 19, and in response to the input operation on the light and shadow enhancement option 21, displays the light and shadow editing interface 27 as shown in 2B.

[0089] like Figure 2B As shown, the light and shadow editing interface 27 includes a light and shadow display area 28 and an image selection area 29. The image that has not yet been subjected to light and shadow effect migration, that is, the target image, is displayed in the light and shadow display area 28. The image used to provide light and shadow effects (light and shadow information) is displayed in the image selection area 29. The image displayed in the image selection area 29 can be a panoramic image or a partial image.

[0090] The images displayed in the image selection area 29 may include pictures in the mobile phone, such as images taken by the mobile phone or images downloaded from the network. In some embodiments, the images displayed in the image selection area 29 may also include image templates provided by mobile phone manufacturers, etc. Image templates may include, but are not limited to: high dynamic range images, panoramic images.

[0091] like Figure 2B As shown, the image selection area 29 displays an image thumbnail 291 and an image thumbnail 292. The image thumbnail 291 corresponds to the first source image, and the image thumbnail 292 corresponds to the second source image.

[0092] In some embodiments, the mobile phone may display more image thumbnails of source images for user selection in response to the user's input operation on the image selection area 29 .

[0093] It will be appreciated that the image selection area 29 may display image thumbnails of a plurality of images simultaneously.

[0094] The mobile phone can select the target source image in response to the user's input operation on the image thumbnail in the image selection area 29, and migrate the light and shadow effects of the target source image to the target image, so that the light and shadow effects of the target image are converted into the light and shadow effects of the target source image.

[0095] Exemplarily, when a user selects image thumbnail 281, the mobile phone can respond to the user's input operation on image thumbnail 281, select the image corresponding to image thumbnail 281 as the target source image, and migrate the light and shadow effects of the target source image to the target image, so that the light and shadow effects of the target image are converted into the light and shadow effects of the target source image.

[0096] For example, if the light source in the target source image of image thumbnail 281 includes electric lights and sunlight coming in from the window, the light and shadow effects of the target image can be edited based on the light source of the target source image so that the lighting information of the target image also includes electric lights and sunlight coming in from the window.

[0097] like Figure 2B As shown, the light and shadow editing interface 27 also includes a light and shadow input box 30. The mobile phone receives the user's input operation on the light and shadow input box 30, and in response to the input operation on the light and shadow input box 30, displays a light and shadow option box 31 as shown in 2C.

[0098] like Figure 2CAs shown, the light and shadow option box 31 includes a butterfly light option, a split light option, a ring light option, a side light option, a backlight option, a top light option and a custom option. The butterfly light option indicates that the light and shadow effect is butterfly light. The mobile phone can add the light and shadow effect of butterfly light to the target image in response to the user's input operation on the butterfly light option. For example, when editing the butterfly light of the target image, the set lighting information includes: the light source is set above the camera to generate a butterfly-shaped shadow under the face. The split light option indicates that the light and shadow effect is split light. The mobile phone can add the light and shadow effect of split light to the target image in response to the user's input operation on the split light option. For example, when editing the split light of the target image, the set lighting information includes: the light source is illuminated from the side, so that one side of the face is bright and the other side is in the shadow. The ring light option indicates that the light and shadow effect is ring light. The mobile phone can add the light and shadow effect of ring light to the target image in response to the user's input operation on the ring light option. For example, when editing the ring light of the target image, the set lighting information includes: the light source is located between the camera and the subject, generating a small shadow on one side of the nose, forming a ring effect. The side light option indicates that the light and shadow effect is side light. The mobile phone can add the light and shadow effect of side light to the target image in response to the user's input operation on the side light option. For example, when editing the side light of the target image, the set lighting information includes: the light source is completely located on one side of the subject. The backlight option indicates that the light and shadow effect is backlight. The mobile phone can add the light and shadow effect of backlight to the target image in response to the user's input operation on the backlight option. For example, when editing the backlight of the target image, the set lighting information includes: the light source is located behind the subject to form a contour light or silhouette effect. The top light option indicates that the light and shadow effect is top light. The mobile phone can add the light and shadow effect of top light to the target image in response to the user's input operation on the top light option. For example, when editing the top light of the target image, the set lighting information includes: the light source is illuminated from above, which can highlight the top of the head and facial structure of the subject. The custom option allows the user to define the light and shadow effect by himself, such as defining the direction of the light source, defining the number of light sources, defining the light of the light source, etc.

[0099] The image processing module provided in the embodiment of the present application can provide convenience for amateur photographers and customers who need background relighting processing, making it easier for them to obtain high-quality portrait photos.

[0100] It is understandable that the above input operation can be clicking, long pressing, etc., and this application does not make any specific limitations on this.

[0101] It can be understood that the above-mentioned method of turning on the image processing module is only an example to illustrate the implementation method of turning on the image processing module in the embodiment of the present application. The different technical solutions formed by other methods of turning on the image processing module in the embodiment of the present application are all within the scope of protection of the present application.

[0102] It should be understood that the above is an example of a scenario and does not limit the scenario of the present application.

[0103] It can be understood that the visual interface elements such as text and controls displayed in the above interface are only examples, and this application does not limit the display position, display style, display size and other parameters of the interface.

[0104] See also Figure 3 , the structure of the image processing module 100 provided in the embodiment of the present application is exemplarily introduced. The image processing module 100 includes an illumination perception model 110 and a generation model 120. The illumination perception model 110 is used to extract illumination features from the first input image, and obtain a panoramic illumination feature vector for describing the illumination information under the panorama, so as to provide a more comprehensive illumination context for the generation model 120. The generation model 120 obtains the second input image and the panoramic illumination feature vector output by the illumination perception model 110, and uses the panoramic illumination feature vector as a conditional vector to guide the generation of a first output image with specific features based on the conditional vector. The first output image is an image generated based on the second input image with the panoramic illumination feature vector output by the illumination perception model 110 as the conditional vector of the generation model 120. The first output image includes the light and shadow effects of the first input image. The illumination information of the first input image is encoded by the illumination perception model 110 and introduced into the generation model to generate a re-lit image with more physical illumination and more realism.

[0105] In an embodiment of the present application, based on the parameters of the lighting perception model 110, the first input image is subjected to lighting feature extraction to obtain a panoramic lighting feature vector corresponding to the first input image. The first input image may include the following lighting training images, panoramic training images, local training images, real lighting images, real panoramic images, real local images, and first images. The panoramic lighting feature vector corresponding to the first input image may include the following: a first panoramic lighting feature vector, a panoramic lighting feature vector corresponding to the panoramic training image, a panoramic lighting feature vector corresponding to the local training image, a panoramic lighting feature vector corresponding to the real lighting image, a panoramic lighting feature vector corresponding to the real panoramic image, a panoramic lighting feature vector corresponding to the real local image, and a target panoramic lighting feature vector.

[0106] In the embodiment of the present application, the lighting perception model 110 includes an image encoder. The image encoder of the lighting perception model 110 is used to extract lighting features from an image to obtain a lighting feature vector.

[0107] In an embodiment of the present application, the image encoder may include a variety of models and algorithms for extracting illumination features from an image and converting them into a numerical code, which is an illumination feature vector. An illumination feature vector is a set of numerical values ​​used to describe illumination properties in an image, and an illumination feature vector encodes illumination information of the image.

[0108] In an embodiment of the present application, the lighting feature vector may include but is not limited to the lighting information in the following image: brightness distribution, light source direction, light source intensity, color temperature, reflection characteristics, lighting uniformity, shadow length and depth, highlight area, ambient lighting, lighting uniformity, lighting color distribution, reflection and refraction caused by lighting, interaction between lighting and object surface, and physical properties of lighting.

[0109] Among them, the brightness distribution indicates the brightness level of different areas in the image, which may include the brightness of the overall image and the local brightness change of the image. The light source direction is the direction of the source of the light, which may include the direct light source direction and / or a combination of multiple light source directions. The light source intensity indicates the intensity or brightness of the light, which may include the intensity of the main light source and the intensity of the auxiliary light source. The color temperature is the color temperature of the light source, which is used to describe the coldness and warmth of the light source color and is expressed in Kelvin (K). The reflection characteristics are the characteristics of the light reflected from the surface, such as the ratio of specular reflection and diffuse reflection. The illumination uniformity is the uniformity of the distribution of the light in the image, whether there is a significant brightness change or uneven illumination. The shadow length and depth may include the length, depth and clarity of the shadow area, which is used to reflect the relative position and angle of the object and the light source. The highlight area is the highlight area in the image caused by the strong reflection of the light. The ambient light includes the intensity and distribution of the ambient light, such as the sky light or the reflected light of the surrounding objects. The illumination uniformity is used to indicate the consistency or degree of change of the illumination in the image. The illumination color distribution is used to describe the distribution of the illumination color in different areas. Reflection and refraction caused by light are used to indicate the reflection and refraction characteristics of light on the surface of an object. The interaction of light with the surface of an object is used to indicate how light affects the color and texture of the surface of an object. The physical properties of light are used to describe the physical characteristics of light, such as wavelength, frequency, etc.

[0110] Image encoders may include but are not limited to the following models and algorithms: traditional image processing algorithms, deep learning models, generative adversarial networks, recurrent neural networks (RNNs), etc., wherein traditional image processing algorithms may include but are not limited to: histogram analysis that estimates illumination information by calculating the brightness histogram of an image, gradient analysis that estimates illumination intensity and direction by calculating the gradient magnitude and direction of an image. Among them, deep learning models may include but are not limited to: convolutional neural networks (CNNs) that can automatically learn hierarchical structural features (including illumination features), autoencoders, and variational autoencoders (VAEs).

[0111] It is understandable that which model is chosen by the image encoder depends on the specific application scenario, data availability, required output format and performance requirements. In practical applications, it may be necessary to combine multiple models and methods to achieve illumination feature extraction.

[0112] In an embodiment of the present application, the lighting perception model 110 includes a first image encoder 111. Specifically, the first image encoder 111 extracts a feature vector related to illumination from the first input image, and the extracted feature vector related to illumination is the illumination feature vector. When the first input image is a panoramic image, the first image encoder 111 extracts a feature vector related to illumination from the first input image, and the extracted feature vector related to illumination is a panoramic illumination feature vector. Wherein, if the first input image is a panoramic image, the first input image may include but is not limited to the following panoramic training image, a real panoramic image, and a target panoramic image, and accordingly, the panoramic illumination feature vector corresponding to the first input image may include but is not limited to the following first panoramic illumination feature vector, a target panoramic illumination feature vector, and a panoramic illumination feature vector corresponding to a real panoramic image.

[0113] In an embodiment of the present application, the lighting perception model 110 includes a second image encoder 112. Specifically, the second image encoder 112 extracts a feature vector related to illumination from the first input image, and the extracted feature vector related to illumination is an illumination feature vector. When the first input image is a local image, the second image encoder 112 extracts a feature vector related to illumination from the first input image, and the extracted feature vector related to illumination is a local illumination feature vector. Wherein, if the first input image is a local image, the first input image may include but is not limited to: the following local training image, real local image, target local image, and accordingly, the local illumination feature vector corresponding to the first input image may include but is not limited to the following local illumination feature vector corresponding to the local training image, the local illumination feature vector corresponding to the real local image, and the local illumination feature vector corresponding to the target local image.

[0114] In an embodiment of the present application, the image encoder in the lighting perception model 110 may include a first image encoder 111 and a second image encoder 112. In some embodiments, the first image encoder 111 and the second image encoder 112 may be the same image encoder or two image encoders. When the first image encoder 111 and the second image encoder 112 are two image encoders, the first image encoder 111 and the second image encoder 112 may be the same or different. When the first image encoder 111 and the second image encoder 112 are the same image encoder, the image encoder in the lighting perception model 110 may extract illumination features for a panoramic image or for a local image.

[0115] In the embodiment of the present application, the lighting perception model 110 includes a first image encoder 111 and a second image encoder 112. The first image encoder 111 is used to extract lighting features from the panoramic image to obtain a panoramic lighting feature vector. The second image encoder 112 is used to extract lighting features from the local image to obtain a local lighting feature vector.

[0116] In an embodiment of the present application, a panoramic illumination feature vector and a local illumination feature vector describe illumination properties within scenes of different ranges. The following exemplarily introduces the difference between a panoramic illumination feature vector and a local illumination feature vector from several directions: a local illumination feature vector describes illumination information of a specific region or object in an image. A panoramic illumination feature vector describes illumination information of the entire scene, and may include illumination information of 360 degrees or at least a very wide viewing angle. A local illumination feature vector may include less illumination change information, while a panoramic illumination feature vector needs to capture more complex illumination changes, such as illumination differences in different regions. A local illumination feature vector may only reflect the direction of a local light source, while a panoramic illumination feature vector may reflect illumination in multiple directions.

[0117] The following uses the second image encoder 112 as a convolutional neural network to exemplarily introduce the process of using the convolutional neural network to obtain the illumination feature vector of the first input image.

[0118] The input layer of a convolutional neural network receives a first input image, which usually contains color channels (such as RGB) and pixel values. One or more convolutional layers of the convolutional neural network use a set of learnable convolution kernels (or filters) to extract local features of the first input image. The output of the convolutional layer usually passes through an activation function, such as ReLU (Rectified Linear Unit), to introduce nonlinearity and help the network learn more complex features. The pooling layer of the convolutional neural network (such as maximum pooling or average pooling) is used to reduce the spatial dimension of the features while increasing the invariance to image displacement. The convolutional neural network gradually extracts higher-level features through the stacking of multiple convolutional and pooling layers, which gradually transition from shallow edges and textures to deep semantic information. At the end of the convolutional neural network, the output of the convolutional layer is flattened and passed through one or more fully connected layers, which convert the features from the spatial dimension to a single feature vector. The final fully connected layer of the convolutional neural network outputs a fixed-length vector, the illumination feature vector, which encodes the illumination information in the first input image.

[0119] In the embodiment of the present application, the lighting perception model 110 further includes a vector extension model 113. The vector extension model 113 is used to extend the local illumination feature vector output by the second image encoder 112 into a panoramic illumination feature vector to obtain a panoramic illumination feature vector.

[0120] In an embodiment of the present application, the process in which the vector extension model 113 extends the local illumination feature vector output by the second image encoder 112 into a panoramic illumination feature vector (panoramic illumination feature vector) may include one or more of the following aspects: 1. Feature extension: The vector extension model 113 extends the local illumination feature vector to a space capable of representing the illumination of the entire scene to cover the illumination characteristics of the panorama. Among them, the space capable of representing the illumination of the entire scene may include, but is not limited to, a panoramic illumination space, a high dynamic range space, etc. The panoramic illumination space is used to describe the space of illumination information in a 360-degree scene, which can capture the illumination information of the entire environment, including direction, intensity, and color temperature. The high dynamic range (HDR) space is used to represent a color space with a wider range of brightness and contrast than the ordinary 8-bit color space. 2. Feature generation: The process in which the vector extension model 113 extends the local illumination feature vector output by the second image encoder 112 into a panoramic illumination feature vector can be considered as a process of generating a vector, in which the vector extension model 113 learns how to infer and generate a panoramic illumination feature vector capable of representing panoramic illumination from local features. 3. Illumination Prediction: Use deep learning models to predict panoramic illumination, which may include predictions of the direction, intensity, and distribution of illumination. 4. Representation Learning: The vector extension model 113 learns how to map local illumination feature vectors to high-level representations of panoramic illumination. 5. Feature Transformation: The vector extension model 113 transforms one feature representation (local illumination feature vector) into another feature representation (panoramic illumination feature vector). 6. Interpolation and Extrapolation: The vector extension model 113 uses mathematical methods to interpolate and extrapolate local features to predict panoramic illumination features. 7. Feature Fusion: The vector extension model 113 fuses multiple local illumination feature vectors to generate a comprehensive panoramic illumination feature vector.

[0121] It is understood that the above methods can be used alone or in combination to achieve the best panoramic illumination feature vector generation effect. It is understood that the choice of model depends on the specific application scenario, data availability, required output format and performance requirements. In practical applications, it may be necessary to combine multiple models and methods to achieve the best expansion of local illumination feature vectors into panoramic illumination feature vectors.

[0122] In some embodiments, the lighting perception model 110 may further include a text generation model 114. The text generation model 114 is used to extract the lighting features of the first input image to obtain a first text feature vector for describing the lighting features of the first input image. In some embodiments, the text generation model 114 may obtain a panoramic lighting feature vector or a local lighting feature vector from an image encoder of the lighting perception model 110, and then generate a corresponding first text feature vector based on the obtained panoramic lighting feature vector or the local lighting feature vector.

[0123] In the embodiment of the present application, the text generation model 114 may include but is not limited to the following models or algorithms: a multimodal model, an image description generation model 120, and a natural language processing model.

[0124] In an embodiment of the present application, the generation model 120 is used to transfer the lighting effects of the first input image to the second input image. The lighting perception model 110 outputs a panoramic lighting feature vector corresponding to the first input image to the generation model 120. The generation model 120 is also used to use the panoramic lighting feature vector output by the lighting perception model 110 as a guiding condition and generate a first output image based on the second input image. The panoramic lighting feature vector of the guiding condition provides additional semantic information that can help the model generate images that conform to specific conditions. By combining the conditional information, the generation model 120 can more accurately capture the lighting characteristics of the first input image.

[0125] In the embodiment of the present application, the second input image may include a training image or a real image. The second input image may include the following first training image, second training image, second image, and a synthetic image output by the synthetic model 130. The first output image may include the following first generated image, second generated image, and target generated image.

[0126] In an embodiment of the present application, the generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123. The feature extraction layer 121 is used to extract image features from the second input image to obtain an image feature vector corresponding to the second input image. The fusion layer 122 is used to fuse the image feature vector output by the feature extraction layer 121 with the first panoramic illumination feature vector output by the illumination perception model 110 to obtain a fused feature vector. The generation layer 123 is used to generate a first output image based on the fused feature vector. The panoramic illumination feature vector output by the illumination perception model 110 is used as a conditional vector and introduced into the generation model 120 through the fusion layer 122, so that the light and shadow effects are encoded into the generation model 120.

[0127] In the embodiment of the present application, the image features extracted by the feature extraction layer 121 may include but are not limited to: edges, corners, textures, shapes, and color distribution. The feature extraction layer 121 extracts image features to facilitate the generation model 120 to recognize and understand the content of the image, thereby improving the generation quality and performance.

[0128] In the embodiment of the present application, the feature extraction layer 121 may include but is not limited to the following models or algorithms: convolutional neural network (CNN), autoencoder.

[0129] In the embodiment of the present application, the panoramic illumination feature vector corresponding to the first input image output by the illumination perception model 110 is used as the conditional vector, and the fusion layer 122 fuses the conditional vector with the image feature vector output by the feature extraction layer 121 .

[0130] In the embodiment of the present application, the method of fusing the conditional vector with the image feature vector output by the feature extraction layer 121 may include but is not limited to the following two methods: (1) direct combination, directly combining the conditional vector with the image feature vector, such as through concatenation, addition or multiplication. (2) indirect guidance: the conditional vector may indirectly affect the generation process by influencing the decision-making process of the model, rather than directly combining with the image feature vector.

[0131] In the embodiment of the present application, the fusion layer 122 directly combines the conditional vector with the image feature vector in the following ways, including but not limited to: 1. Feature concatenation. Feature concatenation is to directly connect multiple feature vectors in series to form a longer feature vector. 2. Point-wise addition. Point-wise addition is to add the corresponding elements of two feature vectors. 3. Adaptive spatial feature fusion (ASFF). ASFF is an adaptive feature fusion method that obtains new fused features by multiplying the features of different layers by weight parameters and adding them.

[0132] In an embodiment of the present application, the fusion layer 122 indirectly guides the conditional vector and the image feature vector in a manner that includes but is not limited to the following: 1. Attention Mechanism. The attention mechanism weights different parts of the input data or feature map by learning the weight distribution. For example, SENet calculates the average pixel value of each channel, normalizes it with the sigmoid function after a series of operations, and obtains the attention weight. 2. Self-Attention Mechanism. The self-attention mechanism can capture the dependencies between different features and perform weighted fusion of features based on these relationships. 3. Multi-Scale Channel Attention Module (MS-CAM). 4. Iterative Attention Feature Fusion (iAFF).

[0133] The following takes the example of introducing the self-attention mechanism into the generation model 120 and the example of the fusion layer 122 including the self-attention layer to introduce the fusion of the image feature vector and the conditional vector.

[0134] The conditional vector and the image feature vector output by the feature extraction layer 121 are input to the self-attention layer. The conditional vector is combined with the input features (i.e., the image feature vector) of the self-attention layer, such as adding the conditional vector to each input feature, concatenating the conditional vector with the input feature, or multiplying the conditional vector by the scaling factor of the input feature. In the self-attention layer, the Query (Q), Key (K), and Value (V) matrices are calculated. These matrices are usually obtained by different linear transformations of the input features. The attention weights are calculated using Q and K adjusted by the conditional vector. The attention weights are usually obtained by calculating the dot product of Q and K and applying the softmax function. The conditional vector can affect the calculation of the attention weights, so that the model pays more attention to the features related to the conditional information when generating feature representations. The V matrix is ​​weighted and summed using the calculated attention weights to generate weighted output features. The weighted output features are combined with the original input features, such as by combining the weighted output features with the original input features through residual connections or direct replacement to obtain a fused feature vector. Then, when the generation layer 123 generates an image based on the fused feature vector, not only the information of the image itself is considered, but also the additional conditional information. The conditional vector can indirectly affect the feature representation through the self-attention mechanism. The combination of the conditional vector and the self-attention mechanism will provide a more powerful guiding capability for the generation model 120, so that the generated image or feature representation is more in line with the given conditions.

[0135] In some embodiments, the generation model 120 may include but is not limited to the following models or algorithms: AIGC (artificial intelligence generated content) generative large model, diffusion model, feature-based transfer learning, style transfer algorithm, multimodal model (Contrastive Language-Image Pre-training, CLIP). The diffusion model gradually adds noise to the data (forward diffusion process), and then learns how to reverse this process (reverse denoising process), thereby recovering the original data distribution from the noisy data and generating new data samples.

[0136] Style transfer based on diffusion model generates images by gradually adding and removing noise, and adds control conditions in the generation process to achieve light and shadow effect transfer. Feature-based transfer learning can map data from the source domain (source image) and the target domain (target image) to a common feature space by learning a pair of mapping functions, and then train the classifier using the source domain and target domain data mapped to the new feature space. The neural style transfer algorithm achieves style transfer by minimizing content loss and style loss. Using the CLIP model, image style transfer is guided by natural language descriptions to achieve style transfer based on text descriptions.

[0137] In the embodiment of the present application, the generation model 120 may be a diffusion model, which includes a feature extraction layer 121 , a fusion layer 122 , and a generation layer 123 .

[0138] The following uses the generation model 120 as a diffusion model as an example to exemplify how the diffusion model obtains a processed image based on the second input image and the output of the illumination perception model 110 .

[0139] The second input image is input into the diffusion model, and the illumination feature of the second input image is extracted based on the parameters of the feature extraction layer 121 to obtain the image feature vector of the second input image. The illumination feature vector of the first input image output by the lighting perception model 110 is used as the conditional vector, and the conditional vector and the image feature vector are input into the fusion layer 122. In the diffusion model, the fusion layer 122 will continuously fuse the information of the conditional vector into the generated image, such as performing dot product, weighted summation and other operations on the conditional vector and the image feature vector. The generation process is iterative. In each step, the fusion layer 122 will update the image according to the currently generated image and the conditional vector. This process will continue until the generated image reaches a certain quality standard or reaches a preset number of iterations. Finally, the generated first output image gradually evolves into a clear image that meets the requirements of the conditional vector. The first output image not only has the image features of the second input image, but also contains the illumination information in the conditional vector.

[0140] In an embodiment of the present application, the generation model 120 can generate an image based on an AIGC generative large model, especially a generation technology based on a diffusion model. The AIGC generative large model includes a feature extraction layer 121, a fusion layer 122 and a generation layer 123.

[0141] The following takes the generation model 120 as an AIGC generative large model as an example to exemplify how the AIGC generative large model obtains a processed image based on the second input image and the output of the lighting perception model 110.

[0142] The second input image is input into the AIGC generative model, and based on the parameters of the feature extraction layer 121, the illumination feature of the second input image is extracted to obtain the image feature vector of the second input image. The illumination feature vector of the first input image output by the illumination perception model 110 is used as the conditional vector, and the conditional vector and the image feature vector are input into the fusion layer 122. In the AIGC generative model, the fusion layer 122 will continuously fuse the information of the conditional vector into the generated image, such as performing dot product, weighted summation and other operations on the conditional vector and the image feature vector. The generation process is iterative. In each step, the fusion layer 122 will update the image according to the currently generated image and the conditional vector. This process will continue until the generated image reaches a certain quality standard or reaches a preset number of iterations. Finally, the generated first output image gradually evolves into a clear image that meets the requirements of the conditional vector. The first output image not only has the image features of the second input image, but also contains the illumination information in the conditional vector.

[0143] In an embodiment of the present application, the generation model 120 may perform illumination estimation on the second input image to determine how to adjust its light and shadow to match the light and shadow effects of the first input image. The generation model 120 may adjust the color balance and saturation of the second input image to match the light and shadow effects of the first input image. The generation model 120 may adjust the brightness and contrast of the second input image to simulate the illumination intensity of the first input image. The generation model 120 may add or modify shadow and highlight areas on the second input image to match the light and shadow distribution of the first input image. The generation model 120 may use image fusion technology to combine the light and shadow effects of the first input image with the content of the second input image. The generation model 120 may use a mask or selection tool to locally apply light and shadow effects on the second input image.

[0144] In the embodiment of the present application, the image processing module 100 may further include a synthesis model 130. The synthesis model 130 is used to merge multiple images or partial regions of images into a new image. The synthesis model 130 may merge N objects of one or more images into a first input image to obtain a second input image, where N is an integer greater than or equal to 1. The synthesis model 130 may be applied in image synthesis scenarios, such as Figures 1A to 1G The image shown is a composite scene.

[0145] See also Figure 4 , taking the synthesis model 130 synthesizing the input image and the background image as an example, the working process of the synthesis model 130 is introduced exemplarily.

[0146] The synthesis model 130 obtains an input image and determines the object or area in the input image that needs to be cut out. After determining the object or area that needs to be cut out, the object that needs to be cut out is the target object, and the area that needs to be cut out is the target image area. The synthesis model 130 uses image segmentation technology to cut out the target object from the input image, which can be achieved by a deep learning-based method (such as GrabCut, Mask R-CNN, etc.) or a traditional image processing technology (such as color similarity, edge detection, etc.). A mask image is generated from the cutout process, in which the area of ​​the target object or the target image area is transparent or marked as selected, and the background is non-transparent or marked as unselected. Among them, the object or area that needs to be cut out in the mask image is marked as selected. Mask image is a commonly used technique in image editing and synthesis, which is used to hide or display specific parts of an image. The mask image is usually a two-dimensional array, in which the value of each element represents the transparency or visibility of the corresponding pixel. Each pixel in the mask image can have a transparency value (usually between 0 and 1), where 0 means completely transparent (invisible) and 1 means completely opaque (visible). The mask image can be used to selectively display or hide certain parts of the image, for example, only displaying the foreground object and hiding the background when synthesizing the image. The area of ​​the target object is the object or area that needs to be cut out. The synthesis model 130 synthesizes the area of ​​the target object into the background image to obtain a synthesized image. The synthesis model 130 synthesizes the cut-out object with the background image, which usually involves applying a mask to the background image and then blending it with the background image.

[0147] In some embodiments, the synthesis model 130 may obtain a synthesized image according to an input image, an image region of a target object, and a background image.

[0148] The generation model 120 obtains the synthesized image output by the synthesis model 130, and transfers the light and shadow effects of the first input image to the synthesized image.

[0149] In the embodiment of the present application, training and reasoning are two stages of the image processing module 100. The training stage refers to the process of determining the model parameters of the image processing module 100 by fitting the image processing module 100 to the rules in the training data set through a certain method based on the training data set. Reasoning is the process of the image processing module 100 performing calculations and predictions based on the data input to the model after the training is completed. During the training stage, the values ​​of the model parameters of the image processing module 100 are changing dynamically. The reasoning stage is the process of the image processing module 100 for completing an reasoning task after the training is completed, and the values ​​of the model parameters of the image processing module 100 determined at the end of the training remain unchanged during the reasoning stage.

[0150] In the embodiment of the present application, the training phase of the image processing module 100 includes pre-training and fine-tuning. Pre-training refers to training a model on a large data set so that it can learn common features. Fine-tuning is to further train the model for a specific task based on pre-training. Through fine-tuning, the model can adjust its learned features to make it more suitable for a specific application scenario or data set.

[0151] It is understood that the image processing module 100 can be trained in an electronic device, and the electronic device is a training device. The training device includes: a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the following pre-training and / or fine-tuning of the image processing module 100. In some embodiments, the training device may include multiple electronic devices.

[0152] The following is an exemplary introduction to the pre-training phase of the image processing module 100 .

[0153] See also Figure 5A The pre-training method of the image processing module 100 is applied to an electronic device, and the electronic device includes an illumination perception model 110 and a generation model 120. The generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123.

[0154] Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above and will not be repeated here.

[0155] The electronic device inputs the first training image of the first training set to the feature extraction layer 121 of the generation model 120 .

[0156] The first training set may include a large-scale data set, such as millions of images, covering multiple categories. The first training image may include multiple training images. The first training image in the first training set is an image that needs to transfer light and shadow effects, which may be images of the same scene under different lighting conditions, or images of different scenes.

[0157] The electronic device extracts image features from the first training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the first training image.

[0158] The electronic device inputs the illumination training image into the illumination perception model 110 , and extracts illumination features from the illumination training image based on the parameters of the illumination perception model 110 , to obtain a first panoramic illumination feature vector corresponding to the illumination training image.

[0159] In the embodiment of the present application, the lighting training image can be regarded as a reference source or source of lighting effects (lighting information). As the basis for the model to learn lighting effects, it provides lighting features (lighting information) that the image processing module 100 needs to learn, and these lighting features are then applied to the first training image to achieve the migration of lighting effects. The lighting training image can include images with different lighting conditions, such as images at different times (early morning, noon, evening) and in different weather (sunny, cloudy, rainy), to provide different lighting information for the model.

[0160] In an embodiment of the present application, the first panoramic lighting feature vector provides lighting information of 360 degrees or at least a very wide viewing angle, including lighting information in all directions of the scene, thereby being able to more comprehensively and realistically reflect the lighting information of the scene.

[0161] The electronic device inputs the image feature vector and the first panoramic illumination feature vector into the fusion layer 122 , and fuses the image feature vector and the first panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector.

[0162] The electronic device inputs the fused feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a first generated image according to the fused feature vector, wherein the first generated image is an image generated based on the first training image with the first panoramic illumination feature vector as the conditional vector of the generation model 120.

[0163] The electronic device adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first generation image to obtain the trained generation model 120 and the lighting perception model 110 .

[0164] In the embodiment of the present application, adjusting the parameters of the generation model 120 and the lighting perception model 110 includes: adjusting the parameters of the feature extraction layer 121 , the parameters of the fusion layer 122 , the parameters of the generation layer 123 , and the parameters of the lighting perception model 110 .

[0165] In the embodiments of the present application, the electronic device may use supervised learning (Supervised Learning) or unsupervised learning (Unsupervised Learning) to train the generation model 120 and the lighting perception model 110, or may use a combination of supervised learning and unsupervised learning, and the present application does not make any specific limitations on this.

[0166] In an embodiment of the present application, in supervised learning, the first training set may include image pairs, and the image pairs include a first training image and a first target image corresponding to the first training image, that is, each pair of images includes a first training image and a corresponding first target image. The first training image is an image whose lighting effect is desired to be changed to a target lighting effect, and the first target image is an image that provides a target lighting effect. The paired images in the first training set provide direct supervisory signals for the generation model 120, and the generation model 120 will learn how to extract lighting features from the first target image, and adjust the lighting information of the first training image according to these features during the generation process.

[0167] Among them, the first training image can be an original image with certain lighting conditions. In order to ensure the generalization ability of the model, the collected first training images can include multiple lighting conditions (such as front light, side light, back light, etc.), scenes, lighting attributes (such as light source direction, intensity, color, etc.) and different objects (if it is a general lighting migration model, the objects should be diversified; if it is a lighting migration of a specific object such as a face, the objects should remain consistent). The first target image has a similar scene or object to the first training image, but the lighting conditions are different. In the context of lighting migration, the first target image is an image with specific lighting effects. It is the starting point of lighting migration and provides lighting features that the model needs to learn. The first training image is the image to which the lighting effects of the first target image are expected to be migrated. It is similar to the first target image in content, but has different lighting information. During the training process, the first target image serves as a reference after migrating the lighting effects.

[0168] In an embodiment of the present application, in supervised learning, image pairs can be annotated. When annotating image pairs, it is necessary to ensure that the first training image and the first target image are paired, that is, they represent the same scene or object, but the lighting conditions are different. In this way, the model can learn the mapping relationship between different lighting conditions. The annotation process may include specifying the correspondence between the first training image and the first target image, and assigning appropriate labels or lighting parameters to them. These labels or parameters will serve as supervisory signals during the training process to guide the model to learn the lighting migration task. Specifically, the label may include parameters such as lighting direction, lighting intensity, and lighting color, which describe the changes in lighting of the target image relative to the source image. The model will try to learn the mapping relationship between these parameters and image pairs so that when a new first input image is given, a first output image with desired lighting conditions can be generated.

[0169] In an embodiment of the present application, in supervised learning, the first target image corresponding to the first training image includes an illumination training image. The illumination training image provides a conditional vector for the generation model 120, and the illumination training image can also be used as the first target image corresponding to the first training image to provide a direct supervision signal for the generation model 120.

[0170] Specifically, when conducting supervised learning, an illumination training image set and a training image set are obtained. The illumination training image set includes illumination training images under different illumination conditions, such as images at different times (early morning, noon, evening) and in different weather conditions (sunny, cloudy, rainy days). These images will serve as the basis for the generation model 120 to learn light and shadow effects. The training image set includes a first training image to which the light and shadow effects need to be transferred, which may be an image of the same scene under different illumination conditions, or an image of different scenes. Prepare a pairing set of the illumination training image set and the training image set to obtain a first training set. Each pair of images in the first training set is similar in content, but has different illumination conditions. Thus, the first training set may include illumination training images in the illumination training image set and the first training image in the training image set.

[0171] In some embodiments, for supervised learning, it is necessary to provide annotation information for the training images. In the context of light and shadow effect transfer, this may include, but is not limited to: description of the light and shadow effect (e.g., "strong light", "soft light", "shadow", etc.). Light direction (e.g., "from the top left", "from the front", etc.). Light intensity (possibly measured by the brightness, contrast, etc. of the image).

[0172] In an embodiment of the present application, during the pre-training process, when training the model in a supervised learning manner, a CG (computer graphics) synthetic data set can be used for pre-training. The image pairs included in the CG synthetic data set are usually a combination of images with a specific relationship generated by computer graphics technology. The image pair may include a first training image and a first target image. In rendering generation: using computer graphics technology to render and generate images with specific light and shadow effects. By adjusting lighting parameters, scene layout, material properties, etc., a large number of image pairs with different lighting information can be generated. Automated annotation: Since the CG synthetic data set is generated by a computer algorithm, the lighting migration relationship between image pairs can be automatically annotated without manual intervention.

[0173] In an embodiment of the present application, during unsupervised learning, the model will not require paired labeled data (ie, no explicitly labeled first training image and first target image pair and their corresponding lighting condition labels) to learn how to migrate lighting.

[0174] The following description uses the method of supervised learning to train the image processing module 100 during the pre-training process. It is understandable that unsupervised learning can also be used to fine-tune the image processing module 100.

[0175] See also Figure 5B , taking the example that the image processing module 100 includes an illumination perception model 110 and a generation model 120, the illumination perception model 110 includes a first image encoder 111, and the generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123, the pre-training process of the image processing module 100 is exemplarily introduced. Figure 5B and Figure 5A The differences include: Figure 5B The lighting perception model 110 includes a first image encoder 111, and the lighting training image includes a panoramic training image.

[0176] Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above and will not be repeated here.

[0177] The illumination training image includes a panoramic training image, and the first training set includes a first training image and a first target image. The relevant contents of the first training set, the first training data, and the first target image can be referred to above, and will not be repeated here.

[0178] In an embodiment of the present application, the illumination training image includes a panoramic training image. The panoramic training image provides a 360-degree full or at least very wide viewing angle, that is, the panoramic training image includes illumination information from all directions in the scene, so that the illumination information of the scene can be more comprehensively and realistically reflected, helping the model to better understand the overall environment of the scene, including the background, illumination conditions, the relationship between objects, etc., which helps the training model to maintain consistent illumination effects between different parts, especially in the task of light and shadow effect migration. The electronic device inputs the first training image of the first training set into the feature extraction layer 121 of the generation model 120. The electronic device extracts image features from the first training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the first training image. The electronic device inputs the panoramic training image into the first image encoder 111, and extracts illumination features from the panoramic training image based on the parameters of the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the panoramic training image, which is used as a guiding condition for the generation model, that is, the panoramic illumination feature vector is input into the fusion layer 122 as a conditional vector of the fusion layer 122. The electronic device inputs the image feature vector and the panoramic illumination feature vector into the fusion layer 122, and fuses the image feature vector and the panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector. The electronic device inputs the fused feature vector into the generation layer 123, and obtains a first generated image according to the fused feature vector based on the parameters of the generation layer 123, wherein the first generated image is an image generated based on the first training image with the panoramic illumination feature vector as the condition vector of the generation model 120. The electronic device constructs a first loss function based on the first generated image and the first target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function to obtain the trained generation model 120 and the lighting perception model 110.

[0179] Among them, the relevant contents of the image feature vector, the panoramic illumination feature vector, and the fused feature vector can be referred to above.

[0180] In the embodiment of the present application, the first target image includes the above-mentioned panoramic training image, that is, the first target image is a panoramic training image corresponding to the first training image.

[0181] In the embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the first image encoder 111, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0182] In the embodiment of the present application, the first loss function can be used to measure the difference between the model output and the real light and shadow effect, which may include but is not limited to: pixel-level loss (such as mean square error), perceptual loss (such as VGG feature loss), style loss (such as Gram matrix loss). The first loss function may include but is not limited to: mean square error (MSE), cosine similarity, feature alignment loss (Feature Alignment Loss).

[0183] See also Figure 5C , taking the image processing module 100 including the lighting perception model 110 and the generation model 120, the lighting perception model 110 including the second image encoder 112 and the vector extension model 113, and the generation model 120 including the feature extraction layer 121, the fusion layer 122, and the generation layer 123 as an example, the pre-training process of the image processing module 100 is exemplarily introduced. Figure 5C and Figure 5B The differences include: the lighting perception model 110 includes a second image encoder 112 and a vector extension model 113, and the lighting training image includes a local training image. Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above, and will not be repeated here. The lighting training image includes a local training image. The first training set includes a first training image and a first target image. The relevant contents of the first training image, the first training image and the first target image can be referred to above, and will not be repeated here.

[0184] In the embodiment of the present application, the illumination training image includes a local training image. The local training image includes an image of a specific part or area in a scene. When the image processing module 100 is applied to an inference task, the user can more easily provide a local image, thereby training the model based on the local training image to improve the generalization ability of the model for different scenes and tasks.

[0185] The electronic device inputs the first training image of the first training set to the feature extraction layer 121 of the generation model 120. The electronic device extracts image features from the first training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the first training image. The electronic device inputs the local training image to the second image encoder 112, and extracts illumination features from the local training image based on the parameters of the second image encoder 112 to obtain a local illumination feature vector corresponding to the local training image. The electronic device inputs the local illumination feature vector corresponding to the local training image to the vector extension model 113, and expands the local illumination feature vector into a panoramic illumination feature vector based on the parameters of the vector extension model 113 to obtain a panoramic illumination feature vector corresponding to the local training image. The electronic device inputs the panoramic illumination feature vector and the image feature vector corresponding to the local training image to the fusion layer 122, and fuses the panoramic illumination feature vector and the image feature vector corresponding to the local training image based on the parameters of the fusion layer 122 to obtain a fused feature vector. The electronic device inputs the fused feature vector to the generation layer 123, and obtains a first generated image according to the fused feature vector based on the parameters of the generation layer 123, wherein the first generated image is an image generated based on the first training image using the panoramic illumination feature vector corresponding to the local training image as the condition vector of the generation model 120. The electronic device constructs a first loss function according to the first generated image and the first target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function, to obtain the trained generation model 120 and the lighting perception model 110.

[0186] Among them, the relevant contents of the image feature vector, the local illumination feature vector, the panoramic illumination feature vector, the fused feature vector, and the first loss function can be referred to above.

[0187] In the embodiment of the present application, the first target image includes the above-mentioned local training image, that is, the first target image is a local training image corresponding to the first training image.

[0188] In an embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the second image encoder 112, the parameters of the vector extension model 113, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0189] See also Figure 5D, taking the example that the image processing module 100 includes an illumination perception model 110 and a generation model 120, the illumination perception model 110 includes a first image encoder 111, a second image encoder 112 and a vector extension model 113, and the generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123, the pre-training process of the image processing module 100 is exemplarily introduced. Figure 5D and Figure 5C The differences include: the lighting perception model 110 further includes a first image encoder 111, and the lighting training image further includes a panoramic training image.

[0190] Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above and will not be repeated here.

[0191] The illumination training image includes a panoramic training image and a local training image. The first training set includes a first training image and a first target image. The relevant contents of the panoramic training image, the local training image, the first training image and the first target image can be referred to above and will not be repeated here.

[0192] The electronic device inputs the first training image of the first training set to the feature extraction layer 121 of the generation model 120. Based on the illumination training image including the panoramic training image and the local training image, at this time, the first training image may include the panoramic image and the local image.

[0193] The electronic device performs image feature extraction on the first training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the first training image. The electronic device inputs the panoramic training image into the first image encoder 111, and performs illumination feature extraction on the panoramic training image based on the parameters of the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the panoramic training image. The electronic device inputs the local training image into the second image encoder 112, and performs illumination feature extraction on the local training image based on the parameters of the second image encoder 112 to obtain a local illumination feature vector corresponding to the local training image. The electronic device inputs the local illumination feature vector corresponding to the local training image into the vector extension model 113, and expands the local illumination feature vector into a panoramic illumination feature vector based on the parameters of the vector extension model 113 to obtain a panoramic illumination feature vector corresponding to the local training image. The electronic device selects any one of the panoramic illumination feature vectors corresponding to the panoramic training image and the panoramic illumination feature vector corresponding to the local training image as the condition vector, that is, inputs the panoramic illumination feature vector corresponding to the panoramic training image or the panoramic illumination feature vector corresponding to the local training image into the fusion layer 122, and fuses the input panoramic illumination feature vector and the image feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector. The electronic device inputs the fused feature vector into the generation layer 123, and obtains a first generated image based on the fused feature vector based on the parameters of the generation layer 123, wherein the first generated image is an image generated based on the first training image with the panoramic illumination feature vector input to the fusion layer 122 as the condition vector of the generation model 120. The electronic device constructs a first loss function based on the first generated image and the first target image. The electronic device constructs a second loss function based on the panoramic illumination feature vector corresponding to the panoramic training image and the panoramic illumination feature vector corresponding to the local training image. The electronic device adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function and the second loss function to obtain the trained generation model 120 and the lighting perception model 110.

[0194] In an embodiment of the present application, the second loss function can be implemented by calculating the Euclidean distance or cosine similarity between the two (the panoramic lighting feature vector corresponding to the panoramic training image and the panoramic lighting feature vector corresponding to the local training image).

[0195] In other embodiments, the losses of the panoramic illumination feature vector corresponding to the panoramic training image and the panoramic illumination feature vector corresponding to the local training image are combined, and different weights can be assigned to the panoramic illumination feature vectors of the panoramic training image and the local training image to balance their contributions to the total loss.

[0196] It is understandable that the first loss function and the second loss function can be constructed according to actual conditions, and this application does not specifically limit this. Among them, the image feature vector, the local illumination feature vector, the panoramic illumination feature vector, the fused feature vector, and the related contents of the first loss function can refer to the above.

[0197] In an embodiment of the present application, the first target image includes the above-mentioned panoramic training image and the local training image, that is, the first target image is the panoramic training image and the local training image corresponding to the first training image. For example, when the first training image is a local image, the first target image corresponding to the first training image is the local training image. When the first training image is a panoramic image, the first target image corresponding to the first training image is the panoramic training image.

[0198] In an embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the first image encoder 111, the parameters of the second image encoder 112, the parameters of the vector extension model 113, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0199] In the embodiment of the present application, using panoramic training images and local training images for training at the same time can help the model achieve good performance at both the global and local levels. Constructing a second loss function to achieve supervised training of the panoramic illumination feature vector corresponding to the local training image and the panoramic illumination feature vector corresponding to the panoramic training image, and keep them consistent, so as to ensure better rationality of illumination physics.

[0200] See also Figure 5E The following takes the lighting perception model 110 including the first image encoder 111, the second image encoder 112, the vector extension model 113 and the text generation model 114 as an example to exemplarily introduce the pre-training of the image processing module 100. Figure 5E and Figure 5D The differences include: the lighting perception model 110 also includes a text generation model 114, and the conditional vector of the generation model 120 also includes a first text feature vector.

[0201] Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122, the generation layer 123 and the text generation model 114 can be referred to above and will not be repeated here.

[0202] The illumination training image includes a panoramic training image and a local training image. The first training set includes a first training image and a first target image. The relevant contents of the panoramic training image, the local training image, the first training image and the first target image can be referred to above and will not be repeated here.

[0203] The electronic device inputs the first training image of the first training set to the feature extraction layer 121 of the generation model 120 .

[0204] The electronic device extracts image features from the first training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the first training image.

[0205] The electronic device inputs the panoramic training image to the first image encoder 111, and extracts illumination features of the panoramic training image based on the parameters of the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the panoramic training image.

[0206] The electronic device inputs the local training image to the second image encoder 112, and extracts illumination features from the local training image based on the parameters of the second image encoder 112 to obtain a local illumination feature vector corresponding to the local training image.

[0207] The electronic device inputs the local illumination feature vector corresponding to the local training image into the vector extension model 113, and based on the parameters of the vector extension model 113, expands the local illumination feature vector into a panoramic illumination feature vector to obtain the panoramic illumination feature vector corresponding to the local training image.

[0208] The electronic device inputs the local training image into the text generation model 114 , and obtains a first text feature vector corresponding to the local training image based on the parameters of the text generation model 114 .

[0209] In some embodiments, the electronic device inputs the panoramic training image into the text generation model 114 , and obtains a first text feature vector corresponding to the panoramic training image based on the parameters of the text generation model 114 .

[0210] In some embodiments, the electronic device inputs the panoramic training image and the local training image into the text generation model 114, and obtains a first text feature vector corresponding to the panoramic training image and a first text feature vector corresponding to the local training image based on the parameters of the text generation model 114.

[0211] The electronic device selects any one of the panoramic illumination feature vectors corresponding to the panoramic training image and the panoramic illumination feature vectors corresponding to the local training image as the conditional vector, that is, inputs the panoramic illumination feature vector corresponding to the panoramic training image or the panoramic illumination feature vector corresponding to the local training image into the fusion layer 122, and inputs the first text feature vector into the fusion layer 122, and based on the parameters of the fusion layer 122, fuses the input panoramic illumination feature vector, the image feature vector and the first text feature vector to obtain a fused feature vector.

[0212] In the embodiment of the present application, the electronic device may input the first text feature vector corresponding to the panoramic training image and / or the first text feature vector corresponding to the local training image into the fusion layer 122 .

[0213] The electronic device inputs the fused feature vector to the generation layer 123, and obtains a first generated image based on the fused feature vector based on the parameters of the generation layer 123, wherein the first generated image is an image generated based on the first training image using the panoramic illumination feature vector and the first text feature vector input to the fusion layer 122 as conditional vectors of the generation model 120.

[0214] The electronic device constructs a first loss function according to the first generated image and the first target image.

[0215] The electronic device constructs a second loss function according to the first text feature vector, the panoramic illumination feature vector corresponding to the panoramic training image, and the panoramic illumination feature vector corresponding to the local training image.

[0216] The electronic device adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function and the second loss function to obtain the trained generation model 120 and the lighting perception model 110.

[0217] Among them, the relevant contents of the image feature vector, the local illumination feature vector, the panoramic illumination feature vector, the fused feature vector, the first loss function, and the second loss function can be referred to above.

[0218] In an embodiment of the present application, the first target image includes the above-mentioned panoramic training image and the local training image, that is, the first target image is the panoramic training image and the local training image corresponding to the first training image. For example, when the first training image is a local image, the first target image corresponding to the first training image is the local training image. When the first training image is a panoramic image, the first target image corresponding to the first training image is the panoramic training image.

[0219] In an embodiment of the present application, adjusting the parameters of the generation model 120 and the lighting perception model 110 includes: adjusting the parameters of the first image encoder 111, the parameters of the second image encoder 112, the parameters of the vector extension model 113, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, the parameters of the generation layer 123, and the parameters of the text generation model 114.

[0220] In some embodiments, the lighting perception model 110 includes a first image encoder 111 and a text generation model 114. The panoramic training image is input to the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the panoramic training image. The panoramic training image is input to the text generation model 114 to obtain a first text feature vector corresponding to the panoramic training image. The first text feature vector and the panoramic illumination feature vector corresponding to the panoramic training image are input to the fusion layer 122 to obtain a fused feature vector. The electronic device constructs a second loss function based on the first text feature vector and the panoramic illumination feature vector corresponding to the panoramic training image.

[0221] In some embodiments, the lighting perception model 110 includes a second image encoder 112, a vector extension model 113, and a text generation model 114. The local training image is input to the second image encoder 112 and the vector extension model 113 to obtain a panoramic illumination feature vector corresponding to the panoramic training image. The panoramic training image is input to the text generation model 114 to obtain a first text feature vector corresponding to the panoramic training image. The first text feature vector and the panoramic illumination feature vector corresponding to the panoramic training image are input to the fusion layer 122 to obtain a fused feature vector. The electronic device constructs a second loss function based on the first text feature vector and the panoramic illumination feature vector corresponding to the local training image.

[0222] In an embodiment of the present application, by obtaining a first panoramic illumination feature vector of an illumination training image, such as a panoramic illumination feature vector of a panoramic training image or a panoramic illumination feature vector of a local training image, the illumination-aware portrait relighting in the inference stage is achieved without relying on an environment map (High-Dynamic Range Image, HDRI). The illumination representation extracted from the local image is converted into a representation learned from the environment map (panorama), so that the aligned illumination representation (first panoramic illumination feature vector) is more robust and physically more reasonable.

[0223] In an embodiment of the present application, the training data used when fine-tuning may include real lighting data. Specifically, when performing supervised learning, a real lighting image set and a real image set are obtained, and the real lighting image set includes real lighting images of different lighting conditions taken by a camera device, such as the real panoramic image and the real local image described below. The real image set includes a second training image taken by a camera device, which needs to migrate the light and shadow effects. Prepare a pairing set of the real lighting image set and the real image set to obtain a second training set, that is, each pair of images is similar in content but different in lighting conditions. Thus, the second training set may include the real lighting images in the real lighting image set and the second training images in the real image set.

[0224] The second training set may be a training set for a specific task. The second training set may be the same as or different from the first training set. Exemplarily, the first training set may be a CG synthetic data set, and the second training set may be an image set of real images.

[0225] The following description uses the supervised learning method to train the image processing module 100 during the fine-tuning process. It is understandable that the image processing module 100 can also be fine-tuned using unsupervised learning.

[0226] See also Fig. 6A , taking the example that the image processing module 100 includes the illumination perception model 110 and the generation model 120, the illumination perception model 110 includes the first image encoder 111, and the generation model 120 includes the feature extraction layer 121, the fusion layer 122, and the generation layer 123, the fine-tuning process of the image processing module 100 is exemplarily introduced. Among them, the relevant contents of the illumination perception model 110, the first image encoder 111, the generation model 120, the feature extraction layer 121, the fusion layer 122, and the generation layer 123 can be referred to above, and will not be repeated here.

[0227] The electronic device inputs the second training image of the second training set to the feature extraction layer 121 of the generation model 120 .

[0228] The electronic device extracts image features from the second training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the second training image.

[0229] The electronic device inputs the real panoramic image into the first image encoder 111, and extracts illumination features of the real panoramic image based on the parameters of the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the real panoramic image. The panoramic illumination feature vector is used as a guiding condition for generating a model, that is, the panoramic illumination feature vector is input into the fusion layer 122 as a conditional vector of the fusion layer 122.

[0230] The electronic device inputs the panoramic illumination feature vector and the image feature vector corresponding to the real panoramic image into the fusion layer 122 , and fuses the image feature vector and the panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector.

[0231] The electronic device inputs the fused feature vector to the generation layer 123, and obtains a second generated image based on the parameters of the generation layer 123 according to the fused feature vector, wherein the second generated image is an image generated based on the second training image using the panoramic illumination feature vector corresponding to the real panoramic image as the conditional vector of the generation model 120.

[0232] The electronic device constructs a first loss function according to the second generated image and the second target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function to obtain the trained generation model 120 and the lighting perception model 110. The trained generation model 120 and the lighting perception model 110 can be inferred.

[0233] In the embodiment of the present application, the second target image includes the above-mentioned real panoramic image, that is, the second target image is a real panoramic image corresponding to the second training image.

[0234] In the embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the first image encoder 111, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0235] Among them, the relevant contents of the image feature vector, the panoramic illumination feature vector, the fused feature vector, and the first loss function can be referred to above.

[0236] The above-mentioned fine-tuning uses real panoramic images. Real panoramic images usually contain more extensive background information and context, which helps the model understand the overall structure and layout of the scene. Real panoramic images are real images captured by cameras, etc., which improve the authenticity of the generated images.

[0237] See also Figure 6B , taking the image processing module 100 including the lighting perception model 110 and the generation model 120, the lighting perception model 110 including the second image encoder 112 and the vector extension model 113, and the generation model 120 including the feature extraction layer 121, the fusion layer 122, and the generation layer 123 as an example, the pre-training process of the image processing module 100 is exemplarily introduced. Figure 6B and Fig. 6A The differences include: the lighting perception model 110 includes a second image encoder 112 and a vector extension model 113, and the image input to the lighting perception model 110 is a real local image.

[0238] Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above and will not be repeated here.

[0239] The electronic device inputs the second training image of the second training set to the feature extraction layer 121 of the generation model 120 .

[0240] The electronic device extracts image features from the second training image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the second training image.

[0241] The electronic device inputs the real local image to the second image encoder 112, and extracts illumination features of the real local image based on the parameters of the second image encoder 112 to obtain a local illumination feature vector corresponding to the real local image.

[0242] The electronic device inputs the local illumination feature vector corresponding to the real local image into the vector extension model 113, and based on the parameters of the vector extension model 113, expands the local illumination feature vector into a panoramic illumination feature vector to obtain a panoramic illumination feature vector corresponding to the real local image.

[0243] The electronic device inputs the panoramic illumination feature vector and the image feature vector corresponding to the real local image into the fusion layer 122, and based on the parameters of the fusion layer 122, fuses the panoramic illumination feature vector and the image feature vector corresponding to the real local image to obtain a fused feature vector.

[0244] The electronic device inputs the fused feature vector to the generation layer 123, and obtains a second generated image based on the parameters of the generation layer 123 according to the fused feature vector, wherein the second generated image is an image generated based on the second training image using the panoramic illumination feature vector corresponding to the real local image as the conditional vector of the generation model 120.

[0245] The electronic device constructs a first loss function according to the second generated image and the second target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function to obtain the trained generation model 120 and the lighting perception model 110. The trained generation model 120 and the lighting perception model 110 can be inferred.

[0246] In the embodiment of the present application, the second target image includes the above-mentioned real partial image, that is, the second target image is a real partial image corresponding to the first training image.

[0247] In an embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the second image encoder 112, the parameters of the vector extension model 113, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0248] Among them, the relevant contents of the image feature vector, the panoramic illumination feature vector, the fused feature vector, and the first loss function can be referred to above.

[0249] The above-mentioned fine-tuning uses a real local image, which is a type of local image commonly used by the application image processing module 100. The real local image is a real image captured by a camera device, etc., which improves the authenticity of the generated image.

[0250] The following takes the application of the image processing module 100 to the synthetic image task as an example to exemplify the fine-tuning of the image processing module 100. The fine-tuning can be performed using a supervised learning method and / or an unsupervised learning method, which is not specifically limited in this application.

[0251] The electronic device obtains a first mask image corresponding to the real portrait image based on the real portrait image (such as the first portrait image described below), wherein the image area where the target object is located in the first mask image is marked as a selected state. The electronic device inputs the first mask image and the real illumination image into the synthesis model 130, and based on the synthesis model 130, synthesizes the target object in the first mask image into the real illumination image to obtain a second training image. The electronic device inputs the second training image into the trained generation model 120 and the real illumination image into the trained lighting perception model 110 for fine-tuning.

[0252] See also Fig. 7A , taking the example that the image processing module 100 includes the illumination perception model 110, the generation model 120 and the synthesis model 130, the illumination perception model 110 includes the first image encoder 111, and the generation model 120 includes the feature extraction layer 121, the fusion layer 122, and the generation layer 123, the fine-tuning process of the image processing module 100 is exemplarily introduced. Among them, the illumination perception model 110, the first image encoder 111, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123, the real illumination image, and the real panoramic image can be referred to above, and will not be repeated here.

[0253] The electronic device obtains a first mask image corresponding to the first portrait image according to the first portrait image, wherein an image area where the target object is located in the first mask image is marked as a selected state.

[0254] The electronic device inputs the first mask image and the real panoramic image into the synthesis model 130, and synthesizes the target object in the first mask image into the real panoramic image based on the synthesis model 130 to obtain a synthesized image. The second training image includes the synthesized image.

[0255] The electronic device inputs the synthesized image to the feature extraction layer 121 of the generation model 120 .

[0256] The electronic device extracts image features from the synthesized image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the synthesized image.

[0257] The electronic device inputs the real panoramic image into the first image encoder 111, and extracts illumination features of the real panoramic image based on the parameters of the first image encoder 111 to obtain a panoramic illumination feature vector corresponding to the real panoramic image. The panoramic illumination feature vector is used as a guiding condition for generating a model, that is, the panoramic illumination feature vector is input into the fusion layer 122 as a conditional vector of the fusion layer 122.

[0258] The electronic device inputs the image feature vector and the panoramic illumination feature vector into the fusion layer 122 , and fuses the image feature vector and the panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector.

[0259] The electronic device inputs the fused feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a second generated image according to the fused feature vector, wherein the second generated image is an image generated based on the synthetic image using the panoramic illumination feature vector as the conditional vector of the generation model 120.

[0260] The electronic device constructs a first loss function according to the second generated image and the second target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function to obtain the trained generation model 120 and the lighting perception model 110.

[0261] In the embodiment of the present application, the second target image includes the above-mentioned real panoramic image, that is, the second target image is a real panoramic image corresponding to the synthetic image.

[0262] In the embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the first image encoder 111, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0263] Among them, the relevant contents of the image feature vector, the panoramic illumination feature vector, the fused feature vector, and the first loss function can be referred to above.

[0264] The above-mentioned method of using real panoramic images to synthesize a synthetic image and using the synthetic image to perform fine-tuning provides the image processing module 100 with the ability to perform image synthesis tasks.

[0265] See also Figure 7B , taking the example that the image processing module 100 includes an illumination perception model 110, a generation model 120 and a synthesis model 130, the illumination perception model 110 includes a second image encoder 112 and a vector extension model 113, and the generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123, the pre-training process of the image processing module 100 is exemplarily introduced. Figure 7B and Fig. 7A The differences include: the lighting perception model 110 includes a second image encoder 112 and a vector extension model 113, and the image input to the lighting perception model 110 is a real local image.

[0266] The electronic device obtains a first mask image corresponding to the first portrait image according to the first portrait image, wherein an image area where the target object is located in the first mask image is marked as a selected state.

[0267] The electronic device inputs the first mask image and the real partial image into the synthesis model 130, and synthesizes the target object in the first mask image into the real partial image based on the synthesis model 130 to obtain a synthesized image. The second training image includes the synthesized image.

[0268] The electronic device inputs the synthesized image to the feature extraction layer 121 of the generation model 120 .

[0269] The electronic device extracts image features from the synthesized image based on the parameters of the feature extraction layer 121 to obtain an image feature vector corresponding to the synthesized image.

[0270] The electronic device inputs the real local image to the second image encoder 112, and extracts illumination features of the real local image based on the parameters of the second image encoder 112 to obtain a local illumination feature vector corresponding to the real local image.

[0271] The electronic device inputs the local illumination feature vector corresponding to the real local image into the vector extension model 113, and based on the parameters of the vector extension model 113, expands the local illumination feature vector into a panoramic illumination feature vector to obtain a panoramic illumination feature vector corresponding to the real local image.

[0272] The electronic device inputs the panoramic illumination feature vector and the image feature vector corresponding to the real local image into the fusion layer 122, and fuses the image feature vector and the panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused feature vector.

[0273] The electronic device inputs the fused feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a second generated image according to the fused feature vector, wherein the second generated image is an image generated based on the synthetic image using the panoramic illumination feature vector as the conditional vector of the generation model 120.

[0274] The electronic device constructs a first loss function according to the second generated image and the second target image, and adjusts the parameters of the generation model 120 and the lighting perception model 110 according to the first loss function to obtain the trained generation model 120 and the lighting perception model 110.

[0275] In the embodiment of the present application, the second target image includes the above-mentioned real partial image, that is, the second target image is the real partial image corresponding to the synthetic image.

[0276] In an embodiment of the present application, adjusting the parameters of the generation model 120 and the illumination perception model 110 includes: adjusting the parameters of the second image encoder 112, the parameters of the vector extension model 113, the parameters of the feature extraction layer 121, the parameters of the fusion layer 122, and the parameters of the generation layer 123.

[0277] Among them, the relevant contents of the image feature vector, the panoramic illumination feature vector, the fused feature vector, and the first loss function can be referred to above.

[0278] The above-mentioned synthesis of the synthetic image using the real local image and the use of the synthetic image for fine-tuning provide the image processing module 100 with the ability to perform image synthesis tasks.

[0279] It is understandable that the pre-trained generation model 120 and the lighting perception model 110 can be fine-tuned according to actual conditions to generate images that are more in line with specific conditions or tasks, and this application does not make specific limitations on this.

[0280] In an embodiment of the present application, high-quality pixel-aligned training pairs of photographed images can be used to fine-tune the lighting perception model and the generation model to fine-tune the generated photo realism.

[0281] It is understood that the stop training conditions in the above-mentioned pre-training and fine-tuning stages may include but are not limited to: a preset number of iterations, a preset time. Exemplarily, before starting training, a preset number of iterations or number of cycles may be set. The stop training conditions in the pre-training and fine-tuning stages may be the same, such as both being a preset number of iterations, or may be different. The embodiments of the present application do not specifically limit the stop training conditions in the above-mentioned pre-training and fine-tuning stages.

[0282] The image processing module 100 obtained by the training device can be applied to different systems or devices, such as terminals, such as mobile terminals, tablet computers, laptop computers, augmented reality (AR) / virtual reality (VR), vehicle terminals, televisions, etc., and can also be servers or cloud computing, etc.

[0283] The following introduces the reasoning stage of the image processing module 100 provided in the embodiment of the present application.

[0284] See also Fig. 8A, the reasoning of the image processing module 100 is applied to an electronic device, and the electronic device includes an illumination perception model 110 and a generation model 120. The generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123. Among them, the relevant contents of the illumination perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122, and the generation layer 123 can be referred to above, and will not be repeated here.

[0285] The electronic device receives a second image input by a user.

[0286] like Figure 1C to Figure 1E As shown, the electronic device determines that the image corresponding to the photo thumbnail 16 is the second image in response to the user's input operation on the photo thumbnail 16 and in response to the user's input operation on the image synthesis option 20.

[0287] like Figure 1C as well as Figure 2A As shown, the electronic device determines that the image corresponding to the photo thumbnail 16 is the second image in response to the user's input operation on the photo thumbnail 16 and in response to the user's input operation on the light and shadow enhancement option 21.

[0288] It is understandable that the electronic device can also obtain the second image input by the user in other ways, and the embodiments of the present application do not specifically limit this.

[0289] The electronic device inputs the first image into the lighting perception model 110 , and extracts lighting features of the first image based on the parameters of the lighting perception model 110 , to obtain a target panoramic lighting feature vector corresponding to the first image.

[0290] like Figure 1F As shown, the electronic device determines that the image corresponding to the image thumbnail 251 is the first image in response to the user's input operation on the image thumbnail 251, so as to transfer the light and shadow effects of the image corresponding to the image thumbnail 251 to the image displayed in the image display area 23.

[0291] For example, Figure 2B The electronic device can respond to the user's input operation on the image thumbnail 291 in the image selection area 29 and determine that the image corresponding to the image thumbnail 291 is the first image, so as to migrate the light and shadow effects of the image corresponding to the image thumbnail 291 to the image displayed in the light and shadow display area 28.

[0292] The electronic device inputs the second image to the feature extraction layer 121 of the generation model 120 , and extracts image features of the second image based on the parameters of the feature extraction layer 121 , to obtain a target image feature vector corresponding to the second image.

[0293] The electronic device inputs the target image feature vector and the target panoramic illumination feature vector into the fusion layer 122 , and fuses the target image feature vector and the target panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused target feature vector.

[0294] The electronic device inputs the fused target feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a target generated image according to the fused target feature vector, wherein the target generated image is an image generated based on the second image using the target panoramic illumination feature vector as a conditional vector of the generation model 120.

[0295] In an embodiment of the present application, the generation model 120 is based on the AIGC generative large model, especially the generation technology based on the diffusion model, to solve the problems of complex processes and insufficient knowledge in portrait re-lighting. The generation model 120 can realize the generation of re-lighted portraits based on background introduction by learning a large amount of CG data. Compared with the prior art, the image processing module 100 has the following advantages: the image processing module 100 can generate images with photo-realistic feeling, which benefits from the generation ability of the generation model and the introduction of lighting perception in the actual scene, that is, the introduction of the panoramic lighting feature vector obtained according to the image. The image processing module 100 has a strong generalization ability and can realize the synthesis of portrait re-lighting pictures in diverse and complex and changeable data scenes. The image processing module 100 does not require a large amount of manually annotated data for training, thereby reducing the cost and difficulty of obtaining training data.

[0296] See also Figure 8B , the reasoning of the image processing module 100 is applied to an electronic device, and the electronic device includes an illumination perception model 110, a generation model 120, and a synthesis model 130. The generation model 120 includes a feature extraction layer 121, a fusion layer 122, and a generation layer 123. Among them, the relevant contents of the illumination perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122, and the generation layer 123 can be referred to above, and will not be repeated here. Figure 8B and Fig. 8A The differences include: Figure 8B The image processing module 100 further includes a synthesis model 130 .

[0297] The electronic device obtains the target portrait image. In image synthesis, such as Figure 1E As shown, the electronic device determines that the image displayed in the image display area 23 is a target portrait image in response to the user's input operation on the image synthesis option 20.

[0298] The electronic device obtains a target mask image corresponding to the target portrait image according to the target portrait image, wherein the image area where the target object is located in the target mask image is marked as a selected state. Figure 1F As shown, the image region 24 where the human body object is located is the image region where the target object is located.

[0299] The electronic device inputs the target mask image and the first image into the synthesis model 130, and based on the synthesis model 130, synthesizes the target object in the target mask image into the first image to obtain the second image. Figure 1F As shown, the electronic device responds to the user's input operation on the image thumbnail 251, determines that the image corresponding to the image thumbnail 251 is the first image, and synthesizes the image area 24 where the human body object is located into the image corresponding to the image thumbnail 251, and obtains the following Figure 1G The composite image shown in the image composite display area 26. The composite image displayed in the image composite display area 26 is the second image.

[0300] The electronic device inputs the first image into the lighting perception model 110 , and extracts lighting features of the first image based on the parameters of the lighting perception model 110 , to obtain a target panoramic lighting feature vector corresponding to the first image.

[0301] The electronic device inputs the second image to the feature extraction layer 121 of the generation model 120 , and extracts image features of the second image based on the parameters of the feature extraction layer 121 , to obtain a target image feature vector corresponding to the second image.

[0302] The electronic device inputs the target image feature vector and the target panoramic illumination feature vector into the fusion layer 122 , and fuses the target image feature vector and the target panoramic illumination feature vector based on the parameters of the fusion layer 122 to obtain a fused target feature vector.

[0303] The electronic device inputs the fused target feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a target generated image according to the fused target feature vector, wherein the target generated image is an image generated based on the second image using the target panoramic illumination feature vector as a conditional vector of the generation model 120.

[0304] In an embodiment of the present application, the electronic device determines at least one target portrait image in response to the first operation. The electronic device can select multiple target portrait images at the same time. When the selected at least one target portrait image includes two or more objects, the electronic device determines at least one target object in the at least one target portrait image in response to the second operation. The electronic device determines the first image in response to the third operation.

[0305] Exemplarily, the electronic device determines the target portrait image A in response to the first operation. In the case where the determined target portrait image A includes an object, the electronic device can directly determine the object as the target object. In response to the third operation, the electronic device determines the first image to synthesize the target object into the first image.

[0306] In another exemplary embodiment, the electronic device determines the target portrait image A in response to the first operation. When the determined target portrait image A includes M objects, M is an integer greater than 1. The electronic device determines at least one target object from the M objects included in the target portrait image A in response to the second operation. The electronic device determines the first image in response to the third operation to synthesize the at least one target object into the first image.

[0307] In another exemplary embodiment, the electronic device determines the target portrait image A and the target portrait image B in response to the first operation. When the determined target portrait image A and the target portrait image B include M objects, M is an integer greater than 1. The electronic device determines at least one target object from the M objects in response to the second operation. The electronic device determines the first image in response to the third operation to synthesize the at least one target object into the first image.

[0308] In an embodiment of the present application, the image processing module 100 can enhance the portrait light and shadow based on the AIGC generative large model, and can seamlessly synthesize the subject background and foreground portrait into a new image while maintaining authenticity and aesthetic uniformity in terms of lighting and color. The generation module 100 generates a photo-level re-lit portrait image such as a diffusion model (stable Diffusion). First, the subject is segmented from the original image and synthesized into a new background. Then, the lighting information of the background image is encoded into the diffusion model through the lighting perception module, so as to make detailed adjustments to the foreground based on the diffusion model so that it is aligned with the new background in terms of color, brightness, saturation, and lighting conditions.

[0309] See also Fig.9A and Fig. 9B , the reasoning of the image processing module 100 is applied to an electronic device, and the electronic device includes an illumination perception model 110 and a generation model 120. The generation model 120 includes a feature extraction layer 121 (not shown in the figure), a fusion layer 122 (not shown in the figure), and a generation layer 123 (not shown in the figure). The illumination perception model 110 includes a second image encoder 112 and a vector extension model 113. Among them, the relevant contents of the illumination perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122, and the generation layer 123 can be referred to above, and will not be repeated here.

[0310] The electronic device obtains a target portrait image.

[0311] For example, in image synthesis, Figure 1E As shown, the electronic device determines that the image displayed in the image display area 23 is a target portrait image in response to the user's input operation on the image synthesis option 20.

[0312] The electronic device obtains a target mask image corresponding to the target portrait image according to the target portrait image, wherein the image area where the target object is located in the target mask image is marked as a selected state. Fig.9A and Fig. 9B As shown, the image area where the target object is located is white in the target mask image.

[0313] like Figure 1F As shown, the image region 24 where the human body object is located is the image region where the target object is located.

[0314] The first image includes a target partial image, such as a real partial image. Figure 1F As shown, the image corresponding to the image thumbnail 251 is a real partial image. The electronic device responds to the user's input operation on the image thumbnail 251 and determines that the image corresponding to the image thumbnail 251 is the first image. Figure 1G As shown, the image corresponding to the image thumbnail 251 is a real partial image. In response to the user's input operation on the image thumbnail 251, the electronic device determines that the image corresponding to the image thumbnail 251 is the first image.

[0315] The electronic device obtains a target mask image corresponding to the target portrait image based on the target portrait image, wherein the image area where the target object is located in the target mask image is marked as a selected state. The electronic device inputs the target mask image and the first image into the synthesis model 130, and based on the synthesis model 130, synthesizes the target object in the target mask image into the first image to obtain a second image. The electronic device inputs the second image into the feature extraction layer 121 of the generation model 120, and based on the parameters of the feature extraction layer 121, performs image feature extraction on the second image to obtain a target image feature vector corresponding to the second image. The electronic device inputs the target local image into the second image encoder 112, and based on the parameters of the second image encoder 112, performs illumination feature extraction on the target local image to obtain a local illumination feature vector corresponding to the target local image.

[0316] The electronic device inputs the local illumination feature vector corresponding to the target local image into the vector extension model 113, and based on the parameters of the vector extension model 113, expands the local illumination feature vector into a panoramic illumination feature vector, and obtains a panoramic illumination feature vector corresponding to the target local image, wherein the target panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the target local image or a panoramic illumination feature vector corresponding to the target panoramic image.

[0317] The electronic device inputs the image feature vector and the panoramic illumination feature vector corresponding to the target local image into the fusion layer 122, and based on the parameters of the fusion layer 122, fuses the panoramic illumination feature vector and the image feature vector corresponding to the local training image to obtain a fused feature vector.

[0318] The electronic device inputs the fused target feature vector to the generation layer 123, and based on the parameters of the generation layer 123, obtains a target generated image according to the fused target feature vector, wherein the target generated image is an image generated based on the second image using the target panoramic illumination feature vector as a conditional vector of the generation model 120.

[0319] like Fig.9A As shown, the target object and the background in the second image synthesized by the synthesis model 130 are not harmonious. Fig.9A The target generated image generated by the generation model 120 has high overall visual coordination and is more realistic. Fig. 9B As shown, the target object and the background in the second image synthesized by the synthesis model 130 are not harmonious. Fig. 9B The target generated image generated by the generation model 120 has high overall visual coordination and is more realistic.

[0320] In some embodiments, the lighting perception model 110 further includes a text generation model 114. The electronic device inputs the first image into the text generation model 114, and obtains a target text feature vector for describing the lighting characteristics of the first image based on the text generation model 114. The electronic device inputs the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector into the fusion layer 122, and fuses the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector based on the parameters of the fusion layer 122 to obtain a fused target feature vector.

[0321] See also Fig. 9C , the reasoning of the image processing module 100 is applied to an electronic device, and the electronic device includes a lighting perception model 110 and a generation model 120. The generation model 120 includes a feature extraction layer 121 (not shown in the figure), a fusion layer 122 (not shown in the figure) and a generation layer 123 (not shown in the figure). The lighting perception model 110 includes a first image encoder 111. Among them, the relevant contents of the lighting perception model 110, the generation model 120, the feature extraction layer 121, the fusion layer 122 and the generation layer 123 can be referred to above and will not be repeated here. The electronic device obtains a target mask image corresponding to the target portrait image based on the target portrait image, wherein the image area where the target object is located in the target mask image is marked as selected. As Fig. 9CAs shown, the image area where the target object is located is white in the target mask image. The electronic device inputs the target mask image and the first image into the synthesis model 130, and synthesizes the target object in the target mask image into the first image based on the synthesis model 130 to obtain the second image. The first image includes the target panoramic image, such as a real panoramic image.

[0322] like Fig. 1I As shown, the third background image and the third background image displayed in the background image display area 25 are real panoramic images. The electronic device can determine that the image corresponding to the image thumbnail 253 is the first image in response to the user's input operation on the image thumbnail 253.

[0323] The electronic device inputs the target panoramic image into the first image encoder 111, and based on the parameters of the first image encoder 111, extracts the illumination features of the target panoramic image to obtain a panoramic illumination feature vector corresponding to the target panoramic image. The electronic device inputs the image feature vector and the panoramic illumination feature vector corresponding to the target panoramic image into the fusion layer 122, and based on the parameters of the fusion layer 122, fuses the panoramic illumination feature vector corresponding to the local training image and the image feature vector to obtain a fused feature vector. The electronic device inputs the fused target feature vector into the generation layer 123, and based on the parameters of the generation layer 123, obtains a target generated image according to the fused target feature vector, wherein the target generated image is an image generated based on the second image using the target panoramic illumination feature vector as the conditional vector of the generation model 120.

[0324] like Fig. 9C As shown, the target object and the background in the second image synthesized by the synthesis model 130 are not harmonious. Fig.9A The target generated image generated by the generation model 120 has high overall visual coordination and is more realistic.

[0325] In some embodiments, the lighting perception model 110 further includes a text generation model 114. The electronic device inputs the first image into the text generation model 114, and based on the text generation model 114, obtains a target text feature vector for describing the lighting characteristics of the first image, and the electronic device inputs the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector into the fusion layer 122, and based on the parameters of the fusion layer 122, fuses the target image feature vector, the target panoramic lighting feature vector, and the target text feature vector to obtain a fused target feature vector.

[0326] For electronic devices such as mobile phones, please refer to Fig.10 , exemplarily introduces the structure of the electronic device provided in the embodiments of the present application.

[0327] It should be understood that Fig.10The electronic device 200 shown is only one example of an electronic device, and the electronic device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Fig.10 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0328] See also Fig.10 , the electronic device 200 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Among them, the electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor, etc. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information. The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1. The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 294, and the touch sensor 180K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 200, which is different from the position of the display screen 294.

[0329] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 200.

[0330] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here. The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned method embodiments. The embodiment of the present application also provides a computer storage medium, including computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the image processing method as in the above embodiment. Among them, the electronic device, computer storage medium, computer program product or chip system provided in the embodiment of the present application are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0331] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0332] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the module or unit is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be a physical unit or multiple physical units, that is, it can be located in one place, or it can be distributed in multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0333] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0334] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A training model method, characterized in that: Applied to an electronic device, the electronic device includes a lighting perception model and a generation model, the generation model includes a feature extraction layer, a fusion layer and a generation layer, and the method includes: Inputting the illumination training image into the illumination perception model, and extracting illumination features from the illumination training image based on parameters of the illumination perception model to obtain a first panoramic illumination feature vector corresponding to the illumination training image; Inputting a first training image of a first training set into the feature extraction layer, and performing image feature extraction on the first training image based on parameters of the feature extraction layer to obtain an image feature vector corresponding to the first training image; Inputting the image feature vector and the first panoramic illumination feature vector into the fusion layer, and fusing the image feature vector and the first panoramic illumination feature vector based on the parameters of the fusion layer to obtain a fused feature vector; Inputting the fused feature vector into the generation layer, and obtaining a first generated image according to the fused feature vector based on the parameters of the generation layer, wherein the first generated image is an image generated based on the first training image with the first panoramic illumination feature vector as the condition vector of the generation model; According to the first generated image, the parameters of the generation model and the lighting perception model are adjusted to obtain the trained generation model and lighting perception model.

2. The training model method according to claim 1, characterized in that: The illumination training image includes a panoramic training image, the illumination perception model includes a first image encoder, and the step of inputting the illumination training image into the illumination perception model, extracting illumination features from the illumination training image based on parameters of the illumination perception model, and obtaining a first panoramic illumination feature vector corresponding to the illumination training image includes: The panoramic training image is input into the first image encoder, and based on the parameters of the first image encoder, illumination features are extracted for the panoramic training image to obtain a panoramic illumination feature vector corresponding to the panoramic training image, wherein the first panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the panoramic training image.

3. The training model method according to claim 2, characterized in that: The illumination training image includes a local training image, the illumination perception model includes a second image encoder and a vector extension model, and the step of inputting the illumination training image into the illumination perception model, extracting illumination features from the illumination training image based on parameters of the illumination perception model, and obtaining a first panoramic illumination feature vector corresponding to the illumination training image includes: Inputting the local training image into the second image encoder, and extracting illumination features of the local training image based on parameters of the second image encoder to obtain a local illumination feature vector corresponding to the local training image; The local illumination feature vector corresponding to the local training image is input into the vector extension model, and based on the parameters of the vector extension model, the local illumination feature vector is expanded into a panoramic illumination feature vector to obtain a panoramic illumination feature vector corresponding to the local training image, wherein the first panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the local training image or a panoramic illumination feature vector corresponding to the panoramic training image.

4. The training model method according to claim 3, characterized in that: The first training set also includes a first target image corresponding to the first training image, and the parameters of the generation model and the lighting perception model are adjusted according to the first generation image to obtain the trained generation model and lighting perception model, including: Constructing a first loss function according to the first target image and the first generated image; constructing a second loss function according to the panoramic illumination feature vector corresponding to the local training image and the panoramic illumination feature vector corresponding to the panoramic training image; According to the first loss function and the second loss function, the parameters of the generation model and the lighting perception model are adjusted to obtain the trained generation model and the lighting perception model.

5. The training model method according to any one of claims 1 to 4, characterized in that: The electronic device further includes a synthesis model, and the method further includes: Obtaining a first mask image corresponding to the real portrait image according to the real portrait image, wherein an image area where a target object is located in the first mask image is marked as a selected state; Inputting the first mask image and the real light training image into the synthesis model, and based on the synthesis model, synthesizing the target object in the first mask image into the real light training image to obtain a second training image; The second training image is input into the trained generative model and the real lighting training image is input into the trained lighting perception model for fine-tuning.

6. The training model method according to any one of claims 1 to 5, characterized in that: The lighting perception model also includes a text generation model, and the method further includes: Inputting the illumination training image into the text generation model, and obtaining a first text feature vector for describing illumination features of the illumination training image based on the text generation model; The step of inputting the image feature vector and the first panoramic illumination feature vector into the fusion layer, and fusing the image feature vector and the first panoramic illumination feature vector based on the parameters of the fusion layer to obtain a fused feature vector comprises: The image feature vector, the first panoramic illumination feature vector and the first text feature vector are input into the fusion layer, and based on the parameters of the fusion layer, the image feature vector, the first panoramic illumination feature vector and the text feature vector are fused to obtain a fused feature vector.

7. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a lighting perception model and a generation model, the generation model includes a feature extraction layer, a fusion layer and a generation layer, and the method includes: Inputting the first image into the lighting perception model, and extracting lighting features of the first image based on parameters of the lighting perception model to obtain a target panoramic lighting feature vector corresponding to the first image; Inputting the second image into the feature extraction layer, performing image feature extraction on the second image based on the parameters of the feature extraction layer, and obtaining a target image feature vector corresponding to the second image; Inputting the target image feature vector and the target panoramic illumination feature vector into the fusion layer, and fusing the target image feature vector and the target panoramic illumination feature vector based on the parameters of the fusion layer to obtain a fused target feature vector; The fused target feature vector is input into the generation layer, and based on the parameters of the generation layer, a target generated image is obtained according to the fused target feature vector, wherein the target generated image is an image generated based on the second image with the target panoramic illumination feature vector as the conditional vector of the generation model.

8. The image processing method according to claim 7, characterized in that: The first image includes a target panoramic image, the lighting perception model includes a first image encoder, and the first image is input into the lighting perception model, and based on the parameters of the lighting perception model, the lighting feature extraction is performed on the first image to obtain the target panoramic lighting feature vector corresponding to the first image, which includes: The target panoramic image is input into the first image encoder, and based on the parameters of the first image encoder, illumination features of the target panoramic image are extracted to obtain a panoramic illumination feature vector corresponding to the target panoramic image, wherein the target panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the target panoramic image.

9. The image processing method according to claim 7 or 8, characterized in that: The first image includes a local image of the target, the lighting perception model includes a second image encoder and a vector extension model, and the first image is input into the lighting perception model, and based on the parameters of the lighting perception model, the lighting feature extraction is performed on the first image to obtain the target panoramic lighting feature vector corresponding to the first image, which includes: Inputting the target local image into the second image encoder, and extracting illumination features of the target local image based on parameters of the second image encoder to obtain a local illumination feature vector corresponding to the target local image; The local illumination feature vector corresponding to the target local image is input into the vector extension model, and based on the parameters of the vector extension model, the local illumination feature vector is expanded into a panoramic illumination feature vector to obtain a panoramic illumination feature vector corresponding to the target local image, wherein the target panoramic illumination feature vector includes a panoramic illumination feature vector corresponding to the target local image or a panoramic illumination feature vector corresponding to the target panoramic image.

10. The image processing method according to any one of claims 7 to 9, characterized in that: The lighting perception model also includes a text generation model, and the method further includes: Inputting the first image into the text generation model, and obtaining a target text feature vector for describing the illumination features of the first image based on the text generation model; The step of inputting the target image feature vector and the target panoramic illumination feature vector into the fusion layer, and fusing the target image feature vector and the target panoramic illumination feature vector based on the parameters of the fusion layer to obtain the fused target feature vector comprises: The target image feature vector, the target panoramic illumination feature vector and the target text feature vector are input into the fusion layer, and based on the parameters of the fusion layer, the target image feature vector, the target panoramic illumination feature vector and the target text feature vector are fused to obtain a fused target feature vector.

11. The image processing method according to any one of claims 7 to 9, characterized in that: The electronic device further includes a synthesis model, and before inputting the second image into the feature extraction layer, the method further includes: Obtaining a target mask image corresponding to the target portrait image according to the target portrait image, wherein an image region where the target object is located in the target mask image is marked as a selected state; The target mask image and the first image are input into the synthesis model, and based on the synthesis model, the target object in the target mask image is synthesized into the first image to obtain the second image.

12. The image processing method according to claim 11, characterized in that: The method further comprises: In response to the first operation, determining at least one target portrait image; In a case where the at least one target portrait image includes two or more objects, in response to a second operation, determining at least one target object in the at least one target portrait image; In response to the third operation, the first image is determined.

13. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the training model method as described in any one of claims 1 to 6 or the image processing method as described in any one of claims 7 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed, the training model method according to any one of claims 1 to 6 or the image processing method according to any one of claims 7 to 12 is executed.

Citation Information

Patent Citations

  • Illumination remapping method and device for image synthesis, storage medium and processor

    CN110288512A

  • Picture re-illumination method based on inverse rendering and illumination controllable image generation

    CN117496041A

Cited By

  • Training method of image generation model, image generation method and related equipment

    CN119810587A

  • A training method of an image generation model, an image generation method, and related devices

    CN119810587B

  • Fusion segmentation AI product graph enhancement method

    CN121280297A

  • An AI product image enhancement method fusing segmentation

    CN121280297B

  • AI coordination method for image light and shadow harmony

    CN121329841A