Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image processing method and device, electronic equipment and storage medium, the method comprising: obtaining an original image, the original image including a target subject; based on the original image, obtaining a segmented subject image and a to-be-processed image, the to-be-processed image including a to-be-repaired region corresponding to the target subject; based on the subject image, determining a spherical harmonic illumination corresponding to the target subject; based on the to-be-processed image, determining a corresponding to-be-fused panorama; based on the spherical harmonic illumination and the to-be-fused panorama, determining a target panorama through an image processing model. Through the technical solutions of the embodiments of the present disclosure, in the case that the environment part in the original image is blocked, the effect of environment illumination estimation can be improved, thereby providing users with a more realistic augmented reality experience.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of AR (Augmented Reality), in order to make virtual objects more in line with the real environment, lighting information and / or environmental information in the corresponding scene can be used to render virtual objects so that they conform to the characteristics of brightness, shadows and other features of their location.

[0003] However, most current ambient lighting estimation methods are designed for a single location within the captured scene, without considering application scenarios where a large proportion of non-environmental features (such as human figures) exist in the image. In such scenarios, the environment is severely occluded, making it difficult to guarantee the final ambient lighting estimation effect, thus affecting the implementation of subsequent augmented reality technologies. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, electronic device and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, the method comprising:

[0006] Acquire the original image, which includes the target subject;

[0007] Based on the original image, a segmented subject image and a to-be-processed image are obtained, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject;

[0008] Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject;

[0009] Based on the image to be processed, determine the corresponding panoramic image to be fused;

[0010] Based on the spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined by an image processing model.

[0011] Secondly, embodiments of this disclosure also provide an image processing apparatus, the apparatus comprising:

[0012] The original image acquisition module is used to acquire an original image, wherein the original image includes the target subject;

[0013] The original image segmentation module is used to obtain a segmented subject image and a to-be-processed image based on the original image, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject;

[0014] A spherical harmonic illumination determination module is used to determine the spherical harmonic illumination corresponding to the target subject based on the subject image;

[0015] The panorama determination module is used to determine the corresponding panorama to be merged based on the image to be processed.

[0016] The target panoramic image determination module is used to determine the target panoramic image based on the spherical harmonic illumination and the panoramic image to be fused, using an image processing model.

[0017] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0018] One or more processors;

[0019] Storage device for storing one or more programs.

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of this disclosure.

[0021] Fourthly, a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image processing method as described in any of the embodiments of this disclosure.

[0022] The technical solution of this disclosure involves acquiring an original image, obtaining a segmented subject image and an image to be processed based on the original image, determining spherical harmonic illumination corresponding to the target subject based on the subject image, determining a corresponding panoramic image to be fused based on the image to be processed, and finally determining the target panoramic image through an image processing model based on the spherical harmonic illumination and the panoramic image to be fused. By separating the subject image from the original image and performing illumination estimation and environmental information restoration based on the segmented subject and environment parts, the problem of the large proportion of the subject in the original image and the severe occlusion of the environment part affecting the environmental illumination estimation effect is solved. At the same time, the final target panoramic image combines environmental and illumination information, which can provide a more realistic rendering effect for the realization of augmented reality technology. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0024] Figure 1This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0025] Figure 2 This is a schematic flowchart of another image processing method provided in an embodiment of the present disclosure;

[0026] Figure 3 This is a flowchart illustrating a method for determining a panoramic image to be processed, provided in an embodiment of this disclosure.

[0027] Figure 4 This is a flowchart illustrating a method for determining a panoramic image to be fused, provided in an embodiment of this disclosure.

[0028] Figure 5 This is a schematic flowchart of another image processing method provided in an embodiment of the present disclosure;

[0029] Figure 6 This is a flowchart illustrating a target illumination estimation model training method provided in an embodiment of the present disclosure.

[0030] Figure 7 This is a schematic flowchart of another image processing method provided in an embodiment of the present disclosure;

[0031] Figure 8 This is a flowchart illustrating an image processing model training method provided in an embodiment of the present disclosure.

[0032] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0033] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0034] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0038] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0039] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0040] Before introducing the technical solutions of each disclosed embodiment, let's briefly introduce the application scenarios. When the original image contains a large proportion of non-environmental features, such as when the original image is a user's selfie, the large proportion of the human figure leads to inaccurate estimation of ambient lighting, resulting in insufficient realism in subsequent AR technology implementation. The technical solution of this disclosed embodiment can separate the main body (e.g., the human figure) and the environment (e.g., non-human figures), and perform lighting estimation and environmental information restoration on the segmented main body and environment, ultimately obtaining an environment map representing ambient lighting information in the form of a panoramic image, which is convenient for subsequent use. An image processing program can be generated according to the technical solution of the disclosed embodiment, so that when the image processing program is applied, ambient lighting estimation can be performed on ordinary images containing the main body. The technical solution of this disclosed embodiment can also be integrated into existing image processing software. When the panoramic image processing function is triggered, an ordinary image containing the human figure can be acquired and ambient lighting estimation can be performed to obtain a panoramic image containing lighting and environmental information. Furthermore, the panoramic image can be applied to augmented reality technology. For example, it can be used in virtual operations targeting human figures to cleverly integrate virtual information with the real world.

[0041] Figure 1This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. This embodiment is applicable to situations where ambient lighting is estimated from an original image containing a target subject to obtain a panoramic image containing lighting and environmental information. The method can be executed by an image processing device, which can be implemented in software and / or hardware, or optionally by an electronic device, such as a mobile terminal, a PC, or a server.

[0042] like Figure 1 As shown, the method specifically includes the following steps:

[0043] S110, Obtain the original image.

[0044] The original image can be a directly acquired image, such as one obtained through shooting, uploading, or downloading. The original image includes the target subject, which can be considered a human figure.

[0045] In this embodiment, the original image can be obtained in any of the following ways:

[0046] In some examples, the current video frame containing the target subject is used as the original image. The current video frame can be any image frame from a video file.

[0047] In some examples, images capturing the target subject, taken with a camera device, are used as the original images. The camera device can be any device with a shooting function, such as a digital camera, camcorder, mobile phone with a camera function, or surveillance camera.

[0048] In some examples, the uploaded image including the target subject is used as the original image.

[0049] Specifically, the original image can be obtained through local upload. Users can select an image containing the target subject from the map library by triggering the image upload control, and the uploaded image will be used as the original image.

[0050] In some examples, images containing the target subject retrieved from a database are used as the original images. The database can be an image library.

[0051] Specifically, the original image can be obtained by selecting from a database. Users can trigger an online image selection control to choose an image from the database that includes the target subject. Then, the selected image is retrieved and used as the original image.

[0052] S120. Based on the original image, obtain the segmented main image and the image to be processed. The image to be processed includes the region to be repaired corresponding to the target main image.

[0053] The region to be repaired corresponds to the target subject; that is, the region in the original image where the target subject is located. The subject image can be the part of the original image that corresponds to the target subject. The image to be processed can be the part remaining after removing the subject image from the original image; the region formed by the removed part in the image to be processed is the region to be repaired. For example, at least part of the subject image can be removed from the original image, and the remaining part can be used as the image to be processed; alternatively, the entire subject image can be removed from the original image, and the remaining part can be used as the image to be processed.

[0054] For example, the region formed by the pixels of the image to be processed is an irregular ring-shaped region, and the region formed by the outer edge of the image to be processed includes the region to be repaired; or, after removing the main image from the original image, the pixels in the region where the main image is located are uniformly set to a preset value, and the resulting image is the image to be processed. In this case, the image to be processed can be considered to include the region to be repaired.

[0055] Specifically, the acquired original image is segmented, the part containing the target subject is determined as the subject image, and the subject image is removed from the original image. The removed area is used as the area to be repaired, and the remaining part of the original image after removing the subject image is determined as the image to be processed.

[0056] In this embodiment of the disclosure, the subject image and the image to be processed can be obtained by segmenting the original image in any of the following ways:

[0057] In some examples, based on a subject segmentation model, the target subject in the original image is determined, resulting in a subject image and an image to be processed, including the region to be repaired.

[0058] The subject segmentation model can be a trained network model used to identify the target subject in the original image, such as a neural network model.

[0059] Specifically, the original image is input into a pre-trained subject segmentation model to identify and segment the target subject. The output of the model can be used to determine the subject image and the image to be processed containing the region to be repaired.

[0060] In some examples, based on a subject segmentation algorithm, the target subject in the original image is segmented to obtain the subject image and the image to be processed, including the region to be repaired.

[0061] Among them, the subject segmentation algorithm can be an image segmentation algorithm used to distinguish the target subject, such as a threshold segmentation algorithm, a region segmentation algorithm, or a histogram segmentation algorithm.

[0062] Specifically, the original image is processed using a subject segmentation algorithm to identify the target subject in the original image, and the image portion corresponding to the target subject is segmented out as the subject image. The image portion corresponding to the target subject in the original image is then removed to obtain the image to be processed, which includes the area to be repaired.

[0063] S130. Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject.

[0064] Spherical harmonic lighting can include information about the ambient light surrounding the target subject. For example, the ambient light around the target subject can be sampled, the sampled information can be divided into a preset number of coefficients, and then the lighting can be represented using the preset number of coefficients through image rendering technology. The acquired coefficient representation information can be used as the spherical harmonic lighting of the target subject. Spherical harmonic lighting can include information such as the geometric information, color, energy, and speed of light.

[0065] Specifically, spherical harmonic illumination analysis is performed on the subject image to obtain the spherical harmonic illumination corresponding to the target subject.

[0066] S140. Based on the image to be processed, determine the corresponding panoramic image to be fused.

[0067] Specifically, the image to be processed is processed, such as image preprocessing, image restoration, etc., and then the processed image can be used as a panoramic image to be fused.

[0068] In this embodiment of the disclosure, the area to be repaired in the image to be processed can be repaired to obtain the image to be used; and a panoramic image to be fused corresponding to the image to be used can be determined based on the reference point located in the area to be repaired.

[0069] The reference point can be a pixel located in the area to be repaired, which is required for subsequent panoramic processing. For example, the intersection of the horizontal centerline and the vertical centerline of the area to be repaired can be used as the reference point. This disclosure does not limit this. The panoramic image to be fused can be an image obtained by performing panoramic processing on the image to be used at the reference point.

[0070] Specifically, since the image to be processed contains areas to be repaired, these areas need to be repaired, i.e., the repaired areas need to be completed. This involves determining and filling in the pixel values ​​of each pixel in the repaired area to restore the occluded environmental information, thereby eliminating the potential impact of the target subject on the estimation of ambient lighting. Based on the shape, position, and contour information of the areas to be repaired, reference points for these areas can be determined through mathematical calculations. Using these reference points located in the repaired areas as a reference, the image to be used is subjected to panoramic processing, i.e., converted into a panoramic image. This converted panoramic image is then used as the corresponding panoramic image to be fused. Because the reference point for panoramic processing of the image to be used is located in the repaired areas—that is, in the area where the target subject is located in the original image—the resulting panoramic image can better map the environmental information surrounding the target subject. Subsequent effects processing for the target subject in the original image can be achieved by rendering the effects using the panoramic image obtained in the above manner, resulting in a more realistic effect.

[0071] In this embodiment of the disclosure, the area to be repaired in the image to be processed can be repaired to obtain an image to be used by: determining the pixel information to be applied to the edge contour of the area to be repaired; and determining the image to be used by interpolating the pixels to be applied.

[0072] The pixel information to be applied may include the pixel values ​​of each pixel in the edge contour of the region to be repaired, and may also include the position information of each pixel in the edge contour of the region to be repaired.

[0073] Specifically, the edge contour of the area to be repaired can be determined, and then the pixel value and position information of the pixels to be applied to that edge contour can be determined. Based on the pixels to be applied, the pixel value of each pixel in the area to be repaired can be determined through interpolation, thus repairing the area and using the repaired image as the final image. For example, for each pixel in the area to be repaired, its pixel value can be determined by interpolation based on the pixel values ​​of its neighboring pixels. Interpolation methods can include nearest neighbor interpolation, bilinear interpolation, higher-order interpolation, etc.

[0074] S150. Based on spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined through an image processing model.

[0075] The image processing model can be used to fuse spherical harmonic lighting and the panoramic image to be fused. The target panoramic image is an HDR (High Dynamic Range) panoramic image, which can also be understood as an HDR environment map, containing complete lighting and environmental information, and can be used to render virtual objects to enhance their realism.

[0076] Specifically, the spherical harmonic illumination and the panoramic image to be fused are input into a pre-trained image processing model to overlay the illumination information onto the panoramic image to be fused, thereby obtaining an output image. The output image is then used as the target panoramic image corresponding to the image to be processed.

[0077] In some examples, after obtaining the target panoramic image, special effects processing can be applied to the original image. This processing includes adding effects to the original image and rendering those effects based on the target panoramic image. Furthermore, the effects can be augmented reality effects, specifically:

[0078] In response to the detection of a trigger operation to add effects to the original image, the effect is rendered based on the target panorama to obtain an effect image corresponding to the original image.

[0079] The special effects can be user-selected effects or pre-set effects to be added. The effect image can be an image obtained by rendering the effect onto the original image.

[0080] In some examples, the triggering of special effects can be based on user selection. For instance, special effects elements to be added can be pre-set on the relevant application interface, and the user can choose whether to add them.

[0081] Specifically, for the original image, effects can be triggered either by user selection or by pre-setting the effects to be added on the relevant application interface. In response to the aforementioned effect triggering action or command, the effects are rendered based on the target panoramic image and added to the original image to obtain an image with added effects.

[0082] In some examples, the special effects processing includes: special effects switching operations, that is, switching the special effects element in the special effects image from the first special effects element to the second special effects element. Specifically, the second special effects element is rendered using the target panoramic image, and the rendered second special effects element replaces the first special effects element to obtain the corresponding second special effects image.

[0083] The target subject can be a human figure, and the special effects elements can be virtual clothing. Correspondingly, the special effects switching process can be a virtual dress-up operation for the target subject.

[0084] The virtual clothing change operation is performed on the target subject. Specifically, in response to the detection of a clothing change trigger operation corresponding to the target subject, the virtual clothing to be added is rendered based on the target panoramic image to obtain a special effects image corresponding to the original image.

[0085] The "clothing change" operation can be a special effect trigger operation that virtually changes the clothing of the target subject. The special effect image can be the image of the target subject in the original image after the clothing change operation.

[0086] Specifically, users can select a target subject in the original image and choose to perform a clothing change operation for that subject. When a clothing change operation corresponding to the target subject is triggered, the virtual clothing to be added is rendered based on the target panoramic image, and the rendered virtual clothing is added to the corresponding position of the target subject in the original image to realize the clothing change operation. The image after the clothing change operation is then identified as the special effects image corresponding to the original image.

[0087] The technical solution of this disclosure involves acquiring an original image, obtaining a segmented subject image and an image to be processed based on the original image, determining spherical harmonic illumination corresponding to the target subject based on the subject image, determining a corresponding panoramic image to be fused based on the image to be processed, and finally determining the target panoramic image through an image processing model based on the spherical harmonic illumination and the panoramic image to be fused. By separating the subject image from the original image and performing illumination estimation and environmental information restoration based on the segmented subject and environment parts, the problem of the large proportion of the subject in the image and the severe occlusion of the environment part affecting the environmental illumination estimation effect is solved. At the same time, the final target panoramic image combines environmental and illumination information, which can provide a more realistic rendering effect for the realization of augmented reality technology.

[0088] Figure 2 This is a schematic flowchart illustrating another image processing method provided in this embodiment. In some examples, the panoramic image to be fused corresponding to the image to be used can be determined by depth estimation. For specific implementation details, please refer to the detailed description of this technical solution. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0089] like Figure 2 As shown, the method includes:

[0090] S210. Obtain the original image. Based on the original image, obtain the segmented main image and the image to be processed.

[0091] S220. Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject.

[0092] S230. Repair the areas to be repaired in the image to be processed to obtain the image to be used.

[0093] S240. Based on the image to be used, obtain a reference depth map including the depth information of each pixel, and determine the reference depth value of the reference point.

[0094] The depth information can be the distance from each pixel to the camera. The reference depth map can be a map composed of the depth information of each pixel. The baseline depth value can be the depth information of the reference point in the area to be repaired.

[0095] Specifically, the image to be used is processed, for example, through depth estimation to obtain the distance information from each pixel in the image to the camera. Then, the distance information from each pixel in the image to the camera is used as depth information. A reference depth map is constructed based on the depth information of each pixel, and the depth information of the pixel corresponding to the reference point of the area to be repaired is determined as the reference depth value within the reference depth map.

[0096] S250: Based on the baseline depth value, reference depth map, and the image to be used, determine the panoramic image to be processed.

[0097] The panoramic image to be processed can be a panoramic image of the image to be used, based on panoramic processing.

[0098] Specifically, using a reference depth value as the baseline, pixels with depth information less than the reference depth value will be invisible. Therefore, pixels in the reference depth map with depth information less than the reference depth value can be removed, while pixels with depth information greater than or equal to the reference depth value are retained. Based on the pixels retained in the reference depth map, the corresponding pixels in the image to be used are determined, and panoramic processing is performed based on these pixels to obtain the panoramic image to be processed.

[0099] In some examples, it can be based on Figure 3 The steps shown are used to determine the panorama to be processed based on the baseline depth value, the reference depth map, and the image to be used:

[0100] S2501. Based on the baseline depth value and the reference depth map, determine the depth map to be processed for the panorama.

[0101] The depth map to be processed in the panorama can be the part of the reference depth map whose depth information is greater than or equal to the reference depth value.

[0102] Specifically, the depth information of each pixel in the reference depth map is compared with the baseline depth value. The image composed of pixels whose depth information is greater than or equal to the baseline depth value is used as the depth map to be processed in the panorama.

[0103] S2502. Based on the depth map to be processed and the image to be used, determine the map to be processed.

[0104] The image to be processed in the panorama can be a portion of the image to be used that corresponds to the depth map to be processed in the panorama.

[0105] Specifically, based on each pixel in the depth map to be processed, the corresponding pixels in the image to be used are determined, and these pixels in the image to be used are combined to form the panoramic image to be processed.

[0106] S2503. By performing distortion processing on the panoramic image to be processed, a distortion diagram is obtained, and the panoramic image to be processed is obtained based on the distortion diagram.

[0107] Distortion processing can include left-right distortion, which can be understood as distortion processing performed on the horizontal direction of the original image. For example, the process of obtaining the image from the target viewpoint through projection transformation based on the viewpoint and intrinsic parameters is distortion processing. The distortion diagram can be the output image after distortion processing of the panoramic image to be processed, that is, the image after left-right distortion of the panoramic image to be processed. The pixel values ​​of the pixels outside the distortion diagram in the panoramic image to be processed are set values. The set values ​​can be pre-set pixel values, such as the pixel value corresponding to black.

[0108] Specifically, the panoramic image to be processed can be distorted horizontally. Based on the preset size information of the panoramic image to be processed, a set of target pixels corresponding to the distortion diagram in the panoramic image to be processed is determined. The pixel values ​​of all pixels in the target pixel set are then updated one by one to the corresponding pixel values ​​in the distortion diagram. Pixels in the panoramic image to be processed other than the target pixel set can be considered irrelevant pixels, and their pixel values ​​are adjusted to set values, such as 0 (white) or 255 (black), to distinguish these irrelevant pixels from the pixels in the distortion diagram. Then, the adjusted panoramic image to be processed is determined, so that the distortion diagram is considered part of the panoramic image to be processed.

[0109] S260. By completing the panoramic image to be processed, a panoramic image to be fused corresponding to the image to be used is obtained.

[0110] Specifically, since the panoramic image to be processed contains pixels with set values, these pixels need to be padded to ensure the integrity and authenticity of the information in the panoramic image. Therefore, the panoramic image to be processed can be padded to fill in the pixels with set values, and then the padded image can be determined as the panoramic image to be fused corresponding to the image to be used.

[0111] In some examples, such as Figure 4 As shown, a pre-trained panoramic frame graph generation model can be used to complete the panoramic image to be processed, resulting in the panoramic image to be fused.

[0112] S310. Input the panoramic image to be processed into the panoramic frame image generation model to obtain the panoramic frame image.

[0113] The panoramic frame diagram can be the image output by the panoramic frame diagram generation model that corresponds to the panoramic image to be processed. The panoramic frame diagram only shows the general structure of the panoramic image and does not contain detailed information.

[0114] Specifically, the panoramic image to be processed is input into a pre-trained panoramic frame image generation model to obtain an output image corresponding to the panoramic image to be processed, that is, a panoramic frame image corresponding to the panoramic image to be processed.

[0115] S320. Perform gridding on the panoramic frame image to obtain at least one panoramic grid image.

[0116] Among them, the panoramic grid image can be an image in any grid after the panoramic frame image has been gridded.

[0117] Specifically, according to image processing requirements, the panoramic frame image can be divided into at least one panoramic grid image. For example, it can be divided into a number of panoramic grid images such as 3×3 or 16×9. The specific division form and number are not specifically limited in this embodiment.

[0118] For example, the panoramic frame image A can be divided into panoramic grid images A1, A2, A3, A4, A5, A6, A7, A8 and A9 according to the 3×3 division rule.

[0119] S330. Based on at least one panoramic mesh image, obtain the corresponding refined mesh image through a local image generation model.

[0120] S340. Based on the panoramic frame map and at least one refined grid map, a panoramic image to be fused is obtained through a panoramic image generation model, which corresponds to the image to be used.

[0121] Specifically, there are n panoramic mesh images, where 0 < i ≤ n, and i and n are both positive integers. The panoramic frame image can be used as the current panoramic frame image, and the following operations can be performed:

[0122] The i-th panoramic mesh image is used as input to the local image generation model to obtain a locally processed refined mesh image. The refined mesh image and the current panoramic frame image are used as input to the panoramic image generation model, and the output is the refined panoramic image corresponding to the i-th refined mesh image. The i-th refined panoramic image output by the panoramic image generation model is used as the new current panoramic frame image.

[0123] Traverse n panoramic mesh images and use the refined panoramic image output by the panoramic image generation model in the nth operation as the panoramic image to be fused.

[0124] The local image generation model can be a network model used to locally process the current mesh image, or a pre-trained neural network model, such as a patch-based UET network. The refined mesh image can be the output image of the local image generation model, which is the image after adding local information to the input panoramic mesh image. The panoramic image generation model can be a network model used to fuse the refined mesh image and the current panoramic frame image, or a pre-trained neural network model, such as a merging network. The refined panoramic image corresponds to the input refined mesh image.

[0125] The above example will now be further illustrated.

[0126] In this example, the panoramic frame image A is divided into panoramic mesh images A1, A2, A3, A4, A5, A6, A7, A8, and A9. Panoramic mesh image A1 is used as the i-th panoramic mesh image, and panoramic frame image A is used as the current panoramic frame image. The i-th panoramic mesh image A1 is input into the local image generation model to obtain a refined mesh image A1'. The refined mesh image A1' and the current panoramic frame image A are then input into the panoramic image generation model to obtain a refined panoramic image A' formed by fusing the refined mesh image A1' and the current panoramic frame image A.

[0127] The refined panoramic image A' is updated to the current panoramic frame image. Furthermore, panoramic mesh image A2 can be selected as the (i+1)th panoramic mesh image. Based on the local image generation model, the (i+1)th panoramic mesh image A2 is processed to obtain the refined mesh image A2'. A2' and A' are input into the panoramic image generation model to obtain the refined panoramic image A'". At this point, the refined panoramic image A' contains the refined features of panoramic mesh images A1 and A2. Next, the refined panoramic image A' is updated to the current panoramic frame image. Furthermore, panoramic mesh image A3 can be selected as the (i+2)th panoramic mesh image. The above steps are repeated until A4-A9 are also processed. The final refined panoramic image is then used as the panoramic image to be fused, A_final. A_final contains the refined features of all panoramic mesh images A1-A9.

[0128] In some examples, the panorama to be processed can also be completed in any of the following ways to obtain the panorama to be fused corresponding to the image to be used:

[0129] Method 1: Based on a preset image completion algorithm, complete the panoramic image to be processed to obtain the panoramic image to be fused.

[0130] The preset image completion algorithm can be a pre-selected image completion algorithm. This algorithm can be one that completes the missing areas in the panoramic image to be processed based on the panoramic image itself, making the completed image look natural and difficult to distinguish from the undamaged image. For example, the image completion algorithm can be a texture synthesis method, a depth-based completion method, etc.

[0131] Specifically, a preset image completion algorithm is used to complete the panoramic image to be processed, which completes the parts of the panoramic image with pixel values ​​of a set value, and the completed image is determined as the panoramic image to be fused.

[0132] Method 2: Based on at least one pre-stored panoramic image to be matched, determine the panoramic image to be fused that matches the panoramic image to be processed.

[0133] The panoramic image to be matched can be a panoramic image stored in an image library.

[0134] Specifically, the similarity between the panoramic image to be processed and at least one pre-stored panoramic image to be matched can be calculated. The panoramic image to be matched that meets the similarity criteria is determined as the panoramic image to be fused. The similarity criteria can be that the similarity reaches a predetermined similarity threshold, or that the similarity value is at its maximum, etc.

[0135] S270. Based on spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined through an image processing model.

[0136] The technical solution of this disclosure embodiment obtains a reference depth map including the depth information of each pixel based on the image to be used, and determines the reference depth value of the reference point. Based on the reference depth value, the reference depth map and the image to be used, a panoramic image to be processed is determined. By completing the panoramic image to be processed, a panoramic image to be fused corresponding to the image to be used is obtained. On the existing basis, the environmental information is further completed and the environmental information is represented in the form of a panoramic image, which facilitates the subsequent rendering operation of virtual objects.

[0137] Figure 5 This is a flowchart illustrating another image processing method provided in this embodiment. In some examples, the spherical harmonic illumination corresponding to the target subject can be determined by a pre-trained target illumination estimation model. An initial environment map can also be determined based on the spherical harmonic illumination, and the initial environment map and the panoramic image to be fused can be fused using an image processing model to generate the target panoramic image, i.e., the final HDR environment map. Specific implementation details can be found in the detailed description of this technical solution. Explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0138] like Figure 5 As shown, the method includes:

[0139] S410, Obtain the original image.

[0140] S420. Based on the original image, obtain the segmented main image and the image to be processed.

[0141] S430. Input the subject image into the target illumination estimation model to determine the spherical harmonic illumination corresponding to the target subject.

[0142] The target illumination estimation model can be a pre-trained model used to determine the spherical harmonic illumination of the subject image.

[0143] Specifically, the subject image is input into a pre-trained target illumination estimation model, and the model's output and the illumination estimation information corresponding to the target subject are used as spherical harmonic illumination.

[0144] In this embodiment, before using the pre-trained target illumination estimation model, it is possible to... Figure 6 The steps shown are used to train the illumination estimation model to obtain the target illumination estimation model:

[0145] S4301. Obtain multiple training samples.

[0146] The training samples can be used for subsequent model training. The training samples include training images and viewpoints. The training images can be images containing illumination estimation information, that is, images containing actual spherical harmonic illumination. The viewpoints can be the viewpoints when the training images were taken.

[0147] Specifically, multiple training samples can be obtained through methods such as shooting, uploading, and downloading.

[0148] S4302. For each training sample, input the training image in the current training sample into the illumination estimation model to obtain the actual spherical harmonic illumination corresponding to the current training sample.

[0149] It should be noted that for each training sample, the following steps can be used to train it to obtain the desired target illumination estimation model.

[0150] The actual spherical harmonic illumination is the spherical harmonic illumination output after the training images in the current training samples are input into the illumination estimation model.

[0151] Specifically, after inputting the training images from the current training samples into the illumination estimation model, the illumination estimation model can obtain the actual spherical harmonic illumination corresponding to the current training samples.

[0152] It should be noted that the model parameters in the illumination estimation model do not meet the expected requirements. Therefore, there is a certain difference between the actual spherical harmonic illumination output based on the model parameters at this time and the theoretical spherical harmonic illumination. Therefore, the corresponding error loss value can be determined based on the actual spherical harmonic illumination and the theoretical spherical harmonic illumination corresponding to each training image.

[0153] In this embodiment, the illumination estimation model can be a ResNet network model. It should be noted that any model type is acceptable as long as it can provide spherically harmonic illumination corresponding to the image to be processed; that is, the specific model type is not limited.

[0154] S4303. Based on the first preset loss function, determine the loss value according to the actual spherical harmonic illumination and viewing angle of the current training sample, and optimize the model parameters in the illumination estimation model based on the loss value.

[0155] It should be noted that the training parameters can be set to default values ​​before training the illumination estimation model. During training, the training parameters can be adjusted based on the model's output; that is, the target illumination estimation model can be obtained by modifying the loss function in the illumination estimation model. Each training image has a corresponding loss value, which is determined based on the actual spherical harmonic illumination and viewpoint of each training image.

[0156] The first preset loss function can be a function used to measure the degree of loss between actual spherical harmonic illumination and theoretical spherical harmonic illumination.

[0157] Specifically, the actual spherical harmonic illumination and viewing angle are processed according to the first preset loss function in the illumination estimation model to determine the theoretical spherical harmonic illumination corresponding to the viewing angle, and the loss value between the theoretical spherical harmonic illumination and the actual spherical harmonic illumination is determined, that is, the loss value corresponding to the training image. The backpropagation method can be used to optimize the model parameters in the illumination estimation model.

[0158] S4304. The convergence of the first preset loss function is taken as the optimization objective to obtain the target illumination estimation model.

[0159] Specifically, the training error of the first preset loss function, i.e., the loss parameter, can be used as a condition to detect whether the first preset loss function has reached convergence. For example, this could be whether the training error is less than the preset error, whether the error trend is stable, or whether the current number of iterations equals the preset number. If the convergence condition is met, such as the training error of the first preset loss function being less than the preset error or the error trend being stable, it indicates that the illumination estimation model training is complete, and iterative training can be stopped. If the convergence condition is not met, training samples can be further obtained to train the illumination estimation model until the training error of the loss function is within the preset range. When the training error of the loss function converges, the current illumination estimation model can be used as the target illumination estimation model.

[0160] S440. Based on the image to be processed, determine the corresponding panoramic image to be fused.

[0161] S450: Based on spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined through an image processing model.

[0162] The technical solution of this disclosure achieves accurate estimation of illumination information by inputting the subject image into the target illumination estimation model to determine the spherical harmonic illumination corresponding to the target subject.

[0163] Figure 7 This is a schematic flowchart illustrating another image processing method provided in this embodiment. In some examples, an initial environment map can be determined based on spherical harmonic illumination, and the initial environment map and the panoramic image to be fused can be fused using an image processing model. Specific implementation details can be found in the detailed description of this technical solution. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0164] like Figure 7 As shown, the method includes:

[0165] S510. Obtain the original image. Based on the original image, obtain the segmented main image and the image to be processed.

[0166] S520. Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject.

[0167] S530. Based on the image to be processed, determine the corresponding panoramic image to be fused.

[0168] S540, Determine the initial environment texture based on spherical harmonic lighting.

[0169] The initial environment map can be a light map based on spherical harmonic lighting simulation.

[0170] Specifically, the initial environment texture is simulated and constructed based on the information carried by the spherical harmonic lighting.

[0171] S550: The initial environment texture and the panoramic image to be fused are stitched together to obtain the input of the image processing model, and the target panoramic image is output based on the image processing model.

[0172] The panorama to be fused is an LDR (Low Dynamic Range) image, while the target panorama is an HDR image. Compared to HDR images, LDR images contain less illumination information.

[0173] Specifically, the initial environment texture and the panoramic image to be fused are stitched together to convert the LDR image into an HDR image. Then, the stitched image is input into an image processing model, and the output image of the image processing model is used as the target panoramic image.

[0174] In this embodiment, before using the image processing model, it can be done through, as follows: Figure 8 The steps shown are used to train and obtain the image processing model:

[0175] S610: Obtain multiple training panoramic images.

[0176] The training panorama is an HDR image. The training panorama can be a real panoramic image with spherical harmonic illumination information.

[0177] S620. For each training panoramic image, extract the theoretical spherical harmonic illumination of the current training panoramic image, and obtain the input panoramic image associated with the current training panoramic image based on the theoretical spherical harmonic illumination.

[0178] The input panoramic image is an LDR image. The theoretical spherical harmonic illumination can be obtained by analyzing the spherical harmonic illumination of the training panoramic image. The input panoramic image can be obtained by removing the theoretical spherical harmonic illumination from the training panoramic image.

[0179] Specifically, the same processing can be performed on each training panorama: analyze the current training panorama to obtain its theoretical spherical harmonic illumination; extract the theoretical spherical harmonic illumination from the training panorama; and determine the remaining portion that does not contain the theoretical spherical harmonic illumination as the input panorama associated with the current training panorama.

[0180] S630. Based on each training panoramic image, the input panoramic image associated with the training panoramic image, and the theoretical spherical harmonic illumination, determine multiple training samples.

[0181] Specifically, for each training panorama, the training panorama, the input panorama associated with the training panorama, and the theoretical spherical harmonic illumination are used as one of multiple training samples.

[0182] After determining the training samples, the image processing model can be obtained through the following steps:

[0183] S640. For each training sample, determine the input environment texture based on the theoretical spherical harmonic illumination in the current training sample, and after stitching the input environment texture and the input panoramic image, input it into the image processing model to obtain the output panoramic image.

[0184] The input environment map can be a lighting map simulated by theoretical spherical harmonic illumination based on the current training samples. The output panorama can be the output of an image processing model, or a panorama predicted from an image stitched together from the input environment map and the input panorama.

[0185] Specifically, the current training sample is determined from multiple training samples. This current training sample includes a training panorama, an input panorama associated with the training panorama, and theoretical spherical harmonic illumination. Based on the theoretical spherical harmonic illumination in the current training sample, an input environment map can be simulated and constructed. Then, the input environment map and the input panorama are stitched together to convert the LDR image into an HDR image. Finally, the stitched image is used as input to an image processing model, which calculates and produces the output image, i.e., the output panorama.

[0186] S650. Determine the output spherical harmonic illumination of the output panoramic image, and process the output panoramic image based on the output spherical harmonic illumination to obtain the discriminative panoramic image.

[0187] The discrimination panorama is an LDR image, while the output panorama is an HDR image. The output spherical harmonic illumination can be obtained by analyzing the spherical harmonic illumination based on the output panorama. The discrimination panorama can be the image obtained by removing the output spherical harmonic illumination from the output panorama.

[0188] Specifically, the output panoramic image is analyzed to obtain the output spherical harmonic illumination. The output spherical harmonic illumination is then removed from the output panoramic image to obtain the discriminative panoramic image.

[0189] S660. Based on the loss function corresponding to the image processing model, the loss value is determined according to the output spherical harmonic illumination, theoretical spherical harmonic illumination, discriminative panoramic image and training panoramic image, and the model parameters in the image processing model are optimized based on the loss value.

[0190] Specifically, the loss function corresponding to the image processing model can be used to determine the loss value between the output spherically harmonic illumination and the theoretical spherically harmonic illumination, as well as the loss value between the discriminative panorama and the training panorama. Based on these two loss values, the model performance of the image processing model can be determined. Therefore, the model parameters in the image processing model can be optimized in reverse based on these two loss values ​​to improve the model performance, that is, to make the output spherically harmonic illumination generated by the image processing model more similar to the theoretical spherically harmonic illumination, and the discriminative panorama more closely similar to the training panorama.

[0191] For example, the loss value between the output spherically harmonic illumination and the theoretical spherically harmonic illumination can be converted to the Lab color space to obtain a new loss value. Furthermore, the loss value between the discrimination panorama and the training panorama is determined. Based on these two loss values, the model parameters in the image processing model are optimized.

[0192] S670. Taking the convergence of the loss function in the image processing model as the optimization objective, the image processing model is obtained.

[0193] Specifically, when the loss function in the image processing model converges, such as when the training error is less than the preset error or the error trend becomes stable, or when the current number of iterations equals the preset number, it can be considered that the effect of the image processing model can meet the usage requirements. At this time, model training is stopped, and the current image processing model is used as the image processing model for subsequent use.

[0194] The technical solution of this disclosure determines the initial environment map based on spherical harmonic lighting, stitches the obtained initial environment map and the panoramic image to be fused together to obtain the input of the image processing model, and outputs the target panoramic image based on the image processing model, thereby realizing the fusion of environmental information and lighting information. The final HDR environment map (i.e. the target panoramic image) can effectively improve the rendering realism of virtual objects.

[0195] Figure 9 This is a schematic diagram of an image processing apparatus provided in Embodiment 4 of this disclosure. The image processing apparatus provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal and / or server to implement the image processing method in this disclosure. The image processing apparatus provided in this embodiment is suitable for performing ambient lighting processing on original images containing target subjects to obtain panoramic images.

[0196] like Figure 9 As shown, the image processing device includes: an original image acquisition module 710, an original image segmentation module 720, a spherical harmonic illumination determination module 730, a panoramic image determination module 740 to be fused, and a target panoramic image determination module 750.

[0197] The system includes: an original image acquisition module 710 for acquiring an original image, which includes a target subject; an original image segmentation module 720 for obtaining a segmented subject image and a to-be-processed image based on the original image, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject; a spherical harmonic illumination determination module 730 for determining the spherical harmonic illumination corresponding to the target subject based on the subject image; a panorama to be fused determination module 740 for determining the corresponding panorama to be fused based on the to-be-processed image; and a target panorama determination module 750 for determining the target panorama based on the spherical harmonic illumination and the panorama to be fused using an image processing model.

[0198] The technical solution of this disclosure involves acquiring an original image, obtaining a segmented subject image and an image to be processed based on the original image, determining spherical harmonic lighting corresponding to the target subject based on the subject image, determining a corresponding panoramic image to be fused based on the image to be processed, and finally determining the target panoramic image based on the image processing model, spherical harmonic lighting, and the panoramic image to be fused. By separating the subject image from the original image and performing lighting estimation and environmental information restoration based on the segmented subject and environment parts, the problem of the large proportion of the subject in the image and the severe occlusion of the environment part affecting the environmental lighting estimation effect is solved. At the same time, the final target panoramic image combines environmental and lighting information, which can provide a more realistic rendering effect for the realization of augmented reality technology.

[0199] Based on the above technical solution, the panorama determination module 740 is further used to repair the repair area of ​​the image to be processed to obtain the image to be used; and to determine the panorama to be fused corresponding to the image to be used based on the reference point located in the repair area.

[0200] Based on the above technical solution, the target panoramic image determination module 750 is further configured to determine an initial environment map based on the spherical harmonic illumination; stitch the initial environment map and the panoramic image to be fused together to obtain the input of the image processing model, and output the target panoramic image based on the image processing model; wherein, the panoramic image to be fused is a low dynamic range image, and the target panoramic image is a high dynamic range image.

[0201] Based on the above technical solutions, the device further includes: a first special effects rendering module, configured to, in response to a trigger operation that detects adding special effects to the original image, render the special effects based on the target panoramic image to obtain a special effects image corresponding to the original image. Furthermore, the special effects can be augmented reality effects.

[0202] Based on the above technical solutions, the target subject is a human portrait, and the device further includes: a second special effects rendering module, used to render virtual clothing to be added based on the target panoramic image when a clothing change trigger operation corresponding to the target subject is detected, so as to obtain a special effects image corresponding to the original image.

[0203] Based on the above technical solution, the panorama determination module 740 is further configured to obtain a reference depth map including depth information of each pixel based on the image to be used, and determine the reference depth value of the reference point; determine the panorama to be processed based on the reference depth value, the reference depth map and the image to be used; and obtain the panorama to be fused corresponding to the image to be used by completing the panorama to be processed.

[0204] Based on the above technical solution, the panorama determination module 740 is further configured to determine the panorama processing depth map based on the reference depth value and the reference depth map; determine the panorama processing image based on the panorama processing depth map and the image to be used; obtain a distortion diagram by distorting the panorama processing image, and display the distortion diagram on the panorama processing image; wherein, the pixel values ​​of the pixels outside the distortion diagram in the panorama processing image are set values.

[0205] Based on the above technical solution, the panorama determination module 740 is further configured to input the panorama to be processed into a panorama frame image generation model to obtain a panorama frame image; perform gridding processing on the panorama frame image to obtain at least one panorama grid image; obtain a corresponding refined grid image through a local image generation model based on at least one panorama grid image; and obtain a panorama to be fused corresponding to the image to be used through a panorama image generation model based on the panorama frame image and at least one refined grid image.

[0206] Based on the above technical solution, the spherical harmonic illumination determination module 730 is further used to input the subject image into the target illumination estimation model to determine the spherical harmonic illumination corresponding to the target subject.

[0207] Based on the above technical solution, the device further includes: a target illumination estimation model training module, used to acquire multiple training samples; wherein, the training samples include training images and viewpoints; for each training sample, the training image in the current training sample is input into the illumination estimation model to obtain the actual spherical harmonic illumination corresponding to the current training sample; based on a first preset loss function, a loss value is determined according to the actual spherical harmonic illumination and viewpoint of the current training sample, and the model parameters in the illumination estimation model are optimized based on the loss value.

[0208] Based on the above technical solution, the device further includes: a training sample determination module, used to acquire multiple training panoramic images; for each training panoramic image, extract the theoretical spherical harmonic illumination of the current training panoramic image, and obtain an input panoramic image associated with the current training panoramic image based on the theoretical spherical harmonic illumination; wherein, the input panoramic image is a low dynamic range image, and the training panoramic image is a high dynamic range image; and based on each training panoramic image, the input panoramic image associated with the training panoramic image, and the theoretical spherical harmonic illumination, determine multiple training samples.

[0209] Based on the above technical solution, the device further includes: an image processing model training module, used for determining the input environment map based on the theoretical spherical harmonic illumination in the current training sample for each training sample, and stitching the input environment map and the input panoramic image together before inputting them into the image processing model to obtain an output panoramic image; determining the output spherical harmonic illumination of the output panoramic image, and processing the output panoramic image based on the output spherical harmonic illumination to obtain a discriminative panoramic image; wherein the discriminative panoramic image is a low dynamic range image and the output panoramic image is a high dynamic range image; and determining a loss value based on the loss function corresponding to the image processing model, according to the output spherical harmonic illumination, the theoretical spherical harmonic illumination, the discriminative panoramic image, and the training panoramic image, and optimizing the model parameters in the image processing model based on the loss value.

[0210] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0211] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0212] Figure 10 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. Refer to the following... Figure 10 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 10 The diagram below shows the structure of the terminal device or server 800. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0213] like Figure 10 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An edit / output (I / O) interface 805 is also connected to the bus 804.

[0214] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0215] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0216] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0217] The electronic device provided in this embodiment and the image processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0218] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image processing method provided in the above embodiments.

[0219] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0220] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0221] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0222] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0223] Acquire the original image, which includes the target subject;

[0224] Based on the original image, a segmented subject image and a to-be-processed image are obtained, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject;

[0225] Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject;

[0226] Based on the image to be processed, determine the corresponding panoramic image to be fused;

[0227] Based on the spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined by an image processing model.

[0228] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0229] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0230] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0231] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0232] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0233] According to one or more embodiments of this disclosure, an image processing method is provided, the method comprising:

[0234] Acquire the original image, which includes the target subject;

[0235] Based on the original image, a segmented subject image and a to-be-processed image are obtained, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject;

[0236] Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject;

[0237] Based on the image to be processed, determine the corresponding panoramic image to be fused;

[0238] Based on the spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined by an image processing model.

[0239] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0240] Optionally, determining the corresponding panoramic image to be fused based on the image to be processed includes:

[0241] The areas to be repaired in the image to be processed are repaired to obtain the image to be used;

[0242] Based on the reference point located in the area to be repaired, determine the panoramic image to be fused that corresponds to the image to be used.

[0243] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0244] Optionally, determining the target panoramic image based on the spherical harmonic illumination and the panoramic image to be fused using an image processing model includes:

[0245] Based on the spherical harmonic lighting, determine the initial environment texture;

[0246] The initial environment texture and the panoramic image to be fused are stitched together to obtain the input of the image processing model, and the target panoramic image is output based on the image processing model.

[0247] The panoramic image to be fused is a low dynamic range image, and the target panoramic image is a high dynamic range image.

[0248] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0249] Optional, also includes:

[0250] In response to a triggered operation that adds effects to the original image, the effects are rendered based on the target panorama to obtain an effect image corresponding to the original image.

[0251] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0252] Optionally, the target subject is a human image, and the system further includes:

[0253] In response to the detection of a clothing change trigger operation corresponding to the target subject, the virtual clothing to be added is rendered based on the target panoramic image to obtain a special effects image corresponding to the original image.

[0254] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0255] Optionally, determining the panoramic image to be fused corresponding to the image to be used, based on a reference point located in the area to be repaired, includes:

[0256] Based on the image to be used, a reference depth map including depth information of each pixel is obtained, and the reference depth value of the reference point is determined.

[0257] Based on the baseline depth value, the reference depth map, and the image to be used, a panoramic image to be processed is determined.

[0258] By completing the panoramic image to be processed, a panoramic image to be fused corresponding to the image to be used is obtained.

[0259] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0260] Optionally, determining the panoramic image to be processed based on the baseline depth value, the reference depth map, and the image to be used includes:

[0261] Based on the baseline depth value and the reference depth map, determine the depth map to be processed for the panorama;

[0262] Based on the depth map to be processed and the image to be used, determine the map to be processed.

[0263] By performing distortion processing on the panoramic image to be processed, a distortion diagram is obtained, and the distortion diagram is displayed on the panoramic image to be processed; wherein, the pixel values ​​of the pixels outside the distortion diagram in the panoramic image to be processed are set values.

[0264] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0265] Optionally, the step of completing the panoramic image to be processed to obtain a panoramic image to be fused corresponding to the image to be used includes:

[0266] The panoramic image to be processed is input into the panoramic frame image generation model to obtain the panoramic frame image;

[0267] The panoramic frame image is processed into a grid to obtain at least one panoramic grid image;

[0268] Based on at least one of the panoramic mesh images, a corresponding refined mesh image is obtained through a local image generation model;

[0269] Based on the panoramic frame map and at least one of the refined mesh maps, a panoramic image to be fused, corresponding to the image to be used, is obtained through a panoramic image generation model.

[0270] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0271] Optionally, determining the spherical harmonic illumination corresponding to the target subject based on the subject image includes:

[0272] The subject image is input into the target illumination estimation model to determine the spherical harmonic illumination corresponding to the target subject.

[0273] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0274] Optional, also includes:

[0275] Acquire multiple training samples; wherein, the training samples include training images and viewpoints;

[0276] For each training sample, the training image in the current training sample is input into the illumination estimation model to obtain the actual spherical harmonic illumination corresponding to the current training sample;

[0277] Based on the first preset loss function, the loss value is determined according to the actual spherical harmonic illumination and viewing angle of the current training sample, and the model parameters in the illumination estimation model are optimized based on the loss value.

[0278] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0279] Optional, also includes:

[0280] Obtain multiple training panoramas;

[0281] For each training panoramic image, the theoretical spherical harmonic illumination of the current training panoramic image is extracted, and the input panoramic image associated with the current training panoramic image is obtained based on the theoretical spherical harmonic illumination; wherein, the input panoramic image is a low dynamic range image, and the training panoramic image is a high dynamic range image.

[0282] Based on each training panoramic image, the input panoramic image associated with the training panoramic image, and the theoretical spherical harmonic illumination, multiple training samples are determined.

[0283] According to one or more embodiments of this disclosure, an image processing method is provided, further comprising:

[0284] Optional, also includes:

[0285] For each training sample, the environment texture to be input is determined based on the theoretical spherical harmonic illumination in the current training sample. The environment texture to be input and the panoramic image to be input are then stitched together and input into the image processing model to obtain the output panoramic image.

[0286] The output spherical harmonic illumination of the output panoramic image is determined, and the output panoramic image is processed based on the output spherical harmonic illumination to obtain a discriminative panoramic image; wherein, the discriminative panoramic image is a low dynamic range image, and the output panoramic image is a high dynamic range image;

[0287] Based on the loss function corresponding to the image processing model, the loss value is determined according to the output spherical harmonic illumination, theoretical spherical harmonic illumination, discriminative panorama, and training panorama, and the model parameters in the image processing model are optimized based on the loss value.

[0288] According to one or more embodiments of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0289] The original image acquisition module is used to acquire an original image, wherein the original image includes the target subject;

[0290] The original image segmentation module is used to obtain a segmented subject image and a to-be-processed image based on the original image, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject;

[0291] A spherical harmonic illumination determination module is used to determine the spherical harmonic illumination corresponding to the target subject based on the subject image;

[0292] The panorama determination module is used to determine the corresponding panorama to be merged based on the image to be processed.

[0293] The target panoramic image determination module is used to determine the target panoramic image based on the spherical harmonic illumination and the panoramic image to be fused, using an image processing model.

[0294] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0295] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0296] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, characterized in that, include: Acquire the original image, which includes the target subject; Based on the original image, a segmented subject image and a to-be-processed image are obtained, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject; Based on the subject image, determine the spherical harmonic illumination corresponding to the target subject; Based on the image to be processed, determine the corresponding panoramic image to be fused; Based on the spherical harmonic illumination and the panoramic image to be fused, the target panoramic image is determined by an image processing model. The step of determining the target panoramic image based on the spherical harmonic illumination and the panoramic image to be fused using an image processing model includes: Based on the spherical harmonic lighting, determine the initial environment texture; The initial environment texture and the panoramic image to be fused are stitched together to obtain the input of the image processing model, and the target panoramic image is output based on the image processing model. The panoramic image to be fused is a low dynamic range image, and the target panoramic image is a high dynamic range image.

2. The method according to claim 1, characterized in that, The step of determining the corresponding panoramic image to be fused based on the image to be processed includes: The areas to be repaired in the image to be processed are repaired to obtain the image to be used; Based on the reference point located in the area to be repaired, determine the panoramic image to be fused that corresponds to the image to be used.

3. The method according to claim 1, characterized in that, Also includes: In response to a triggered operation that adds effects to the original image, the effects are rendered based on the target panorama to obtain an effect image corresponding to the original image.

4. The method according to claim 1, characterized in that, The target subject is a human image, and also includes: In response to the detection of a clothing change trigger operation corresponding to the target subject, the virtual clothing to be added is rendered based on the target panoramic image to obtain a special effects image corresponding to the original image.

5. The method according to claim 2, characterized in that, The step of determining the panoramic image to be fused, corresponding to the image to be used, based on a reference point located in the area to be repaired, includes: Based on the image to be used, a reference depth map including depth information of each pixel is obtained, and the reference depth value of the reference point is determined. Based on the baseline depth value, the reference depth map, and the image to be used, a panoramic image to be processed is determined. By completing the panoramic image to be processed, a panoramic image to be fused corresponding to the image to be used is obtained.

6. The method according to claim 5, characterized in that, The process of determining the panoramic image to be processed based on the baseline depth value, the reference depth map, and the image to be used includes: Based on the baseline depth value and the reference depth map, determine the depth map to be processed for the panorama; Based on the depth map to be processed and the image to be used, determine the map to be processed. By performing distortion processing on the panoramic image to be processed, a distortion diagram is obtained, and the distortion diagram is displayed on the panoramic image to be processed; wherein, the pixel values ​​of the pixels outside the distortion diagram in the panoramic image to be processed are set values.

7. The method according to claim 5, characterized in that, The step of completing the panoramic image to be processed to obtain a panoramic image to be fused corresponding to the image to be used includes: The panoramic image to be processed is input into the panoramic frame image generation model to obtain the panoramic frame image; The panoramic frame image is processed into a grid to obtain at least one panoramic grid image; Based on at least one of the panoramic mesh images, a corresponding refined mesh image is obtained through a local image generation model; Based on the panoramic frame map and at least one of the refined mesh maps, a panoramic image to be fused, corresponding to the image to be used, is obtained through a panoramic image generation model.

8. The method according to claim 1, characterized in that, The step of determining the spherical harmonic illumination corresponding to the target subject based on the subject image includes: The subject image is input into the target illumination estimation model to determine the spherical harmonic illumination corresponding to the target subject.

9. The method according to claim 8, characterized in that, Also includes: Acquire multiple training samples; wherein, the training samples include training images and viewpoints; For each training sample, the training image in the current training sample is input into the illumination estimation model to obtain the actual spherical harmonic illumination corresponding to the current training sample; Based on the first preset loss function, the loss value is determined according to the actual spherical harmonic illumination and viewing angle of the current training sample, and the model parameters in the illumination estimation model are optimized based on the loss value.

10. The method according to claim 1, characterized in that, Also includes: Obtain multiple training panoramas; For each training panoramic image, the theoretical spherical harmonic illumination of the current training panoramic image is extracted, and the input panoramic image associated with the current training panoramic image is obtained based on the theoretical spherical harmonic illumination; wherein, the input panoramic image is a low dynamic range image, and the training panoramic image is a high dynamic range image. Based on each training panoramic image, the input panoramic image associated with the training panoramic image, and the theoretical spherical harmonic illumination, multiple training samples are determined.

11. The method according to claim 10, characterized in that, Also includes: For each training sample, the environment texture to be input is determined based on the theoretical spherical harmonic illumination in the current training sample. The environment texture to be input and the panoramic image to be input are then stitched together and input into the image processing model to obtain the output panoramic image. The output spherical harmonic illumination of the output panoramic image is determined, and the output panoramic image is processed based on the output spherical harmonic illumination to obtain a discriminative panoramic image; wherein, the discriminative panoramic image is a low dynamic range image, and the output panoramic image is a high dynamic range image; Based on the loss function corresponding to the image processing model, the loss value is determined according to the output spherical harmonic illumination, theoretical spherical harmonic illumination, discriminative panorama, and training panorama, and the model parameters in the image processing model are optimized based on the loss value.

12. An image processing apparatus, characterized in that, include: The original image acquisition module is used to acquire an original image, wherein the original image includes the target subject; The original image segmentation module is used to obtain a segmented subject image and a to-be-processed image based on the original image, wherein the to-be-processed image includes a region to be repaired corresponding to the target subject; A spherical harmonic illumination determination module is used to determine the spherical harmonic illumination corresponding to the target subject based on the subject image; The panorama determination module is used to determine the corresponding panorama to be merged based on the image to be processed. The target panoramic image determination module is used to determine the target panoramic image based on the spherical harmonic illumination and the panoramic image to be fused, using an image processing model. The target panoramic image determination module is specifically used to determine the initial environment texture based on the spherical harmonic illumination; The initial environment texture and the panoramic image to be fused are stitched together to obtain the input of the image processing model, and the target panoramic image is output based on the image processing model. The panoramic image to be fused is a low dynamic range image, and the target panoramic image is a high dynamic range image.

13. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-11.

14. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any one of claims 1-11.

Images