Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium. The application embodiment can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises the following steps: if a target object exists in a target image, segmenting the target image to obtain a foreground region comprising the target object; determining a target adjustment pixel value corresponding to each pixel point in the foreground region from a preset lookup table according to a pixel value of each pixel point; adjusting the pixel value of each pixel point in the foreground region to the target adjustment pixel value corresponding to the pixel point to obtain an adjusted foreground region; and performing fusion processing on the adjusted foreground region and a target background image to obtain a fusion image. In the application, a shadow region and a non-shadow region do not need to be rendered according to the target image, a large amount of rendering time is saved, the processing efficiency of the foreground region is improved, and the acquisition efficiency of the fusion image is improved.

Description

Technical Field

[0001] This application relates to the field of Internet information processing technology, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Human-background blending refers to using computer vision technology to blend a human image in an image into a specified background image to obtain a blended image, where the lighting of the human image in the background image is consistent with the lighting of the background image.

[0003] Currently, it's possible to acquire a portrait image, its lighting conditions, and a depth map. Based on the depth map and lighting conditions, the shadow and non-shadow areas of the portrait image are rendered. Then, the image is relit using these shadow and non-shadow areas to obtain a relit image, thus adjusting the background of the portrait image. However, this method has low processing efficiency for portrait images, resulting in low efficiency in generating the fused image. Summary of the Invention

[0004] In view of this, embodiments of this application propose an image processing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: if a target object exists in a target image, segmenting the target image to obtain a foreground region including the target object; determining a target adjustment pixel value corresponding to each pixel in the foreground region from a preset lookup table based on the pixel value of each pixel, the preset lookup table including adjustment pixel values ​​corresponding to different pixel values, the preset lookup table being obtained after performing illumination estimation on the target background image to be fused; adjusting the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel to obtain an adjusted foreground region; using the target background image as the background of the target image, fusing the adjusted foreground region and the target background image to obtain a fused image.

[0006] Secondly, embodiments of this application provide an image processing apparatus, comprising: a segmentation module, configured to segment the target image to obtain a foreground region including the target object if a target object exists in the target image; a determination module, configured to determine a target adjustment pixel value corresponding to each pixel in the foreground region from a preset lookup table, the preset lookup table including adjustment pixel values ​​corresponding to different pixel values, the preset lookup table being obtained after performing illumination estimation on the target background image to be fused; an adjustment module, configured to adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel, to obtain an adjusted foreground region; and a fusion module, configured to use the target background image as the background of the target image, and perform fusion processing on the adjusted foreground region and the target background image to obtain a fused image.

[0007] Optionally, the fusion module is further configured to obtain a visual contour image of the corresponding foreground region based on the target image; perform Gaussian blur processing on the visual contour image to obtain the first blur parameter of the corresponding visual contour image; and obtain a fused image based on the blur parameter of the visual contour image, the target background image, and the foreground region.

[0008] Optionally, the blur parameters of the visualized contour image include the blur parameters corresponding to each pixel in the visualized contour image; the fusion module is further configured to determine the blur parameters of each pixel in the target background image and the blur parameters of each pixel in the foreground region based on the blur parameters corresponding to each pixel in the visualized contour image; replace the pixel value of each pixel in the target background image with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed background image of the corresponding target background image; replace the pixel value of each pixel in the foreground region with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed foreground region of the corresponding foreground region; and perform fusion processing on the preprocessed background image and the preprocessed foreground region to obtain a fused image.

[0009] Optionally, the device further includes a lookup table acquisition module for acquiring a target background image; performing spherical mapping on the target background image to obtain a spherical background image; determining spherical harmonic basis parameters based on the spherical background image and a preset spherical harmonic basis function; and obtaining a preset lookup table based on the spherical harmonic basis parameters and the preset spherical harmonic basis function.

[0010] Optionally, the lookup table module is also used to construct a spherical model, the number of spherical points in the spherical model being the same as the number of pixels in the target background image; determine the spherical points corresponding to each pixel in the target background image from the spherical model; and adjust the pixel value of each spherical point to the pixel value of the corresponding pixel in the target background image to obtain the spherical background image.

[0011] Optionally, the lookup table acquisition module is also used to obtain the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; determine the yaw angle and pitch angle of each pixel in the target background image based on the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; and determine the spherical point corresponding to the pixel from the spherical model based on the yaw angle and pitch angle of each pixel in the target background image.

[0012] Optionally, the lookup table acquisition module is also used to determine the function operation result of each spherical point in the spherical background image through a preset spherical harmonic basis function; calculate the product of the function operation result of each spherical point in the spherical background image and the pixel value of the spherical point as the first product result of the spherical point; and sum the first product results of all spherical points in the spherical background image to obtain the spherical harmonic basis parameters.

[0013] Optionally, the lookup table acquisition module is also used to acquire a natural color lookup table; and to obtain a preset lookup table based on the preset spherical harmonic basis functions, spherical harmonic basis parameters and the natural color lookup table.

[0014] Optionally, the natural color lookup table includes adjusted pixel values ​​corresponding to multiple pixel values; the lookup table acquisition module is also used to obtain the normal vector of the corresponding target background image; obtain the direction vector corresponding to each spherical point in the spherical background image; determine the operation result of each spherical point in the spherical background image through a preset spherical harmonic basis function; calculate the product of the operation result of each spherical point in the spherical background image, the direction vector of the spherical point, the spherical harmonic basis parameter, and the normal vector, as the second product result of the spherical point; sum the second product results of all spherical points in the spherical background image to obtain the summation result; calculate the product of the summation result and the adjusted pixel value of each pixel value in the natural color lookup table, as the preset adjusted pixel value of the pixel value; and obtain the preset lookup table based on each pixel value in the natural color lookup table and the preset adjusted pixel value corresponding to the pixel value.

[0015] Optionally, the segmentation module is further configured to segment the target image using a preset semantic segmentation network if a target object exists in the target image, thereby obtaining a foreground region including the target object.

[0016] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0019] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium. The method directly determines the target adjustment pixel value from a preset lookup table using the pixel value of each pixel in the foreground region, adjusts the pixel value of each pixel in the foreground region to the corresponding target adjustment pixel value, and obtains an adjusted foreground region. Then, the adjusted foreground region is fused with the target background image to obtain a fused image. This method does not require rendering shadow and non-shadow areas based on the depth map and lighting conditions of the target image, saving significant rendering time and improving the processing efficiency of the foreground region, thereby increasing the efficiency of obtaining the fused image. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of a system architecture proposed in an embodiment of this application is shown;

[0022] Figure 2 A flowchart of an image processing method according to an embodiment of this application is shown;

[0023] Figure 3 A schematic diagram of a target image according to an embodiment of this application is shown;

[0024] Figure 4 It shows Figure 3 A schematic diagram of the foreground region corresponding to the target image in the image;

[0025] Figure 5 It shows Figure 3 A schematic diagram of the background region corresponding to the target image in the image;

[0026] Figure 6 A flowchart of an image processing method according to yet another embodiment of this application is shown;

[0027] Figure 7 It shows Figure 3 A schematic diagram of the visual contour image corresponding to the target image in the diagram;

[0028] Figure 8A flowchart of an image processing method according to another embodiment of this application is shown;

[0029] Figure 9 A schematic diagram of a target background image in an embodiment of this application is shown;

[0030] Figure 10 It shows Figure 9 A schematic diagram of the spherical background image corresponding to the target background image in the diagram;

[0031] Figure 11 A schematic diagram of the image processing procedure in an embodiment of this application is shown;

[0032] Figure 12 A schematic diagram of yet another target background image in an embodiment of this application is shown;

[0033] Figure 13 A schematic diagram of yet another target image in an embodiment of this application is shown;

[0034] Figure 14 It shows Figure 13 A schematic diagram of the foreground region corresponding to the target image in the image;

[0035] Figure 15 It shows Figure 12 Target background image and Figure 13 A schematic diagram of the fused image after the target image is fused;

[0036] Figure 16 A block diagram of an image processing apparatus according to one embodiment of this application is shown;

[0037] Figure 17 A structural block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0039] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0041] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0043] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0044] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0045] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0046] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0047] To address the issues with background technology, electronic devices can perform intrinsic decomposition on the original portrait image to obtain material information, and then use rendering techniques to obtain shadow and non-shadow areas under specified lighting. The original portrait image, the shadow and non-shadow areas are then combined to obtain the final blended image, which shows a change in the background compared to the original portrait image. However, this method requires significant time and resources to render the shadow and non-shadow areas under specified lighting, resulting in a long rendering time and low efficiency in obtaining the blended image.

[0048] Based on this, the inventors proposed an image processing method, apparatus, electronic device, and computer-readable storage medium. The method directly determines the target adjustment pixel value from a preset lookup table using the pixel value of each pixel in the foreground region, and adjusts the pixel value of each pixel in the foreground region to the corresponding target adjustment pixel value to obtain the adjusted foreground region. Then, the adjusted foreground region is fused with the target background image to obtain a fused image. This method does not require rendering shadow and non-shadow areas based on the depth map and lighting conditions of the target image, saving significant rendering time and improving the processing efficiency of the foreground region, thereby increasing the efficiency of obtaining the fused image.

[0049] The solutions provided in this application involve artificial intelligence technologies such as semantic segmentation and image fusion, which are specifically illustrated through the following embodiments:

[0050] The image processing method provided in this application can be applied to, for example... Figure 1 The system shown may include a server 10 and a user terminal 20.

[0051] The server 10 in the system can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data, artificial intelligence, intelligent transportation, and assisted driving platforms.

[0052] The user terminal 20 in the system can be a smartphone, tablet, laptop, desktop computer, game console, in-vehicle equipment, aircraft, wearable device, and television, but is not limited to these.

[0053] To enable real-time communication between user terminals 20, user terminals 20 and server 10 can be connected via a network. The network can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0054] In this embodiment of the application, the user terminal 20 can send the acquired target background image to be fused to the server 10. The server 10 performs illumination estimation on the target background image to obtain a preset lookup table. The server 10 then returns the preset lookup table and the target background image to the user terminal 20. The user terminal 20 processes the target image according to the preset lookup table and the target background image to obtain the fused image.

[0055] The target background image can refer to an image used as a background. It can be a photographed image of a building, natural landscape, interior scene, or an artificially synthesized image used as a background. The target image can refer to the image to be processed. It can include the target object and its background area, which are distinct from the target background image. The target object can refer to a person, object, or animal; for example, a cat, a person, or a car.

[0056] In another embodiment, the target background image to be fused can be obtained by the user terminal 20, the user terminal 20 performs illumination estimation on the target background image to obtain a preset lookup table, and then the user terminal 20 processes the target image according to the preset lookup table and the target background image to obtain the fused image.

[0057] In another embodiment, the target background image to be fused can be obtained through server 10 (or the target background image to be fused can be obtained through user terminal 20 and sent to server 10), and server 10 can perform illumination estimation on the target background image to obtain a preset lookup table; when user terminal 20 obtains the target image, it sends the target image to server 10, and server 10 processes the target image according to the preset lookup table and the target background image to obtain the fused image, and feeds back the fused image to user terminal 20.

[0058] For ease of description, the following embodiments are illustrated using the example of an image processing method being executed by an electronic device.

[0059] The solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0060] Please see Figure 2 , Figure 2 This application illustrates a flowchart of an image processing method according to an embodiment of the present application. This method can be applied to an electronic device, which can be... Figure 1 The method, which includes the user terminal 20 or server 10 shown, comprises:

[0061] S110. If a target object exists in the target image, segment the target image to obtain a foreground region that includes the target object.

[0062] In this embodiment, the target image can be a single image or a video frame determined from video information. For example, if the duration of the video information is 60 seconds and each second includes 40 frames, then the target image can refer to the 2400 frames included in the video information.

[0063] For each acquired target image, determine whether a target object exists in the image. This can be done by using a semantic segmentation model to identify the target image and determine whether a target object exists. The semantic segmentation model can be obtained by training a neural network model based on sample images containing target objects. The semantic segmentation model can also be a general model such as DeeLab V3, FCN, UNet, SegNet, etc.

[0064] If a target object exists in the target image, the target image is segmented to obtain a foreground region including the target object and a background region excluding the target object. This segmentation can be performed using a pre-defined semantic segmentation network, where the pre-defined semantic segmentation network can refer to the aforementioned semantic segmentation model.

[0065] For example, the target object is user A, and the target image is an image taken by user A in valley B, such as... Figure 3 As shown, at this point, the target image is segmented using a semantic segmentation network, resulting in a foreground region that includes only user A and a background region that does not include user A. The foreground region is as follows: Figure 4 As shown, the background area is as follows Figure 5 As shown.

[0066] S120. Based on the pixel value of each pixel in the foreground region, determine the target adjustment pixel value corresponding to each pixel from a preset lookup table. The preset lookup table includes the adjustment pixel values ​​corresponding to different pixel values. The preset lookup table is obtained after performing illumination estimation on the target background image to be fused.

[0067] Illumination estimation can be performed on the target background image to obtain a preset lookup table. Illumination estimation can refer to mapping the target background image to a spherical background image, and then processing the spherical background image to obtain the preset lookup table.

[0068] The preset lookup table can include multiple pixel values ​​and their corresponding adjustment pixel values. For example, the preset lookup table may include 256 pixel values ​​from 0 to 255, with each pixel value corresponding to an adjustment pixel value. The adjustment pixel value corresponding to each pixel in the foreground region can be determined from the preset lookup table as the target adjustment pixel value based on its pixel value. For example, if pixel value B is b1, and the adjustment pixel value corresponding to pixel value b1 in the preset lookup table is b2, then the target adjustment pixel value for pixel B is b2.

[0069] In one implementation, both the target image and the target background image are RGB images. Each pixel is composed of R pixels (red pixels), G pixels (green pixels), and B pixels (blue pixels). The pixel value of each pixel includes the pixel values ​​corresponding to the R pixels, G pixels, and B pixels respectively. The preset lookup table includes the adjusted pixel value corresponding to each R pixel, the adjusted pixel value corresponding to each G pixel, and the adjusted pixel value corresponding to each B pixel. Therefore, the determined target adjusted pixel value includes the target adjusted pixel values ​​corresponding to the R pixels, G pixels, and B pixels respectively.

[0070] For example, if pixel c1 has pixel values ​​d1 for R, d2 for G, and d3 for B, in the preset lookup table, the adjusted pixel value corresponding to pixel value d1 for R is e1, the adjusted pixel value corresponding to pixel value d2 for G is e2, and the adjusted pixel value corresponding to pixel value d3 for B is e3. Then the target adjusted pixel values ​​for pixel c1 are e1 for R, e2 for G, and e3 for B.

[0071] S130. Adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel to obtain the adjusted foreground region.

[0072] For each pixel in the foreground region, the pixel value of that pixel is adjusted to the corresponding target adjustment pixel value. By traversing all pixels in the foreground region, the adjusted foreground region is obtained. The lighting conditions of the adjusted foreground region are matched with the lighting conditions of the target background image, thereby realizing the relighting processing of the foreground region.

[0073] For example, the foreground region includes 5000 pixels, each pixel including R pixels, G pixels, and B pixels. A preset lookup table includes the adjustment pixel value corresponding to each R pixel, the adjustment pixel value corresponding to each G pixel, and the adjustment pixel value corresponding to each B pixel. For pixel F, based on the pixel value of the pixel's R pixel, the corresponding target adjustment pixel value f1 for the R pixel is determined; based on the pixel value of the pixel's G pixel, the corresponding target adjustment pixel value f2 for the G pixel is determined; and based on the pixel value of the pixel's B pixel, the corresponding target adjustment pixel value f3 for the B pixel is determined. Then, the pixel value of the pixel's R pixel is adjusted to f1, the pixel value of the pixel's G pixel is adjusted to f2, and the pixel value of the pixel's B pixel is adjusted to f3 to obtain the adjusted pixel value corresponding to pixel F. This process is repeated for all 5000 pixels in the foreground region to obtain the adjusted foreground region.

[0074] S140. Using the target background image as the background of the target image, the adjusted foreground area and the target background image are fused to obtain a fused image.

[0075] The process of blending the adjusted foreground region with the target background image refers to directly superimposing the adjusted foreground region onto the target background image, where the pixel values ​​of the superimposed parts are taken from the pixel values ​​of the adjusted foreground region, and the pixel values ​​of the non-superimposed parts are taken from the pixel values ​​of the target background image.

[0076] As one implementation, before S140, setting position information for the foreground region of the target object can be received, and the superposition position information of the adjusted foreground region in the target background image can be determined according to the setting position information. The adjusted foreground region and the target background image can be fused according to the superposition position information so that the target object is located at the position corresponding to the superposition position information in the fused image.

[0077] For example, if the location information is set to the upper right corner of the target background image, then the adjusted foreground area and the target background image are fused together according to the set location information to obtain a fused image, in which the target object is located at the upper right corner.

[0078] In this embodiment, the target adjustment pixel value is directly determined from a preset lookup table using the pixel value of each pixel in the foreground region. The pixel value of each pixel in the foreground region is then adjusted to the corresponding target adjustment pixel value to obtain the adjusted foreground region. The adjusted foreground region is then fused with the target background image to obtain a fused image. This eliminates the need to render shadow and non-shadow areas based on the depth map and lighting conditions of the target image, saving a significant amount of rendering time and thus improving the processing efficiency of the foreground region, which in turn improves the efficiency of obtaining the fused image.

[0079] Compared to existing portrait relighting techniques, this application does not rely on the depth information of the target image, thus avoiding the high noise levels caused by depth information processing and reducing the impact of noise on the fused image. It also eliminates the need for manually designing ambient lighting maps, reducing manual processing steps and improving the efficiency of obtaining the fused image. Furthermore, it saves the cost of manually designing ambient lighting maps, thereby reducing the overall cost of obtaining the fused image.

[0080] Please see Figure 6 , Figure 6 This application illustrates a flowchart of an image processing method according to another embodiment of the present application. This method can be applied to an electronic device, which can be... Figure 1 The method, which includes the user terminal 20 or server 10 shown, comprises:

[0081] S210. If a target object exists in the target image, segment the target image to obtain a foreground region that includes the target object.

[0082] S220. Based on the pixel value of each pixel in the foreground region, determine the target adjustment pixel value corresponding to each pixel from a preset lookup table. The preset lookup table includes the adjustment pixel values ​​corresponding to different pixel values. The preset lookup table is obtained after performing illumination estimation on the target background image to be fused.

[0083] S230. Adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel to obtain the adjusted foreground region.

[0084] The descriptions of S210-S230 are the same as those of S110-S130 above, and will not be repeated here.

[0085] S240. Based on the target image, obtain the visual contour image of the corresponding foreground region; and perform Gaussian blur processing on the visual contour image to obtain the blur parameters of the corresponding visual contour image.

[0086] A visualized contour image can be divided into a foreground region and a background region by using the contour line of the target object as the dividing line. The pixel value of all pixels in the foreground region is a fixed value x (e.g., 255), and the pixel value of all pixels in the background region is y (0), and x and y are different.

[0087] For example, target image such as Figure 3 As shown, the target object is a person. Figure 3 The foreground region corresponding to the target image in the image is as follows: Figure 4 As shown, the background area is as follows Figure 5 As shown, the visualized contour image is Figure 7 As shown.

[0088] The target image can be processed using the semantic segmentation model described above to obtain a foreground region of the target object and a visualized contour image.

[0089] Gaussian blurring is applied to the visualized contour image corresponding to the foreground region to obtain the blur parameters of the visualized contour image. These blur parameters can include the blur parameters corresponding to each pixel in the visualized contour image. The blur parameter for each pixel far from the contour line in the foreground portion (the region corresponding to the foreground region) of the visualized contour image is typically 1. The blur parameter for each pixel far from the contour line in the background portion (the region corresponding to the background region) of the visualized contour image is typically 0. The blur parameter for pixels close to the contour line is typically a value between 0 and 1.

[0090] S250. Based on the blur parameters of the visualized contour image, the target background image, and the foreground region, a fused image is obtained.

[0091] The target background image can be processed according to the blur parameters of the visualized contour image to obtain the processed target background image. The foreground region can also be processed according to the blur parameters of the visualized contour image to obtain the processed foreground region. Then, the processed target background image and the processed foreground region are fused to obtain a fused image. The process of fusion between the processed target background image and the processed foreground region is similar to the fusion process in S140 above, and will not be described again.

[0092] In one implementation, S250 includes: determining the blur parameters of each pixel in the target background image and the blur parameters of each pixel in the foreground region based on the blur parameters corresponding to each pixel in the visualized contour image; replacing the pixel value of each pixel in the target background image with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed background image corresponding to the target background image; replacing the pixel value of each pixel in the foreground region with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed foreground region corresponding to the foreground region; and performing a fusion process on the preprocessed background image and the preprocessed foreground region to obtain a fused image.

[0093] In a visualized contour image, the pixels in the foreground part correspond one-to-one with the pixels in the foreground region. The blur parameter of each pixel in the foreground part of the visualized contour image is used as the blur parameter of the corresponding pixel in the foreground region.

[0094] Based on the location information of the foreground area, the target background area corresponding to the background area can be determined from the target background image.

[0095] If the target background image and the target image are the same size, then the pixels in the background part of the visualized contour image correspond one-to-one with the pixels in the target background region. In this case, the difference between a fixed value (e.g., 1) and the blur parameter of each pixel in the background part of the visualized contour image is used as the blur parameter of the pixel in the target background region corresponding to that pixel in the background part.

[0096] If the target background image is larger than the target image size, then the pixels in the target background region that correspond one-to-one with the pixels in the background part of the visualized contour image are determined. The difference between a fixed value (e.g., 1) and the blur parameter of each pixel in the foreground part of the visualized contour image is used as the blur parameter of the pixel in the target background region that corresponds to that pixel in the background part. The blur parameter of the pixels in the target background region that do not correspond to the pixels in the background part of the visualized contour image is determined to be 1.

[0097] If the target background image is smaller than the target image size, then the pixels in the target background region that correspond one-to-one with the pixels in the background part of the visualized contour image are determined. The difference between a fixed value (e.g., 1) and the blur parameter of each pixel in the foreground part of the visualized contour image is used as the blur parameter of the pixel in the target background region that corresponds to that pixel in the background part.

[0098] The process of processing the target background image and foreground region by visualizing the blur parameters of the contour image can be expressed as Formula 1, as follows:

[0099] Formula 1 is:

[0100] I=(1-α)I bg +αI relight

[0101] Among them, I bg For the target background image, I relight Let α be the foreground region, α be the blur parameter of the visualized contour image, and I be the fused image.

[0102] The pixel value of each pixel in the foreground region is replaced by the product of the pixel value and the corresponding blur parameter, resulting in a preprocessed foreground region. Similarly, the pixel value of each pixel in the target background image is replaced by the product of the pixel value and the corresponding blur parameter, resulting in a preprocessed background image. Then, the preprocessed background image and the preprocessed foreground region are fused to obtain a fused image. The process of fusing the preprocessed background image and the preprocessed foreground region is similar to the fusion process in S140 and will not be described further.

[0103] In this embodiment, by using the first blur parameter of the visualized contour image, the target background image and the foreground region are processed, thereby achieving edge blurring of the foreground region of the target object. This makes the transition between the foreground and background parts in the fused image more harmonious, eliminates the sense of separation between the foreground and background parts in the fused image, and improves the fusion effect of the fused image.

[0104] Please see Figure 8 , Figure 8 This invention illustrates a flowchart of an image processing method according to another embodiment of the present application. This method can be applied to an electronic device, which can be... Figure 1 The method, which includes the user terminal 20 or server 10 shown, comprises:

[0105] S310. Obtain the target background image; perform spherical mapping on the target background image to obtain a spherical background image.

[0106] The target background image is a planar image. Spherical mapping of the target background image can refer to mapping the target background image to a spherical image, which is then used as the spherical background image.

[0107] In one implementation, S310 includes: acquiring a target background image; constructing a spherical model, wherein the number of spherical points in the spherical model is the same as the number of pixels in the target background image; determining spherical points from the spherical model corresponding to each pixel in the target background image; and adjusting the pixel value of each spherical point to the pixel value of the pixel corresponding to that spherical point in the target background image to obtain a spherical background image.

[0108] For example, the target background image includes 1000 pixels, and the constructed spherical model includes 1000 spherical points. Then, the spherical points corresponding to each pixel in the target background image are determined from the spherical model to establish a one-to-one correspondence between pixels in the target background image and spherical points in the spherical model. Based on this correspondence, the pixel value of each spherical point is adjusted to the pixel value of its corresponding pixel in the target background image, thus obtaining the spherical background image. Figure 9 As shown, the target background image is a planar image. Mapping it to a spherical image yields the following result: Figure 10 The spherical background image shown.

[0109] Determining the spherical point corresponding to each pixel in the target background image from the spherical model includes: obtaining the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; determining the yaw angle and pitch angle of each pixel in the target background image based on the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; and determining the spherical point corresponding to each pixel from the spherical model based on the yaw angle and pitch angle of each pixel in the target background image.

[0110] The process of determining the spherical point corresponding to each pixel in the target background image from the spherical model can be expressed as Formula 2, which is:

[0111]

[0112] Where, x p and y p θ represents the x-coordinate and y-coordinate of the p-th pixel in the target background image, respectively. q and φ q are the pitch angle and yaw angle of the q-th spherical point in the spherical model, respectively. The q-th spherical point is the spherical point corresponding to the p-th pixel. w is the width of the background image and h is the height of the background image.

[0113] According to Formula 2 above, the θ corresponding to the p-th pixel in the background image can be determined. q and φ q , θ q and φ q The spherical point represented is used as the spherical point corresponding to the p-th pixel.

[0114] S320. Determine the spherical harmonic basis parameters based on the spherical background image and the preset spherical harmonic basis functions.

[0115] The spherical background image is processed by a preset spherical harmonic basis function to obtain the spherical harmonic basis parameters. In this application, the preset spherical harmonic basis function is as follows:

[0116]

[0117]

[0118]

[0119] Where m is the order of the preset spherical harmonic basis function, l is the degree of the preset spherical harmonic basis function, and θ and φ are the pitch angle and yaw angle of any spherical point in the spherical background image, respectively. In this application, l takes values ​​of 0, 1, and 2, and m takes values ​​of [-l, l]. That is, in this embodiment, the preset spherical harmonic basis function includes 9 sets of spherical harmonic basis functions.

[0120] In one implementation, S320 may include: determining the function operation result of each spherical point in the spherical background image by using a preset spherical harmonic basis function; calculating the product of the function operation result of each spherical point in the spherical background image and the pixel value of that spherical point as the first product result of that spherical point; and summing the first product results of all spherical points in the spherical background image to obtain the spherical harmonic basis parameters.

[0121] The process of determining the spherical harmonic basis parameters described above can be expressed as Formula 3, as follows:

[0122]

[0123] Among them, y l,m Let θ be the spherical harmonic basis parameter. i Let φ be the pitch angle of the i-th spherical point in the spherical background image. i Let P be the yaw angle of the i-th spherical point, N be the total number of spherical points in the spherical background image, and P be the yaw angle of the i-th spherical point. l,m For the pre-defined spherical harmonic basis functions, P l,m (θ i ,φ i ) represents the result of the operation of the preset spherical harmonic basis function on the i-th spherical point, m represents the order of the preset spherical harmonic basis function, l represents the degree of the preset spherical harmonic basis function, and I represents the degree of the preset spherical harmonic basis function. sphere (θ i ,φ i ) represents the pixel value of the i-th spherical point.

[0124] S330. Based on the spherical harmonic basis parameters and the preset spherical harmonic basis functions, a preset lookup table is obtained.

[0125] After obtaining the spherical harmonic basis parameters, a preset lookup table is obtained using the spherical harmonic basis parameters and the preset spherical harmonic basis functions. Specifically, the natural color lookup table can be obtained first, and then the preset lookup table can be obtained based on the preset spherical harmonic basis functions, the spherical harmonic basis parameters, and the natural color lookup table.

[0126] The natural color lookup table refers to a color lookup table in its natural state. The natural color lookup table includes multiple pixel values ​​and their corresponding adjustment pixel values. In the natural color lookup table, the adjustment pixel value corresponding to any pixel value is the pixel value itself. For example, the adjustment pixel value corresponding to pixel value g1 in the natural color lookup table is also g1.

[0127] It is understandable that if both the target image and the target background image are RGB images, and each pixel is composed of R pixels, G pixels and B pixels, the natural color lookup table includes the adjusted pixel value corresponding to each R pixel, the adjusted pixel value corresponding to each G pixel and the adjusted pixel value corresponding to each B pixel.

[0128] As one implementation method, a preset lookup table is obtained based on a preset spherical harmonic basis function, spherical harmonic basis parameters, and a natural color lookup table. This includes: obtaining the normal vector of the corresponding target background image; obtaining the direction vector corresponding to each spherical point in the spherical background image; determining the calculation result for each spherical point in the spherical background image using the preset spherical harmonic basis function; calculating the product of the calculation result, the direction vector, the spherical harmonic basis parameters, and the normal vector for each spherical point in the spherical background image, as the second product result for that spherical point; summing the second product results for all spherical points in the spherical background image to obtain a summation result; calculating the product of the summation result and the adjusted pixel value for each pixel value in the natural color lookup table, as the preset adjusted pixel value for that pixel value; and obtaining the preset lookup table based on each pixel value in the natural color lookup table and the corresponding preset adjusted pixel value.

[0129] The process of determining the above-mentioned preset lookup table can be expressed as Formula 4, as follows:

[0130]

[0131] Among them, y l,m Let θ be the spherical harmonic basis parameter. i Let φ be the pitch angle of the i-th spherical point in the spherical background image. i Let P be the yaw angle of the i-th spherical point, N be the total number of spherical points in the spherical background image, and P be the yaw angle of the i-th spherical point. l,m For the pre-defined spherical harmonic basis functions, P l,m (θ i ,φ i) represents the result of the operation of the preset spherical harmonic basis function on the i-th spherical point, l represents the degree of the preset spherical harmonic basis function, n represents the normal vector (in this application, n can be (0, 0, 1)), ω i Let i be the direction vector of the i-th point on the sphere. For natural lookup tables, I LUT This is a preset lookup table.

[0132] S340. If a target object exists in the target image, segment the target image to obtain a foreground region that includes the target object.

[0133] S350. Based on the pixel value of each pixel in the foreground region, determine the target adjustment pixel value corresponding to each pixel from a preset lookup table. The preset lookup table includes the adjustment pixel values ​​corresponding to different pixel values. The preset lookup table is obtained after performing illumination estimation on the target background image to be fused.

[0134] S360. Adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel to obtain the adjusted foreground region.

[0135] S370. Using the target background image as the background of the target image, the adjusted foreground area and the target background image are fused together to obtain a fused image.

[0136] The descriptions of S340-S370 are the same as those of S110-S140 above, and will not be repeated here.

[0137] In this embodiment, the spherical background image is processed according to the preset spherical harmonic basis function to obtain the spherical harmonic basis parameters. Then, a preset lookup table is obtained according to the preset spherical harmonic basis function, the spherical harmonic basis parameters, and the natural color lookup table. The preset lookup table can accurately reflect the adjusted pixel values ​​corresponding to each pixel value when the target background image is used as the background. This makes the adjusted foreground area obtained according to the preset lookup table more consistent with the lighting conditions of the target background image, thereby improving the fusion effect of the fused image.

[0138] In this embodiment, starting with an RGB target background image, an automated method for spherical harmonic illumination estimation based on RGB images is proposed to obtain high-quality data on complex illumination. This makes the preset lookup table more suitable for the illumination conditions of the target background image, resulting in a more accurate fused image.

[0139] To facilitate understanding of the solutions provided in the embodiments of this application, the image processing method of this application will be described below with reference to an exemplary scenario and accompanying drawings. In this example scenario, the target object is a person wearing glasses.

[0140] like Figure 11As shown, in this example scenario, the image processing method of this application is implemented through an illumination estimation module and a portrait background fusion module. The illumination estimation module is used to obtain a preset LUT (i.e., the preset lookup table in the above embodiment), and the portrait background fusion module is used to obtain a fused image.

[0141] 1. Processing procedure of the illumination estimation module

[0142] Obtain the background image: Obtain the image corresponding to valley M as the target background image m1, as shown in the image below. Figure 12 As shown.

[0143] Illumination parameter estimation: Construct a spherical model, and determine the spherical point corresponding to each pixel in the target background image in the spherical model using Formula 2 above; then adjust the pixel value of each spherical point to the pixel value of the pixel corresponding to that spherical point to obtain the spherical background image m2. The spherical points in the spherical background image m2 correspond one-to-one with the pixels in the background image m1; then, process the spherical background image m2 using the preset spherical harmonic basis function and Formula 3 to obtain the spherical harmonic basis parameters, which serve as the illumination parameters corresponding to valley M.

[0144] Spherical harmonic LUT rendering: First, obtain the natural color lookup table, the normal vector of the background image m1, and the direction vector of each spherical point in the spherical background image m2. Then, according to Formula 4, the preset spherical harmonic basis function, the spherical harmonic basis parameter, the natural color lookup table, the direction vector, and the normal vector, obtain the preset lookup table, which is the preset LUT obtained by rendering.

[0145] 2. Processing procedure of the portrait background blending module

[0146] Acquire portrait video stream: Acquire the portrait video stream NL to be processed;

[0147] Semantic segmentation: A pre-built semantic segmentation network is used to segment video frames N (which can be any video frame containing a human image) within the human image video stream NL. Figure 13 Semantic segmentation is performed on the image region n1 (as shown), resulting in the segmented image region n1 (the image region is the image region corresponding to the person wearing glasses; this image region is the foreground region segmented in the above embodiment, and image region n1 is as shown). Figure 14 (as shown);

[0148] LUT Transformation: Based on the pixel value of each pixel in the portrait region n1, the target adjustment pixel value of each pixel in the corresponding portrait region n1 is determined from the preset LUT, and the pixel value of each pixel in the portrait region n1 is adjusted to the target adjustment pixel value corresponding to that pixel to obtain the adjusted portrait region n2.

[0149] Background fusion: Gaussian blurring is applied to the visualized contour image n3 corresponding to the portrait region n1 to obtain the first blur parameter corresponding to n3. Based on the first blur parameter, Formula 1, the target background image m1, and the portrait region n2, the fused image is obtained, as shown in the figure. Figure 15 As shown, this fused image serves as the final portrait fusion result.

[0150] The above process is performed on each video frame containing a human image in the human image video stream to obtain the human image fusion result corresponding to each video frame, so as to obtain the background replacement video stream of the corresponding human image video stream. In the background replacement video stream, the video frames containing human images are all based on the background image m1.

[0151] Please see Figure 16 , Figure 16 This illustration shows a block diagram of an image processing apparatus according to one embodiment of the present application. The electronic device may be... Figure 1 The device 1100 includes: a user terminal 20 or a server 10.

[0152] The segmentation module 1110 is used to segment the target image to obtain a foreground region including the target object if a target object exists in the target image;

[0153] The determination module 1120 is used to determine the target adjustment pixel value corresponding to each pixel in the foreground region from a preset lookup table based on the pixel value of each pixel in the foreground region. The preset lookup table includes the adjustment pixel values ​​corresponding to different pixel values. The preset lookup table is obtained after performing illumination estimation on the target background image to be fused.

[0154] The adjustment module 1130 is used to adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel, so as to obtain the adjusted foreground region.

[0155] The fusion module 1140 is used to use the target background image as the background of the target image, and to perform fusion processing on the adjusted foreground area and the target background image to obtain a fused image.

[0156] Optionally, the fusion module 1140 is further configured to obtain a visual contour image of the corresponding foreground region based on the target image; perform Gaussian blur processing on the visual contour image to obtain the first blur parameter of the corresponding visual contour image; and obtain a fused image based on the blur parameter of the visual contour image, the target background image, and the foreground region.

[0157] Optionally, the blur parameters of the visualized contour image include the blur parameters corresponding to each pixel in the visualized contour image; the fusion module 1140 is further configured to determine the blur parameters of each pixel in the target background image and the blur parameters of each pixel in the foreground region based on the blur parameters corresponding to each pixel in the visualized contour image; replace the pixel value of each pixel in the target background image with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed background image of the corresponding target background image; replace the pixel value of each pixel in the foreground region with the product of the pixel value and the blur parameter corresponding to that pixel to obtain a preprocessed foreground region of the corresponding foreground region; and perform fusion processing on the preprocessed background image and the preprocessed foreground region to obtain a fused image.

[0158] Optionally, the device further includes a lookup table acquisition module for acquiring a target background image; performing spherical mapping on the target background image to obtain a spherical background image; determining spherical harmonic basis parameters based on the spherical background image and a preset spherical harmonic basis function; and obtaining a preset lookup table based on the spherical harmonic basis parameters and the preset spherical harmonic basis function.

[0159] Optionally, the lookup table module is also used to construct a spherical model, the number of spherical points in the spherical model being the same as the number of pixels in the target background image; determine the spherical points corresponding to each pixel in the target background image from the spherical model; and adjust the pixel value of each spherical point to the pixel value of the corresponding pixel in the target background image to obtain the spherical background image.

[0160] Optionally, the lookup table acquisition module is also used to obtain the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; determine the yaw angle and pitch angle of each pixel in the target background image based on the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; and determine the spherical point corresponding to the pixel from the spherical model based on the yaw angle and pitch angle of each pixel in the target background image.

[0161] Optionally, the lookup table acquisition module is also used to determine the function operation result of each spherical point in the spherical background image through a preset spherical harmonic basis function; calculate the product of the function operation result of each spherical point in the spherical background image and the pixel value of the spherical point as the first product result of the spherical point; and sum the first product results of all spherical points in the spherical background image to obtain the spherical harmonic basis parameters.

[0162] Optionally, the lookup table acquisition module is also used to acquire a natural color lookup table; and to obtain a preset lookup table based on the preset spherical harmonic basis functions, spherical harmonic basis parameters and the natural color lookup table.

[0163] Optionally, the natural color lookup table includes adjusted pixel values ​​corresponding to multiple pixel values; the lookup table acquisition module is also used to obtain the normal vector of the corresponding target background image; obtain the direction vector corresponding to each spherical point in the spherical background image; determine the operation result of each spherical point in the spherical background image through a preset spherical harmonic basis function; calculate the product of the operation result of each spherical point in the spherical background image, the direction vector of the spherical point, the spherical harmonic basis parameter, and the normal vector, as the second product result of the spherical point; sum the second product results of all spherical points in the spherical background image to obtain the summation result; calculate the product of the summation result and the adjusted pixel value of each pixel value in the natural color lookup table, as the preset adjusted pixel value of the pixel value; and obtain the preset lookup table based on each pixel value in the natural color lookup table and the preset adjusted pixel value corresponding to the pixel value.

[0164] Optionally, the segmentation module 1110 is further configured to segment the target image using a preset semantic segmentation network if a target object exists in the target image, thereby obtaining a foreground region including the target object.

[0165] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0166] Figure 17 This diagram illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device may be... Figure 1 User terminal 20 or server 10, Figure 17 The computer system 1300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 17 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0168] In some embodiments, the following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0169] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0170] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0172] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0173] In another aspect, this application also provides a computer-readable storage medium, which may be included in the user terminal and server described in the above embodiments; or it may exist independently and not assembled into the user terminal and server. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the methods in any of the above embodiments.

[0174] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a user terminal or server reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the user terminal or server to perform the methods of any of the above embodiments.

[0175] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0176] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a user terminal or server to execute the method according to the embodiments of this application.

[0177] It should be noted that in this application, information such as object interaction data, object identity information, and object subscription information requires the authorization of the object. After obtaining the object's authorization for the object interaction data, object identity information, and object subscription information, the above information can be processed, thereby complying with the relevant legal provisions.

[0178] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: If a target object exists in the target image, the target image is segmented to obtain a foreground region including the target object; Based on the pixel value of each pixel in the foreground region, a target adjustment pixel value corresponding to each pixel is determined from a preset lookup table. The preset lookup table includes adjustment pixel values ​​corresponding to different pixel values. The method for obtaining the preset lookup table includes: obtaining a target background image; performing spherical mapping on the target background image to obtain a spherical background image; determining spherical harmonic basis parameters based on the spherical background image and a preset spherical harmonic basis function; and obtaining the preset lookup table based on the spherical harmonic basis parameters and the preset spherical harmonic basis function. The pixel value of each pixel in the foreground region is adjusted to the target adjustment pixel value corresponding to that pixel to obtain the adjusted foreground region; The target background image is used as the background of the target image, and the adjusted foreground region is fused with the target background image to obtain a fused image.

2. The method according to claim 1, characterized in that, The step of fusing the adjusted foreground region with the target background image to obtain a fused image includes: Based on the target image, obtain the visual contour image corresponding to the foreground region; Gaussian blur processing is applied to the visualized contour image to obtain the blur parameters corresponding to the visualized contour image; A fused image is obtained based on the blur parameters of the visualized contour image, the target background image, and the foreground region.

3. The method according to claim 2, characterized in that, The blur parameters of the visualized contour image include the blur parameters corresponding to each pixel in the visualized contour image; obtaining the fused image based on the blur parameters of the visualized contour image, the target background image, and the foreground region includes: Based on the blur parameters corresponding to each pixel in the visualized contour image, determine the blur parameters of each pixel in the target background image and the blur parameters of each pixel in the foreground region. The pixel value of each pixel in the target background image is replaced by the product of the pixel value and the blur parameter corresponding to that pixel, so as to obtain the preprocessed background image corresponding to the target background image. The pixel value of each pixel in the foreground region is replaced by the product of the pixel value and the blur parameter corresponding to that pixel, thus obtaining the preprocessed foreground region corresponding to the foreground region. The preprocessed background image and the preprocessed foreground region are fused together to obtain a fused image.

4. The method according to claim 1, characterized in that, The step of performing spherical mapping on the target background image to obtain a spherical background image includes: Construct a spherical model, wherein the number of spherical points in the spherical model is the same as the number of pixels in the target background image; Determine the spherical point corresponding to each pixel in the target background image from the spherical model; The pixel value of each spherical point is adjusted to the pixel value of the corresponding pixel in the target background image to obtain the spherical background image.

5. The method according to claim 4, characterized in that, Determining the spherical point corresponding to each pixel in the target background image from the spherical model includes: Obtain the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image; Based on the coordinate information of each pixel in the target background image, the width of the target background image, and the height of the target background image, determine the yaw angle and pitch angle corresponding to each pixel in the target background image; Based on the yaw angle and pitch angle of each pixel in the target background image, a spherical point corresponding to that pixel is determined from the spherical model.

6. The method according to claim 1, characterized in that, The step of determining the spherical harmonic basis parameters based on the spherical background image and the preset spherical harmonic basis functions includes: The function operation result for each spherical point in the spherical background image is determined using the preset spherical harmonic basis function; Calculate the product of the function operation result of each spherical point in the spherical background image and the pixel value of that spherical point, and use it as the first product result of that spherical point; The spherical harmonic basis parameters are obtained by summing the first product of all spherical points in the spherical background image.

7. The method according to claim 1, characterized in that, The step of obtaining the preset lookup table based on the spherical harmonic basis parameters and the preset spherical harmonic basis functions includes: Obtain a natural color lookup table; the natural color lookup table includes multiple pixel values ​​and the adjustment pixel values ​​corresponding to each of the multiple pixel values, and in the natural color lookup table, the adjustment pixel value corresponding to any pixel value is the pixel value itself; The preset lookup table is obtained based on the preset spherical harmonic basis functions, the spherical harmonic basis parameters, and the natural color lookup table.

8. The method according to claim 7, characterized in that, The step of obtaining the preset lookup table based on the preset spherical harmonic basis functions, the spherical harmonic basis parameters, and the natural color lookup table includes: Obtain the normal vector corresponding to the target background image; Obtain the direction vector corresponding to each spherical point in the spherical background image; The calculation result for each spherical point in the spherical background image is determined using the preset spherical harmonic basis function; The product of the operation result of each spherical point in the spherical background image, the direction vector of the spherical point, the spherical harmonic basis parameter, and the normal vector is calculated as the second product result of the spherical point. Summing the second product of all spherical points in the spherical background image yields the summation result. The summation result is multiplied by the adjusted pixel value of each pixel value in the natural color lookup table, and this product is used as the preset adjusted pixel value for that pixel value. The preset lookup table is obtained by using each pixel value in the natural color lookup table and the preset adjusted pixel value corresponding to that pixel value.

9. The method according to claim 1, characterized in that, If a target object exists in the target image, the target image is segmented to obtain a foreground region including the target object, including: If a target object exists in the target image, the target image is segmented using a preset semantic segmentation network to obtain a foreground region that includes the target object.

10. An image processing apparatus, characterized in that, The device includes: The segmentation module is used to segment the target image if a target object exists in the target image, thereby obtaining a foreground region including the target object; A determining module is configured to determine the target adjustment pixel value corresponding to each pixel in the foreground region from a preset lookup table based on the pixel value of each pixel. The preset lookup table includes adjustment pixel values ​​corresponding to different pixel values. The method for obtaining the preset lookup table includes: obtaining a target background image; performing spherical mapping on the target background image to obtain a spherical background image; determining spherical harmonic basis parameters based on the spherical background image and a preset spherical harmonic basis function; and obtaining the preset lookup table based on the spherical harmonic basis parameters and the preset spherical harmonic basis function. An adjustment module is used to adjust the pixel value of each pixel in the foreground region to the target adjustment pixel value corresponding to that pixel, so as to obtain the adjusted foreground region; The fusion module is used to use the target background image as the background of the target image, and to perform fusion processing on the adjusted foreground region and the target background image to obtain a fused image.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images