Target foreground and replacement background fusion method, system and device and medium
The three-dimensional model is reconstructed through RGBD images and simulated different lighting conditions, which solves the problem of inconsistent lighting in the cutout technology, and achieves an efficient and stable background replacement effect, improving the visual experience.
Patent Information
- Application Number
- CN202310547081.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
When the existing cutout technology is replaced in the background, the foreground lighting is inconsistent with the new background lighting, resulting in disharmoniousness of the synthetic picture and affecting the user's visual experience. The neural network-based method is computationally large and unstable.
The three-dimensional point cloud and mesh model are reconstructed using RGBD images, and the background lighting conditions are simulated by light sources of different lighting intensities, the target foreground and replacement background are irradiated, and projected to the two-dimensional plane to achieve consistency of lighting conditions.
It realizes the heavy lighting effect with small calculation volume, real-time calculation, high accuracy and strong stability, ensuring consistent lighting of the synthetic picture and improving the visual experience.
Smart Images

Figure CN119006508A_ABST
Abstract
Description
Background Art
[0002] In recent years, with the rapid development of the Internet, logistics, and e-commerce, short video production, live streaming, and entertainment live streaming have gradually become part of people's daily lives. Many anchors will choose to replace the background with a green screen to improve the live broadcast effect. The background replacement method uses the cutout technology, which extracts the foreground of the target of interest in a picture and synthesizes it into a new background image. Unlike image segmentation technology, cutout can make the synthesized edge more natural. Cutout can be further classified into semantic cutout and trimap cutout. Semantic cutout selects the cutout object based on the semantics in the image, while trimap cutout uses additional trimap input to select the cutout object. Using depth to generate the trimap RGBD cutout is one of the ideal trimap cutout technologies. Cutout can achieve arbitrary background replacement, but directly replacing the background will bring new problems, that is, the foreground lighting conditions are inconsistent with the lighting conditions of the new background, resulting in an unharmonious synthesized picture, affecting the user's visual experience.
[0003] At present, many scholars and researchers have studied the problem of the disharmony between the foreground and the background caused by cutouts, and proposed relighting technology, which is to correct the lighting conditions of the foreground based on the lighting conditions of the background. Some scholars have used neural networks and RGB cutouts to relight portraits. This method can only relight portraits, and the method based on neural networks has the problems of large amount of calculation, inability to be real-time, and unstable effect. The difficulty of this problem lies in the fact that the inaccuracy of the cutout range leads to the inaccuracy of the lighting range, and the three-dimensional reconstruction technology based only on RGB has large errors, which affects the final visual effect.
[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0005] To this end, the present invention uses RGBD images to obtain the target foreground, and then reconstructs the three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the target foreground and the mesh model that replaces the background are illuminated, and then projected into a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0006] In a first aspect, the present invention provides a method for fusing a target foreground with a replacement background, characterized in that it comprises the following steps:
[0007] Step S1: Obtain the target foreground and replace the background in the RGBD image;
[0008] Step S2: Reconstructing the three-dimensional point clouds of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information, and establishing mesh models of the target foreground and the replacement background;
[0009] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the replacement background or an equivalent light source;
[0010] Step S4: Project the mesh model onto a two-dimensional plane to obtain a replacement image.
[0011] Optionally, the method for fusing a target foreground with a replacement background is characterized in that step S1 comprises:
[0012] Step S11: Acquire a first RGB image, a first depth image and a replacement background;
[0013] Step S12: obtaining a first target range on the first depth image using a preset distance parameter;
[0014] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range;
[0015] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0016] Optionally, the method for fusing a target foreground with a replacement background is characterized in that step S2 comprises:
[0017] Step S21: Reconstructing the three-dimensional point cloud of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information;
[0018] Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud;
[0019] Step S23: using surface reconstruction technology to obtain mesh models of the target foreground and the replacement background.
[0020] Optionally, the method for fusing a target foreground with a replacement background is characterized in that, in step S21, if the replacement background is an RGB image, a first deep neural network is used to obtain a depth value of the replacement background.
[0021] Optionally, the method for fusing a target foreground with a replacement background is characterized in that step S3 comprises:
[0022] Step S31: determining the positional relationship between the target foreground and the replacement background;
[0023] Step S32: obtaining the lighting conditions of the replacement background;
[0024] Step S33: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the replacement background at a preset position;
[0025] Step S34: using n light sources to illuminate the mesh model simultaneously.
[0026] Optionally, the method for fusing a target foreground with a replacement background is characterized in that step S31 comprises:
[0027] Step S311: adjusting the size of the replacement background so that the display ratio of the replacement background matches that of the target foreground;
[0028] Step S312: adjusting the distance between the replacement background and the target foreground so that there is no overlap between the replacement background and the target foreground.
[0029] Optionally, the method for fusing a target foreground with a replacement background is characterized in that the light source is at least one of a point light source, a surface light source, and a line light source.
[0030] In a second aspect, the present invention provides a system for fusing a target foreground with a replacement background, which is used to implement any of the above methods for fusing a target foreground with a replacement background, and is characterized by comprising:
[0031] The acquisition module is used to obtain the target foreground and replace the background in the RGBD image;
[0032] A three-dimensional reconstruction module, used to reconstruct the three-dimensional point clouds of the target foreground and the replacement background by using the one-to-one corresponding RGB color information and depth information, and to establish mesh models of the target foreground and the replacement background;
[0033] An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source;
[0034] The synthesis module is used to project the mesh model into a two-dimensional plane to obtain a replacement image.
[0035] In a third aspect, the present invention provides a device for fusing a target foreground with a replacement background, characterized in that it comprises:
[0036] processor;
[0037] a memory storing executable instructions of the processor;
[0038] The processor is configured to execute the steps of any one of the above-mentioned methods for fusing the target foreground with the replacement background by executing the executable instructions.
[0039] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of the method for fusing a target foreground with a replacement background as described in any one of the above items are implemented.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] The present invention illuminates the target foreground and the replacement background simultaneously, so that the illumination of the target foreground is the same as that of the replacement background, and the interaction between the target foreground and the replacement background is achieved, so that the shadow of the background is on the target foreground and the shadow of the target foreground is on the replacement background.
[0042] The present invention uses RGBD cutout technology with a more accurate range to achieve more realistic and accurate relighting. Depth information can make the target and range of cutout more clearly defined, so as to obtain more stable cutout content. With the support of depth information, the target foreground obtained by RGBD cutout can be accurately reconstructed in three-dimensional space, providing a better three-dimensional model for subsequent relighting.
[0043] The present invention utilizes a plurality of light sources with different illumination intensities to illuminate the mesh model of the target foreground in three-dimensional space, so as to obtain a more realistic illumination result, and at the same time has the characteristics of small calculation amount and fast operation speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:
[0045] Figure 1 A flowchart of a method for fusing a target foreground with a replacement background according to an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram of a target foreground depth map in an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of a target foreground mesh model in an embodiment of the present invention;
[0048] Figure 4 A flowchart of steps for obtaining a target foreground and replacing a background in an embodiment of the present invention;
[0049] Figure 5 A flowchart of the steps of establishing a mesh model in an embodiment of the present invention;
[0050] Figure 6 This is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention;
[0051] Figure 7 A flowchart of the steps of determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention;
[0052] Figure 8 This is a schematic diagram of the interaction between a target foreground and a replacement background in an embodiment of the present invention;
[0053] Fig. 9 A schematic diagram of the structure of a system for fusing a target foreground with a replacement background in an embodiment of the present invention;
[0054] Fig.10 A schematic diagram of the structure of a device for fusing a target foreground with a replacement background in an embodiment of the present invention; and
[0055] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0057] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0058] The embodiment of the present invention provides a method for fusing a target foreground with a replacement background, aiming to solve the problems existing in the prior art.
[0059] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0060] The present invention uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the target foreground and the mesh model that replaces the background are illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0061] Figure 1 FIG. 1 is a flow chart of the steps of a method for fusing a target foreground with a replacement background in an embodiment of the present invention. Figure 1 As shown, the steps of a method for fusing a target foreground with a replacement background in an embodiment of the present invention include:
[0062] Step S1: Get the target foreground and replace the background in the RGBD image.
[0063] In this step, a neural network model for RGBD cutout is trained using the data collected by the RGBD camera. The RGB (color image), Depth (depth image) and depth image collected by the RGBD camera are used as the input of the RGBD cutout model to obtain the alpha corresponding to the target foreground. The RGBD cutout model can adopt any neural network model in the prior art, and this embodiment does not limit this. The replacement background is a pre-prepared background, which can be an RGB image or an RGBD image. Figure 2As shown, the depth map can accurately identify the target foreground.
[0064] Step S2: Reconstruct the three-dimensional point clouds of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information, and establish mesh models of the target foreground and the replacement background.
[0065] In this step, the color image and the depth image are one-to-one corresponding, so each pixel has unique RGB color information and unique depth information. Based on the RGB color information and depth information, three-dimensional reconstruction can be performed to obtain a three-dimensional point cloud of the target foreground and establish a mesh model of the target foreground. When the replacement background is an RGBD image, the same method can be used to establish a mesh model of the replacement background. When the replacement background is an RGB image, a first deep neural network is used to obtain the depth value of the replacement background, and then the previous steps are used to establish a mesh model of the replacement background. The first deep neural network obtains the depth information of the RGB image by training the RGB image and the RGBD image. The first deep neural network is an adversarial generative network, including a generator and a discriminator. The discriminator makes a judgment based on the depth value in the RGBD image. When performing surface reconstruction to obtain a mesh model of the target foreground, a variety of algorithms can be used, such as Alpha shapes, Ball pivoting, Poisson surface reconstruction, etc. Such as Figure 3 As shown in Figure 3, the mesh model reconstructs the target foreground in three-dimensional space and can be used to better simulate the characteristics of the target foreground.
[0066] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position.
[0067] In this step, the light source is the light source of the replacement background or an equivalent light source. The light source of the replacement background may be one or more light sources, and the types of light sources may be different. At the same time, the light source may be far away from the replacement background and cannot be directly simulated in the reconstructed three-dimensional space. Therefore, the light source used in this step can be the same as the light source of the replacement background or different. When this step uses n light sources with different light intensities, each light source is at a preset position. The preset position is not a fixed position, but an interval, such as the outermost surface of the three-dimensional space. Depending on the lighting conditions of the background, the value of n will vary, such as 3, 4, 5, etc.
[0068] Step S4: Project the mesh model onto a two-dimensional plane to obtain a replacement image.
[0069] In this step, the illuminated mesh model is projected onto a two-dimensional plane, thereby obtaining a two-dimensional image under the same lighting as the background. Then, it is synthesized with the replacement background to obtain a replacement image. The lighting conditions of the foreground in the replacement image are the same as those of the background, thereby making the synthesized image more realistic and avoiding the visual confusion caused by inconsistent lighting. This embodiment can also make the shadow of the target foreground displayed on the replacement background, and the shadows of some objects in the replacement background will also be displayed on the target foreground, greatly increasing the realism of the synthesized image.
[0070] Figure 4 FIG. 1 is a flowchart of steps for obtaining a target foreground and replacing a background in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a target foreground and replacing a background includes:
[0071] Step S11: Acquire a first RGB image, a first depth image, and a replacement background.
[0072] In this step, the first RGB image and the first depth image are aligned at the pixel level. In some scenarios, the first RGB image and the first depth image are described as RGBD images. This embodiment describes them separately to more clearly illustrate the processing of image data. The first frame of image is captured by the depth camera.
[0073] Step S12: obtaining a first target range on the first depth image using a preset distance parameter.
[0074] In this step, the preset distance parameter is a constant used to extract the effective area. For example, if the preset distance parameter is 1.2m, then all pixels with depth values less than 1.2m on the first depth image constitute the first target range. For the same video, the preset distance parameter is a fixed value. For different videos, the preset distance parameter can be different values. The setting of the preset distance can be selected according to the depth camera and the specific application scenario.
[0075] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range.
[0076] In this step, the UNet structure includes three components: encoder, decoder and Skip-Connection in the same layer. The encoder performs downsampling. For example, the input image size is 572*572, the feature map size after two rounds of 3*3 convolution is 568*568, and the output size after 22 maximum pooling is 284*284. The decoder performs upsampling. For example, the decoder is composed of 4 groups of 2*2 transposed convolutions, 3*3 convolutions and a ReLU activation function, and a 1*1 convolution is added to the final output layer. Finally, there is Skip-Connection in the same layer, which is to crop the output of each layer during downsampling and connect it to the upsampling layer of the same layer for fusion. This step also processes the image in combination with the temporal features of the first frame to ensure the stability of target recognition between different frames. To some extent, temporal features can also be regarded as a kind of deep supervision.
[0077] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0078] In this step, different from the screening performed in step S12, this step resets the area within part of the depth value range to increase the selected area and obtain a third target range. The third target range is larger than the second target range. When determining the area to be reset, a fixed depth value range or a depth value range related to the target area can be used. When a fixed depth value range is used, the depth value within 1m can be reset to obtain the third target range. When a depth value range is used that is related to the target area, if the average depth value of the target area is 0.8m, the depth value of 0.6m-1m is reset to obtain the third target range.
[0079] This embodiment uses a preset distance to obtain a first target range, and then uses the UNet network and the Skip-Connection of different dimensional features, combined with the timing characteristics, to identify the second portrait image, and then uses the preset distance parameters to adjust the second target range. It can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, which can realize real-time processing of the video.
[0080] Figure 5 FIG. 1 is a flow chart of steps for establishing a mesh model in an embodiment of the present invention. Figure 5 As shown, a step of establishing a mesh model in an embodiment of the present invention includes:
[0081] Step S21: Reconstruct the three-dimensional point cloud of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information.
[0082] In this step, 3D reconstruction is performed using point cloud. This step simultaneously reconstructs the target foreground and the replacement background. This step projects the pixels according to the depth value of each pixel to obtain the distribution of each pixel in the 3D space. Since the RGB color information contains the lighting information of the current target foreground, the 3D point cloud obtained in this step is also affected by the lighting.
[0083] Step S22: filtering the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud.
[0084] In this step, the filtering process includes de-flickering, de-flying point, missing area filling and other processes. Since there are flying points and missing areas in the depth map. This step can improve the quality of the depth image and ensure the integrity of the image content, making the extraction of the foreground more complete. This step also calculates the reflectivity of each point in the three-dimensional point cloud. The reflectivity can be calculated using any method in the prior art, and this embodiment does not limit this. The reflectivity can be used for calculation when irradiating the mesh model in step S3 to improve the authenticity of the illumination effect.
[0085] Step S23: using surface reconstruction technology to obtain mesh models of the target foreground and the replacement background.
[0086] In this step, the 3D point cloud is sampled, and the surface reconstruction technology is used to obtain the mesh model of the target foreground and the replacement background. Sampling the 3D point cloud can further reduce the amount of calculation and improve processing efficiency. When sampling the 3D point cloud, the sampling ratio varies according to the resolution. The higher the resolution, the lower the sampling ratio. The lower the resolution, the higher the sampling ratio. The mesh model removes the influence of lighting, which can make the lighting in subsequent steps more obvious. Surface reconstruction technology can be selected according to different objects. For example, when using the Ball-Pivoting algorithm, the normal of the point cloud is used for calculation, which can be applied to most scenes of the point cloud and achieve good results.
[0087] This embodiment performs 3D reconstruction through 3D point cloud, processes the point cloud, improves subsequent processing efficiency, and uses surface reconstruction technology to obtain a mesh model, thereby removing the influence of illumination on the target foreground, making subsequent illumination of the mesh model more realistic.
[0088] Figure 6 FIG. 1 is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of irradiating a mesh model includes:
[0089] Step S31: Determine the positional relationship between the target foreground and the replacement background.
[0090] In this step, the positional relationship between the target foreground and the replacement background can be adjusted in various ways, such as adjusting the background position, etc. For example, the replacement background is regarded as an environment map, and the target foreground is located at the center of a cube formed by the environment map.
[0091] Step S32: Acquire the lighting conditions of the replacement background.
[0092] In this step, the illumination condition of the replacement background may be known or unknown. If the illumination condition of the replacement background is known, relevant data is directly obtained. If the illumination condition of the replacement background is unknown, the image is analyzed using parameters such as left-right contrast, top-bottom contrast, average brightness, contrast, covariance matrix, etc. to obtain detailed information of the illumination.
[0093] Step S33: According to the conversion function, n light sources with different illumination intensities are used to simulate the illumination condition of the replacement background at a preset position.
[0094] In this step, the lighting conditions of the replacement background are fitted based on the spherical harmonic function, and then according to the measured conversion function, n light sources with different lighting intensities are used to simulate the lighting conditions of the replacement background at a preset position. The lighting model can adopt Lambert Lighting - Lambert lighting model, Half-Lambert Lighting - Lambert lighting model, Phong Lighting - Feng lighting model, Blinn-Phong Lighting - Blinn-Feng lighting model or other arbitrary models. The specific selection of the lighting model can also be selected according to the application scenario. For example, for the application scenario focusing on portraits, Phong Lighting - Feng lighting model is selected to highlight the highlights and form a better visual impact effect; for the application scenario focusing on product display, Half-Lambert Lighting - Lambert lighting model is selected to highlight the product more completely. In this step, multiple different light sources are used to simulate the lighting conditions of the replacement background. n can be any positive integer. The light source is at least one of a point light source, a surface light source, and a line light source. For example, if four light sources are used for simulation, two point light sources, one surface light source and one line light source can be used for simulation. The intensity of each light source can be different. The area of the same light source can also be different.
[0095] Step S34: using n light sources to illuminate the mesh model simultaneously.
[0096] In this step, according to the illumination model and rendering formula, n light sources are applied to the mesh models of the target foreground and the replacement background. For the target foreground, the illumination is new illumination. For the replacement background, only the shadow part of the target foreground needs to be calculated, and other parts do not need to be changed. Through the illumination in this step, the shadow, color change, etc. of the target foreground can be obtained.
[0097] This embodiment simulates the lighting conditions of the replacement background at a preset position through multiple light sources with different lighting intensities, so as to better simulate the lighting conditions of the replacement background, so that the lighting of the mesh model can be the same as that of the replacement background, and the corresponding lighting model is selected according to different application scenarios to improve the effect of the synthetic image.
[0098] Figure 7 The present invention is a flowchart of a step of determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention.
[0099] like Figure 7 As shown, in an embodiment of the present invention, a step of determining the position relationship between the target foreground and the replacement background includes:
[0100] Step S311: adjusting the size of the replacement background so that the display ratio of the replacement background matches that of the target foreground.
[0101] In this step, the size of the replacement background is adjusted according to the size of the target foreground so that the display ratio of the replacement background matches the target foreground. This step adjusts the size of the replacement background to a preset first ratio. The first ratio is a more appropriate ratio between the replacement background and the target foreground, but it is not the final ratio.
[0102] Step S312: adjusting the distance between the replacement background and the target foreground so that there is no overlap between the replacement background and the target foreground.
[0103] In this step, since the image sizes of the replacement background and the target foreground have been determined, but the ratio of the two may be inconsistent, the distance between the replacement background and the target foreground is adjusted by using the pinhole imaging principle so that the ratio of the target foreground to the replacement background is consistent in the camera coordinate system, which is more in line with the visual characteristics of the human eye. This step adjusts the depth distance between the replacement background and the target foreground so that the ratio of the replacement background to the target foreground on the image changes to reach a second ratio. The difference between the second ratio and the first ratio is small, and no excessive visual conflict will occur.
[0104] This embodiment adjusts the size of the replacement background and the distance between the target foreground and the replacement background so that the ratio of the target foreground to the replacement background is more coordinated, realistic, and in line with the visual characteristics of the human eye.
[0105] Figure 8 FIG. 1 is a schematic diagram of the interaction between a target foreground and a replacement background in an embodiment of the present invention. Figure 8 It can be seen that the target foreground is a human body, the replacement background is a square with a tree, and the light is incident from the right side of the image. The shadow of the human body is reflected on the ground of the replacement background. At the same time, the shadow of the tree is also reflected on the head of the human body. After the replacement background, the target foreground interacts better with the objects in the replacement background, the lighting effect is more obvious, the integrity is stronger, and it is more realistic.
[0106] Fig. 9 FIG. 1 is a schematic diagram of a system for fusing a target foreground with a replacement background in an embodiment of the present invention. Fig. 9 As shown, a system for fusing a target foreground with a replacement background in an embodiment of the present invention includes:
[0107] The acquisition module is used to obtain the target foreground and replace the background in the RGBD image.
[0108] The three-dimensional reconstruction module is used to reconstruct the three-dimensional point clouds of the target foreground and the replacement background by using the one-to-one corresponding RGB color information and depth information, and to establish mesh models of the target foreground and the replacement background.
[0109] An illumination module is used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source.
[0110] The synthesis module is used to project the mesh model into a two-dimensional plane to obtain a replacement image.
[0111] Specifically, the acquisition module obtains the target foreground and replacement background of the RGBD image. The target foreground obtained using depth information is more accurate and can be used for adjustment in the subsequent synthesis process. The three-dimensional reconstruction module reconstructs the three-dimensional information of the target foreground and replacement background, and establishes a mesh model, removing the lighting effect in the RGBD image to obtain accurate information. The illumination module uses the same light source as the replacement background or an equivalent light source to illuminate the mesh model, so that the lighting conditions of the target foreground are the same as the replacement background. The synthesis module projects the mesh model into a two-dimensional plane, and finally generates a two-dimensional image that is well integrated with the replacement background, greatly improving the quality of the fused image.
[0112] This embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the target foreground and the mesh model that replaces the background are illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0113] The embodiment of the present invention also provides a device for fusing a target foreground with a replacement background, comprising a processor and a memory in which executable instructions of the processor are stored. The processor is configured to execute the steps of a method for fusing a target foreground with a replacement background by executing the executable instructions.
[0114] As mentioned above, this embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the target foreground and the mesh model that replaces the background are illuminated, and then projected into a two-dimensional plane. It has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.
[0115] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".
[0116] Fig.10 Schematic diagram of the structure of a device for merging a target foreground with a replacement background in an embodiment of the present invention. Fig.10 The electronic device 600 according to this embodiment of the present invention is described. Fig.10 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0117] like Fig.10 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0118] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method of merging a target foreground with a replacement background. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .
[0119] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0120] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0121] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0122] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.
[0123] In an embodiment of the present invention, a computer-readable storage medium is further provided for storing a program, and when the program is executed, the steps of a method for fusing a target foreground with a replacement background are implemented. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for fusing a target foreground with a replacement background section of this specification.
[0124] As shown above, this embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, and the target foreground and the mesh model that replaces the background are illuminated and then projected onto a two-dimensional plane. It has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.
[0125] Fig.11Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.11 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0126] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0127] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0128] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0129] This embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the target foreground and the mesh model that replaces the background are illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0130] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.
[0131] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A method for fusing a target foreground with a replacement background, characterized in that: The steps include: Step S1: Obtain the target foreground and replace the background in the RGBD image; Step S2: Reconstructing the three-dimensional point clouds of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information, and establishing mesh models of the target foreground and the replacement background; Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the replacement background or an equivalent light source; Step S4: Project the mesh model onto a two-dimensional plane to obtain a replacement image.
2. The method for fusing a target foreground with a replacement background according to claim 1, characterized in that: Step S1 includes: Step S11: Acquire a first RGB image, a first depth image and a replacement background; Step S12: obtaining a first target range on the first depth image using a preset distance parameter; Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range; Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
3. The method for fusing a target foreground with a replacement background according to claim 1, characterized in that: Step S2 includes: Step S21: Reconstructing the three-dimensional point cloud of the target foreground and the replacement background using the one-to-one corresponding RGB color information and depth information; Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud; Step S23: using surface reconstruction technology to obtain mesh models of the target foreground and the replacement background.
4. The method for fusing a target foreground with a replacement background according to claim 3, characterized in that: In step S21, if the replacement background is an RGB image, a first deep neural network is used to obtain a depth value of the replacement background.
5. The method for fusing a target foreground with a replacement background according to claim 1, characterized in that: Step S3 includes: Step S31: determining the positional relationship between the target foreground and the replacement background; Step S32: obtaining the lighting conditions of the replacement background; Step S33: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the replacement background at a preset position; Step S34: using n light sources to illuminate the mesh model simultaneously.
6. A method for fusing a target foreground with a replacement background according to claim 5, characterized in that: The step S31 comprises: Step S311: adjusting the size of the replacement background so that the display ratio of the replacement background matches that of the target foreground; Step S312: adjusting the distance between the replacement background and the target foreground so that there is no overlap between the replacement background and the target foreground.
7. The method for fusing a target foreground with a replacement background according to claim 5, characterized in that: The light source is at least one of a point light source, a surface light source, and a line light source.
8. A system for fusing a target foreground with a replacement background, used to implement the method for fusing a target foreground with a replacement background as claimed in any one of claims 1 to 7, characterized in that: include: The acquisition module is used to obtain the target foreground and replace the background in the RGBD image; A three-dimensional reconstruction module, used to reconstruct the three-dimensional point clouds of the target foreground and the replacement background by using the one-to-one corresponding RGB color information and depth information, and to establish mesh models of the target foreground and the replacement background; An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source; The synthesis module is used to project the mesh model into a two-dimensional plane to obtain a replacement image.
9. A device for fusing a target foreground with a replacement background, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the method for fusing a target foreground with a replacement background as described in any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the method for fusing the target foreground with the replacement background described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method for calculating lighting parameters and carrying out relighting rendering based on image and model
CN105447906A
Face data enhancement method and device, computer equipment and storage medium
CN115239861A
Illumination adjustment method and device for fused image, equipment and medium
CN115293960A
