Method, system and device for changing target foreground illumination condition and medium
Reconstructing a three-dimensional model through RGBD images and simulating light sources with different lighting intensities is solved, and the problem of inconsistent lighting conditions in the cutout technology is achieved, with small calculations, real-time and accurate re-lighting effect, and the visual effect of background replacement is improved.
Patent Information
- Application Number
- CN202310547002.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
When the existing cutout technology is replaced in the background, the foreground lighting conditions are inconsistent with the new background lighting conditions, resulting in disharmoniousness of the synthetic picture and affecting the user's visual experience. The RGB-based re-lighting method is computationally large and unstable.
The three-dimensional point cloud and mesh model are reconstructed using RGBD images, and the background lighting conditions are simulated by light sources of different lighting intensities, the target foreground is irradiated, and projected to the two-dimensional plane to adjust the lighting conditions.
It realizes a heavy lighting effect with small calculation volume, real-time, high accuracy and strong stability, ensuring consistent lighting of the synthetic picture and improving the visual experience.
Smart Images

Figure CN119006691A_ABST
Abstract
Description
Background Art
[0002] In recent years, with the rapid development of the Internet, logistics, and e-commerce, short video production, live streaming, and entertainment live streaming have gradually become part of people's daily lives. Many anchors will choose to replace the background with a green screen to improve the live broadcast effect. The background replacement method uses the cutout technology, which extracts the foreground of the target of interest in a picture and synthesizes it into a new background image. Unlike image segmentation technology, cutout can make the synthesized edge more natural. Cutout can be further classified into semantic cutout and trimap cutout. Semantic cutout selects the cutout object based on the semantics in the image, while trimap cutout uses additional trimap input to select the cutout object. Using depth to generate the trimap RGBD cutout is one of the ideal trimap cutout technologies. Cutout can achieve arbitrary background replacement, but directly replacing the background will bring new problems, that is, the foreground lighting conditions are inconsistent with the lighting conditions of the new background, resulting in an unharmonious synthesized picture, affecting the user's visual experience.
[0003] At present, many scholars and researchers have studied the problem of the disharmony between the foreground and the background caused by cutouts, and proposed relighting technology, which is to correct the lighting conditions of the foreground based on the lighting conditions of the background. Some scholars have used neural networks and RGB cutouts to relight portraits. This method can only relight portraits, and the method based on neural networks has the problems of large amount of calculation, inability to be real-time, and unstable effect. The difficulty of this problem lies in the fact that the inaccuracy of the cutout range leads to the inaccuracy of the lighting range, and the three-dimensional reconstruction technology based only on RGB has large errors, which affects the final visual effect.
[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0005] To this end, the present invention uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0006] In a first aspect, the present invention provides a method for changing a target foreground illumination condition, characterized in that it comprises the following steps:
[0007] Step S1: Obtain the target foreground and replace the background in the RGBD image;
[0008] Step S2: Reconstructing the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establishing a mesh model of the target foreground;
[0009] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the replacement background or an equivalent light source;
[0010] Step S4: Project the mesh model into a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
[0011] Optionally, the method for changing the target foreground illumination condition is characterized in that step S1 comprises:
[0012] Step S11: Acquire a first RGB image, a first depth image and a replacement background;
[0013] Step S12: obtaining a first target range on the first depth image using a preset distance parameter;
[0014] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range;
[0015] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0016] Optionally, the method for changing the target foreground illumination condition is characterized in that step S2 comprises:
[0017] Step S21: Reconstructing a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information;
[0018] Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud;
[0019] Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.
[0020] Optionally, the method for changing the target foreground illumination condition is characterized in that step S3 comprises:
[0021] Step S31: determining the positional relationship between the target foreground and the replacement background;
[0022] Step S32: obtaining the lighting conditions of the replacement background;
[0023] Step S33: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the replacement background at a preset position;
[0024] Step S34: using n light sources to illuminate the mesh model simultaneously.
[0025] Optionally, the method for changing the target foreground lighting condition is characterized in that the light source is at least one of a point light source, a surface light source, and a line light source.
[0026] Optionally, the method for changing the target foreground illumination condition is characterized in that step S31 comprises:
[0027] Step S311: obtaining the three-dimensional information of the replacement background;
[0028] Step S312: adjusting the distance between the replacement background and the target foreground so that the replacement background and the target foreground have the same ratio.
[0029] Optionally, the method for changing the target foreground illumination condition is characterized by further comprising:
[0030] Step S313: adjusting the position of the replacement background so that the replacement background does not block the target foreground in the illumination direction.
[0031] In a second aspect, the present invention provides a system for changing a target foreground illumination condition, which is used to implement any of the above methods for changing a target foreground illumination condition, and is characterized by comprising:
[0032] The acquisition module is used to obtain the target foreground and replace the background in the RGBD image;
[0033] A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground;
[0034] An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source;
[0035] The synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
[0036] In a third aspect, the present invention provides a device for changing a target foreground illumination condition, characterized in that it comprises:
[0037] processor;
[0038] a memory storing executable instructions of the processor;
[0039] The processor is configured to execute any one of the steps of the above-mentioned method for changing the target foreground lighting condition by executing the executable instructions.
[0040] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned methods for changing the target foreground lighting conditions are implemented.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The present invention attempts to use RGBD cutout technology with a more accurate range to achieve more realistic and accurate relighting. Depth information can make the target and range of cutout more clearly defined, so as to obtain more stable cutout content. With the support of depth information, the target foreground obtained by RGBD cutout can be accurately reconstructed in three-dimensional space, providing a better three-dimensional model for subsequent relighting.
[0043] The present invention utilizes a plurality of light sources with different illumination intensities to illuminate the mesh model of the target foreground in three-dimensional space, so as to obtain a more realistic illumination result, and at the same time has the characteristics of small calculation amount and fast operation speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:
[0045] Figure 1 A flowchart of a method for changing the illumination condition of a target foreground according to an embodiment of the present invention;
[0046] Figure 2 is a schematic diagram of a depth map in an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of a mesh model in an embodiment of the present invention;
[0048] Figure 4 A flowchart of steps for obtaining a target foreground and replacing a background in an embodiment of the present invention;
[0049] Figure 5 A flowchart of the steps of establishing a mesh model in an embodiment of the present invention;
[0050] Figure 6 This is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention;
[0051] Figure 7 A flowchart of the steps of determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention;
[0052] Figure 8 Another flow chart of steps for determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention;
[0053] Fig. 9 It is a structural schematic diagram of a system for changing the illumination condition of a target foreground in an embodiment of the present invention;
[0054] Fig.10 is a schematic diagram of the structure of a device for changing the illumination condition of a target foreground according to an embodiment of the present invention; and
[0055] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0057] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0058] The embodiment of the present invention provides a method for changing the illumination condition of a target foreground, aiming to solve the problems existing in the prior art.
[0059] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0060] The present invention uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0061] Figure 1 FIG. 1 is a flow chart of steps of a method for changing the illumination condition of a target foreground according to an embodiment of the present invention. Figure 1 As shown, the steps of a method for changing the target foreground illumination condition in an embodiment of the present invention include:
[0062] Step S1: Get the target foreground and replace the background in the RGBD image.
[0063] In this step, a neural network model for RGBD cutout is trained using the data collected by the RGBD camera. The RGB (color image), Depth (depth image) and depth image collected by the RGBD camera are used as the input of the RGBD cutout model to obtain the alpha corresponding to the target foreground. The RGBD cutout model can adopt any neural network model in the prior art, and this embodiment does not limit this. The replacement background is a pre-prepared background, which can be an RGB image or an RGBD image. Figure 2 As shown, the depth map can accurately identify the target foreground.
[0064] Step S2: Reconstruct the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establish a mesh model of the target foreground.
[0065] In this step, the color map and the depth map are one-to-one corresponding, so each pixel has unique RGB color information and unique depth information. Based on the RGB color information and depth information, three-dimensional reconstruction can be performed to obtain a three-dimensional point cloud of the target foreground and establish a mesh model of the target foreground. When performing surface reconstruction to obtain a mesh model of the target foreground, a variety of algorithms can be used, such as Alpha shapes, Ball pivoting, Poisson surface reconstruction, etc. Figure 3As shown in Figure 3, the mesh model reconstructs the target foreground in three-dimensional space and can be used to better simulate the characteristics of the target foreground.
[0066] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position.
[0067] In this step, the light source is the light source of the replacement background or an equivalent light source. The light source of the replacement background may be one or more light sources, and the types of light sources may be different. At the same time, the light source may be far away from the replacement background and cannot be directly simulated in the reconstructed three-dimensional space. Therefore, the light source used in this step can be the same as the light source of the replacement background or different. When this step uses n light sources with different light intensities, each light source is at a preset position. The preset position is not a fixed position, but an interval, such as the outermost surface of the three-dimensional space. Depending on the lighting conditions of the background, the value of n will vary, such as 3, 4, 5, etc.
[0068] Step S4: Project the mesh model into a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
[0069] In this step, the illuminated mesh model is projected onto a two-dimensional plane to obtain a two-dimensional image under the same lighting as the background. Then it is synthesized with the replacement background to obtain a replacement image. The lighting conditions of the foreground in the replacement image are the same as the background, making the synthesized image more realistic and avoiding visual confusion caused by inconsistent lighting.
[0070] Figure 4 FIG. 1 is a flowchart of steps for obtaining a target foreground and replacing a background in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a target foreground and replacing a background includes:
[0071] Step S11: Acquire a first RGB image, a first depth image, and a replacement background.
[0072] In this step, the first RGB image and the first depth image are aligned at the pixel level. In some scenarios, the first RGB image and the first depth image are described as RGBD images. This embodiment describes them separately to more clearly illustrate the processing of image data. The first frame of image is captured by the depth camera.
[0073] Step S12: obtaining a first target range on the first depth image using a preset distance parameter.
[0074] In this step, the preset distance parameter is a constant used to extract the effective area. For example, if the preset distance parameter is 1.2m, then all pixels with depth values less than 1.2m on the first depth image constitute the first target range. For the same video, the preset distance parameter is a fixed value. For different videos, the preset distance parameter can be different values. The setting of the preset distance can be selected according to the depth camera and the specific application scenario.
[0075] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range.
[0076] In this step, the UNet structure includes three components: encoder, decoder and Skip-Connection in the same layer. The encoder performs downsampling. For example, the input image size is 572*572, the feature map size after two rounds of 3*3 convolution is 568*568, and the output size after 22 maximum pooling is 284*284. The decoder performs upsampling. For example, the decoder is composed of 4 groups of 2*2 transposed convolutions, 3*3 convolutions and a ReLU activation function, and a 1*1 convolution is added to the final output layer. Finally, there is Skip-Connection in the same layer, which is to crop the output of each layer during downsampling and connect it to the upsampling layer of the same layer for fusion. This step also processes the image in combination with the temporal features of the first frame to ensure the stability of target recognition between different frames. To some extent, temporal features can also be regarded as a kind of deep supervision.
[0077] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
[0078] In this step, different from the screening performed in step S12, this step resets the area within part of the depth value range to increase the selected area and obtain a third target range. The third target range is larger than the second target range. When determining the area to be reset, a fixed depth value range or a depth value range related to the target area can be used. When a fixed depth value range is used, the depth value within 1m can be reset to obtain the third target range. When a depth value range is used that is related to the target area, if the average depth value of the target area is 0.8m, the depth value of 0.6m-1m is reset to obtain the third target range.
[0079] This embodiment uses a preset distance to obtain a first target range, and then uses the UNet network and the Skip-Connection of different dimensional features, combined with the timing characteristics, to identify the second portrait image, and then uses the preset distance parameters to adjust the second target range. It can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, which can realize real-time processing of videos.
[0080] Figure 5 FIG. 1 is a flowchart of the steps of establishing a mesh model in an embodiment of the present invention. Figure 5 As shown, a step of establishing a mesh model in an embodiment of the present invention includes:
[0081] Step S21: Reconstruct the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information.
[0082] In this step, 3D reconstruction is performed using point cloud. This step only performs 3D reconstruction on the target foreground. This step projects the pixels according to the depth value of each pixel to obtain the distribution of each pixel in 3D space. Since RGB color information contains the illumination information of the current target foreground, the 3D point cloud obtained in this step is also affected by the illumination of the target foreground.
[0083] Step S22: filtering the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud.
[0084] In this step, the filtering process includes de-flickering, de-flying point, missing area filling and other processes. Since there are flying points and missing areas in the depth map. This step can improve the quality of the depth image and ensure the integrity of the image content, making the extraction of the foreground more complete. This step also calculates the reflectivity of each point in the three-dimensional point cloud. The reflectivity can be calculated using any method in the prior art, and this embodiment does not limit this. The reflectivity can be used for calculation when irradiating the mesh model in step S3 to improve the authenticity of the illumination effect.
[0085] Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.
[0086] In this step, the three-dimensional point cloud is sampled, and the mesh model of the target foreground is obtained using surface reconstruction technology. Sampling the three-dimensional point cloud can further reduce the amount of calculation and improve processing efficiency. When sampling the three-dimensional point cloud, the sampling ratio varies according to the resolution. The higher the resolution, the lower the sampling ratio. The lower the resolution, the higher the sampling ratio. The mesh model removes the lighting effect of the target foreground, which can make the lighting in subsequent steps more obvious. Surface reconstruction technology can be selected according to different objects. For example, when using the Ball-Pivoting algorithm, the point cloud normal is used for calculation, which can be applied to most scenes of the point cloud and achieve good results.
[0087] This embodiment performs 3D reconstruction through 3D point cloud, processes the point cloud, improves subsequent processing efficiency, and uses surface reconstruction technology to obtain a mesh model of the target foreground, thereby removing the influence of illumination on the target foreground and making subsequent illumination of the mesh model more realistic.
[0088] Figure 6 FIG. 1 is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of irradiating a mesh model includes:
[0089] Step S31: Determine the positional relationship between the target foreground and the replacement background.
[0090] In this step, the positional relationship between the target foreground and the replacement background can be adjusted in various ways, such as adjusting the background position, etc. For example, the replacement background is regarded as an environment map, and the target foreground is located at the center of a cube formed by the environment map.
[0091] Step S32: Acquire the lighting conditions of the replacement background.
[0092] In this step, the illumination condition of the replacement background may be known or unknown. If the illumination condition of the replacement background is known, relevant data is directly obtained. If the illumination condition of the replacement background is unknown, the image is analyzed using parameters such as left-right contrast, top-bottom contrast, average brightness, contrast, covariance matrix, etc. to obtain detailed information of the illumination.
[0093] Step S33: According to the conversion function, n light sources with different illumination intensities are used to simulate the illumination condition of the replacement background at a preset position.
[0094] In this step, the lighting conditions of the replacement background are fitted based on the spherical harmonic function, and then according to the measured conversion function, n light sources with different lighting intensities are used to simulate the lighting conditions of the replacement background at a preset position. The lighting model can adopt Lambert Lighting, Half-Lambert Lighting, Phong Lighting, Blinn-Phong Lighting, or any other model. The specific selection of the lighting model can also be selected according to the application scenario. For example, for the application scenario focusing on portraits, Phong Lighting is selected to highlight the highlights and form a better visual impact effect; for the application scenario focusing on product display, Half-Lambert Lighting is selected to highlight the product more completely. In this step, multiple different light sources are used to simulate the lighting conditions of the replacement background. n can be any positive integer. The light source is at least one of a point light source, a surface light source, and a line light source. For example, if four light sources are used for simulation, two point light sources, one surface light source, and one line light source can be used for simulation. The intensity of each light source can be different. The area of the same light source can also be different.
[0095] Step S34: using n light sources to illuminate the mesh model simultaneously.
[0096] In this step, according to the illumination model and rendering formula, n light sources are applied to the mesh model of the target foreground. Through the illumination in this step, shadows, color changes, etc. of the target foreground can be obtained.
[0097] This embodiment simulates the lighting conditions of the replacement background at a preset position through multiple light sources with different lighting intensities, so as to better simulate the lighting conditions of the replacement background, so that the lighting of the mesh model can be the same as that of the replacement background, and the corresponding lighting model is selected according to different application scenarios to improve the effect of the synthetic image.
[0098] Figure 7 The present invention is a flowchart of a step of determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention.
[0099] like Figure 7 As shown, in an embodiment of the present invention, a step of determining the position relationship between the target foreground and the replacement background includes:
[0100] Step S311: Obtaining three-dimensional information of the replacement background.
[0101] In this step, the three-dimensional information of the replacement background can be known or unknown. When the three-dimensional information of the replacement background is known, the three-dimensional data of the replacement background is directly obtained. For example, when the replacement background is RGBD, the three-dimensional information of the replacement background can be directly obtained. When the three-dimensional information of the replacement background is unknown, such as the replacement background is a newly obtained RGB image, it is necessary to analyze the replacement background using a recognition model to obtain the three-dimensional information of the replacement background. The recognition model is a deep neural network model, which obtains the depth data corresponding to each pixel by learning the RGB image. The recognition model can adopt any deep neural network model in the prior art, and this embodiment does not limit this.
[0102] Step S312: adjusting the distance between the replacement background and the target foreground so that the replacement background and the target foreground have the same ratio.
[0103] In this step, since the image sizes of the replacement background and the target foreground have been determined, but the ratio of the two may be inconsistent, the pinhole imaging principle is used to adjust the distance between the replacement background and the target foreground so that in the camera coordinate system, the ratio of the target foreground to the replacement background is consistent, which is more in line with the visual characteristics of the human eye.
[0104] This embodiment obtains three-dimensional information of the replacement background and adjusts the distance between the target foreground and the replacement background so that the ratio of the target foreground to the replacement background is more coordinated and realistic, which conforms to the visual characteristics of the human eye.
[0105] Figure 8 FIG. 1 is another flow chart of steps for determining the position relationship between the target foreground and the replacement background in an embodiment of the present invention. Figure 8 As shown, compared with the above-mentioned embodiment, another step of determining the position relationship between the target foreground and the replacement background in the embodiment of the present invention further includes:
[0106] Step S313: adjusting the position of the replacement background so that the replacement background does not block the target foreground in the illumination direction.
[0107] In this step, since there may be relatively high objects in the replacement background, some objects in the replacement background may block the target foreground in the direction of illumination in three-dimensional space. Therefore, the replacement background needs to be adjusted, for example, the distance between the replacement background and the target foreground may be adjusted, the replacement background may be rotated, flipped, or enlarged.
[0108] This embodiment avoids the possible influence of the replacement background on the foreground by adjusting the replacement background, avoids the conflict between the target foreground and the replacement background, and further ensures the consistency of the foreground and the background.
[0109] Fig. 9FIG. 1 is a schematic diagram of a system for changing the illumination condition of a target foreground according to an embodiment of the present invention. Fig. 9 As shown, a system for changing target foreground illumination conditions in an embodiment of the present invention includes:
[0110] The acquisition module is used to obtain the target foreground and replace the background in the RGBD image;
[0111] A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground;
[0112] An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source;
[0113] The synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
[0114] Specifically, the acquisition module obtains the target foreground and replacement background of the RGBD image. The target foreground obtained using depth information is more accurate and can be used for adjustment in the subsequent synthesis process. The three-dimensional reconstruction module reconstructs the three-dimensional information of the target foreground and establishes a mesh model, removes the lighting effect in the RGBD image, and obtains accurate original target foreground information. The illumination module uses the same light source as the replacement background or an equivalent light source to illuminate the mesh model, so that the lighting conditions of the target foreground are the same as the replacement background. The synthesis module projects the mesh model into a two-dimensional plane and finally generates a two-dimensional image that is well integrated with the replacement background, greatly improving the quality of the fused image.
[0115] This embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0116] The present invention also provides a device for changing the illumination condition of a target foreground, comprising a processor and a memory storing executable instructions of the processor, wherein the processor is configured to execute the steps of a method for changing the illumination condition of a target foreground by executing the executable instructions.
[0117] As mentioned above, this embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. It has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.
[0118] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".
[0119] Fig.10 Schematic diagram of a device for changing the illumination condition of a target foreground according to an embodiment of the present invention. Fig.10 The electronic device 600 according to this embodiment of the present invention is described. Fig.10 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0120] like Fig.10 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0121] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method for changing the illumination condition of the target foreground. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .
[0122] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202 , and may further include a read-only storage unit (ROM) 6203 .
[0123] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0124] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0125] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.
[0126] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a method for changing the target foreground lighting condition are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for changing the target foreground lighting condition section of this specification.
[0127] As shown above, this embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. It has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.
[0128] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.11As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0129] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0130] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0131] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0132] This embodiment uses RGBD images to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected onto a two-dimensional plane. It has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.
[0133] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.
[0134] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A method for changing the illumination condition of a target foreground, characterized in that: The steps include: Step S1: Obtain the target foreground and replace the background in the RGBD image; Step S2: Reconstructing the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establishing a mesh model of the target foreground; Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the replacement background or an equivalent light source; Step S4: Project the mesh model onto a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
2. A method for changing target foreground illumination conditions according to claim 1, characterized in that: Step S1 includes: Step S11: Acquire a first RGB image, a first depth image and a replacement background; Step S12: obtaining a first target range on the first depth image using a preset distance parameter; Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first RGB image and the first depth image, the first target range is identified to obtain the second target range; Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.
3. A method for changing target foreground illumination conditions according to claim 1, characterized in that: Step S2 includes: Step S21: Reconstructing a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information; Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud; Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.
4. The method for changing the target foreground illumination condition according to claim 1, characterized in that: Step S3 includes: Step S31: determining the positional relationship between the target foreground and the replacement background; Step S32: obtaining the lighting conditions of the replacement background; Step S33: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the replacement background at a preset position; Step S34: using n light sources to illuminate the mesh model simultaneously.
5. The method for changing the target foreground illumination condition according to claim 1, characterized in that: The light source is at least one of a point light source, a surface light source, and a line light source.
6. A method for changing target foreground illumination conditions according to claim 4, characterized in that: The step S31 comprises: Step S311: obtaining three-dimensional information of the replacement background; Step S312: adjusting the distance between the replacement background and the target foreground so that the replacement background and the target foreground have the same ratio.
7. A method for changing target foreground illumination conditions according to claim 6, characterized in that: Also includes: Step S313: adjusting the position of the replacement background so that the replacement background does not block the target foreground in the illumination direction.
8. A system for changing target foreground lighting conditions, used to implement the method for changing target foreground lighting conditions according to any one of claims 1 to 7, characterized in that: include: The acquisition module is used to obtain the target foreground and replace the background in the RGBD image; A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground; An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source for replacing the background or an equivalent light source; The synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the replacement background to obtain a replacement image.
9. A device for changing the illumination condition of a target foreground, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the method for changing the target foreground lighting conditions as described in any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the method for changing the target foreground lighting conditions as described in any one of claims 1 to 7 are implemented.
