Method, system and device for changing target foreground illumination condition in video and medium

The three-dimensional model is reconstructed through RGBD images and different lighting conditions are simulated, which solves the problem of inconsistent lighting in the cutout background replacement, and accurately adjusts the target foreground lighting in the video, improving the visual experience.

CN119006507APending Publication Date: 2024-11-22SHENZHEN GUANGJIAN TECH CO LTD +2
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310547047.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When replacing the cutout background in the prior art, the foreground lighting conditions are inconsistent with the new background lighting conditions, resulting in discord in the synthetic picture, affecting the user's visual experience, and the calculation amount of neural network-based methods is large and unstable.

Method used

By acquiring RGBD images, reconstructing three-dimensional point cloud and mesh models, using light sources with different lighting intensities to simulate background lighting conditions, illuminate the target foreground, and project it onto a two-dimensional plane, and combining the influence of the previous frame image, the lighting conditions are adjusted.

Benefits of technology

Light adjustments with small calculation volume, real-time calculation, high accuracy and strong stability are achieved, ensuring that the target foreground lighting conditions in the video are consistent with the background, and improving the visual effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006507A_ABST
    Figure CN119006507A_ABST
Patent Text Reader

Abstract

The method for changing the illumination condition of the target foreground in the video is characterized by comprising the following steps: S1, acquiring the target foreground and a first replacement background in a first RGBD image in the video; wherein the first replacement background and a second replacement background of a previous frame are adjusted according to a preset condition; s2, reconstructing a three-dimensional point cloud of the target foreground by using the RGB color information and the depth information which are in one-to-one correspondence, and establishing a mesh model of the target foreground; s3, n light sources with different illumination intensities are adopted to irradiate the mesh model at a preset position; wherein the light source is a light source of the first replacement background or an equivalent light source; and S4, projecting the mesh model into a two-dimensional plane, and synthesizing the mesh model with the first replacement background to obtain a replacement image. The method has the characteristics of small calculation amount, real-time calculation, high accuracy and high stability.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] In recent years, with the rapid development of the Internet, logistics, and e-commerce, short video production, live streaming, and entertainment live streaming have gradually become part of people's daily lives. Many anchors will choose to replace the background with a green screen to improve the live broadcast effect. The background replacement method uses the cutout technology, which extracts the foreground of the target of interest in a picture and synthesizes it into a new background image. Unlike image segmentation technology, cutout can make the synthesized edge more natural. Cutout can be further classified into semantic cutout and trimap cutout. Semantic cutout selects the cutout object based on the semantics in the image, while trimap cutout uses additional trimap input to select the cutout object. Using depth to generate the trimap RGBD cutout is one of the ideal trimap cutout technologies. Cutout can achieve arbitrary background replacement, but directly replacing the background will bring new problems, that is, the foreground lighting conditions are inconsistent with the lighting conditions of the new background, resulting in an unharmonious synthesized picture, affecting the user's visual experience.

[0003] At present, many scholars and researchers have studied the problem of the disharmony between the foreground and the background caused by cutouts, and proposed relighting technology, which is to correct the lighting conditions of the foreground based on the lighting conditions of the background. Some scholars have used neural networks and RGB cutouts to relight portraits. This method can only relight portraits, and the method based on neural networks has the problems of large amount of calculation, inability to be real-time, and unstable effect. The difficulty of this problem lies in the fact that the inaccuracy of the cutout range leads to the inaccuracy of the lighting range, and the three-dimensional reconstruction technology based only on RGB has large errors, which affects the final visual effect.

[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0005] To this end, the present invention uses the first RGBD image to obtain the target foreground, and then reconstructs the three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. The influence of the previous frame image is taken into account. The present invention has the characteristics of small computational complexity, real-time calculation, high accuracy and strong stability.

[0006] In a first aspect, the present invention provides a method for changing the illumination condition of a target foreground in a video, characterized in that it comprises the following steps:

[0007] Step S1: obtaining a target foreground and a first replacement background in a first RGBD image in a video; wherein the first replacement background and the second replacement background of a previous frame are adjusted according to preset conditions;

[0008] Step S2: Reconstructing the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establishing a mesh model of the target foreground;

[0009] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the first replacement background or an equivalent light source;

[0010] Step S4: Projecting the mesh model into a two-dimensional plane and synthesizing it with the first replacement background to obtain a replacement image.

[0011] Optionally, the method for changing the illumination condition of the target foreground in the video is characterized in that step S1 comprises:

[0012] Step S11: Acquire a first frame image and a first replacement background in the video; wherein the first frame image is an RGBD image, including a first RGB image and a first depth image; the first replacement background and the second replacement background of the previous frame are adjusted according to preset conditions;

[0013] Step S12: obtaining a first target range on the first depth image using a preset distance parameter;

[0014] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first target range is identified to obtain the second target range;

[0015] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.

[0016] Optionally, the method for changing the illumination condition of the target foreground in the video is characterized in that step S2 comprises:

[0017] Step S21: Reconstructing a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information;

[0018] Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud;

[0019] Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.

[0020] Optionally, the method for changing the illumination condition of the target foreground in the video is characterized in that step S3 comprises:

[0021] Step S31: determine whether the first replacement background and the second replacement background are the same; if they are the same, obtain the light source of the second replacement background for illumination; if they are not the same, execute step S32;

[0022] Step S32: determining the positional relationship between the target foreground and the first replacement background;

[0023] Step S33: obtaining the lighting condition of the first replacement background;

[0024] Step S34: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the first replacement background at a preset position;

[0025] Step S35: using n light sources to illuminate the mesh model simultaneously.

[0026] Optionally, the method for changing the lighting conditions of a target foreground in a video is characterized in that the switching between the first replacement background and the second replacement background is related to the posture of the target foreground.

[0027] Optionally, the method for changing the illumination condition of the target foreground in the video is characterized in that step S32 comprises:

[0028] Step S321: obtaining three-dimensional information of the first replacement background;

[0029] Step S322: adjusting the distance between the first replacement background and the target foreground so that the first replacement background and the target foreground have a consistent ratio.

[0030] Optionally, the method for changing the illumination condition of the target foreground in the video is characterized by further comprising:

[0031] Step S323: adjusting the position of the first replacement background so that the first replacement background does not block the target foreground in the lighting direction.

[0032] In a second aspect, the present invention provides a system for changing the illumination condition of a target foreground in a video, which is used to implement any of the above methods for changing the illumination condition of a target foreground in a video, and is characterized by comprising:

[0033] An acquisition module, used to acquire a target foreground and a first replacement background in a first RGBD image in a video; wherein the first replacement background and the second replacement background of a previous frame are adjusted according to a preset condition;

[0034] A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground;

[0035] An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source of the first replacement background or an equivalent light source;

[0036] A synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the first replacement background to obtain a replacement image.

[0037] In a third aspect, the present invention provides a device for changing the illumination condition of a target foreground in a video, characterized in that it comprises:

[0038] processor;

[0039] a memory storing executable instructions of the processor;

[0040] The processor is configured to execute any one of the steps of the above-mentioned method for changing the foreground lighting condition of a target in a video by executing the executable instructions.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned methods for changing the foreground lighting conditions of a target in a video are implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The replacement background in the present invention can be adjusted according to preset conditions, ensuring that the background can be changed according to the subject content when the video is played, which helps to improve the video effect.

[0044] The present invention attempts to use RGBD cutout technology with a more accurate range to achieve more realistic and accurate relighting. Depth information can make the target and range of cutout more clearly defined, so as to obtain more stable cutout content. With the support of depth information, the target foreground obtained by RGBD cutout can be accurately reconstructed in three-dimensional space, providing a better three-dimensional model for subsequent relighting.

[0045] The present invention utilizes a plurality of light sources with different illumination intensities to illuminate the mesh model of the target foreground in three-dimensional space, so as to obtain a more realistic illumination result, and at the same time has the characteristics of small calculation amount and fast operation speed.

[0046] When the replacement background is the same as the original background, the present invention can also improve the lighting conditions of the target foreground without replacing the background, making the target foreground more prominent, and effectively solving the problem of large difference in brightness between the foreground and the background, such as the problem of the foreground being too dark when the background brightness is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0048] Figure 1 A flowchart of a method for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of a depth map in an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a mesh model in an embodiment of the present invention;

[0051] Figure 4 A flowchart of steps for obtaining a target foreground and a first replacement background in an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of background replacement transformation in an embodiment of the present invention;

[0053] Figure 6 A flowchart of the steps of establishing a mesh model in an embodiment of the present invention;

[0054] Figure 7 This is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention;

[0055] Figure 8 A flowchart of a step of determining a position relationship between a target foreground and a first replacement background in an embodiment of the present invention;

[0056] Fig. 9 This is another flow chart of steps for determining the position relationship between the target foreground and the first replacement background in an embodiment of the present invention;

[0057] Fig.10 Schematic diagram of the structure of a system for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention;

[0058] Fig.11 is a schematic diagram of the structure of a device for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention; and

[0059] Fig.12 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0061] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] The embodiment of the present invention provides a method for changing the illumination condition of a target foreground in a video, aiming to solve the problems existing in the prior art.

[0063] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0064] The present invention uses the first RGBD image to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. The influence of the previous frame image is taken into consideration. The present invention has the characteristics of small calculation amount, real-time calculation, high accuracy and strong stability.

[0065] Figure 1 FIG. 1 is a flow chart of steps of a method for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention. Figure 1 As shown, the steps of a method for changing the illumination condition of a target foreground in a video in an embodiment of the present invention include:

[0066] Step S1: Obtain a target foreground and a first replacement background in a first RGBD image in a video.

[0067] In this step, the first replacement background and the second replacement background of the previous frame are adjusted according to preset conditions. The preset conditions refer to the pre-set conditions for switching the replacement background, the range of replacement background selection, and the order of changing the replacement background. The current frame image obtained is the first RGBD image. The replacement background in the image obtained before the current frame image is the second replacement background. The second replacement background can be the replacement background of the previous frame image, or it can be a general term for the replacement backgrounds of the previous multiple frames of images. Using the data collected by the RGBD camera, a neural network model for RGBD cutout is trained. The first replacement background can be the same as or different from the second replacement background. The RGB (color image), Depth (depth map) and depth image collected by the RGBD camera are used as the input of the RGBD cutout model to obtain the alpha corresponding to the target foreground. The RGBD cutout model can adopt any neural network model in the prior art, and this embodiment does not limit this. The first replacement background is a pre-prepared background, which can be an RGB image or a first RGBD image. Figure 2 As shown, the depth map can accurately identify the target foreground.

[0068] Step S2: Reconstruct the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establish a mesh model of the target foreground.

[0069] In this step, the color map and the depth map are one-to-one corresponding, so each pixel has unique RGB color information and unique depth information. Based on the RGB color information and depth information, three-dimensional reconstruction can be performed to obtain a three-dimensional point cloud of the target foreground and establish a mesh model of the target foreground. When performing surface reconstruction to obtain a mesh model of the target foreground, a variety of algorithms can be used, such as Alpha shapes, Ball pivoting, Poisson surface reconstruction, etc. Figure 3 As shown in Figure 3, the mesh model reconstructs the target foreground in three-dimensional space and can be used to better simulate the characteristics of the target foreground.

[0070] Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position.

[0071] In this step, the light source is the light source of the first replacement background or an equivalent light source. The light source of the first replacement background may be one or more light sources, and the types of light sources may be different. At the same time, the light source may be far away from the first replacement background and cannot be directly simulated in the reconstructed three-dimensional space. Therefore, the light source used in this step can be the same as or different from the light source of the first replacement background. When this step uses n light sources with different light intensities, each light source is at a preset position. The preset position is not a fixed position, but an interval, such as on the outermost surface of the three-dimensional space. Depending on the lighting conditions of the background, the value of n will vary, such as 3, 4, 5, etc.

[0072] Step S4: Projecting the mesh model into a two-dimensional plane and synthesizing it with the first replacement background to obtain a replacement image.

[0073] In this step, the illuminated mesh model is projected onto a two-dimensional plane to obtain a two-dimensional image under the same lighting as the background. Then, it is synthesized with the first replacement background to obtain a replacement image. The lighting conditions of the foreground in the replacement image are the same as those of the background, so that the synthesized image is more realistic and avoids visual confusion caused by inconsistent lighting.

[0074] Figure 4 FIG. 1 is a flow chart of steps for obtaining a target foreground and a first replacement background in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a target foreground and a first replacement background includes:

[0075] Step S11: Acquire the first frame image and the first replacement background in the video.

[0076] In this step, the first frame image is an RGBD image, including a first RGB image and a first depth image. The first replacement background and the second replacement background of the previous frame are adjusted according to preset conditions. Figure 5 As shown, the background replacement can replace the three images in sequence, such as replacing a with the first continuous 1s in the video, replacing b with the second continuous 1s, replacing c with the third continuous 1s, and then replacing a with the fourth continuous 1s in the video, and so on, until all the backgrounds in the video are replaced. The first RGB image and the first depth image are aligned at the pixel level.

[0077] In some embodiments, the switching between the first replacement background and the second replacement background is related to the posture of the target foreground. When the proportion, orientation, etc. of the target foreground changes, a replacement background matching it is required. When the parameters related to the posture of the target foreground reach a preset value, the replacement background is changed to keep the replacement background matching the replacement foreground.

[0078] Step S12: obtaining a first target range on the first depth image using a preset distance parameter.

[0079] In this step, the preset distance parameter is a constant used to extract the effective area. For example, if the preset distance parameter is 1.2m, then all pixels with depth values ​​less than 1.2m on the first depth image constitute the first target range. For the same video, the preset distance parameter is a fixed value. For different videos, the preset distance parameter can be different values. The setting of the preset distance can be selected according to the depth camera and the specific application scenario.

[0080] Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first target range is identified to obtain the second target range.

[0081] In this step, the UNet structure includes three components: encoder, decoder and Skip-Connection in the same layer. The encoder performs downsampling. For example, the input image size is 572*572, the feature map size after two rounds of 3*3 convolution is 568*568, and the output size after 22 maximum pooling is 284*284. The decoder performs upsampling. For example, the decoder is composed of 4 groups of 2*2 transposed convolutions, 3*3 convolutions and a ReLU activation function, and a 1*1 convolution is added to the final output layer. Finally, there is Skip-Connection in the same layer, which is to crop the output of each layer during downsampling and connect it to the upsampling layer of the same layer for fusion. This step also processes the image in combination with the temporal features of the first frame to ensure the stability of target recognition between different frames. To some extent, temporal features can also be regarded as a kind of deep supervision.

[0082] Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.

[0083] In this step, different from the screening performed in step S12, this step resets the area within part of the depth value range to increase the selected area and obtain a third target range. The third target range is larger than the second target range. When determining the area to be reset, a fixed depth value range or a depth value range related to the target area can be used. When a fixed depth value range is used, the depth value within 1m can be reset to obtain the third target range. When a depth value range is used that is related to the target area, if the average depth value of the target area is 0.8m, the depth value of 0.6m-1m is reset to obtain the third target range.

[0084] This embodiment uses a preset distance to obtain a first target range, and then uses the UNet network and the Skip-Connection of different dimensional features, combined with the timing characteristics, to identify the second portrait image, and then uses the preset distance parameters to adjust the second target range. It can completely and effectively identify the human body and interactive objects, and generate high-quality and high-resolution images, which can realize real-time processing of videos.

[0085] Figure 6 FIG. 1 is a flowchart of the steps of establishing a mesh model in an embodiment of the present invention. Figure 6 As shown, a step of establishing a mesh model in an embodiment of the present invention includes:

[0086] Step S21: Reconstruct the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information.

[0087] In this step, 3D reconstruction is performed using point cloud. This step only performs 3D reconstruction on the target foreground. This step projects the pixels according to the depth value of each pixel to obtain the distribution of each pixel in 3D space. Since RGB color information contains the illumination information of the current target foreground, the 3D point cloud obtained in this step is also affected by the illumination of the target foreground.

[0088] Step S22: filtering the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud.

[0089] In this step, the filtering process includes de-flickering, de-flying point, missing area filling and other processes. Since there are flying points and missing areas in the depth map. This step can improve the quality of the depth image and ensure the integrity of the image content, making the extraction of the foreground more complete. This step also calculates the reflectivity of each point in the three-dimensional point cloud. The reflectivity can be calculated using any method in the prior art, and this embodiment does not limit this. The reflectivity can be used for calculation when irradiating the mesh model in step S3 to improve the authenticity of the illumination effect.

[0090] Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.

[0091] In this step, the three-dimensional point cloud is sampled, and the mesh model of the target foreground is obtained using surface reconstruction technology. Sampling the three-dimensional point cloud can further reduce the amount of calculation and improve processing efficiency. When sampling the three-dimensional point cloud, the sampling ratio varies according to the resolution. The higher the resolution, the lower the sampling ratio. The lower the resolution, the higher the sampling ratio. The mesh model removes the lighting effect of the target foreground, which can make the lighting in subsequent steps more obvious. Surface reconstruction technology can be selected according to different objects. For example, when using the Ball-Pivoting algorithm, the point cloud normal is used for calculation, which can be applied to most scenes of the point cloud and achieve good results.

[0092] This embodiment performs 3D reconstruction through 3D point cloud, processes the point cloud, improves subsequent processing efficiency, and uses surface reconstruction technology to obtain a mesh model of the target foreground, thereby removing the influence of illumination on the target foreground and making subsequent illumination of the mesh model more realistic.

[0093] Figure 7 FIG. 1 is a flow chart of steps for irradiating a mesh model in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of irradiating a mesh model includes:

[0094] Step S31: determine whether the first replacement background and the second replacement background are the same; if they are the same, obtain the light source of the second replacement background for illumination; if they are not the same, execute step S32;

[0095] In this step, if the first replacement background is the same as the second replacement background, the light source obtained in the second replacement background can be directly used for irradiation, and there is no need to perform the subsequent steps of this embodiment, thereby improving efficiency. It should be noted that the second replacement background in this step does not only refer to the replacement background in the previous frame image, but refers to the replacement background in the previous multiple frames. Only when the replacement background in any frame in the previous multiple frames is the same as the first replacement background, the previous light source can be used for irradiation. If the first replacement background is different from the second replacement background, step S32 is performed.

[0096] Step S32: Determine the positional relationship between the target foreground and the first replacement background.

[0097] In this step, the positional relationship between the target foreground and the first replacement background can be adjusted in various ways, such as adjusting the background position, etc. For example, the first replacement background is regarded as an environment map, and the target foreground is located at the center of a cube formed by the environment map.

[0098] Step S33: Acquire the lighting condition of the first replacement background.

[0099] In this step, the illumination condition of the first replacement background may be known or unknown. If the illumination condition of the first replacement background is known, relevant data is directly obtained. If the illumination condition of the first replacement background is unknown, the image is analyzed using parameters such as left-right contrast, top-bottom contrast, average brightness, contrast, covariance matrix, etc. to obtain detailed information of the illumination.

[0100] Step S34: according to the conversion function, using n light sources with different illumination intensities to simulate the illumination condition of the first replacement background at a preset position.

[0101] In this step, the lighting conditions of the first replacement background are fitted based on the spherical harmonic function, and then according to the measured conversion function, n light sources with different lighting intensities are used to simulate the lighting conditions of the first replacement background at a preset position. The lighting model can adopt Lambert Lighting, Half-Lambert Lighting, Phong Lighting, Blinn-Phong Lighting or any other model. The specific selection of the lighting model can also be selected according to the application scenario. For example, for the application scenario focusing on portraits, Phong Lighting is selected to highlight the highlights and form a better visual impact effect; for the application scenario focusing on product display, Half-Lambert Lighting is selected to highlight the product more completely. In this step, multiple different light sources are used to simulate the lighting conditions of the first replacement background. n can be any positive integer. The light source is at least one of a point light source, a surface light source, and a line light source. For example, if four light sources are used for simulation, two point light sources, one surface light source and one line light source can be used for simulation. The intensity of each light source can be different. The area of ​​the same light source can also be different.

[0102] Step S35: using n light sources to illuminate the mesh model simultaneously.

[0103] In this step, according to the illumination model and rendering formula, n light sources are applied to the mesh model of the target foreground. Through the illumination in this step, shadows, color changes, etc. of the target foreground can be obtained.

[0104] This embodiment simulates the lighting conditions of the first replacement background at a preset position through multiple light sources with different lighting intensities, so as to better simulate the lighting conditions of the first replacement background, so that the lighting of the mesh model can be the same as that of the first replacement background, and the corresponding lighting model is selected according to different application scenarios to improve the effect of the composite image.

[0105] Figure 8 FIG. 1 is a flow chart of a step of determining the position relationship between the target foreground and the first replacement background in an embodiment of the present invention. Figure 8 As shown, in an embodiment of the present invention, a step of determining a position relationship between a target foreground and a first replacement background includes:

[0106] Step S321: Obtaining three-dimensional information of the first replacement background.

[0107] In this step, the three-dimensional information of the first replacement background can be known or unknown. When the three-dimensional information of the first replacement background is known, the three-dimensional data of the first replacement background is directly obtained. For example, when the first replacement background is RGBD, the three-dimensional information of the first replacement background can be directly obtained. When the three-dimensional information of the first replacement background is unknown, such as the first replacement background is a newly obtained RGB image, it is necessary to analyze the first replacement background using a recognition model to obtain the three-dimensional information of the first replacement background. The recognition model is a deep neural network model, which obtains the depth data corresponding to each pixel by learning the RGB image. The recognition model can adopt any deep neural network model in the prior art, and this embodiment does not limit this.

[0108] Step S322: adjusting the distance between the first replacement background and the target foreground so that the first replacement background and the target foreground have a consistent ratio.

[0109] In this step, since the image sizes of the first replacement background and the target foreground have been determined, but the ratio between the two may be inconsistent, the pinhole imaging principle is used to adjust the distance between the first replacement background and the target foreground so that in the camera coordinate system, the ratio of the target foreground to the first replacement background is consistent, which is more in line with the visual characteristics of the human eye.

[0110] This embodiment obtains the three-dimensional information of the first replacement background and adjusts the distance between the target foreground and the first replacement background so that the ratio of the target foreground to the first replacement background is more coordinated and realistic, which conforms to the visual characteristics of the human eye.

[0111] Fig. 9 FIG. 1 is another flow chart of steps for determining the position relationship between the target foreground and the first replacement background in an embodiment of the present invention. Fig. 9 As shown, compared with the above-mentioned embodiment, another step of determining the position relationship between the target foreground and the first replacement background in the embodiment of the present invention further includes:

[0112] Step S323: adjusting the position of the first replacement background so that the first replacement background does not block the target foreground in the lighting direction.

[0113] In this step, since there may be relatively high objects in the first replacement background, some objects in the first replacement background block the target foreground in the illumination direction in the three-dimensional space. Therefore, the first replacement background needs to be adjusted, for example, the distance between the first replacement background and the target foreground is adjusted, the first replacement background is rotated, the first replacement background is flipped, the first replacement background is enlarged, etc.

[0114] This embodiment avoids the possible influence of the first replacement background on the foreground by adjusting the first replacement background, avoids the conflict between the target foreground and the first replacement background, and further ensures the consistency of the foreground and the background.

[0115] Fig.10 FIG. 1 is a schematic diagram of a system for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention. Fig.10 As shown, a system for changing the illumination condition of a target foreground in a video in an embodiment of the present invention includes:

[0116] An acquisition module, used to acquire a target foreground and a first replacement background in a first RGBD image in a video; wherein the first replacement background and the second replacement background of a previous frame are adjusted according to a preset condition;

[0117] A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground;

[0118] An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source of the first replacement background or an equivalent light source;

[0119] A synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the first replacement background to obtain a replacement image.

[0120] Specifically, the acquisition module obtains the target foreground and the first replacement background of the first RGBD image. The target foreground obtained using depth information is more accurate and can be used for adjustment in the subsequent synthesis process. The three-dimensional reconstruction module reconstructs the three-dimensional information of the target foreground and establishes a mesh model, removes the lighting effect in the first RGBD image, and obtains accurate original target foreground information. The illumination module uses the same light source as the first replacement background or an equivalent light source to illuminate the mesh model, so that the lighting conditions of the target foreground are the same as those of the first replacement background. The synthesis module projects the mesh model into a two-dimensional plane, and finally generates a two-dimensional image that is well integrated with the first replacement background, greatly improving the quality of the fused image.

[0121] This embodiment uses the first RGBD image to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. The influence of the previous frame image is taken into account. This embodiment has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.

[0122] The present invention also provides a device for changing the illumination condition of a target foreground in a video, comprising a processor and a memory storing executable instructions of the processor. The processor is configured to execute the steps of a method for changing the illumination condition of a target foreground in a video by executing the executable instructions.

[0123] As mentioned above, this embodiment uses the first RGBD image to obtain the target foreground, and then reconstructs the three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. The influence of the previous frame image is taken into account. It has the characteristics of small computational complexity, real-time calculation, high accuracy and strong stability.

[0124] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0125] Fig.11 Schematic diagram of a device for changing the illumination condition of a target foreground in a video according to an embodiment of the present invention. Fig.11 The electronic device 600 according to this embodiment of the present invention is described. Fig.11 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0126] like Fig.11 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0127] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method for changing the illumination condition of the foreground of a target in a video. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0128] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0129] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0130] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0131] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.11 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0132] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, steps of a method for changing the illumination condition of a target foreground in a video are implemented. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for changing the illumination condition of a target foreground in a video section of this specification.

[0133] As shown above, this embodiment uses the first RGBD image to obtain the target foreground, and then reconstructs the three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane, and the influence of the previous frame image is taken into account. It has the characteristics of small computational complexity, real-time calculation, high accuracy and strong stability.

[0134] Fig.12 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.12 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0135] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0136] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0137] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0138] This embodiment uses the first RGBD image to obtain the target foreground, and then reconstructs a three-dimensional point cloud and mesh model. The lighting conditions of the background are simulated by light sources with different lighting intensities, the mesh model of the target foreground is illuminated, and then projected into a two-dimensional plane. The influence of the previous frame image is taken into account. This embodiment has the characteristics of small computational complexity, real-time calculation, high accuracy, and strong stability.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0140] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for changing the illumination condition of a target foreground in a video, characterized in that: The steps include: Step S1: obtaining a target foreground and a first replacement background in a first RGBD image in a video; wherein the first replacement background and the second replacement background of a previous frame are adjusted according to preset conditions; Step S2: Reconstructing the three-dimensional point cloud of the target foreground using the one-to-one corresponding RGB color information and depth information, and establishing a mesh model of the target foreground; Step S3: using n light sources with different illumination intensities to illuminate the mesh model at a preset position; wherein the light source is the light source of the first replacement background or an equivalent light source; Step S4: Projecting the mesh model into a two-dimensional plane and synthesizing it with the first replacement background to obtain a replacement image.

2. A method for changing the illumination condition of a target foreground in a video according to claim 1, characterized in that: Step S1 includes: Step S11: Acquire a first frame image and a first replacement background in the video; wherein the first frame image is an RGBD image, including a first RGB image and a first depth image; the first replacement background and the second replacement background of the previous frame are adjusted according to preset conditions; Step S12: obtaining a first target range on the first depth image using a preset distance parameter; Step S13: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first target range is identified to obtain the second target range; Step S14: using the preset distance parameter to adjust the second target range to obtain a third target range.

3. The method for changing the illumination condition of the target foreground in a video according to claim 1, characterized in that: Step S2 includes: Step S21: Reconstructing a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information; Step S22: performing filtering processing on the three-dimensional point cloud and calculating the reflectivity of the three-dimensional point cloud; Step S23: Obtaining a mesh model of the target foreground using surface reconstruction technology.

4. A method for changing the illumination condition of a target foreground in a video according to claim 1, characterized in that: Step S3 includes: Step S31: determine whether the first replacement background and the second replacement background are the same; if they are the same, obtain the light source of the second replacement background for illumination; if they are not the same, execute step S32; Step S32: determining the positional relationship between the target foreground and the first replacement background; Step S33: obtaining the lighting condition of the first replacement background; Step S34: according to the conversion function, using n light sources with different light intensities to simulate the lighting conditions of the first replacement background at a preset position; Step S35: using n light sources to illuminate the mesh model simultaneously.

5. The method for changing the illumination condition of the target foreground in a video according to claim 1, characterized in that: The switching between the first replacement background and the second replacement background is related to the posture of the target foreground.

6. A method for changing the illumination condition of a target foreground in a video according to claim 4, characterized in that: The step S32 comprises: Step S321: obtaining three-dimensional information of the first replacement background; Step S322: adjusting the distance between the first replacement background and the target foreground so that the first replacement background and the target foreground have a consistent ratio.

7. A method for changing the illumination condition of a target foreground in a video according to claim 6, characterized in that: Also includes: Step S323: adjusting the position of the first replacement background so that the first replacement background does not block the target foreground in the lighting direction.

8. A system for changing the illumination condition of a target foreground in a video, used to implement the method for changing the illumination condition of a target foreground in a video according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire a target foreground and a first replacement background in a first RGBD image in a video; wherein the first replacement background and the second replacement background of a previous frame are adjusted according to a preset condition; A three-dimensional reconstruction module, used to reconstruct a three-dimensional point cloud of the target foreground using one-to-one corresponding RGB color information and depth information, and to establish a mesh model of the target foreground; An illumination module, used to illuminate the mesh model at a preset position using n light sources with different illumination intensities; wherein the light source is the light source of the first replacement background or an equivalent light source; A synthesis module is used to project the mesh model into a two-dimensional plane and synthesize it with the first replacement background to obtain a replacement image.

9. A device for changing the illumination condition of a target foreground in a video, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the method for changing the foreground lighting conditions of a target in a video as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the method for changing the foreground lighting conditions of a target in a video as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Target object relighting method and device, storage medium and background replacement method

    CN115131492A

  • Face data enhancement method and device, computer equipment and storage medium

    CN115239861A

  • Illumination adjustment method and device for fused image, equipment and medium

    CN115293960A