A method for replacing a video background using a depth camera

Abstract

A method for replacing video background by using a depth camera, comprising the following steps: S1: obtaining a pth frame of RGBD image, identifying a target object, marking a region where the target object is located as a target object region, and then replacing a background; wherein the RGBD image is obtained by a depth camera, and p is a positive integer; S2: expanding the target object region outward by n pixels to obtain an expanded region; S3: obtaining a (p+1)th frame of RGBD image, and determining a target region corresponding to the expanded region of the pth frame; S4: confirming a target boundary according to depth information in the target region on the (p+1)th frame of RGBD image, and obtaining an accurate target object, and then replacing the background. The present application can obtain a high-precision target object, reduce the processing amount of subsequent video frames, respond quickly, have a low requirement on computing power, and realize real-time processing of video.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video image processing, in particular, to a method for replacing video background by using a depth camera. BACKGROUND

[0002] In many scenarios of work and life, it is necessary to extract target objects in images and replace picture images, such as ID photos, promotional photos, etc. In many application scenarios, continuous processing of videos is required, such as anchors, TV programs, video advertisements, etc. By replacing the background, the corresponding production cost can be greatly reduced, and the time and workload required for preparation work can be saved.

[0003] In the prior art, the foreground and background are identified by judging the RGB image to realize the extraction of target objects.

[0004] A certain patent discloses a video image enhancement processing method, device and electronic equipment. The method comprises: obtaining target video data, the target video data containing multiple frames of video images; dividing the video images in the target video data to obtain multiple image groups, each image group containing at least one frame of the video images, the video images divided into the same image group being continuous and satisfying a preset mutation division condition; selecting at least one reference image for each frame of the video images in the image groups; and performing enhancement processing on each frame of the video images by using the respective corresponding reference image to obtain the corresponding enhanced image of the video image.

[0005] A certain invention provides a depth map post-processing method capable of effectively improving the accuracy of depth map information. The technical solution comprises the following steps: first, obtaining a current frame image and a previous frame image in an original two-dimensional video; using an optical flow method to detect the motion of the current frame image and the previous frame image to obtain a motion vector of the current frame image, and determining whether the current frame image and the previous frame image have undergone a scene change; if yes, performing smoothing filtering to obtain a new depth map of the current frame image; otherwise, obtaining a motion parameter of the current frame image and an existing depth map of the previous frame image to obtain a virtual depth map of the current frame image; performing occlusion detection on the virtual depth map and the existing depth map of the current frame image; and performing smoothing filtering to obtain a new depth map of the current frame image. The present application is suitable for post-processing of depth maps.

[0006] The prior art solution adopts per-frame processing and multi-frame interaction technologies, and has large data processing amount, large delay in real-time processing, and high resource consumption, and it is difficult to realize large-area real-time video background replacement. SUMMARY

[0007] To this end, the present invention identifies the target object in the p-th frame image and reduces the processing range in the p+1-th frame by pixel expansion. Then, it optimizes the target boundary using the depth information in the p+1-th frame to obtain the accurate target object. While obtaining the high-precision target object, it also reduces the processing amount of subsequent video frames, enabling fast response, low computing power requirements, and real-time video processing.

[0008] In a first aspect, the present invention provides a method for replacing the background of a video using a depth camera, characterized by comprising the following steps:

[0009] S1: Acquire the p-th frame RGBD image, identify the target object, mark the area where the target object is located as the target object area, and then replace the background; wherein, the RGBD image is obtained by a depth camera, and p is a positive integer;

[0010] S2: Extend the target object region outward by n pixels to obtain the extended region;

[0011] S3: Acquire the RGBD image of frame p+1 and determine the target region corresponding to the extended region of frame p;

[0012] S4: In the target area of ​​the p+1th frame RGBD image, the target boundary is confirmed based on the depth information, and the precise target object is obtained, and then the background is replaced.

[0013] Optionally, the method for replacing the video background using a depth camera is characterized in that step S1 includes:

[0014] S11: Obtain the image to be replaced; wherein, the image to be replaced is an RGBD image;

[0015] S12: Identify the target object through the image to be replaced;

[0016] S13: Establish a pixel-level correspondence between the image to be replaced and the replacement image; wherein, the replacement image is an RGBD image;

[0017] S14: Adjust the overall depth value of the replacement image or target object to match the depth value of the target object;

[0018] S15: Place the target object in the replacement image to obtain an image with a changed background.

[0019] Optionally, the method for replacing the video background using a depth camera is characterized in that step S12 includes:

[0020] S121: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions;

[0021] S122: Add the three target object regions on the image to be replaced to obtain the target object regions;

[0022] S123: Extend a certain number of pixels to both sides along the outline of the target object region to obtain an edge region; the edge region includes both the information of the target object and the information of the background;

[0023] S124: In the edge region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, the pixel is determined as a boundary.

[0024] S125: Crop the region within the boundary in the RGBD image to obtain the target object.

[0025] Optionally, the method for replacing the video background using a depth camera is characterized in that step S14 includes:

[0026] S141: Obtain the depth range a of the target object;

[0027] S142: Obtain the depth range b of the part in the replacement image that corresponds to the target object;

[0028] S143: If a is less than b and there is no intersection, then increase a or decrease b so that the largest element in a is equal to the smallest element in b;

[0029] S144: If a is greater than b and there is no intersection, then decrease a or increase b so that the largest element in a is equal to the smallest element in b.

[0030] S145: If a and b have an intersection, adjust a so that the largest element in a is equal to the smallest element in b.

[0031] Optionally, the method for replacing the video background using a depth camera is characterized in that step S15 includes:

[0032] S151: Align the replacement image with the image to be replaced;

[0033] S152: Extract the target object;

[0034] S153: Replace the background information in the image to be replaced with the aligned replacement image;

[0035] S154: Place the target object on a layer above the replacement image.

[0036] Optionally, the method for replacing the video background using a depth camera is characterized in that step S15 includes:

[0037] S151: Align the replacement image with the image to be replaced;

[0038] S152: Extract the target object;

[0039] S155: Reassemble the target object and the replacement image based on depth information.

[0040] Optionally, the method for replacing video background using a depth camera is characterized in that, in S155, if the target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object.

[0041] Optionally, the method for replacing video background using a depth camera is characterized in that, in S155, if there are regional influencing factors in the replacement image, a response feature is added to the target object; the regional influencing factors refer to factors that can affect other objects in three-dimensional space.

[0042] Secondly, the present invention provides a device for replacing the background using an RGBD image, characterized in that it comprises:

[0043] processor;

[0044] A memory in which executable instructions of the processor are stored;

[0045] The processor is configured to perform the steps of any of the above-described methods for changing the video background using an RGBD image by executing the executable instructions.

[0046] Thirdly, the present invention provides a computer-readable storage medium for storing a program, characterized in that, when the program is executed, it implements the steps of the method for changing the video background using an RGBD image as described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention uses RGBD images for processing, which makes the identification and extraction of target objects more accurate, and can obtain higher quality and better results. At the same time, the replacement image also uses RGBD images, and the data type is consistent, which is conducive to better processing.

[0049] This invention identifies target objects only in the p-th frame of the image, while identifying them in subsequent frames by extending the target object. This reduces the area that needs to be processed, thereby reducing the amount of data processing, greatly improving processing efficiency, shortening the resource consumption of processing, and facilitating real-time data processing.

[0050] This invention identifies target boundaries in the p+1th frame image based on depth map information, thereby reducing the amount of data processing, making data processing faster, improving processing efficiency, and facilitating real-time data processing.

[0051] This invention utilizes depth data to optimize target objects, thereby reducing the accuracy requirements for target object recognition, lowering algorithm requirements, increasing computational speed, and enhancing data processing capabilities, thus adapting to real-time video processing. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This is a flowchart illustrating the steps of a method for changing the background of a video using a depth camera, as described in an embodiment of the present invention.

[0054] Figure 2 This is a video illustration in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart illustrating a step of replacing the background of the p-th frame image in an embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating a step in identifying a target object using a diagram to be replaced, according to an embodiment of the present invention.

[0057] Figure 5 This is a flowchart illustrating a step in adjusting and replacing the overall depth value of an image according to an embodiment of the present invention;

[0058] Figure 6 This is a flowchart illustrating a step in placing a target object within the replacement image according to an embodiment of the present invention;

[0059] Figure 7 This is a flowchart illustrating another step in an embodiment of the present invention of placing a target object in the replacement image;

[0060] Figure 8 This is a schematic diagram of a device for changing video backgrounds using a depth camera, as described in an embodiment of the present invention.

[0061] Figure 9This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0063] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0064] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] This invention provides a method for changing the background of a video using a depth camera, which aims to solve the problems existing in the prior art.

[0066] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0067] Figure 1 This is a flowchart illustrating the steps of a method for replacing the background of a video using a depth camera, as described in an embodiment of the present invention. Figure 1 As shown in the figure, an embodiment of the present invention provides a method for changing the background of a video using a depth camera, comprising the following steps:

[0068] S1: Acquire the p-th frame RGBD image, identify the target object, mark the area where the target object is located as the target object area, and then replace the background.

[0069] In this step, the RGBD image is acquired by a depth camera, and p is a positive integer. The letter p represents the initial frame from which background replacement begins. p can be 1 or any other integer specified by the user. RGBD images are three-dimensional images, unlike traditional two-dimensional images. RGBD images also include depth data, thus forming a three-dimensional space. When replacing the background, the background image can be either an RGBD image or a two-dimensional RGB image. If the background image is an RGBD image, the image after background replacement will be an RGBD image. If the background image is an RGB image, the image after background replacement will be an RGB image.

[0070] S2: Extend the target object region outward by n pixels to obtain the extended region.

[0071] In this step, the target object region on both the RGB image and the depth map is expanded outward by n pixels, and the expanded region remains aligned. The number of pixels n expanded depends on the size of the target object's area m, and generally does not exceed [a certain value]. When the target object consists of two or more parts, the expanded pixels are calculated as independent regions. For example, if the target object region consists of region A and region B, with areas s and t respectively, then the expansion is performed outwards for each region. and Each region is 10 pixels. If the extended region exceeds the image boundary, the image boundary is used as the boundary of the extended region.

[0072] S3: Obtain the RGBD image of frame p+1 and determine the target region corresponding to the extended region of frame p.

[0073] In this step, since the RGBD image always maintains a consistent size, the target region of the RGBD image in the p+1th frame can be directly obtained from the extended region on the p-th frame of the RGBD image.

[0074] S4: In the target area of ​​the p+1th frame RGBD image, the target boundary is confirmed based on the depth information, and the precise target object is obtained, and then the background is replaced.

[0075] In this step, the boundary is optimized based on the depth data of the target area in the RGBD image to obtain an accurate target boundary, thereby obtaining the precise target object in the RGBD image.

[0076] After S4 is executed, the RGBD image of frame p+2 and the RGBD image of frame p+2 are processed by S3 and S4, and then the RGB images and corresponding depth maps of subsequent frames are processed in sequence until a stop signal is received.

[0077] Figure 2This is a video illustration in an embodiment of the present invention. Frames M1 and M2 are used as examples. Frames M1 and M2 are spaced several frames apart. When the video is acquired, the method described in the preceding embodiment is applied to each frame. In this embodiment, the target object is a person. In the original video, the person is indoors and standing. The processed video shows the person outdoors throughout the day, with the sun rising in the east and setting in the west. Frame M1 is at 9:39, and frame M2 is at 9:40. The sun is at different directions and altitudes, resulting in shadows with different angles in the processed image, and interacting with objects in the background to produce different effects. In this embodiment, when the background image is an RGBD image, depth data can also be used to interact with the background information and the target object, such as generating shadows for the target object, making the replaced video more realistic and consistent with human cognition. Furthermore, it eliminates the need for motion-sensing interaction devices, thereby reducing costs.

[0078] Figure 3 This is a flowchart illustrating a step in replacing the background of the p-th frame image according to an embodiment of the present invention. Figure 3 As shown in the figure, an embodiment of the present invention provides a method for changing the background of a video using a depth camera, comprising the following steps:

[0079] S11: Get the image to be replaced.

[0080] In this step, the image to be replaced is an RGBD image. The RGBD image is captured by a depth camera, which can be obtained using any of the following methods: a structured light camera, a stereo camera, or a Time-of-Flight (TOF) camera. The depth camera captures the RGBD image, obtaining both the RGB image and the depth image separately, and then combines the two images to form the final RGBD image. The RGBD image is both visually perceptible to the human eye and contains depth information.

[0081] S12: Identify the target object through the image to be replaced.

[0082] In this step, target object identification can be performed using a combination of manually generated features and a classifier, such as using Haar features and AdaBoost. Alternatively, features can be obtained through deep learning, classified using a classifier, and the obtained reference object region can be adjusted using regression algorithms, such as cascaded convolutional neural networks. Furthermore, the detection accuracy of the obtained reference object region can be adjusted according to requirements, making it more flexible and easier to apply. Target object identification can also be performed using a pre-trained object detection model based on a convolutional neural network, and so on. This embodiment does not limit the method used for target object identification.

[0083] S13: Establish the pixel-level correspondence between the image to be replaced and the replacement image.

[0084] In this step, the replacement image is an RGBD image. The replacement image is a complete image, and its size is not smaller than the image to be replaced. Preferably, the image size of the replacement image is the same as the size of the image to be replaced. The original image size of the replacement image can be larger than or equal to the size of the image to be replaced. When the original image size of the replacement image is larger than the size of the image to be replaced, the replacement image needs to be cropped or otherwise reduced in size to make its final size the same as the size of the image to be replaced. This step establishes a pixel-level one-to-one correspondence between the image to be replaced and the replacement image, so that each pixel in the image to be replaced has a corresponding unique pixel in the replacement image. The replacement image in this step contains depth data, thus providing three-dimensional spatial information. The replacement image can be composed of a color image and a depth image, and all operations on the replacement image are performed simultaneously on both images so that the data of the replacement image can ultimately correspond directly to the image to be replaced.

[0085] S14: Adjust the overall depth value of the replacement image or target object to match the depth value of the target object.

[0086] In this step, the overall depth value of the replacement image or target object is adjusted to make the overall depth value of the replaced image more reasonable. If the depth of field of the partially replaced image is similar to that of the image to be replaced (e.g., two adjacent rooms where the goal is to update decorations), then minimal adjustment to the overall depth value of the replacement image or target object is needed. However, if the depth of field of the partially replaced image differs significantly from that of the image to be replaced (e.g., replacing an indoor scene with an outdoor scene), the large difference in depth of field can easily lead to visual conflict, requiring adjustment of the depth values ​​of both. When adjusting, the overall depth value of the replacement image or target object can be adjusted individually or simultaneously to achieve a reasonable depth of field.

[0087] S15: Place the target object in the replacement image to obtain an image with a changed background.

[0088] In this step, the target object is placed within the replacement image. The target object will then cover the area overlapping with it on the two-dimensional plane, resulting in an image with a changed background. Since the image to be replaced and the replacement image are the same size, the image with the changed background is also the same size as the image to be replaced. This step preserves all information about the target object and the replacement image, especially the three-dimensional data of the overlapping portion on the two-dimensional plane. This makes the data more comprehensive, allowing the image with the changed background to better match the characteristics observed by the human eye, resulting in a better effect.

[0089] This embodiment optimizes the image to be replaced using depth data, improving the accuracy of data recognition, especially for edge recognition. By adding three-dimensional information to two-dimensional planar data, accuracy is significantly enhanced, achieving effects impossible with two-dimensional data alone, and making the target object information more prominent. Simultaneously, this embodiment adjusts the image after background replacement using depth information, making the image more reasonable, more complete in three-dimensional space, conforming to human visual characteristics, achieving better results, and enabling interaction between the target object and the background.

[0090] Figure 4 This is a flowchart illustrating a step in identifying a target object using an image to be replaced, according to an embodiment of the present invention. Figure 4 As shown in the figure, an embodiment of the present invention provides a method for identifying a target object through an image to be replaced, comprising the following steps:

[0091] S121: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions.

[0092] In this step, data from three channels—R, G, and B—are extracted from the image to be replaced. Each channel has corresponding data features. Since the target object regions differ across the three channels, the target objects in each channel need to be processed separately. Target object identification is performed on each of the three channels, resulting in three distinct target object regions. The identification method is the same as described in the previous embodiment.

[0093] S122: Add the three target object regions together on the image to be replaced to obtain the target object regions.

[0094] In this step, the three target object regions are converted into regions of corresponding pixels in the image to be replaced. Then, an OR operation is performed on the three target object regions to obtain a total selection area that includes all three target object regions; this is the target object. This step ensures that all target objects identified on all three channels are recognized, thus guaranteeing more complete identification.

[0095] S123: Extend a certain number of pixels to both sides along the outline of the target object area to obtain the edge area.

[0096] In this step, the edge region includes both information about the target object and information about the background. The contour line is the boundary line of the target object region. The contour line can be a single closed curve or composed of multiple closed curves. The edge region includes both information about the target object and information about the background. The specified pixel value can be adjusted according to the image size to ensure that the true boundary is within the edge region.

[0097] S124: In the edge region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, determine the pixel as a boundary.

[0098] In this step, the depth change rate is calculated from the inside out along the perpendicular line to the tangent of the contour line. When the depth change rate reaches or exceeds a preset value, the pixel is defined as a boundary. The boundary is part of the target object. The preset value varies depending on the target object and the application scenario. For example, if the target object is relatively smooth and the application scenario has a large depth, the preset value can be set to a smaller value; conversely, it needs to be set to a larger value. In some embodiments, for areas where pixels are not adjacent, a Laplacian Smooth algorithm is applied to adjust the boundary positions in those areas to ensure that all boundaries form a closed line.

[0099] S125: Crop the region within the boundary in the RGBD image to obtain the target object.

[0100] In this step, the boundary range determined in step 204 is extracted from the RGBD image to obtain the target object.

[0101] This embodiment utilizes the target object identified in the image to be replaced, expands the edge region, and then uses depth data for identification to obtain the target object. This embodiment greatly improves the accuracy of identification by utilizing depth information, making the obtained target object more accurate. At the same time, since only the edge region needs to be calculated, the amount of computation involved is small, and real-time image output can be achieved.

[0102] Figure 5 This is a flowchart illustrating a step in adjusting and replacing the overall depth value of an image according to an embodiment of the present invention. Figure 5 As shown, the method for adjusting and replacing the overall depth value of an image includes the following steps:

[0103] S141: Obtain the depth range a of the target object.

[0104] In this step, the depth values ​​at different locations of the target object are different, so it is necessary to obtain its depth range 'a'. Generally, the depth change of a target object is continuous, but for some target objects, such as regular trapezoids, there are indeed discontinuous cases. Regardless of whether the depth change of the target object is continuous or discontinuous, its minimum and maximum values ​​are taken, and these minimum and maximum values ​​are used as the two endpoints of the depth range. For example, if the minimum depth value of the target object is 1050 and the maximum value is 1520, then its depth range 'a' is [1050, 1520].

[0105] S142: Obtain the depth range b of the part in the replacement image that corresponds to the target object.

[0106] In this step, similar to the previous step, the maximum and minimum values ​​of the parts of the image corresponding to the target object are taken as the two endpoints of the depth range b, thus obtaining the depth range b.

[0107] S143: If a is less than b and they have no intersection, then increase a or decrease b so that the largest element in a is equal to the smallest element in b.

[0108] In this step, if 'a' is less than 'b' and they have no intersection, the target object and the replacement image are detached, with the target object in front and the replacement image behind, presenting a "floating" state with poor visual effect. It's necessary to bring the two objects closer together. For example, if 'b' is [2200, 2250] and 'a' is [1050, 1520], the difference between the maximum value of 'a' and the minimum value of 'b' is 680. Therefore, add 680 to 'a', resulting in an adjusted 'a' of [1730, 2200]. Now, 'a' and 'b' are connected, resulting in a better visual effect.

[0109] S144: If a is greater than b and there is no intersection, then decrease a or increase b so that the largest element in a is equal to the smallest element in b.

[0110] In this step, if a is greater than b and they have no intersection, the target object is separated from the replacement image, and the composite image is occluded by the background image due to the depth data. For example, if b is [200, 250] and a is [1050, 1520], the minimum value of a differs from the maximum value of b by 800. Therefore, 800 is added to b as a whole, resulting in an adjusted b of [1000, 1050]. At this point, a and b are connected, resulting in a better visual effect.

[0111] S145: If a and b have an intersection, adjust a so that the largest element in a is equal to the smallest element in b.

[0112] In this step, if a and b intersect, it means the target object is similar to the replacement image but misaligned, potentially causing partial occlusion of the target object. In this case, adjust a so that the largest element in a equals the smallest element in b. For example, if b is [1200, 1250] and a is [1050, 1520], the difference between the largest element of a and the smallest element of b is 320. Subtract 320 from a to obtain the adjusted a as [730, 1200]. Now a and b are connected, resulting in a better visual effect.

[0113] It should be noted that in this embodiment, if b is adjusted, the same operation needs to be performed on the depth data of other areas of the replacement image simultaneously, so that the replacement image itself maintains good consistency.

[0114] This embodiment adjusts the depth values ​​of the target object and the replacement image to keep their depth ranges adjacent, ensuring that the target object is not occluded and is located in front. Furthermore, the target object and the replacement image remain close in depth map, resulting in good depth consistency and a good visual effect.

[0115] Figure 6 This is a flowchart illustrating a step in placing a target object within a replacement image according to an embodiment of the present invention. Figure 6 As shown in the figure, an embodiment of the present invention provides a method for placing a target object in the replacement image, comprising the following steps:

[0116] S151: Align the replacement image with the image to be replaced.

[0117] In this step, the replacement image is aligned pixel-wise with the image to be replaced. The angle of the replacement image is close to the angle of the image to be replaced, and the aligned replacement image needs to be the same size as the image to be replaced. If the size of the aligned replacement image is different from the size of the image to be replaced, the replacement image needs to be processed to make its size the same as the image to be replaced.

[0118] S152: Extract the target object.

[0119] In this step, the target object from the aforementioned embodiments is obtained to facilitate the processing of background information.

[0120] S153: Replace the background information in the image to be replaced with the aligned replacement image.

[0121] S154: Place the target object on a layer above the replacement image.

[0122] In this step, considering that some scenarios require the target object to stand out more, it is necessary to make the target object more obvious than the replacement image. Therefore, the target object needs to be placed on the upper layer to present a certain highlighting effect.

[0123] This embodiment extracts the target object by aligning the replacement image with the image to be replaced, replaces the background, and then adjusts the depth information to obtain a more reasonable depth map, resulting in a better visual experience. By adjusting the depth data, this embodiment produces an image with better visual effects, placing the target object in a prominent position to highlight it. This approach can be widely applied in various scenarios.

[0124] Figure 7 This is a flowchart illustrating another step in an embodiment of the present invention: placing the target object within the replacement image.Figure 7 As shown, another method for placing a target object in the replacement image in an embodiment of the present invention includes the following steps:

[0125] S151: Align the replacement image with the image to be replaced.

[0126] S152: Extract the target object.

[0127] S155: Reassemble the target object and the replacement image based on depth information.

[0128] In this step, instead of simply replacing the target object's data with the replacement image, the target object and the replacement image are reconstructed based on depth information. This means the target object is placed within the 3D space shown by the replacement image, thereby generating a new 3D image. For example, if the target object is a person at a distance of 3m-3.2m, and the replacement image is 5m away from the target object, the person will be in the foreground of the reconstructed image, similar to the previous embodiment. If the replacement image is 2.5m away from the target object, this portion will occlude the person, and this occlusion will appear as such in the final image.

[0129] In some embodiments, if the target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object. Since the overlap between the target object and the replacement image in three-dimensional space would cause the final image to violate natural principles, it is necessary to squeeze the overlapping parts. The squeezing in this embodiment is three-dimensional squeezing. The squeezing uses a mesh squeezing algorithm, which can deform adjacent areas to a certain extent, resulting in a more natural appearance. This embodiment does not need to consider the material of the background object; instead, the background area actively adapts to the target object, ensuring the effect of the target object.

[0130] In some embodiments, if regional influencing factors exist in the replacement image, response features are added to the target object. These regional influencing factors refer to factors that can affect other objects in three-dimensional space, such as the sun, lamplight, and wind. Since regional influencing factors affect not only the background but also the target object, it is necessary to process the target object accordingly and add response features based on the type of regional influencing factor. For example, when there is sun or lamplight, facial shadows and shadows are added to the target object.

[0131] This embodiment places the target object within the three-dimensional space of the replacement image and considers the relationships between them, increasing interactive influencing factors to make the three-dimensional image effect more reasonable and realistic. This embodiment maximizes the realism of the image after replacement, allowing the target object to be placed in various scenes and meeting more market demands.

[0132] Figure 8 This is a schematic diagram of a device for changing the background of a video using a depth camera, according to an embodiment of the present invention. Figure 8 As shown in the figure, an embodiment of the present invention provides a device for changing the background using an RGBD image, comprising:

[0133] At least one processor 710, at least one memory 720, a bus 730 connecting different platform components (including memory 720 and processor 710), a display 740, etc.

[0134] The memory 720 stores program code that can be executed by the processor 710, causing the processor 710 to perform the steps described in the section on a method for replacing the background using an RGBD image, based on various exemplary embodiments of the present invention. For example, the processor 710 can perform actions such as... Figure 1 The steps are shown in the figure.

[0135] The memory 720 may include a readable medium in the form of volatile memory cells, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include read-only memory (ROM) 7203.

[0136] The memory 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0137] Bus 730 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0138] A device for changing the background using an RGBD image can communicate with one or more external devices 700 (including depth cameras, and may also be auxiliary devices such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable the user to interact with this embodiment, and / or with any device that enables this embodiment to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, this embodiment can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of this embodiment via bus 730. It should be understood that, although... Figure 8 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with this embodiment, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0139] Figure 9 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. (Reference) Figure 9 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0142] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0144] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for replacing the background of a video using a depth camera, characterized in that, Includes the following steps: S1: Acquire the p-th frame RGBD image, identify the target object, mark the area where the target object is located as the target object area, and then replace the background; wherein, the RGBD image is obtained by a depth camera, and p is a positive integer; S2: Extend the target object region outward by n pixels to obtain the extended region; S3: Acquire the RGBD image of frame p+1 and determine the target region corresponding to the extended region of frame p; S4: In the target area of ​​the (p+1)th frame RGBD image, the target boundary is confirmed based on the depth information, and the precise target object is obtained, and then the background is replaced; S1 includes: S11: Obtain the image to be replaced; wherein, the image to be replaced is an RGBD image; S12: Identify the target object through the image to be replaced; S13: Establish a pixel-level correspondence between the image to be replaced and the replacement image; wherein, the replacement image is an RGBD image; S14: Adjust the overall depth value of the replacement image or target object to match the depth value of the target object; S15: Place the target object in the replacement image to obtain an image with a changed background; S12 includes: S121: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions; S122: Add the three target object regions on the image to be replaced to obtain the target object regions; S123: Extend a certain number of pixels to both sides along the outline of the target object region to obtain an edge region; the edge region includes both the information of the target object and the information of the background; S124: In the edge region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, the pixel is determined as a boundary. S125: Crop the region within the boundary in the RGBD image to obtain the target object.

2. The method for changing the background of a video using a depth camera according to claim 1, characterized in that, S14 includes: S141: Obtain the depth range a of the target object; S142: Obtain the depth range b of the part in the replacement image that corresponds to the target object; S143: If a is less than b and there is no intersection, then increase a or decrease b so that the largest element in a is equal to the smallest element in b; S144: If a is greater than b and there is no intersection, then decrease a or increase b so that the largest element in a is equal to the smallest element in b. S145: If a and b have an intersection, adjust a so that the largest element in a is equal to the smallest element in b.

3. A method for replacing the background of a video using a depth camera, characterized in that, Includes the following steps: S1: Acquire the p-th frame RGBD image, identify the target object, mark the area where the target object is located as the target object area, and then replace the background; wherein, the RGBD image is obtained by a depth camera, and p is a positive integer; S2: Extend the target object region outward by n pixels to obtain the extended region; S3: Acquire the RGBD image of frame p+1 and determine the target region corresponding to the extended region of frame p; S4: In the target area of ​​the (p+1)th frame RGBD image, the target boundary is confirmed based on the depth information, and the precise target object is obtained, and then the background is replaced; S1 includes: S11: Obtain the image to be replaced; wherein, the image to be replaced is an RGBD image; S12: Identify the target object through the image to be replaced; S13: Establish a pixel-level correspondence between the image to be replaced and the replacement image; wherein, the replacement image is an RGBD image; S14: Adjust the overall depth value of the replacement image or target object to match the depth value of the target object; S15: Place the target object in the replacement image to obtain an image with a changed background; S15 includes: S151: Align the replacement image with the image to be replaced; S152: Extract the target object; S155: Reassemble the target object and the replacement image based on depth information; In S155, if the target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object.

4. The method for changing the background of a video using a depth camera according to claim 3, characterized in that, In S155, if there are regional influencing factors in the replacement image, a response feature is added to the target object; the regional influencing factors refer to factors that can affect other objects in three-dimensional space.

5. A device for replacing video backgrounds using a depth camera, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the method for changing the video background using a depth camera as described in any one of claims 1 to 4 by executing the executable instructions.

6. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the method for changing the video background using a depth camera as described in any one of claims 1 to 4.

Images