A method for replacing background using rgb image and depth image

Abstract

A method for replacing background by using RGB image and depth image, characterized in that, comprising the following steps: S1: obtaining the RGB image and the depth image of the same scene; S2: identifying the target object through the RGB image; S3: establishing the pixel-level correspondence between the RGB image and the replacement picture; S4: optimizing the edge of the target object according to the depth image to obtain the accurate target object; S5: placing the accurate target object in the replacement picture to obtain the image after replacing the background. The present application can quickly respond while obtaining the high-precision target object, has low requirement on computing power, and can process the video in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a method for replacing background using RGB image and depth map. BACKGROUND

[0002] In many scenarios of work and life, it is necessary to extract target objects in images and replace picture images, such as ID photos, promotional photos, etc. In many application scenarios, continuous processing of videos is required, such as anchors, TV programs, video advertisements, etc. Through replacement of the background, the corresponding production cost can be greatly reduced, and the time and workload required for preparation work can be saved.

[0003] In the prior art, the foreground and background are identified by judging the RGB image to realize extraction of the target object.

[0004] A certain specification embodiment discloses a face recognition method and device and electronic equipment, and the method comprises: acquiring a to-be-recognized face image; obtaining a plurality of first images respectively used to represent the diffusion speed of pixel point channel values of each channel according to the to-be-recognized face image; determining a second image used to represent the gray amount diffusion speed of the to-be-recognized face image based on the plurality of first images; and recognizing the to-be-recognized face image according to the second image. In the recognition process, whether the to-be-recognized face image is a living body can be recognized according to the gray amount diffusion speed image by using a preset deep learning model.

[0005] A certain application proposes an image recognition method, device, computer equipment and storage medium, wherein the method comprises: acquiring an image to be recognized, and using a trained first convolutional neural network model to perform image recognition on the to-be-recognized image to determine the object displayed in the to-be-recognized image, wherein the first convolutional neural network model comprises a first path convolutional neural network for extracting global features of an image and a second path convolutional neural network for extracting local features of an image. The to-be-recognized image is recognized by using the trained first convolutional neural network model.

[0006] The image recognition effect obtained by the prior art scheme is poor, especially for edge and target recognition. Some prior art uses a large amount of calculation to improve the recognition accuracy, but the data processing amount is greatly increased, which makes it difficult to process a large amount of data in real time, and it is even impossible to process videos in real time. SUMMARY

[0007] Therefore, the present application identifies the target object by RGB image, and quickly optimizes it by depth map, so as to obtain high-precision target objects while quickly responding, and has low computing power requirement, and can process videos in real time.

[0008] In a first aspect, the present invention provides a method for changing the background using an RGB image and a depth image, characterized by comprising the following steps:

[0009] S1: Obtain an RGB image and a depth image of the same scene; wherein the RGB image and the depth image are captured from similar angles;

[0010] S2: Identify the target object using the RGB image;

[0011] S3: Establish the pixel-level correspondence between the RGB image and the replacement image;

[0012] S4: Optimize the edges of the target object based on the depth map to obtain the accurate target object;

[0013] S5: Place the precise target object into the replacement image to obtain an image with a changed background.

[0014] Optionally, the method for changing the background using an RGB image and a depth image is characterized in that step S2 includes:

[0015] S201: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions;

[0016] S202: Add the three target object regions together on the RGB image to obtain the target object.

[0017] Optionally, the method for changing the background using an RGB image and a depth image is characterized in that step S4 includes:

[0018] S401: Extend a certain number of pixels to both sides along the outline of the target object to obtain an edge region; the edge region includes both the information of the target object and the information of the background;

[0019] S402: Determine the depth region in the depth map that corresponds to the edge region;

[0020] S403: In the depth region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, the pixel is determined as a boundary.

[0021] S404: Based on the boundary in the depth region, determine the corresponding pixel in the RGB image and mark it as the boundary;

[0022] S405: Crop the image region within the boundaries of the RGB image and the depth image to obtain the precise target object in the RGB image and the depth image.

[0023] Optionally, the method for changing the background using an RGB image and a depth image is characterized in that step S5 includes:

[0024] S501: Align the replacement image with the RGB image;

[0025] S502: Extract the precise target object;

[0026] S503: Replace the background information in the RGB image and the depth image with the aligned replacement image;

[0027] S504: Place the precise target object on a layer above the replacement image.

[0028] Optionally, the method for changing the background using an RGB image and a depth image is characterized in that step S5 includes:

[0029] S501: Align the replacement image with the RGB image;

[0030] S502: Extract the precise target object;

[0031] S505: Reassemble the precise target object and the replacement image based on depth information.

[0032] Optionally, the method for replacing the background using RGB and depth maps is characterized in that, in step S505, if the precise target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object.

[0033] Optionally, the method for replacing a background using an RGB image and a depth image is characterized in that, in step S505, if there are regional influencing factors in the replacement image, a response feature is added to the precise target object; the regional influencing factors refer to factors that can affect other objects in three-dimensional space.

[0034] Secondly, the present invention provides a method for changing the background of a video using RGB images and depth images, characterized in that the above method is performed on each frame of the video.

[0035] Thirdly, the present invention provides a device for changing the background using an RGB image and a depth image, characterized in that it includes:

[0036] processor;

[0037] A memory in which executable instructions of the processor are stored;

[0038] The processor is configured to perform the steps of any of the above-described methods for changing the background using an RGB image and a depth image by executing the executable instructions.

[0039] Fourthly, the present invention provides a computer-readable storage medium for storing a program, characterized in that, when the program is executed, it implements the steps of the method for changing the background using an RGB image and a depth image as described above.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention uses RGB images for feature extraction, leveraging various mature algorithms and models already available in the technology. It can be easily integrated with various software, improving system usability. Furthermore, this invention can acquire images using existing RGB cameras, eliminating the need for a complete set of RGB and depth maps, thus minimizing costs and facilitating its widespread application.

[0042] This invention utilizes depth data to optimize target objects, thereby reducing the accuracy requirements for target object recognition, lowering algorithm requirements, increasing computational speed, and enhancing data processing capabilities, thus adapting to real-time video processing.

[0043] This invention optimizes RGB images using depth data, improving the accuracy of data recognition, especially for edge recognition. By adding three-dimensional information to two-dimensional planar data, the accuracy is significantly improved, achieving effects that cannot be achieved with two-dimensional data, and making the information of the target object more obvious. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 This is a flowchart illustrating the steps of a method for changing the background using an RGB image and a depth image, as described in an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating a step for identifying a target object using an RGB image according to an embodiment of the present invention.

[0047] Figure 3This is a flowchart illustrating the steps of optimizing the edge of a target object using a depth map, as described in an embodiment of the present invention.

[0048] Figure 4 This is a flowchart illustrating a step in an embodiment of the present invention to place a precise target object within the replacement image;

[0049] Figure 5 This is a flowchart illustrating another step in an embodiment of the present invention for placing a precise target object in the replacement image;

[0050] Figure 6 This is a video illustration in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of a device for changing the background using RGB and depth maps, as described in an embodiment of the present invention.

[0052] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0054] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0056] This invention provides a method for changing the background using RGB and depth maps, aiming to solve the problems existing in the prior art.

[0057] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart illustrating the steps of a method for changing the background using an RGB image and a depth image, as described in an embodiment of the present invention. Figure 1 As shown in the figure, an embodiment of the present invention provides a method for changing the background using an RGB image and a depth image, comprising the following steps:

[0059] Step S1: Obtain the RGB image and depth image of the same scene.

[0060] In this step, the RGB image and the depth image are captured from similar angles, and the resolution of the depth image is not less than the resolution of the RGB image. Preferably, the difference between the capture angles of the RGB image and the depth image does not exceed 9.5 degrees. If the difference between the capture angles of the RGB image and the depth image exceeds 9.5 degrees, the alignment quality of the two images is poor, and only the target object located in the center can obtain a good alignment effect. The depth image can be obtained by any of a structured light camera, a stereo camera, or a TOF camera.

[0061] Step S2: Identify the target object using the RGB image.

[0062] In this step, target object identification can be performed using a combination of manually generated features and a classifier, such as using Haar features and AdaBoost. Alternatively, features can be obtained through deep learning, classified using a classifier, and the obtained reference object region can be adjusted using regression algorithms, such as cascaded convolutional neural networks. Furthermore, the detection accuracy of the obtained reference object region can be adjusted according to requirements, making it more flexible and easier to apply. Target object identification can also be performed using a pre-trained object detection model based on a convolutional neural network, and so on. This embodiment does not limit the method used for target object identification.

[0063] Step S3: Establish the pixel-level correspondence between the RGB image and the replacement image.

[0064] In this step, the replacement image is a complete image, and its size is not smaller than the RGB image. Preferably, the image size of the replacement image is the same as the size of the RGB image. The original image size of the replacement image can be larger than or equal to the size of the RGB image. When the original image size of the replacement image is larger than the size of the RGB image, the replacement image needs to be cropped or otherwise reduced in size so that its final size is the same as the size of the RGB image. This step establishes a one-to-one correspondence between the RGB image and the replacement image at the pixel level, so that each pixel in the RGB image has a corresponding unique pixel in the replacement image. The replacement image in this step contains depth data, thus providing three-dimensional spatial information. The replacement image can be composed of a color image and a depth image, and all operations on the replacement image are performed simultaneously on both images so that the data of the replacement image can ultimately directly correspond to the RGB image. In the following sections of this specification, "depth image" refers only to the "depth image" corresponding to the RGB image.

[0065] Step S4: Optimize the edges of the target object based on the depth map to obtain the accurate target object.

[0066] In this step, depth data is used to optimize the edges of the target object. This is because object recognition in RGB images often has significant errors in edge regions, while depth data can better identify edge regions. Combining the advantages of RGB and depth images allows for the rapid acquisition of accurate target objects. It should be noted that this step yields accurate target objects in both the RGB and depth images, with a pixel-level one-to-one correspondence between the accurate target objects in the two images.

[0067] Step S5: Place the precise target object into the replacement image to obtain an image with a changed background.

[0068] In this step, the precise target object is placed within the replacement image. The precise target object will then cover the area overlapping it on the two-dimensional plane, resulting in an image with a changed background. Since the RGB image and the replacement image have the same size, the image with the changed background also has the same size as the RGB image. This step preserves all information about the precise target object and the replacement image, especially the three-dimensional data of the overlapping portion on the two-dimensional plane. This makes the data more comprehensive, allowing the image with the changed background to better match the characteristics observed by the human eye, resulting in a better effect.

[0069] Figure 2 This is a flowchart illustrating a step in identifying a target object using an RGB image, as described in an embodiment of the present invention. Figure 2As shown in the figure, an embodiment of the present invention provides a method for identifying target objects using RGB images, comprising the following steps:

[0070] S201: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions.

[0071] In this step, since the RGB image is composed of three channels—R, G, and B—each channel has corresponding data features. Because the target object regions differ across the three channels, each target object needs to be processed separately. Target object identification is performed on each of the three channels, resulting in three distinct target object regions. The identification method is the same as described in the previous embodiment.

[0072] S202: Add the three target object regions together on the RGB image to obtain the target object.

[0073] In this step, the three target object regions are converted into corresponding pixel regions in the RGB image, and an OR operation is performed on the three target object regions to obtain a total selection area that includes all three target object regions, which is the target object. This step ensures that the target objects identified on all three channels are recognized, thus guaranteeing more complete recognition.

[0074] Figure 3 This is a flowchart illustrating the steps of optimizing the edges of a target object using a depth map, as described in an embodiment of the present invention. Figure 3 As shown, the method for optimizing the edges of a target object using a depth map includes the following steps:

[0075] S401: Extend a certain number of pixels to both sides along the outline of the target object to obtain the edge region.

[0076] In this step, the contour line is the boundary line of the target object region. The contour line can be a single closed curve or composed of multiple closed curves. The edge region includes information about both the target object and the background. The specified pixel values ​​can be adjusted according to the image size to ensure that the true boundary is within the edge region.

[0077] S402: Determine the depth region in the depth map that corresponds to the edge region.

[0078] In this step, the edge region is a portion of the RGB image. Since the RGB image and the depth image have a pixel-level one-to-one correspondence, the depth region corresponding to the edge region in the depth image can be calculated. Similar to the edge region, the depth region includes both the depth information of the target object and the depth information of the background.

[0079] S403: In the depth region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, determine the pixel as a boundary.

[0080] In this step, the depth change rate is calculated from the inside out along the perpendicular line to the tangent of the contour line. When the depth change rate reaches or exceeds a preset value, the pixel is defined as a boundary. The boundary is part of the target object. The preset value varies depending on the target object and the application scenario. For example, if the target object is relatively smooth and the application scenario has a large depth, the preset value can be set to a smaller value; conversely, it needs to be set to a larger value. In some embodiments, for areas where pixels are not adjacent, a Laplacian Smooth algorithm is applied to adjust the boundary positions in those areas to ensure that all boundaries form a closed line.

[0081] S404: Based on the boundary in the depth region, determine the corresponding pixel in the RGB image and mark it as the boundary.

[0082] In this step, the boundaries in the RGB image are determined based on the one-to-one correspondence between the RGB image and the depth image. The boundaries determined in this step are more precise than those determined by the contour line, and only a small number of areas are adjusted based on the contour line, resulting in less computation.

[0083] S405: Crop the image region within the boundaries of the RGB image and the depth image to obtain the precise target object in the RGB image and the depth image.

[0084] In this step, the boundary ranges determined in steps S403 and S404 are extracted from the RGB image and the depth image respectively, to obtain the precise target object in the depth image and the precise target object in the RGB image.

[0085] This embodiment utilizes the target object identified in the RGB image to expand the edge region, and then uses the depth map for further identification to obtain a precise target object. This embodiment significantly improves the accuracy of identification by utilizing depth information, resulting in a more accurate target object. Furthermore, since only the edge region needs to be calculated, the computational load is small, enabling real-time image output.

[0086] Figure 4 This is a flowchart illustrating a step in an embodiment of the present invention to place a precise target object within the replacement image. Figure 4 As shown in the figure, an embodiment of the present invention provides a method for placing a precise target object in the replacement image, comprising the following steps:

[0087] S501: Align the replacement image with the RGB image.

[0088] In this step, the replacement image is aligned pixel-wise with the RGB image. The angle of the replacement image is close to that of the RGB image, and the aligned replacement image needs to be the same size as the RGB image. If the size of the aligned replacement image is different from the size of the RGB image, the replacement image needs to be processed to make its size the same as the RGB image.

[0089] S502: Extract the precise target object.

[0090] In this step, the precise target object from the aforementioned embodiments is obtained to facilitate the processing of background information. This precise target object includes both RGB image data and depth image data.

[0091] S503: Replace the background information in the RGB image and the depth image with the aligned replacement image.

[0092] In this step, the background information is replaced simultaneously in the RGB image and the depth image, that is, the data outside the boundary of the precise target object is replaced with the information of the replacement image.

[0093] S504: Place the precise target object on a layer above the replacement image.

[0094] In this step, since the previous step included RGB images and depth images, the depth values ​​in some background areas are smaller than the depth values ​​of the precise target object, resulting in visual imbalance. Therefore, it is necessary to adjust the depth values ​​of the precise target object so that it is located in the layer above the replacement image to improve visual balance.

[0095] This embodiment extracts the precise target object by aligning the replacement image with the RGB image, replacing the background, and then adjusting the depth information to obtain a more reasonable depth map, resulting in a better visual experience. By adjusting the depth data, this embodiment achieves better visual effects in the processed image, placing the precise target object in a prominent position to highlight it. This approach can be widely applied in various scenarios.

[0096] Figure 5This is a flowchart illustrating another step in an embodiment of the present invention: placing a precise target object within the replacement image. Figure 5 As shown, another method for placing a precise target object in the replacement image according to an embodiment of the present invention includes the following steps:

[0097] S501: Align the replacement image with the RGB image.

[0098] S502: Extract the precise target object.

[0099] S505: Reassemble the precise target object and the replacement image based on depth information.

[0100] In this step, instead of simply replacing the data of the precise target object with the replacement image, the precise target object and the replacement image are recombined based on depth information. This means the precise target object is placed within the 3D space shown by the replacement image, thereby generating a new 3D image. For example, if the precise target object is a person at a distance of 3m-3.2m, and the replacement image is 5m away from the precise target object, the person will be in the foreground of the recombined image, similar to the previous embodiment. If the replacement image is 2.5m away from the precise target object, this portion will occlude the person, and this occlusion will appear as occlusion in the final image.

[0101] In some embodiments, if the precise target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object. Because the overlap between the precise target object and the replacement image in three-dimensional space can cause the final image to violate natural principles, it is necessary to squeeze the overlapping portion. In this embodiment, the squeezing is in three-dimensional space. The squeezing uses a mesh squeezing algorithm, which can deform adjacent areas to a certain extent, resulting in a more natural appearance. This embodiment does not need to consider the material of the background object; instead, the background area actively adapts to the precise target object, ensuring the effectiveness of the precise target object.

[0102] In some embodiments, if regional influencing factors exist in the replacement image, response features are added to the precise target object. These regional influencing factors refer to factors that can affect other objects in three-dimensional space, such as the sun, lamplight, and wind. Since regional influencing factors not only affect the background but also the precise target object, it is necessary to process the precise target object accordingly, adding response features based on the type of regional influencing factor. For example, when there is sun or lamplight, facial shadows and shadows are added to the precise target object.

[0103] This embodiment places the precise target object within the three-dimensional space of the replacement image, considering the relationships between them and adding interactive influencing factors to make the three-dimensional image effect more reasonable and realistic. This embodiment maximizes the realism of the image after replacement, allowing the target object to be placed in various scenes, meeting more market demands.

[0104] Figure 6 This is a video illustration from an embodiment of the present invention. Figure 6 As shown, a video illustration in an embodiment of the present invention includes multiple frames, and each frame executes the method described in the foregoing embodiment.

[0105] The following explanation uses frames M1 and M2 as examples. Frames M1 and M2 are spaced several frames apart. When acquiring the video, the method described in the previous embodiment is applied to each frame of the video. In this embodiment, the precise target object is a person. In the original video, the person is indoors and standing. The processed video shows the person outdoors throughout the day, with the sun rising in the east and setting in the west. Frame M1 is at 9:39, and frame M2 is at 9:40. The sun is at different directions and altitudes, resulting in shadows with different angles in the processed image, and creating different effects when interacting with objects in the background. This embodiment enables interaction between the target object and the background through video and does not require a motion-sensing interaction device, thus reducing costs.

[0106] Figure 7 This is a schematic diagram of a device for changing the background using RGB and depth maps, according to an embodiment of the present invention. Figure 7 As shown in the figure, an embodiment of the present invention provides a device for changing the background using an RGB image and a depth image, comprising:

[0107] At least one processor 710, at least one memory 720, a bus 730 connecting different platform components (including memory 720 and processor 710), a display 740, etc.

[0108] The memory 720 stores program code that can be executed by the processor 710, causing the processor 710 to perform the steps described in the section on "A Method for Changing Background Using RGB Images and Depth Images" of this specification, based on various exemplary embodiments of the present invention. For example, the processor 710 can perform actions such as... Figure 1 The steps are shown in the figure.

[0109] The memory 720 may include a readable medium in the form of volatile memory cells, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include read-only memory (ROM) 7203.

[0110] The memory 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0111] Bus 730 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0112] A device that uses RGB and depth maps to change the background can communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and also with one or more devices that enable a user to interact with this embodiment, and / or with any device that enables this embodiment to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, this embodiment can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of this embodiment via bus 730. It should be understood that, although... Figure 7 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with this embodiment, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0113] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. (Reference) Figure 8 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0116] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0117] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0118] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for changing the background using an RGB image and a depth image, characterized in that, Includes the following steps: S1: Obtain the RGB image and depth image of the same scene; S2: Identify the target object using the RGB image; S3: Establish the pixel-level correspondence between the RGB image and the replacement image; S4: Optimize the edges of the target object based on the depth map to obtain the accurate target object; S5: Place the precise target object into the replacement image to obtain an image with a changed background; Step S4 includes: S401: Extend a certain number of pixels to both sides along the outline of the target object to obtain an edge region; the edge region includes both the information of the target object and the information of the background; S402: Determine the depth region in the depth map that corresponds to the edge region; S403: In the depth region, calculate the depth change rate from the target object to the background. When the depth change rate reaches or exceeds a preset value, the pixel is determined as a boundary. S404: Based on the boundary in the depth region, determine the corresponding pixel in the RGB image and mark it as the boundary; S405: Crop the image region within the boundaries of the RGB image and the depth image to obtain the precise target object in the RGB image and the depth image.

2. The method for changing the background using an RGB image and a depth image according to claim 1, characterized in that, Step S2 includes: S201: Identify the target object through the R channel, G channel and B channel respectively to obtain three target object regions; S202: Add the three target object regions together on the RGB image to obtain the target object.

3. The method for changing the background using an RGB image and a depth image according to claim 1, characterized in that, Step S5 includes: S501: Align the replacement image with the RGB image; S502: Extract the precise target object; S503: Replace the background information in the RGB image and the depth image with the aligned replacement image; S504: Place the precise target object on a layer above the replacement image.

4. The method for changing the background using an RGB image and a depth image according to claim 1, characterized in that, Step S5 includes: S501: Align the replacement image with the RGB image; S502: Extract the precise target object; S505: Reassemble the precise target object and the replacement image based on depth information.

5. The method for changing the background using an RGB image and a depth image according to claim 4, characterized in that, In step S505, if the precise target object and the replacement image overlap in three-dimensional space, the overlapping content in the replacement image is squeezed away from the target object.

6. The method for changing the background using an RGB image and a depth image according to claim 4, characterized in that, In step S505, if there are regional influencing factors in the replacement image, then a response feature is added to the precise target object; The regional influencing factors refer to factors that can affect other objects in three-dimensional space.

7. A method for changing the background of a video using RGB images and depth images, characterized in that, Perform the method of any one of claims 1-6 on each frame of the video.

8. A device for changing the background using an RGB image and a depth image, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the method for changing the background using an RGB image and a depth image as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the method for changing the background using an RGB image and a depth image as described in any one of claims 1 to 6.

Images