Image rendering method and device

Abstract

The embodiment of the present application discloses an image rendering method and device, the method comprising: obtaining an image to be processed; detecting that a target object in the image to be processed performs a preset action and / or that the target object is blocked by a first object; determining a second virtual object to be rendered corresponding to the preset action and / or a first virtual object to be rendered corresponding to the first object; rendering the first virtual object to be rendered and / or the second virtual object to be rendered in a target virtual background image to obtain a virtual background image to be rendered, wherein the depth value of the interactive object corresponding to the second virtual object to be rendered is greater than the depth value of the target object; performing image rendering according to the virtual background image to be rendered and the main image to obtain a rendered image. The embodiment of the present application can improve the rationality and authenticity of the rendered image, thereby improving the virtual background replacement effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image rendering method and device. Background Art

[0002] With the continuous development of image processing technology, more and more products have the function of virtual background replacement, such as Changlian calls, Zoom video conferencing and Clips.

[0003] Virtual background replacement refers to replacing the background of the original image with another different background. When replacing the virtual background, the original image is generally first segmented into foreground and background to obtain the foreground and original background; then the foreground and virtual background are rendered and fused to obtain the image after the background is replaced.

[0004] In the prior art, the rationality and authenticity of the image after background replacement are poor, resulting in poor background replacement effect. Summary of the invention

[0005] The embodiments of the present application provide an image rendering method and device, which can improve the virtual background replacement effect.

[0006] In a first aspect, an embodiment of the present application provides an image rendering method, which includes: a terminal device first obtains an image to be processed, which may refer to an original video image in a video stream; then detects whether a target object in the image to be processed performs a preset action and / or whether the target object is blocked by a first object; if it is detected that the target object in the image to be processed performs a preset action and / or the target object is blocked by the first object, accordingly, determine a second virtual object to be rendered corresponding to the preset action and / or a first virtual object to be rendered corresponding to the first object; then render the first virtual object to be rendered and / or the second virtual object to be rendered in a target virtual background image to obtain a virtual background image to be rendered, wherein a depth value of an interactive object corresponding to the second virtual object to be rendered is greater than a depth value of the target object; finally, image rendering is performed according to the virtual background image to be rendered and the main image to obtain a rendered image, wherein the main image is extracted from the image to be processed and includes an image of the target object.

[0007] If it is detected that the target object has not performed the preset action and is not obscured; or, the target object is not obscured and the target object has performed the preset action, but the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, then the target virtual background image is used as the background image to be rendered, and then image rendering is performed based on the virtual background image to be rendered and the main image to obtain a rendered image.

[0008] After detecting that the target object is occluded and / or performs a preset action, the embodiment of the present application renders a corresponding virtual object in the virtual background image. That is, when the target object is occluded, the corresponding virtual object is rendered to occlude it; when the target object performs an interactive action, the corresponding interactive virtual object is rendered, so as to avoid unreasonable and unrealistic phenomena in the rendered image as much as possible, improve the rationality and authenticity of the rendered image, and thus improve the virtual background replacement effect.

[0009] Exemplarily, when the target object is a person, that is, when the main image is a portrait main image, if it is detected that the portrait main image has performed the preset action of "sitting down", and it is determined that the second virtual object to be rendered corresponding to the preset action of "sitting down" is a "chair". At this time, assuming that the depth value of the interactive object (such as a stool, etc.) corresponding to the "chair" in the image to be processed is greater than the depth value of the target object, the "chair" is rendered at the corresponding position in the target virtual background image to obtain the virtual background image to be rendered. The virtual background image to be rendered and the main image are then rendered to obtain a rendered image, which is the image after the virtual background is replaced.

[0010] From a visual perspective, the portrait subject in the processed image sits on the interactive object, and the portrait subject in the rendered image will also sit on the "chair", making the image after background replacement and the original video image consistent in terms of interaction, avoiding unreasonable phenomena such as people sitting in the air in the image after background replacement.

[0011] It should be noted that when it is detected that the target object has performed a preset action and is also obscured by the first object, and the depth value of the interactive object of the second virtual object to be rendered is greater than the depth value of the target object, the first virtual object to be rendered and the second virtual object to be rendered are rendered in the target virtual background image to obtain the virtual background image to be rendered.

[0012] When it is detected that the target object has performed a preset action but is not blocked by the first object, and the depth value of the interactive object of the second virtual object to be rendered is greater than the depth value of the target object, the second virtual object to be rendered is rendered in the target virtual background image to obtain the virtual background image to be rendered.

[0013] When it is detected that the target object does not perform the preset action but is blocked by the first object; or, the target object performs the preset action, but the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, and the target object is blocked by the first object, the first virtual object to be rendered is rendered in the target virtual background image to obtain the virtual background image to be rendered.

[0014] In some possible implementations of the first aspect, the above-mentioned process of performing image rendering according to the virtual background image to be rendered and the main image to obtain the rendered image may include: performing underlying consistency rendering based on the virtual background image to be rendered to obtain the main image after underlying consistency rendering; performing image rendering according to the main image after underlying consistency rendering and the virtual background image to be rendered to obtain the rendered image.

[0015] In some possible implementations of the first aspect, the above-mentioned process of performing underlying consistency rendering based on the virtual background image to be rendered to obtain the main image after underlying consistency rendering may include: inputting the low-frequency image of the virtual background image to be rendered and the image to be processed into a pre-trained first style transfer model to obtain the image to be processed after underlying consistency rendering output by the first style transfer model; extracting the main image from the image to be processed after underlying consistency rendering to obtain the main image after underlying consistency rendering.

[0016] In this implementation, low-frequency images are used as the input of the model, and low-level features such as texture in the image can be ignored. The virtual background replacement effect can be further improved through the underlying consistent rendering process.

[0017] In some possible implementations of the first aspect, the training process of the style transfer model may include: obtaining a training data set, the training data set including a first virtual background image and an original video image; inputting the low-frequency image of the first virtual background image and the original video image into a pre-constructed second style transfer model to obtain a forward training result output by the second style transfer model; calculating a first loss value between the forward training result and the low-frequency image of the first virtual background image; inputting the forward training result and the low-frequency image of the original video image into the second style transfer model after forward training to obtain a reverse training result output by the second style transfer model after forward training; calculating a second loss value between the reverse training result and the original video image; calculating a third loss value between the reverse training result and the low-frequency image of the original video image; adjusting the network parameters of the second style transfer model according to the first loss value, and adjusting the network parameters of the second style transfer model after forward training according to the second loss value and the third loss value; repeating the training process, and when a predetermined condition is met, obtaining a trained first style transfer model.

[0018] The predetermined condition can be used to indicate that the loss value of the model tends to be stable, and specifically can refer to that the first loss value, the second loss value, and the third loss value are all stable near a certain value. When the predetermined condition is met, the model training is considered to be completed, and a first style transfer model that has been trained is obtained.

[0019] The first loss value is the loss value of the LAB space, that is, after the forward training result and the low-frequency image of the first virtual background image are transferred to the LAB space, the variance difference and mean difference of the two images in the LAB domain are calculated to constrain the global similarity of color, brightness, saturation, etc. The third loss value is also the loss value of the LAB space.

[0020] In this implementation, the model obtained through the above training process can not only ensure consistency in style, but also ensure consistency in image content. Using the first style transfer model for underlying consistency rendering can further improve the background replacement effect.

[0021] In some possible implementations of the first aspect, the above-mentioned process of performing underlying consistency rendering based on the virtual background image to be rendered to obtain the main image after underlying consistency rendering may also include: converting the virtual background image to be rendered to the LAB color space to obtain a first image; respectively calculating the first standard deviation and the first mean of the L channel, A channel, and B channel of the first image; converting the main image to the LAB color space to obtain a second image; correcting the second image according to the first standard deviation and the first mean to obtain a third image, the difference between the second standard deviation of the L channel, A channel, and B channel of the third image and the first standard is within the first preset threshold range, and the difference between the second mean and the first mean is within the second preset threshold range; converting the third image from the LAB color space to the RGB color space to obtain a fourth image, and the fourth image is the main image after underlying consistency rendering.

[0022] It can be understood that in the LAB color space, each channel has its corresponding variance and mean. According to the standard deviation and mean of each channel in the first image, the standard deviation and mean of the corresponding channel in the second image are corrected. For example, the first standard deviation of the L channel of the first image is A1, and the first mean is B1, the first standard deviation of the A channel is A2, and the first mean is B2, and the first standard deviation of the B channel is A3, and the first mean is B3. The standard deviation of the L channel of the second image is set to A1, and the mean is set to B1; the standard deviation of the A channel of the second image is set to A2, and the mean is set to B2; the standard deviation of the B channel of the second image is set to A3, and the mean is set to B3, so as to correct the second image according to the standard deviation and mean of the first image to obtain a third image.

[0023] The first preset threshold interval and the second preset threshold interval can be set according to actual needs. For example, the first preset threshold interval and the second preset threshold interval can both be set to 0.

[0024] Through the underlying consistent rendering process of this implementation method, the virtual background replacement effect can be further improved.

[0025] In some possible implementations of the first aspect, the process of obtaining a rendered image based on the main image rendered with underlying consistency and the virtual background to be rendered may include: inputting the main image rendered with underlying consistency into a first STN network that has been pre-trained to obtain a first change matrix output by the first STN network; inputting the virtual background image to be rendered into a second STN network that has been pre-trained to obtain a second change matrix output by the second STN network; using the first change matrix to perform image affine transformation on the main image rendered with underlying consistency to obtain a first change image; using the second change matrix to perform image affine transformation on the virtual background image to be rendered to obtain a second change image; and performing image synthesis on the first change image and the second change image to obtain a rendered image.

[0026] In this implementation, the STN network allows the main image to be rendered in a more reasonable position, further improving the rationality and authenticity of the image after background replacement.

[0027] In some possible implementations of the first aspect, the above-mentioned process of placing the second virtual object to be rendered in the target virtual background image may include: determining a first position of an interactive object corresponding to a preset action in the image to be processed based on a semantic segmentation result of the image to be processed; using a second position in the target virtual background image corresponding to the first position as a rendering position of the first virtual object to be rendered; determining that a depth value of the interactive object in the image to be processed is greater than a depth value of the target object; and rendering the second virtual object to be rendered at the rendering position of the target virtual background image.

[0028] In some possible implementations of the first aspect, the above-mentioned process of detecting whether the target object in the image to be processed is obscured by the first object may include: determining the category to which each pixel in the image to be processed belongs based on the semantic segmentation result of the image to be processed; obtaining the depth information of the image to be processed; when it is determined based on the depth information that there is a target pixel whose depth value is less than the depth value of the target object within a preset range of the target object, taking the category corresponding to the target pixel as the first object, and determining that the target object is obscured by the first object.

[0029] It is understandable that the preset range of the target object may refer to the preset range around the pixel corresponding to the target object. After the target pixel is determined, the target pixel is mapped to the semantic segmentation result, that is, the category of the target pixel is determined according to the category of each pixel in the semantic segmentation result, thereby determining the category of the occluded object. On the contrary, if there is no target pixel, it is considered that the target object is not occluded.

[0030] In some possible implementations of the first aspect, before rendering the first virtual object to be rendered and / or the second virtual object to be rendered in the target virtual background image to obtain the virtual background image to be rendered, the method may also include: determining the virtual background image to be recommended based on the similarity between the original background image of the image to be processed and each second virtual background image; and displaying the virtual background image to be recommended.

[0031] In this implementation, based on the similarity between the original background image and the virtual background image, a virtual background image is recommended to the user, so that the virtual background image used for background replacement is more relevant to the original background image.

[0032] In some possible implementations of the first aspect, the process of determining the virtual background image to be recommended based on the similarity between the original background image of the image to be processed and each second virtual background image may include: performing foreground and background segmentation on the image to be processed to obtain the original background image of the image to be processed; performing multi-class semantic segmentation on the original background image to obtain a second semantic segmentation result; performing multi-class semantic segmentation on each second virtual background image to obtain a third semantic segmentation result of each second virtual background image; calculating the IOU value of the original background image and each second virtual background image based on the second semantic segmentation result and the third semantic segmentation result; respectively calculating the first color distribution curve of the original background image and the second color distribution curve of each second virtual background image; calculating the curve similarity between the first color distribution curve and each second color distribution curve; and determining the virtual background image to be recommended from the second virtual background image based on the curve similarity and the IOU value.

[0033] In some possible implementations of the first aspect, the method may further include: if the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, rendering the first virtual object to be rendered in the target virtual background image to obtain the virtual background image to be rendered, or using the target virtual background image as the virtual background image to be rendered;

[0034] At this time, after performing image rendering according to the virtual background image to be rendered and the main image to obtain the rendered image, the method may further include: rendering the second virtual object to be rendered in the rendered image according to the rendering position of the second virtual object to be rendered to obtain the output image.

[0035] That is to say, when it is detected that the target object has performed a preset action, but the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, the second virtual object to be rendered is not rendered in the target virtual background image. Instead, after obtaining the rendered image, the second virtual object to be rendered and the rendered image are fused to obtain the final output image.

[0036] Among them, when the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, the rendering order between the second virtual object to be rendered and the main body is: render the main body first, and then render the second virtual object to be rendered; conversely, when the depth value of the interactive object corresponding to the second virtual object to be rendered is greater than the depth value of the target object, the rendering order between the second virtual object to be rendered and the main body is: render the second virtual object to be rendered first, and then render the main body.

[0037] In a second aspect, an embodiment of the present application provides an image rendering device, which may include:

[0038] An image acquisition module is used to acquire an image to be processed; a detection module is used to detect that a target object in the image to be processed performs a preset action and / or the target object is blocked by a first object; a virtual object determination module is used to determine a second virtual object to be rendered corresponding to the preset action and / or a first virtual object to be rendered corresponding to the first object; a virtual object rendering module is used to render the first virtual object to be rendered and / or the second virtual object to be rendered in a target virtual background image to obtain a virtual background image to be rendered, wherein a depth value of an interactive object corresponding to the second virtual object to be rendered is greater than a depth value of the target object; a rendering module is used to perform image rendering according to the virtual background image to be rendered and a main image to obtain a rendered image, wherein the main image is extracted from the image to be processed and includes an image of the target object.

[0039] In some possible implementations of the second aspect, the rendering module is specifically used to: perform underlying consistency rendering based on the virtual background image to be rendered to obtain a main image after underlying consistency rendering; perform image rendering based on the main image after underlying consistency rendering and the virtual background image to be rendered to obtain a rendered image.

[0040] In some possible implementations of the second aspect, the rendering module is specifically used to: input the low-frequency image of the virtual background image to be rendered and the image to be processed into a pre-trained first style transfer model to obtain the image to be processed after the underlying consistency rendering output by the first style transfer model; extract the main image from the image to be processed after the underlying consistency rendering to obtain the main image after the underlying consistency rendering.

[0041] In some possible implementations of the second aspect, a model training module is also included, which is used to: obtain a training data set, the training data set includes a first virtual background image and an original video image; input the low-frequency image of the first virtual background image and the original video image into a pre-built second style transfer model to obtain a forward training result output by the second style transfer model; calculate a first loss value between the forward training result and the low-frequency image of the first virtual background image; input the forward training result and the low-frequency image of the original video image into the second style transfer model after forward training to obtain a reverse training result output by the second style transfer model after forward training; calculate a second loss value between the reverse training result and the original video image; calculate a third loss value between the reverse training result and the low-frequency image of the original video image; adjust the network parameters of the second style transfer model according to the first loss value, and adjust the network parameters of the second style transfer model after forward training according to the second loss value and the third loss value; repeat the training process, and when a predetermined condition is met, a trained first style transfer model is obtained.

[0042] In some possible implementations of the second aspect, the rendering module is specifically used to: convert the virtual background image to be rendered to the LAB color space to obtain a first image; respectively calculate the first standard deviation and the first mean of the L channel, A channel, and B channel of the first image; convert the main image to the LAB color space to obtain a second image; correct the second image according to the first standard deviation and the first mean to obtain a third image, and the difference between the second standard deviation of the L channel, A channel, and B channel of the third image and the first standard is within a first preset threshold range, and the difference between the second mean and the first mean is within a second preset threshold range; convert the third image from the LAB color space to the RGB color space to obtain a fourth image, and the fourth image is the main image after underlying consistency rendering.

[0043] In some possible implementations of the second aspect, the rendering module is specifically used to: input the main image after the underlying consistency rendering into a pre-trained first STN network to obtain a first change matrix output by the first STN network; input the virtual background image to be rendered into a pre-trained second STN network to obtain a second change matrix output by the second STN network; use the first change matrix to perform image affine transformation on the main image after the underlying consistency rendering to obtain a first change image; use the second change matrix to perform image affine transformation on the virtual background image to be rendered to obtain a second change image; perform image synthesis on the first change image and the second change image to obtain a rendered image.

[0044] In some possible implementations of the second aspect, the virtual object rendering module is specifically used to: determine the first position of the interactive object corresponding to the preset action in the image to be processed according to the semantic segmentation result of the image to be processed; use the second position corresponding to the first position in the target virtual background image as the rendering position of the first virtual object to be rendered; determine that the depth value of the interactive object in the image to be processed is greater than the depth value of the target object; and render the second virtual object to be rendered at the rendering position of the target virtual background image.

[0045] In some possible implementations of the second aspect, the detection module is specifically used to: determine the category to which each pixel in the image to be processed belongs based on the semantic segmentation result of the image to be processed; obtain the depth information of the image to be processed; when it is determined based on the depth information that there is a target pixel whose depth value is less than the depth value of the target object within a preset range of the target object, take the category corresponding to the target pixel as the first object, and determine that the target object is obscured by the first object.

[0046] In some possible implementations of the second aspect, a background recommendation module is further included, which is used to: determine the virtual background image to be recommended based on the similarity between the original background image of the image to be processed and each second virtual background image; and display the virtual background image to be recommended.

[0047] In some possible implementations of the second aspect, the background recommendation module is specifically used to: perform foreground and background segmentation on the image to be processed to obtain the original background image of the image to be processed; perform multi-class semantic segmentation on the original background image to obtain a second semantic segmentation result; perform multi-class semantic segmentation on each second virtual background image to obtain a third semantic segmentation result of each second virtual background image; calculate the IOU value of the original background image and each second virtual background image based on the second semantic segmentation result and the third semantic segmentation result; calculate the first color distribution curve of the original background image and the second color distribution curve of each second virtual background image respectively; calculate the curve similarity between the first color distribution curve and each second color distribution curve; and determine the virtual background image to be recommended from the second virtual background image based on the curve similarity and the IOU value.

[0048] In some possible implementations of the second aspect, the virtual object rendering module is also used to: if the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, render the first virtual object to be rendered in the target virtual background image to obtain the virtual background image to be rendered, or use the target virtual background image as the virtual background image to be rendered; render the second virtual object to be rendered in the rendered image to obtain an output image.

[0049] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any method of the first aspect when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of any one of the above-mentioned first aspects is implemented.

[0051] In a fifth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method as described in any one of the first aspects above. The chip system can be a single chip, or a chip module composed of multiple chips.

[0052] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the first aspect above.

[0053] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application;

[0055] Figure 2 A software structure block diagram of the terminal device 100 provided in an embodiment of the present application;

[0056] Figure 3 A schematic block diagram of the process of the image rendering solution provided in the embodiment of the present application;

[0057] Figure 4 A schematic diagram of an interface for a video call scenario provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of a virtual background image provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the interface of the background recommendation process provided in the embodiment of the present application;

[0060] Figure 7 A schematic diagram of the effect of consistent rendering based on structural semantics provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the effect of consistent rendering based on interactive relationships provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of underlying consistency rendering based on a model provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the style transfer model training process provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram of the effect of underlying consistency rendering provided in an embodiment of the present application;

[0065] Fig.12 A schematic diagram of the underlying consistency rendering process based on the image processing algorithm provided in an embodiment of the present application;

[0066] Fig.13 A schematic diagram of the effect of underlying consistency rendering based on an image processing algorithm provided in an embodiment of the present application;

[0067] Fig.14 A schematic diagram of a consistent rendering process based on positional relationships provided in an embodiment of the present application;

[0068] Fig.15 A schematic diagram of the STN network training process provided in an embodiment of the present application;

[0069] Fig.16 A schematic block diagram of the effect of consistent rendering based on positional relationship provided in an embodiment of the present application;

[0070] Fig.17 A schematic diagram of a flow chart of an image rendering process provided in an embodiment of the present application;

[0071] Fig.18 Another schematic diagram of the image rendering process provided by the embodiment of the present application;

[0072] Fig.19 A schematic diagram of another process flow of an image rendering process provided in an embodiment of the present application;

[0073] Fig. 20 A schematic diagram of a video call scenario of a large-screen device provided in an embodiment of the present application;

[0074] Fig.21 A schematic diagram of a change in a virtual background replacement image provided in an embodiment of the present application;

[0075] Fig. 22 A schematic diagram of an interface for a virtual background replacement process in a shooting scenario provided by an embodiment of the present application;

[0076] Fig.23 A schematic diagram of the interface of the virtual background replacement process in the video recording scene provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In the current virtual background replacement process, the correlation between the original background and the virtual background is usually not considered when rendering the image, which leads to poor rationality and authenticity of the image after background replacement, that is, there are unreasonable or even unrealistic phenomena in the image after background replacement. The original background refers to the background in the original image.

[0078] For example, without considering the content and position in the virtual background, people are rendered on top of the tables and chairs in the virtual background, resulting in the unreasonable phenomenon of people hanging on the seats in the image after the background is replaced.

[0079] For another example, whether the human body in the original image is blocked by objects is not considered, but the person is directly rendered in the corresponding position of the virtual background, resulting in an unreasonable phenomenon in which the human body appears suspended in the air or only a part of the human body appears in the image after the background is replaced.

[0080] For another example, when a person in the original image changes from standing to sitting, the human body is directly rendered at the corresponding position of the virtual background, resulting in the unreasonable phenomenon that the person appears to be sitting in the air instead of on a supporting object such as a chair in the image after the background is replaced.

[0081] In addition, in the current virtual background replacement process, the consistency processing method of hue and brightness is single, which makes the chromaticity, brightness, etc. of the foreground inconsistent and inconsistent with the hue, brightness, etc. of the virtual background, thus affecting the fusion effect of the foreground and the virtual background.

[0082] In response to the above problems, an embodiment of the present application provides an image rendering solution, which considers the correlation between the original background and the virtual background during image rendering to improve the rationality and authenticity of the virtual background replacement, thereby improving the virtual background replacement effect.

[0083] Furthermore, the embodiment of the present application also renders the hue, brightness, contrast and color of the foreground and the virtual background consistently, so that the hue, brightness, contrast and color of the foreground are consistent with the hue, brightness, contrast and color of the virtual background, thereby improving the fusion effect of the foreground and the virtual background.

[0084] The image rendering solution provided in the embodiment of the present application can be applied to a terminal device, which can be a portable terminal device such as a mobile phone, a tablet computer, a laptop computer, or a wearable device, and can be an augmented reality (AR) device or a virtual reality (VR) device, or can also be a vehicle-mounted device, a netbook, or a smart screen. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0085] For example, Figure 1 A schematic structural diagram of a terminal device 100 is shown.

[0086] The terminal device 100 may include a processor 110 , a memory 120 , a camera 130 , a display screen 140 , and the like.

[0087] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0089] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0090] In some embodiments, the processor 110 may include one or more interfaces, such as a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and the like.

[0091] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 140 and the camera 130. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 130 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 140 communicate via the DSI interface to implement the display function of the terminal device 100.

[0092] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 130, the display screen 140, etc. It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0093] The terminal device 100 implements the display function through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0094] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 140, where N is a positive integer greater than 1.

[0095] The terminal device 100 can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 140 and the application processor.

[0096] The ISP is used to process the data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0097] The camera 130 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0098] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0099] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0100] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0101] The memory 120 can be used to store computer executable program codes, which include instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the memory 121, and / or instructions stored in a memory provided in the processor.

[0102] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0103] Figure 2 A software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0104] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0105] The application layer can include a series of application packages.

[0106] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and Changlian call.

[0107] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0108] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0109] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0110] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0111] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0112] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0113] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0114] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0115] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0116] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0117] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0118] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0119] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0120] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with a shooting scenario.

[0121] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is a control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a static image or video through the camera 130.

[0122] After the camera 130 captures an image or video, the ISP converts the electrical signal into a digital image signal. The digital image signal is then input into the DSP for processing, and the digital image signal is converted into an image signal in a standard RGB, YUV or other format. Finally, the image is displayed through the GPU, display screen, and application processor.

[0123] After acquiring the image signal, the CPU executes the image rendering scheme provided in the embodiment of the present application to perform image rendering, obtains the image after background replacement, and displays the image after background replacement through the GPU, display screen, application processor, etc.

[0124] According to the following Figure 1 and Figure 2 The terminal device 100 shown exemplarily introduces the image rendering solution provided by the embodiment of the present application.

[0125] See also Figure 3 The schematic flowchart of the image rendering solution is shown, and the image rendering solution may include the following steps:

[0126] Step S301: The terminal device 100 obtains a video stream.

[0127] It is understandable that the terminal device 100 can collect video streams in real time through its own integrated camera, which can be a front camera or a rear camera; can receive video streams transmitted by other terminal devices; or can read pre-recorded and locally stored video streams. The embodiment of the present application does not limit the way in which the terminal device 100 obtains the video stream.

[0128] For example, see Figure 4 The interface diagram of the video call scenario provided by the embodiment of the present application is shown as follows: Figure 4As shown in (a) to (c) in the figure, the terminal device 100 is specifically a mobile phone, and the main interface of the mobile phone includes a phone 41 and a free call 42, as well as applications such as smart life, settings, and an application store. A user can initiate a video call through the phone 41 and the free call 42, and the following is an introduction based on the user initiating a video call through the phone 41.

[0129] First, the user clicks on the phone 41 on the main interface of the mobile phone, and the mobile phone displays the main interface of the phone 41 in response to the user's click operation. Then, the user clicks on the smooth call 43 in the function bar, and the mobile phone displays the smooth call interface 44 in response to the user's click operation. In the smooth call interface 44, the contacts that can be dialed are displayed. Then, the user can click on the control 45, and the mobile phone initiates an online video call to Lucy in response to the user's click operation, and displays the call interface 46. The call interface 46 includes a magic pen 47.

[0130] In this process, when the mobile phone receives a click operation on the control 45, the front camera is called to collect image data, and the mobile phone can obtain a video stream. The collected image signal is then displayed on the display screen through the DSP, GPU, application processor and display screen.

[0131] Step S302: The terminal device 100 extracts a main image from a video image of a video stream.

[0132] It should be noted that the video stream includes multiple frames of images that are continuous in time sequence. The terminal device 100 can extract the main image for each frame of the image, or can extract the main image at intervals of a preset number of frames. The specific details can be determined based on actual application requirements and are not limited here.

[0133] In a specific application, the terminal device 100 may extract the subject image from the video image in any manner. For example, the subject image may be extracted from the video image by semantic segmentation and instance segmentation. In this case, multi-class semantic segmentation is first performed on the video image to obtain a multi-class semantic segmentation result; and then instance segmentation is performed on the multi-class semantic segmentation result to obtain the subject image in the video image.

[0134] Semantic segmentation can give each pixel in an image a semantic label, which is used to identify the category to which the pixel belongs. For example, the label of a person is set as a red pixel, that is, all the pixels of a person in the image are marked as red.

[0135] In other words, the semantic segmentation results can be used to know the categories contained in the original video image, as well as the location and proportion of each category. For example, the multi-class semantic segmentation results can be used to know that the original video image contains two categories, people and trees, as well as the location and proportion of these two categories in the original video image.

[0136] Instance segmentation can distinguish different individuals of the same category based on semantic segmentation. That is, instance segmentation can distinguish different individuals belonging to the same category, for example, distinguishing two individuals belonging to the same person into person 1 and person 2.

[0137] The video image can generally be divided into a foreground and a background, and the subject image can generally be understood as the foreground in the video image. For ease of explanation, the present application regards the video image used to extract the subject image in the video stream as the original video image.

[0138] Step S303: The terminal device 100 determines a target virtual background image.

[0139] It is understandable that the target virtual background image refers to a virtual background used to replace the original background. After the terminal device 100 determines the target virtual background image, it can perform an image rendering process based on the target virtual background image and the main image to obtain an output image after background replacement.

[0140] In some embodiments, the target virtual background image can be selected by the user. At this time, the terminal device 100 can display multiple virtual background images in the virtual background library on the display screen according to a preset display order for the user to select. After the user selects, the terminal device 100 responds to the user's selection operation and uses the virtual background image selected by the user as the target virtual background image.

[0141] For example, see Figure 4 The interface diagram of the video call scenario provided by the embodiment of the present application is shown. Figure 4 (a) to (c) in the figure are the process of initiating a smooth call, which will not be described in detail here.

[0142] like Figure 4 As shown in (d) to (f) in the figure, when Lucy answers the call, the mobile phone displays an answering interface 48. The answering interface 48 also includes a magic pen 47.

[0143] It can be understood that the mobile phone collects video streams through the camera and displays the video streams in real time on the call interface 46 and the answer interface 48. At this time, the call interface 46 and the answer interface 48 display video images without virtual background replacement. Figure 4 The actual background of the video image is not shown.

[0144] Users can use the magic pen 47 to replace the virtual background. Figure 4 In (e), when the user clicks the magic pen 47, the mobile phone displays a window 49 on the answering interface 48 in response to the user's click operation on the magic pen 47. The window 49 includes two options: skin beauty and scene. After the user clicks the scene option, the mobile phone displays virtual background images 411-414 on the window 49 in response to the user's click operation. The virtual background images 411-414 can be as follows: Figure 5 shown.

[0145] The user clicks on the virtual background image 411 in the window 49, and the mobile phone responds to the user's click operation and uses the virtual background image 411 as the target virtual background image. At this point, the mobile phone determines the target virtual background image in the manner selected by the user.

[0146] After the mobile phone determines the target virtual background image, it can further perform image rendering according to the target virtual background image and the main image to obtain an output image after background replacement, and display the output image on the display screen to obtain an interface after background replacement 410. The process of performing image rendering according to the target virtual background image and the main image is described in detail below.

[0147] In this case, the mobile phone does not recommend a virtual background. When the user clicks the scene option in window 49, the mobile phone displays the virtual background images in the virtual background library in the window 49 in sequence according to the default display order. At this time, the default display order is virtual background image 411, virtual background image 412, virtual background image 413, and virtual background image 414.

[0148] It should be noted that in addition to replacing the virtual background through the magic pen 47 in the answering interface 48, the virtual background can also be replaced through the magic pen 47 in the calling interface 47. The process types of the two are the same and will not be repeated here.

[0149] In other embodiments, the terminal device 100 may first determine the virtual background image to be recommended based on the similarity between each virtual background image and the original background image; and then display the virtual background image to be recommended on the display screen to recommend the virtual background to the user.

[0150] Exemplarily, the virtual background recommendation process may be as follows:

[0151] First, the terminal device 100 performs multi-class semantic segmentation on the original background image to obtain a first segmentation result; and performs multi-class semantic segmentation on each virtual background image in the virtual background gallery to obtain a second segmentation result for each virtual background image.

[0152] The original background image refers to the background image in the original video image, which can be obtained by segmenting the foreground and background of the original video image.

[0153] Then, the intersection-over-union (IOU) between the first segmentation result and each second segmentation result is calculated. The IOU value can be used to characterize the similarity between the original background image and the virtual background image in structure and content.

[0154] Next, the terminal device 100 performs color distribution curve statistics on the original background image to obtain a first color distribution curve; performs color distribution curve statistics on each virtual background image in the virtual background library to obtain a second color distribution curve. Then, the curve similarity between the first color distribution curve and each second color distribution curve is calculated. The curve similarity can be used to characterize the color similarity between the original background image and each virtual background image.

[0155] Finally, the terminal device 100 determines the virtual background image to be recommended according to the IOU value and the curve similarity.

[0156] Specifically, a first weight of the IOU value and a second weight of the curve similarity are pre-set. For each virtual background image, the IOU value of the virtual background image is multiplied by the first weight to obtain a first product, and the curve similarity is multiplied by the second weight to obtain a second product; then the first product and the second product of each virtual background image are added together to obtain a recommendation score for each virtual background image. Finally, each virtual background image is sorted according to its recommendation score, and the first K virtual background images are selected as the virtual background images to be recommended, where K is a positive integer.

[0157] For example, suppose the virtual background library includes Figure 5 Through the virtual background recommendation process in the above example, the mobile phone calculates Figure 5 The recommendation scores of the virtual background images are as follows, and the recommendation scores are virtual background image 412, virtual background image 413, virtual background image 414, and virtual background image 411. K=3, that is, the first three virtual background images are selected as virtual background images to be recommended.

[0158] See also Figure 6 The interface diagram of the background recommendation process is shown as Figure 6 As shown, Figure 4Based on the video call scenario shown in , after the mobile phone determines the virtual background image to be recommended, when the user needs to replace the background through the magic pen 47, the mobile phone responds to the user's operation on the magic pen 47 and displays a window 49 on the answering interface. In addition, the mobile phone displays virtual background image 412, virtual background image 413, virtual background image 414 and virtual background image 411 from left to right in window 49 according to the recommendation score. In addition, a box is added to the display position of virtual background image 412, virtual background image 413, and virtual background image 414 to serve as a prompt to the user, that is, virtual background images 412 to 414 are recommended to the user in the form of adding boxes.

[0159] After the user clicks on the virtual background image 412 recommended in window 49, the mobile phone responds to the user's click operation, performs image rendering based on the virtual background image 412 and the main image, obtains an output image after background replacement, and displays the output image on the display screen to obtain an interface 415 after background replacement.

[0160] It should be noted that the virtual background recommendation method is arbitrary and is not limited to Figure 6 For example, different colored boxes can be added to the virtual background images according to the recommendation scores, or arrow indicators can be added to each virtual background image to be recommended to guide the user to select the virtual background image to be recommended.

[0161] In addition to making virtual background recommendations in window 49, virtual background recommendations can also be made through pop-up windows, etc. For example, after the mobile phone determines the virtual background image to be recommended, a window will be actively popped up in the answering interface 48, and the virtual background image to be recommended will be displayed in the window.

[0162] In addition, the virtual background recommendation process is not limited to the process mentioned above. For example, in the process of determining the virtual background image to be recommended according to the IOU value and the curve similarity, the virtual background image with a recommendation score higher than the preset score threshold can be screened out as the virtual background image to be recommended.

[0163] In comparison, if virtual backgrounds are not recommended, the virtual background image selected by the user may differ greatly from the color, structure and content of the original background image, resulting in a poor virtual background replacement effect. Based on the similarity between the original background image and the virtual background image, recommending a virtual background to the user can make the target virtual background image more relevant to the original background image, that is, the color, structure and content of the target virtual background image are closer to the color, structure and content of the original background image, further improving the virtual background replacement effect and the user experience.

[0164] The above-mentioned methods for determining the target virtual background image all require manual selection, while in some other embodiments, the terminal device 100 can also actively determine the target virtual background image without manual participation. At this time, the terminal device 100 can randomly select a virtual background image as the target virtual background image, or calculate the recommendation score of each virtual background image in the virtual background library through the virtual background recommendation process mentioned above, and select the virtual background image with the highest recommendation score as the target virtual background image.

[0165] Step S304: The terminal device 100 performs image rendering according to the subject image and the target virtual background image to obtain an output image, which is an image after background replacement.

[0166] Since the background replacement process involves the image rendering process, the image after background replacement is regarded as the image after image rendering.

[0167] Specifically, after determining the target virtual background image, the terminal device 100 performs consistency rendering according to the main image and the target virtual background image to obtain an image after background replacement, and displays the image on the display screen.

[0168] It should be noted that the consistent rendering in the embodiment of the present application refers to an image rendering process that takes into account the correlation between the original video image and the target virtual background image. The correlation between the original video image and the target virtual background image may include at least one of the following: content, occlusion, position, and interaction.

[0169] The content refers to the image content, that is, the image content of the image after the background replacement is consistent with the image content of the original video image. Specifically, the image after the background replacement can be made consistent with the original video image in terms of image content through the underlying consistency rendering process.

[0170] The underlying features may refer to the color, hue, brightness, contrast and other features of the image, and the rendering for the underlying features may be called underlying consistency rendering. Through underlying consistency rendering, the color, hue, brightness and contrast of the main image can be made consistent with the color, hue, brightness and contrast of the target virtual background image, thereby making the main image part of the image after the background is replaced consistent with the color, hue, brightness and contrast of the background. In addition, the image content of the image after the background is replaced is consistent with the image content of the original video image, thereby further improving the virtual background replacement effect.

[0171] Occlusion refers to whether there is an occlusion relationship between the subject and the object in the original background image. When an occlusion relationship exists, the corresponding occlusion relationship must also be reflected in the image after the background is replaced, so that the image after the background is replaced is consistent with the original video image in terms of the occlusion relationship of the subject.

[0172] In specific applications, the consistent rendering process based on structural semantics can be used to make the image after background replacement consistent with the original video image in terms of subject occlusion. High-level consistent rendering based on structural semantics can determine whether the subject is occluded, and if the subject is occluded, determine the virtual object that needs to be rendered.

[0173] Position refers to the positional relationship between the subject and the object in the image after the background is replaced, which can be specifically reflected in the rendering position of the subject in the virtual background image. Through consistent rendering based on the positional relationship, the position of the subject image in the target virtual background image can be determined, and the subject image can be rendered at a reasonable position in the target virtual background image.

[0174] Interaction refers to the interactive relationship between the subject and the object in the original video image, which can be specifically reflected in whether the subject performs a preset interactive action. If the preset interactive action is performed, there is an interactive relationship between the subject and the corresponding object. Correspondingly, the interactive relationship between the subject and the corresponding object should also be reflected in the image after background replacement, so that the image after background replacement and the original video image are consistent in the interactive relationship. Through consistent rendering based on the interactive relationship, when the subject makes a preset action, the corresponding virtual objects can be added or reduced. For example, when the subject is a person, when the person in the original video image makes a "sit down" action, virtual objects such as chairs or stools are rendered at reasonable positions in the image to increase the realism after background replacement and improve the virtual background replacement effect.

[0175] During the image rendering process, the terminal device 100 may perform consistent rendering based on at least one of underlying features, structural semantics, positional relationships, and interactive relationships.

[0176] In other words, in the process of obtaining the image after background replacement based on the target virtual background image and the main image, the terminal device 100 can perform at least one of the following consistent rendering processes: underlying consistent rendering, consistent rendering based on structural semantics, consistent rendering based on positional relationships, and consistent rendering based on interactive relationships.

[0177] The following introduces each consistent rendering process separately.

[0178] The process of consistent rendering based on structural semantics can be as follows:

[0179] First, the terminal device 100 can perform multi-class semantic segmentation on the original video image to obtain multi-class semantic segmentation results.

[0180] It can be understood that the above original video image generally refers to the original video image in the above step S302.

[0181] Next, the terminal device 100 determines whether the subject is blocked based on the depth information of each category in the original video image. The depth information can characterize the front-to-back relationship of each category, and whether there are other objects in front of the subject can be determined based on the depth information. If it is determined that there are other objects blocking the subject, the categories of the other objects in front of the subject can be further determined based on the multi-category semantic segmentation results.

[0182] For example, if the subject in the original video image is a person, the depth information of the original video image can be used to determine that there are other objects in front of the person in the original video image. Then, the multi-class semantic segmentation result can be used to determine that the other object in front of the person is a table, and it can be determined that the person is blocked by the table, that is, there is an occlusion relationship between the person and the table.

[0183] Among them, the category of each pixel in the original video image can be known through the multi-class semantic segmentation results, and the distance (i.e., depth value) of each pixel from the camera can be known through the depth map of the original video image. After determining the location of the subject and the depth value of the subject through the multi-class semantic segmentation results and depth information, find out whether there are pixels with a depth value less than the subject within a preset range around the subject. If so, map the pixels with a depth value less than the subject's depth value to the multi-class semantic segmentation results, determine the categories of these pixels, and thus determine the category of the occluded object.

[0184] After the terminal device 100 determines that the subject in the original video image is blocked and the type of the object blocking the subject, it determines the first virtual object to be rendered. For example, the subject in the original video image is a person, and there is a table blocking the person in front of him. At this time, the terminal device 100 can recommend a virtual object similar to or related to the table, and the virtual object can be, for example, various styles of tables pre-entered into the virtual object library. The user can select one or more of the recommended virtual objects as the first virtual object to be rendered according to his needs.

[0185] Of course, the terminal device 100 may not execute the virtual object recommendation process, but directly select a relevant virtual object from the virtual object library as the first virtual object to be rendered.

[0186] After the terminal device 100 determines the first virtual object to be rendered, the first virtual object to be rendered can be used as the foreground, and the foreground and background can be merged with the target virtual background image to obtain a new virtual background image. For example, the first virtual object to be rendered is a table, and the table is used as the foreground, and image rendering is performed with the target virtual background image to obtain a new virtual background image.

[0187] In the process of rendering the first virtual object to be rendered to the target virtual background image, the center position of the occluding object in the original background image can be used as the initial rendering position of the first virtual object to be rendered. Of course, the user can also independently determine the rendering position of the first virtual object to be rendered. For example, the user can adjust the rendering position of the first virtual object to be rendered by dragging.

[0188] If the terminal device 100 determines that the subject in the original video image is not blocked by other objects based on the multi-class semantic segmentation results and depth information, it is not necessary to determine the first virtual object to be rendered, and it is not necessary to render the first virtual object to be rendered in the target virtual background image.

[0189] For example, see Figure 7 The schematic diagram of the effect of consistent rendering based on structural semantics is shown as Figure 7 As shown in (a) in the figure, image 71 is the original video image, which includes a subject 72 and a table 73. According to the multi-class semantic segmentation results of image 71 and the depth information, it is determined that there is an occlusion relationship between the subject 72 and the table 73, that is, the subject 72 is partially occluded by the table 73. After determining that there is an occlusion relationship between the subject 72 and the table 73, the first virtual object to be rendered can be further determined.

[0190] See also Figure 7 In (b), image 74 is the target virtual background image. Assume that the virtual object library includes objects 75 to 77, and object 75 is a virtual object recommended by the system, which can be specifically prompted to the user in the form of a box.

[0191] Assume that the user selects object 75 as the first virtual object to be rendered. At this time, the image 74 and the object 75 can be rendered first to obtain a virtual background image to be rendered, and then the background image to be rendered and the subject 72 are synthesized to obtain an image after background replacement. Figure 7 Image 78 in (c). By comparison, it can be seen that there is an occlusion relationship between the subject 72 and the table 73 in image 71, and there is an occlusion relationship between the subject 72 and the object 75 in image 78, so that the image after background replacement is consistent with the original video image in terms of occlusion relationship.

[0192] The process of consistent rendering based on interaction relationships can be as follows:

[0193] The terminal device 100 can perform action recognition based on multiple consecutive frames of original video images. If it is recognized that the subject in the original video image has made a preset action, a second virtual object to be rendered associated with the preset action is determined. The preset action can be set according to the actual application, for example, the preset action is "sitting down" and "holding an object". The action recognition method can be any existing method and is not limited here.

[0194] After identifying the preset action, the terminal device 100 can select an object associated with the preset action from the virtual object library to determine the second virtual object to be rendered. The association relationship between each preset action and the virtual object can be preset, and the virtual object can be selected directly through the preset action. For example, the preset actions include "sitting down" and "holding an object in hand", and the virtual object corresponding to "sitting down" is a chair or a stool, and the virtual object corresponding to "holding an object in hand" is a cup.

[0195] At this time, the user can also pre-set the virtual objects to be used in the initialization stage, and the terminal device 100 automatically selects the corresponding virtual objects from the virtual objects set by the user after recognizing the preset action. For example, before the video call starts, the user pre-sets which virtual objects may be used during the video call; after the setting is completed, during the video call, after the mobile phone recognizes the preset action, the corresponding virtual object is selected from the set virtual objects.

[0196] In some other embodiments, after the terminal device 100 recognizes the preset action, it can also recommend virtual objects to the user, and the user can select the desired virtual object. However, it takes a certain amount of time for the user to select the virtual object, which will cause a certain lag in the rendering of the virtual object. Therefore, in order to ensure that the corresponding virtual object can be rendered in a timely manner at the corresponding position after the subject makes the preset action, the user does not usually need to select the virtual object, but the terminal device independently determines the virtual object to be rendered.

[0197] After the terminal device 100 determines the second virtual object to be rendered, it is necessary to further determine the rendering position of the second virtual object to be rendered. Specifically, the terminal device 100 performs multi-class semantic segmentation on the original video image, and determines the position of the interactive object in the original video image based on the multi-class semantic segmentation results. An interactive object refers to an object corresponding to a preset action. For example, the subject in the original video image is a person, and the person makes a "sit down" action, that is, the person sits on a chair, and the chair is the above-mentioned interactive object.

[0198] The multi-class semantic segmentation results can be used to know the categories contained in the image and the positions of each category. Therefore, the multi-class semantic segmentation results of the original video image can be used to know the positions of the interactive objects in the original video image.

[0199] According to the position of the interactive object in the original video image, the rendering position of the second virtual object to be rendered in the virtual background image to be rendered is determined. For example, the interactive object is a chair, and the pixel position of the chair in the original video image is the first position; the position in the virtual background image to be rendered corresponding to the first position is used as the rendering position.

[0200] At the same time, the terminal device 100 can also determine the front-to-back position relationship between the interactive object and the subject in the original video image according to the depth information of each category in the original video image, that is, determine which of the interactive object and the subject is in front and which is in the back. And according to the front-to-back position relationship, set the rendering order between the second virtual object to be rendered and the subject. For example, if the subject is in front and the interactive object is in the back, the rendering order is: render the interactive object first, then render the subject.

[0201] If the rendering order is: render the interactive object first, then render the subject, after the terminal device 100 determines the second virtual object to be rendered, the second virtual object to be rendered can be used as the foreground and rendered with the target virtual background image to obtain a new virtual background image. If the preset action is not recognized, the second virtual object to be rendered does not need to be rendered in the target virtual background image.

[0202] If the rendering order is: render the main body first, then render the interactive object, after the terminal device 100 determines the second virtual object to be rendered, after the main body image and the virtual background image to be rendered are merged, the second virtual object to be rendered is rendered in the merged image to obtain the final output image.

[0203] See also Figure 8 The effect diagram of consistent rendering based on interactive relationship is shown as Figure 8 As shown, image 81 and image 82 are original video images, and the time sequence of these two images in the original video stream is: image 81 first, and image 82 later.

[0204] Image 83 and image 84 are output images after background replacement obtained by using an existing virtual background replacement method, wherein image 83 corresponds to image 81 , and image 82 corresponds to image 84 .

[0205] Image 85 and image 86 are images after background replacement obtained by using the above-mentioned consistency rendering based on the interactive relationship. Image 85 corresponds to image 81, and image 86 corresponds to image 82.

[0206] With respect to the images 81 and 82 , the person subject 87 in the image 81 is in a standing state; the person subject 87 in the image 82 is in a sitting state, that is, the person subject 87 changes from standing to sitting.

[0207] Taking images 83 and 84 as the first group of images, and images 85 and 86 as the second group of images, by comparing the first group of images with the second group of images, it can be seen that in the first group of images, when the state of the person changes from standing to sitting, the virtual background image does not render a chair at the corresponding position, resulting in the unreasonable phenomenon that the main character 87 sits in the air in the image after the background is replaced. In the second group of images, due to the use of the above-mentioned consistency rendering based on the interactive relationship, when the action of "sitting down" is recognized, a chair 88 is rendered at the corresponding position, so that in the image after the background is replaced, the main character 87 sits on the chair 88, which is more reasonable.

[0208] The underlying consistency rendering process:

[0209] The underlying consistency rendering may include two different implementation methods. The two different underlying consistency rendering methods are introduced below.

[0210] Method 1:

[0211] The terminal device 100 inputs the low-frequency image of the virtual background image to be rendered and the original video image into a pre-trained style transfer model, and the output of the style transfer model is the original video image after underlying consistency rendering.

[0212] Image style transfer refers to using an algorithm to learn the style of a certain picture and then applying this style to another picture, or transferring the style of one picture to another.

[0213] The style transfer model refers to a model used to achieve image style transfer, that is, the style of the virtual background image to be rendered can be transferred to the original video image through the style transfer model.

[0214] See also Fig. 9 The flowchart of the underlying consistency rendering process based on the model is shown, and the process may include the following steps:

[0215] Step S901: The terminal device 100 obtains a low-frequency image of a virtual background image to be rendered.

[0216] In a specific application, after the virtual background image to be rendered is determined, a low-frequency image of the virtual background image to be rendered may be generated.

[0217] Step S902: The terminal device 100 inputs the low-frequency image of the virtual background image to be rendered and the original video image into a pre-trained style transfer model to obtain the original video image after bottom-level consistency rendering output by the style transfer model.

[0218] See also Fig.10The schematic diagram of the style transfer model training process is shown. The training process of the style transfer model can be as follows:

[0219] First, build a style transfer model.

[0220] Next, the style transfer model is forward trained.

[0221] During forward training, the input of the style transfer model is the low-frequency image of the original video image and the virtual background image, and the output is the forward training result. At this time, the virtual background image refers to the image used for background replacement in the training dataset.

[0222] After each positive training result is obtained, the positive training result is transferred to the color-opponent space (Lab colorspace, LAB), and then the first variance and first mean of the positive training result in the LAB domain are calculated; after the virtual background image is transferred to the LAB space, the second variance and second mean of the virtual background image in the LAB domain are calculated. Finally, based on the first variance and the second variance, as well as the first mean and the second mean, the mean difference and variance difference of the positive training result and the virtual background image in the LAB domain are calculated respectively.

[0223] The mean difference and variance difference of the LAB domain are used as the loss value (Loss) between the model output and input, that is, the Loss during forward training is the Loss of the LAB domain. The Loss of the LAB domain can be used to constrain the similarity of color, brightness, saturation, etc. between the forward training results and the virtual background.

[0224] After calculating the loss value, the network parameters of the style transfer model are adjusted according to the loss value. Through forward training, the color, brightness, saturation, etc. of the main image or the original video image can be made consistent with the color, brightness, saturation, etc. of the virtual background image.

[0225] Then, based on the result of the forward training, the style transfer model after the forward training is reversely trained to obtain a trained style transfer model.

[0226] During the reverse training process, the low-frequency image of the original video image and the forward training result are input into the style transfer model after forward training, and the output of the model is the reverse training result.

[0227] After each reverse training result is obtained, the Loss between the reverse training result and the low-frequency image of the original video image in the LAB domain is calculated. The specific calculation process can be as follows: the reverse training result and the low-frequency image of the original video image are transferred to the LAB space, and the variance and mean of the reverse training result in the LAB domain are calculated, and the variance and mean of the low-frequency image of the original video image in the LAB domain are calculated. According to the variance and mean of the reverse training result and the variance and mean of the low-frequency image of the original video image, the mean difference and variance difference between the reverse training result and the low-frequency image of the original video image are calculated, and the mean and variance difference between the reverse training result and the low-frequency image of the original video image in the LAB domain is used as the Loss of the LAB domain. At the same time, the loss value between the reverse training result and the original video image is calculated. Then, the Loss of the LAB domain and the loss value between the reverse training result and the original video image are weighted to obtain a loss value, and then the network parameters of the style transfer model after forward training are adjusted according to the loss value.

[0228] The training is iterated multiple times until the loss values ​​of the forward training and the reverse training tend to be stable. The training is considered complete and the final style transfer model is obtained.

[0229] Through reverse training, the content of the main image or the original video image can be made consistent with the content of the virtual background image to be rendered.

[0230] In the embodiment of the present application, one training includes a forward training process and a reverse training process, that is, in the current training process, after a forward training process is performed, a reverse training process is performed based on the result of the forward training. Similarly, in the next training process, a forward training process is still performed first, and then a reverse training process is performed. The training is iterated multiple times in sequence to obtain a trained style transfer model.

[0231] It should be noted that the training process of the style transfer model can be performed on the terminal device 100, or it can be performed on other devices instead of the terminal device 100, and then loaded to the terminal device 100 after the training is completed.

[0232] In the existing style transfer model training process, the forward training and reverse training models are different, and the weights are different, that is, the reverse training is not based on the model and results obtained by the forward training. In this way, after using the trained model for style transfer, the image content in the style-transferred image is inconsistent with the image content of the original image.

[0233] Moreover, different styles correspond to different style transfer models. For example, if there are three different background images, three different style transfer models are required to transfer the styles of the three background images to the corresponding pictures respectively.

[0234] In the above method 1, the forward training and reverse training models are the same, and the weights are the same, that is, the reverse training is based on the style transfer model after forward training and the forward training results. In this way, during style transfer, not only can the color, brightness, saturation, etc. of the image after style transfer be guaranteed to be consistent with the virtual background image to be rendered, but also the image content in the image after style transfer can be guaranteed to be consistent with the image content of the original video image, resulting in better rendering effect. The image after style transfer refers to the image obtained by underlying consistency rendering.

[0235] In other words, the forward training and reverse training of the prior art are not the same model, resulting in inconsistency between the image content after style transfer and the content of the original video image. However, the forward training and reverse training of the above-mentioned method 1 are the same model, so the image content after style transfer is the same as the image content of the original video image. In addition, in the model application stage, the input of the style transfer model of the prior art is the virtual background image to be rendered, while the model input of the above-mentioned method 1 is the low-frequency image of the original video image and the virtual background image to be rendered.

[0236] In addition, in the above method 1, different styles correspond to the same style transfer model. For example, if there are three different background images, a style transfer model is needed, and the styles of the three background images can be transferred to the corresponding pictures respectively. In other words, the style transfer model obtained by the model training method of the above method 1 can transfer multiple different styles to another picture.

[0237] It should also be noted that the model training process and the underlying consistency rendering process described above are all based on the original video image. In other embodiments, the original video image can also be replaced by the main image, that is, Fig. 9 and Fig.10 The original video image in can be replaced with the subject image. For example, the subject image is extracted from the original video image, and then the subject image and the low-frequency image of the virtual background image to be rendered are input into the style transfer model for forward training. For another example, in the actual application stage of the model, the subject image and the low-frequency image of the virtual background image to be rendered are input into the trained style transfer model to obtain the underlying consistent rendered subject image output by the style transfer model.

[0238] In addition, low-frequency images of the virtual background image to be rendered are used in both the model application stage and the model training stage. The low-frequency images can ignore the underlying features such as texture, and multi-style reverse training can be performed. In addition, the forward training and reverse training are based on the same model, which reduces distortion.

[0239] In comparison, in the prior art, fixed color temperature and fixed lighting are generally used for image rendering, that is, virtual lighting in a fixed direction is rendered for the foreground, or a fixed color temperature is rendered for the foreground. This may cause inconsistencies in the color, color temperature, brightness, contrast, etc. of the subject and background in the image after the background is replaced, and the background replacement effect is poor.

[0240] By using the underlying consistency rendering of the above method 1 or method 2, the color, color temperature, brightness, contrast, etc. of the subject and the background in the image after the background replacement are consistent, and the background replacement effect is better.

[0241] See also Fig.11 The schematic diagram of the effect of underlying consistency rendering is shown as Fig.11 As shown, image 111 is the original video image, image 112 is the virtual background image, image 113 is the image after background replacement obtained by the above method 1, and image 114 is the image after background replacement obtained by the existing method. The background color and hue in image 111 are mainly ocean blue, and the color of the clothes of the person in image 111 is color 1, for example, color 1 is white. The background color and hue in image 112 are mainly sunset yellow.

[0242] Use the above method 1:

[0243] The subject image and the virtual background image can be made consistent in color, tone, contrast and color temperature. Specifically, the color of the clothes of the person in image 113 (i.e., the subject image) is color 2, for example, color 2 is yellow, that is, the color, color temperature, etc. of the person in image 113 are consistent with the color, color temperature, etc. of the background, and the consistency between the subject and the background is high.

[0244] However, using the existing method, the main image and the virtual background image differ greatly in color, tone, contrast, color temperature, etc. Specifically, the color of the clothes of the person in image 114 is color 1, which is consistent with the main image in the original video image. In this way, the main image and the background in image 114 differ greatly in color and color temperature, and the consistency between the main image and the background is poor.

[0245] Method 2:

[0246] See also Fig.12 The schematic diagram of the underlying consistency rendering process based on the image processing algorithm is shown as Fig.12 As shown, first, the terminal device 100 converts the virtual background image to be rendered from the RGB color space to the LAB color space to obtain the first image in the LAB color space, and then calculates the first standard deviation (std) and the third mean (mean) of the three channels L, A, and B of the first image. Each channel has its own corresponding standard deviation and mean.

[0247] Then, the terminal device 100 converts the main image or the original video image to the LAB color space to obtain a second image in the LAB color space, and corrects the standard deviation and mean of the second image according to the first standard deviation and the third mean of the first image to obtain a third image.

[0248] Specifically, the standard deviations of the three channels L, A, and B of the second image are respectively set to the first standard deviations of the corresponding channels in the first image, or the difference between the standard deviations of the three channels L, A, and B of the second image and the first standard deviations of the corresponding channels in the first image is within a preset threshold range; the means of the three channels L, A, and B of the second image are respectively set to the third means of the corresponding channels in the first image, or the difference between the means of the three channels L, A, and B of the second image and the third means of the corresponding channels in the first image is within a preset threshold range.

[0249] That is to say, the standard deviation of the L, A, and B channels of the third image is equal to the first standard deviation of the corresponding channels in the first image or is close to the first standard deviation, and the mean of the L, A, and B channels of the third image is equal to the third mean of the corresponding channels in the first image or is close to the third mean.

[0250] Finally, the third image is converted from the LAB color space to the RGB color space to obtain a fourth image, which is the image after the underlying consistency rendering.

[0251] See also Fig.13 The schematic diagram of the effect of underlying consistency rendering based on image processing algorithm is shown as Fig.13 As shown, image 131 is a virtual background image, image 132 is an original video image, image 133 is an image after background replacement obtained by using the above-mentioned method 2, and image 134 is an image after background replacement obtained by using the prior art.

[0252] By comparing image 133 and image 134, it can be seen that the color, color temperature, brightness and contrast of person 135 (i.e., subject image) in image 133 are consistent with those in image 131, i.e., the consistency of color, brightness, contrast, color temperature, etc. between person and background in image 133 is better, which results in a better background replacement effect. However, the color, color temperature, brightness and contrast of person 135 in image 134 are consistent with the hue, color temperature, brightness and contrast in image 132, which is quite different from the color, color temperature, brightness and contrast of image 131, which results in a large difference in color, contrast, color temperature, etc. between person and background in image 134, i.e., the consistency of person and background is poor, which results in a poor background replacement effect.

[0253] Consistent rendering process based on positional relationship:

[0254] See also Fig.14 The schematic diagram of the consistent rendering process based on the position relationship is shown. The consistent rendering process based on the position relationship can be as follows:

[0255] First, the main image is input into the first spatial transformer network (STN) to obtain a first change matrix output by the first STN network; the virtual background image to be rendered is input into the second STN network to obtain a second change matrix output by the second STN network.

[0256] Among them, if the underlying consistency rendering process is performed, the main image can be the main image after the underlying consistency rendering, and if the result of the underlying consistency rendering process is the original video image after the underlying consistency rendering, the main image is extracted from the original video image to obtain the main image after the underlying consistency rendering.

[0257] Then, the subject image is subjected to image affine transformation (Warp) using the first change matrix to obtain the subject image after Warp. The virtual background image to be rendered is subjected to Warp using the second change matrix to obtain the virtual background image to be rendered after Warp.

[0258] Finally, the warped subject image and the warped virtual background image to be rendered are fused to obtain a composite image.

[0259] Through the STN network, the mutual rotation, translation or scaling of the foreground (i.e., the main image) and the virtual background image to be rendered are adjusted and cropped.

[0260] It is understandable that the first STN network and the second STN network are both pre-trained networks. The training process of the STN network adopts an adversarial learning method, and the specific process can be as follows:

[0261] See also Fig.15 The schematic diagram of the STN network training process is shown in Fig.15 As shown, the subject image for training is input into the pre-built third STN network to obtain the change matrix H0 output by the third STN network; the virtual background image for training is input into the pre-built fourth STN network to obtain the change matrix H1 output by the fourth STN network. At this time, the virtual background image for training refers to the image used for background replacement in the training data. Among them, the subject image can be an image rendered by the underlying consistency, or it can be an image not rendered by the underlying consistency.

[0262] The subject image for training is warped using the change matrix H0, and the background image for training is warped using the change matrix H1; the subject image after warping and the virtual background image after warping are then fused into a foreground and background to obtain a composite image.

[0263] Finally, the synthesized image is input to the discriminator. The discriminator determines the quality of the synthesized image by judging the difference between the synthesized image and the real image. The smaller the difference between the synthesized image and the real image, the better the synthesized image. Conversely, the larger the difference, the worse the synthesized image.

[0264] When the discriminator considers that the input synthetic image is the same as the real image, the STN network training is considered to be completed, and the first STN network and the second STN network are obtained.

[0265] It can be understood that the training process of the STN network can be performed on the terminal device 100 or on other devices.

[0266] See also Fig.16 The schematic diagram of the effect of consistent rendering based on position relationship is shown in FIG. Fig.16 As shown, images 161, 162 and 163 are all images after background replacement, wherein the image rendering process of images 161 and 162 did not perform the consistency rendering process based on positional relationship, while the image rendering process of image 163 performed the above consistency rendering process based on positional relationship. By comparison, it can be seen that in images 161 and 162, the subject 164 is not reasonably rendered on the table 165, resulting in unreasonable phenomena such as the subject being suspended in the image after background replacement. In image 163, the subject 164 is reasonably rendered on the table 165, which is more reasonable and authentic.

[0267] After introducing various consistency rendering processes respectively, the following is an exemplary description of possible image rendering processes.

[0268] The first image rendering process:

[0269] The image rendering process includes consistency rendering based on structural semantics, consistency rendering based on interaction relationship, bottom-level consistency rendering and consistency rendering based on position relationship.

[0270] See also Fig.17 A schematic diagram of a flow chart of an image rendering process is shown, and the image rendering process may include the following steps:

[0271] Step S1701: The terminal device 100 obtains an original video image.

[0272] It can be understood that the original video image is a frame of video image in the video stream.

[0273] Step S1702: The terminal device 100 detects whether there is an occlusion relationship between the target object in the original video image. If there is an occlusion relationship between the target object, the process proceeds to step S1703; if there is no occlusion relationship between the target object, the process proceeds to step S1707.

[0274] It should be noted that the target object mentioned above refers to the subject in the original video image, and usually, the target object is a human subject.

[0275] In a specific application, the terminal device 100 can first perform multi-class semantic segmentation on the original video image to obtain the multi-class semantic segmentation result, and then determine whether the subject is blocked by other objects based on the multi-class semantic segmentation result and the depth information of the original video image. If the subject is blocked by other objects, it is determined that there is an occlusion relationship with the target object. Conversely, if the subject is not blocked by other objects, it is determined that there is no occlusion relationship with the target object.

[0276] Step S1703: The terminal device 100 determines a first virtual object to be rendered corresponding to the blocking object.

[0277] In a specific application, when the terminal device 100 determines that the subject is blocked by other objects based on the depth information, it determines the category of the blocked object and the position of the blocked object in the original video image through the multi-class semantic segmentation results. Then, according to the category of the blocked object, the first virtual object to be rendered corresponding to the blocked object is determined, and then according to the position of the blocked object in the original video image, the rendering position of the first virtual object to be rendered is determined.

[0278] Step S1704: The terminal device 100 detects whether the target object in the original video image performs a preset action. If the target object performs the preset action, the process proceeds to step S1705; if the target object does not perform the preset action, the process proceeds to step S1707.

[0279] Step S1705: The terminal device 100 determines a second virtual object to be rendered corresponding to the preset action.

[0280] Step S1706: The terminal device 100 determines the rendering position and rendering order of the second virtual object to be rendered.

[0281] It should be noted that when the rendering order is: first render the interactive object and then render the subject, the second virtual object to be rendered and the target virtual background image are merged.

[0282] When the rendering order is: rendering the subject first and then rendering the interactive object, the second virtual object to be rendered is not fused with the target virtual background image, but after obtaining the composite image, the second virtual object to be rendered and the composite image are fused with the foreground and background.

[0283] If there are two types of second virtual objects to be rendered at the same time, the rendering order of the first type of second virtual objects to be rendered is: first render the interactive objects, then render the subject, and the rendering order of the second type of second virtual objects to be rendered is: first render the subject, then render the interactive objects. At this time, the first type of second virtual objects to be rendered are fused with the target virtual background image, and the second type of second virtual objects to be rendered are fused with the synthesized image.

[0284] Step S1707: The terminal device 100 determines a virtual background image to be rendered.

[0285] In some embodiments, the target virtual object to be rendered can be used as the foreground, and the foreground and background can be fused with the target virtual background image to obtain the virtual background image to be rendered.

[0286] The target virtual object to be rendered may include a first virtual object to be rendered and / or a second virtual object to be rendered.

[0287] When it is detected that there is an occlusion relationship with the target object and it is detected that the target object does not perform a preset action, the target virtual object to be rendered only includes the first virtual object to be rendered. At this time, according to the position of the occluding object in the original video image, the first virtual object to be rendered is used as the foreground, and the foreground and background are fused with the target virtual background image to obtain the virtual background image to be rendered.

[0288] When it is detected that the target object has an occlusion relationship and the target object performs a preset action, if the rendering order between the second virtual object to be rendered and the subject is: render the interactive object first, then render the subject, the target virtual object to be rendered includes the first virtual object to be rendered and the second virtual object to be rendered. At this time, according to the rendering position of the second virtual object to be rendered and the position of the occluding object in the original video image, the first virtual object to be rendered and the second virtual object to be rendered are used as the foreground, and the foreground and background are fused with the target virtual background image to obtain the virtual background image to be rendered.

[0289] When it is detected that the target object has an occlusion relationship and the target object performs a preset action, if the rendering order between the second virtual object to be rendered and the subject is: render the subject first, then render the interactive object, the target virtual object to be rendered includes the first virtual object to be rendered. At this time, according to the position of the occluding object in the original video image, the first virtual object to be rendered is used as the foreground, and the foreground and background are fused with the target virtual background image to obtain the virtual background image to be rendered.

[0290] When it is detected that there is no occlusion relationship with the target object and it is detected that the target object performs a preset action, if the rendering order between the second virtual object to be rendered and the subject is: first render the interactive object, then render the subject, the target virtual object to be rendered includes the second virtual object to be rendered. At this time, according to the rendering position of the second virtual object to be rendered, the second virtual object to be rendered is used as the foreground, and the foreground and background are fused with the target virtual background image to obtain the virtual background image to be rendered.

[0291] In some other embodiments, the target virtual background image may be directly used as the virtual background image to be rendered.

[0292] When it is detected that there is no occlusion relationship with the target object and the target object is detected to perform a preset action, if the rendering order between the second virtual object to be rendered and the main body is: render the main body first, then render the interactive object, at this time, the target virtual background image is directly used as the virtual background image to be rendered.

[0293] When it is detected that there is no occlusion relationship with the target object and it is detected that the target object does not perform a preset action, the target virtual background image is directly used as the virtual background image to be rendered.

[0294] After the virtual background image to be rendered is determined, image rendering may be performed according to the virtual background image to be rendered and the main image to obtain an image after background replacement.

[0295] Step S1708: The terminal device 100 performs bottom-level consistency rendering according to the virtual background image to be rendered and the original video image.

[0296] After determining the virtual background image to be rendered, the terminal device 100 may perform a bottom-level consistency rendering process. The bottom-level consistency rendering process can be found above and will not be described in detail here.

[0297] Step S1709: The terminal device 100 performs consistent rendering based on the position relationship according to the main image after the underlying consistent rendering and the virtual background image to be rendered, to obtain a composite image.

[0298] It should be noted that the consistency rendering based on the position relationship can be found in the above text and will not be described in detail here.

[0299] When it is detected that the target object does not perform the preset action, or the target object performs the preset action, but the rendering order between the second virtual object to be rendered and the subject is: render the interactive object first and then render the subject, the composite image is the output image after the background is replaced.

[0300] When it is detected that the target object has performed a preset action, and the rendering order between the second virtual object to be rendered and the subject is: render the subject first, then render the interactive object, the composite image is not the output image after the background is replaced. At this time, after obtaining the composite image, it is also necessary to use the second virtual object to be rendered as the foreground and the composite image as the background according to the rendering position of the second virtual object to be rendered, and perform foreground and background fusion to obtain a fused image, which is the output image after the background is replaced. Therefore, in this case, the image rendering process may also include step S1710.

[0301] Optionally, the method may further include step S1710 of fusing the synthesized image with the second virtual object to be rendered, and obtaining a fused image.

[0302] It should be noted that steps S1702 to S1703 belong to a consistent rendering process based on structural semantics, and steps S1704 to S1706 belong to a consistent rendering process based on interactive relationships. The execution order of steps S1702 to S1703 and steps S1704 to S1706 is arbitrary, and can be executed simultaneously or sequentially.

[0303] In comparison, the prior art does not consider the interactive relationship, structural semantics, underlying consistent rendering, and positional relationship during the image rendering process, which results in unreasonable rendering of the image after background replacement. In the embodiment of the present application, the occlusion relationship of the subject is determined through a consistent rendering process based on structural semantics, thereby avoiding unreasonable subject rendering; through a consistent rendering process based on interactive relationships, when rendering the subject, the interactive relationship between the subject and the interactive object is considered, thereby avoiding unreasonable rendering; through STN network changes, the rendering position of the subject image is determined, thereby avoiding rendering the subject in an unreasonable position, thereby optimizing the sense of reality after background replacement, thereby avoiding unrealistic or even unreasonable background replacement affecting the background replacement effect.

[0304] In addition, through the underlying consistency rendering process, the main image and background in the image after background replacement are consistent in color, brightness, contrast, hue, etc., further improving the background replacement effect.

[0305] The second image rendering process:

[0306] The image rendering process includes a consistency rendering process based on structural semantics, an underlying consistency rendering process and a consistency rendering process based on positional relationships.

[0307] See also Fig.18 Another flowchart of an image rendering process is shown, and the image rendering process may include the following steps:

[0308] Step S1801: The terminal device 100 obtains an original video image.

[0309] Step S1802: The terminal device 100 detects whether there is an occlusion relationship between the target object in the original video image. If there is an occlusion relationship between the target object, the process proceeds to step S1803; if there is no occlusion relationship between the target object, the process proceeds to step S1805.

[0310] Step S1803: The terminal device 100 determines a first virtual object to be rendered corresponding to the occluding object.

[0311] Step S1804: The terminal device 100 merges the first virtual object to be rendered and the target virtual background image to obtain the virtual background image to be rendered.

[0312] After obtaining the virtual background image to be rendered, proceed to step S1806.

[0313] Step S1805: The terminal device 100 uses the target virtual background image as the virtual background image to be rendered.

[0314] After the virtual background image to be rendered is determined, image rendering may be performed according to the virtual background image to be rendered and the main image to obtain an image after background replacement.

[0315] Step S1806: The terminal device 100 performs bottom-level consistency rendering according to the virtual background image to be rendered and the original video image.

[0316] Step S1807: The terminal device 100 performs consistent rendering based on the position relationship according to the main image after the underlying consistent rendering and the virtual background image to be rendered, to obtain a composite image.

[0317] During the image rendering process, the synthesized image is the output image after background replacement.

[0318] In the image rendering process, the occlusion relationship of the subject is determined through the consistency rendering process based on structural semantics, avoiding unreasonable subject rendering; the rendering position of the subject image is determined through STN network changes, avoiding the subject being rendered in an unreasonable position, thereby optimizing the realism after background replacement, and thus avoiding the background replacement effect being affected by unrealistic or even unreasonable background replacement. In addition, through the underlying consistency rendering process, the subject image and background in the image after background replacement are consistent in color, brightness, contrast, hue, etc., further improving the background replacement effect.

[0319] The third image rendering process:

[0320] The image rendering process includes a consistency rendering process based on an interaction relationship, an underlying consistency rendering process, and a consistency rendering process based on a position relationship.

[0321] See also Fig.19 Another flowchart of an image rendering process is shown, and the image rendering process may include the following steps:

[0322] Step S1901: The terminal device 100 obtains an original video image.

[0323] Step S1902: The terminal device 100 detects whether the target object in the original video image performs a preset action. If yes, the process proceeds to step S1903; if no, the process proceeds to step S1906.

[0324] Step S1903: The terminal device 100 determines a second virtual object to be rendered corresponding to the preset action.

[0325] Step S1904: The terminal device 100 determines the rendering order and rendering position of the second virtual object to be rendered.

[0326] Among them, if the rendering order between the second virtual object to be rendered and the main body is: render the interactive object first, then render the main body, then enter step S1905, that is, according to the rendering position of the second virtual object to be rendered, take the second virtual object to be rendered as the foreground, and merge the foreground and background with the target virtual background image to obtain the virtual background image to be rendered.

[0327] If the rendering order between the second virtual object to be rendered and the subject is: render the subject first, then render the interactive object, then proceed to step S1906. At this time, the image rendering process also includes step S1909.

[0328] After the virtual background image to be rendered is determined, image rendering may be performed according to the main image and the virtual background image to be rendered to obtain an image after background replacement.

[0329] Step S1905: The terminal device 100 merges the second virtual object to be rendered and the target virtual background image to obtain the virtual background image to be rendered.

[0330] Step S1906: The terminal device 100 uses the target virtual background image as the virtual background image to be rendered.

[0331] After the virtual background image to be rendered is determined, image rendering can be performed according to the virtual background image to be rendered and the main image to obtain an image after background replacement.

[0332] Step S1907: The terminal device 100 performs bottom-level consistency rendering on the virtual background image to be rendered and the original video image.

[0333] Step S1908: The terminal device 100 performs consistent rendering based on the position relationship according to the main image after the underlying consistent rendering and the virtual background image to be rendered, to obtain a composite image.

[0334] Step S1909: The terminal device 100 fuses the foreground and background of the synthesized image with the second virtual object to be rendered to obtain a fused image.

[0335] In the image rendering process, if the rendering order of the second virtual object to be rendered is first, the composite image is the output image after background replacement; if the rendering order of the second virtual object to be rendered is later, the fused image is the output image after background replacement.

[0336] In the image rendering process, through the consistency rendering process based on the interaction relationship, when rendering the subject, the interaction relationship between the subject and the interactive object is taken into account to avoid unreasonable rendering; through the STN network changes, the rendering position of the subject image is determined to avoid rendering the subject in an unreasonable position, thereby optimizing the realism after background replacement, and thus avoiding the background replacement being unrealistic or even unreasonable and affecting the background replacement effect. In addition, through the underlying consistency rendering process, the subject image and the background in the image after background replacement are consistent in color, brightness, contrast, hue, etc., further improving the background replacement effect.

[0337] Other possible image rendering processes:

[0338] In some embodiments, the image rendering process may only include a consistency rendering process based on structural semantics and / or a consistency rendering process based on interaction relationships, excluding the above-mentioned underlying consistency rendering process and consistency rendering process based on position relationships. The image rendering process may be as follows:

[0339] The terminal device 100 first performs the above-mentioned consistency rendering process based on structural semantics and / or the consistency rendering process based on interactive relationship to obtain the virtual background image to be rendered. The specific process can refer to the image rendering process shown above, which will not be repeated here.

[0340] Then, the terminal device 100 can use the existing color tone and brightness processing methods to process the rendered virtual background image, for example, detect the current scene brightness and the virtual background brightness, when the scene brightness is greater than the virtual background brightness, adjust the exposure time, and when the scene brightness is less than the virtual background brightness, add virtual lighting to the virtual background.

[0341] Finally, the terminal device 100 fuses the processed virtual background image to be rendered with the main image and obtains the final output image.

[0342] Alternatively, the terminal device 100 may not process the hue or brightness of the virtual background image to be rendered, but directly fuse the foreground and background of the virtual background image to be rendered with the main image to obtain the final output image.

[0343] In some embodiments, the image rendering process may include not only a consistency rendering process based on structural semantics and / or a consistency rendering process based on interaction relationships, but also an underlying consistency rendering process or a consistency rendering process based on position relationships. In this case, the image rendering process may be as follows:

[0344] The terminal device 100 first performs the above-mentioned consistency rendering process based on structural semantics and / or the consistency rendering process based on interactive relationship to obtain the virtual background image to be rendered. The specific process can refer to the image rendering process shown above, which will not be repeated here.

[0345] Then, the terminal device 100 performs the bottom layer consistency rendering process, that is, based on the virtual background image to be rendered and the subject image, performs the bottom layer consistency rendering process. Finally, the subject image after the bottom layer consistency rendering and the virtual background image to be rendered are fused with the foreground and background to obtain the final output image.

[0346] Alternatively, the terminal device 100 performs a consistent rendering process based on positional relationships, that is, the main image and the virtual background image to be rendered are input into a trained STN network, and then Warp is performed according to the change matrix, and finally the Warped main image and the Warped virtual background image to be rendered are fused with the foreground and background to obtain the final output image.

[0347] It can be understood that, in addition to the several image rendering processes mentioned above, other possible image rendering processes can also be obtained based on the several image rendering processes mentioned above.

[0348] For example, the image rendering process does not perform the consistency rendering process based on structural semantics and the consistency rendering process based on interaction relationships, but only performs the underlying consistency rendering process and the consistency rendering process based on position relationships. At this time, the image rendering process can be as follows:

[0349] The terminal device 100 uses the target virtual background image as the background image to be rendered, and performs an underlying consistency rendering process based on the background image to be rendered and the main image; finally, a consistency rendering process based on the position relationship is performed on the main image after underlying consistency rendering and the virtual background image to be rendered to obtain an output image.

[0350] For another example, the image rendering process may only include the underlying consistency rendering process, without performing the consistency rendering process based on structural semantics, the consistency rendering process based on interaction relationships, and the consistency rendering process based on position relationships. In this case, the image rendering process may be as follows:

[0351] The terminal device 100 uses the target virtual background image as the background image to be rendered, performs an underlying consistency rendering process based on the background image to be rendered and the main image, and finally, fuses the main image after underlying consistency rendering and the virtual background image to be rendered into the foreground and the background to obtain an output image.

[0352] For another example, the image rendering process does not perform the consistency rendering process based on structural semantics, the consistency rendering process based on interaction relationships, the consistency rendering process based on position relationships, and the consistency rendering process based on underlying layers, but only performs the consistency rendering process based on position relationships. In this case, the image rendering process can be as follows:

[0353] The terminal device 100 performs a consistency rendering process based on the position relationship according to the target virtual background image and the main image to obtain an output image.

[0354] Other image rendering processes are not listed one by one here, and the same or similar points of each image rendering process can be referred to each other and will not be described in detail here.

[0355] In comparison, the second image rendering process, the third image rendering process and other possible image rendering processes, although the effects are worse than the background replacement effect of the first image rendering process, can still improve the background replacement effect.

[0356] The above describes the process of replacing a certain frame of original video image with a virtual background based on the original video image or the subject image and the target virtual background image to obtain an output image. Figure 4 In response to the click operation on the virtual background image 411, the mobile phone determines that the target virtual background image is the virtual background image 411, and then performs any of the above-mentioned image rendering processes based on the original video image corresponding to the answering interface 48 and the virtual background image 411 to obtain an output image, and finally displays the output image to obtain the image 410 after the background is replaced. For another example, see Figure 6 In response to the click operation of the virtual background image 412, the mobile phone determines that the target virtual background image is the virtual background image 412, and then performs any of the image rendering processes mentioned above based on the original video image and the virtual background image 412 corresponding to the answering interface 48 to obtain the output image displayed in the interface 415.

[0357] The virtual background replacement process provided in the embodiment of the present application can be applied to video call scenarios on mobile phones as well as video call scenarios on large-screen devices. Fig. 20 The schematic diagram of the virtual background replacement scene of the large-screen device shown in FIG. Fig. 20 As shown, in a home scene, the user makes a video call through a large-screen device 201, on which a Changlian call is installed. The magic pen 202 in the video call interface can be used to call out a virtual background selection window, from which the user can select a background image to be replaced. After the user selects the target virtual background image, the large-screen device performs any of the above-mentioned image rendering processes based on the target virtual background image and the original video image to obtain an image 203 after the background is replaced.

[0358] In a specific application, the terminal device 100 can perform the above-mentioned virtual background replacement process for each frame image in the video stream, or can perform the above-mentioned virtual background replacement process only every 5 frames or every 10 frames, that is, every 5 frames or 10 frames, the original video image is segmented into foreground and background to obtain the main image and the original background image, and then the target virtual background image is determined, and any of the above-mentioned image rendering processes is performed based on the main image and the target virtual background image.

[0359] When the terminal device 100 continues to replace the virtual background of the video stream, if the terminal device 100 recognizes that there is an occlusion relationship and / or an interaction relationship between the subject and the object in the original background image, the corresponding virtual object is rendered in the virtual background image. If the terminal device 100 recognizes that the interaction relationship ends and / or the occlusion relationship ends, the first virtual object to be rendered and / or the second virtual object to be rendered can be removed.

[0360] The end of the interaction relationship refers to the end of the interaction between the subject and the interaction object in the original background image. For example, the subject is a person, the interaction object is a chair in the original background image, and the interaction action is "sit down". When the person stands up from the chair, the interaction action of "sit down" is considered to be over, and the interaction relationship between the person and the chair in the original background image is over. The end of the occlusion relationship refers to the change from the subject being occluded to being unoccluded. For example, the subject is a person. At a certain moment, the person is occluded by the table in the original background image. At the next moment, the person is not occluded by the table, and the occlusion relationship is considered to be over.

[0361] From a visual point of view, when there is an interaction relationship or an occlusion relationship, a corresponding virtual object will exist in the image after the background is replaced, and when the interaction relationship or the occlusion relationship ends, the virtual object in the image after the background is replaced will disappear.

[0362] In terms of specific implementation, the terminal device 100 continuously performs a virtual background replacement process on the video image in the video stream. If at a certain moment, through the above-mentioned consistency rendering process based on structural semantics, it is determined that the main body in the original video image is not obstructed, then the occlusion relationship is considered to be ended, and there is no need to render the first virtual object to be rendered. Similarly, through the above-mentioned consistency rendering process based on the interaction relationship, it is determined that the interaction relationship is ended, and there is no need to render the second virtual object to be rendered. When an action corresponding to the interaction action is identified, the interaction relationship can be considered to be ended. For example, when the interaction action is "sitting down" and the action corresponding to the interaction action is "standing up", that is, when the "standing up" action is identified, the interaction relationship is considered to be ended.

[0363] At this time, since the subject is not blocked and the subject does not perform the preset interactive action, there is no virtual object to be rendered in the virtual background to be rendered. The terminal device 100 performs the subsequent underlying consistency rendering process and the consistency rendering process based on the position relationship based on the virtual background image to be rendered and the subject image, and obtains the output image of the current virtual background replacement process. There is no virtual object in the output image. In this way, from the user's visual perspective, when the interactive relationship ends or the occlusion relationship ends, the virtual objects in the image after the background is replaced will disappear. For example, after a person stands up from a chair, the chair in the image after the background is replaced will also disappear. When a person changes from being blocked to being unblocked, the objects used for blocking in the image after the background is replaced (such as a table) will also disappear.

[0364] For example, see Fig.21 Schematic diagram of the change of the virtual background replacement image shown. Fig.21 As shown, image 211 is an image obtained after the original video image 1 undergoes the first virtual background replacement process. Figure 7 Image 78 in the original video image 1 is Figure 7 The target virtual background image is Figure 7 For more information, see the above Figure 7 The image 211 includes a subject 212 and a subject 213 , and a virtual object 214 .

[0365] Assume that the video stream includes original video image 1, original video image 2, original video image 3, and original video image 4 in sequence.

[0366] Second virtual background replacement process: The terminal device 100 performs virtual background replacement on the original video image 2 to obtain the image 215 after the background replacement. Specifically, the terminal device 100 determines whether there is an occlusion relationship and / or an interaction relationship between the subject 212 and the subject 213 based on the original video image 2. At this time, there is an occlusion relationship between the subject 213, and there is no occlusion relationship between the subject 213, and the virtual object 214 and the target virtual background image are fused with the foreground and the background to obtain the virtual background image to be rendered. Then, according to the virtual background image to be rendered and the original video image 2, the underlying consistency rendering process and the consistency rendering process based on the position relationship are performed in sequence to obtain the image 215.

[0367] The third virtual background replacement process: the terminal device 100 performs a virtual background replacement on the original video image 3 to obtain an image 216 after the background is replaced. Specifically, the terminal device 100 performs a consistency rendering process based on structural semantics and a consistency rendering process based on an interactive relationship on the original video image to determine whether the subject has an occlusion relationship and an interactive relationship. At this time, the subject 212 has an interactive relationship with the chair in the original background, and the subject 213 has an occlusion relationship with the table in the original background, so the virtual objects 217 and 214 that need to be rendered are determined. Next, the virtual objects 217 and 214 are rendered in the target rendered virtual background image to obtain the virtual background image to be rendered, and then based on the virtual background image to be rendered and the original video image 3, the underlying consistency rendering process and the consistency rendering process based on the position relationship are performed in sequence to obtain the image 216.

[0368] The fourth virtual background replacement process: the terminal device 100 performs a virtual background replacement on the original video image 4 to obtain an image 218 after the background is replaced. Specifically, the terminal device 100 performs a consistency rendering process based on structural semantics and a consistency rendering process based on an interactive relationship on the original video image to determine whether the subject has an occlusion relationship and an interaction relationship. At this time, the subject 213 in the original video image 4 does not have an occlusion relationship and an interaction relationship, and the subject 212 has an interaction relationship but no occlusion relationship, so it is determined that the virtual object 217 needs to be rendered. Next, the virtual object 217 and the target virtual background image are fused with the foreground and background to obtain the virtual background image to be rendered. Finally, based on the original video image 4 and the virtual background image to be rendered, the underlying consistency rendering process and the consistency rendering process based on the position relationship are performed in sequence to obtain image 218.

[0369] The terminal device displays image 211, image 215, image 216 and image 218 on the display screen in sequence according to the playback order of the original video images. From the user's visual perspective, when the subject 212 moves from standing to sitting, a chair will be added to the virtual background image accordingly. When the subject 212 and the subject 213 are not blocked, the previously rendered virtual object 214 will also disappear.

[0370] In addition to being applicable to video call scenarios, the virtual background replacement solution provided in the embodiments of the present application can also be applied to virtual background replacement scenarios such as background crossing, special effects production, video conferencing, taking photos, and recording videos.

[0371] The shooting scene and the video recording scene are introduced as examples below.

[0372] See also Fig. 22 FIG. 1 is a schematic diagram of an interface of a virtual background replacement process in a shooting scene. Fig. 22 As shown, the mobile phone responds to the click operation on the camera 221 and displays the preview interface 222. Then, the mobile phone responds to the click operation on the magic pen 223 in the preview interface 22 and pops up a window 224 in the preview interface 222. The window 224 sequentially displays scenes 225 to 228. The scenes 225 to 228 can correspond to Figure 5 The virtual background images 412, 413, 414 and 411 in the image are shown in FIG. 4. The user can select the scene that he needs to replace from the window 224 according to his needs.

[0373] After the mobile phone receives a click operation for scene 225, the mobile phone displays a preview interface 229. At this time, the image corresponding to the preview interface 222 is the original video image, and the image corresponding to the scene 225 is the target virtual background image. The mobile phone first performs a consistency rendering process based on structural semantics and a consistency rendering process based on an interactive relationship based on the original video image to determine whether a virtual object needs to be rendered. If a virtual object needs to be rendered, the virtual object to be rendered and the target virtual background image are merged to obtain a virtual background image to be rendered. In the current case, since there is no occlusion relationship and interaction relationship between the subject and the objects in the original background image, it is not necessary to render a virtual object. Based on the subject image and the virtual background image to be rendered, the underlying consistency rendering process and the consistency rendering process based on the position relationship are performed to obtain an output image after background replacement, which is the image corresponding to the preview interface 229.

[0374] After the mobile phone displays the preview interface 229 , the user can click the control 2210 to take a photo. After receiving the click operation on the control 2210 , the mobile phone saves the image corresponding to the preview interface 229 as a picture and displays the picture in the control 2211 .

[0375] The user can view the captured picture by clicking 2211. At this time, after the mobile phone receives the click operation on the control 2211, the picture preview interface is displayed, and the captured picture 2212 is displayed in the picture preview interface. The picture 2212 is an image after the background is replaced.

[0376] exist Fig. 22 In the illustrated photo-taking process, if the mobile phone recognizes that a person is blocked by an object and / or performs a preset action in the image corresponding to the preview interface 222, a corresponding virtual object will also exist in the captured image 2212. Fig.21 , assuming that during the shooting process, subject 212 and subject 213 make corresponding interactive actions, or change the occlusion relationship, the preview interface also displays images such as image 211, image 215, image 216 and image 218.

[0377] In addition, after the mobile phone displays the preview interface 229, the user can also use the magic pen to replace the virtual background again.

[0378] See also Fig.23 FIG. 1 is a schematic diagram of an interface for a virtual background replacement process in a video recording scene. Fig.23 As shown, after receiving a click operation on the camera 231, the mobile phone displays a preview interface 232. The preview interface 232 includes a magic pen 233, and the preview interface 232 displays the original video image collected by the mobile phone through the camera.

[0379] When the mobile phone receives a click operation on the control 234, the mobile phone starts recording and displays a recording interface 235. The recording interface 235 still displays the original video image collected by the mobile phone through the camera. During the recording process, the user can click the magic pen 233 to replace the virtual background.

[0380] When the mobile phone receives a click operation on the magic pen 233 in the video recording interface 235, the mobile phone pops up a window 236 in the video recording interface 235, and the window 236 displays scenes 237 to 2310 that can be used for background replacement. After the user clicks the scene 237 in the window 236, the mobile phone responds to the click operation, performs a virtual background replacement process, obtains an image after the background replacement, and displays the image after the background replacement in the interface 2311.

[0381] At this time, the image corresponding to scene 237 is the target virtual background image, and the image corresponding to the recording interface 235 is the original video image. Based on the target virtual image and the original video image, the consistency rendering process based on structural semantics, the consistency rendering process based on interaction relationship, the underlying consistency rendering process and the consistency rendering process based on position relationship are performed in sequence to obtain the output image after background replacement.

[0382] If, during the recording process, there is an occlusion relationship and / or an interaction relationship between a person in the original video image and an object in the background, a corresponding virtual object will also exist in the recorded video.

[0383] in addition, Fig.23 The virtual background replacement is performed after the video recording starts. In other embodiments, the virtual background replacement can also be performed before the video recording starts, that is, the window 236 is called out through the magic pen in the preview interface 232, and the corresponding scene is selected.

[0384] Fig. 22 and Fig.23 In the process of , the mobile phone can also perform the above-mentioned virtual background recommendation process. Fig. 22 and Fig.23 For the same or similar parts as above, please refer to the above and will not be repeated here.

[0385] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0386] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0387] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0388] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0389] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0390] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0391] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above-mentioned embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. The reference "one embodiment" or "some embodiments" described in the present application specification means that one or more embodiments of the present application include specific features, structures or characteristics described in combination with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0392] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image rendering method, characterized in that: Applied to a terminal device, the method comprises: Get the image to be processed; Detecting that a target object in the image to be processed performs a preset action and / or that the target object is blocked by a first object; Determine a second virtual object to be rendered corresponding to the preset action and / or a first virtual object to be rendered corresponding to the first object; Rendering the first virtual object to be rendered and / or the second virtual object to be rendered in a target virtual background image to obtain a virtual background image to be rendered, wherein a depth value of an interactive object corresponding to the second virtual object to be rendered is greater than a depth value of the target object; Image rendering is performed according to the virtual background image to be rendered and the main image to obtain a rendered image, wherein the main image is extracted from the image to be processed and includes the target object.

2. The method according to claim 1, characterized in that Performing image rendering according to the virtual background image to be rendered and the main image to obtain a rendered image includes: Performing bottom-layer consistency rendering based on the virtual background image to be rendered to obtain a main image after bottom-layer consistency rendering; Image rendering is performed according to the main image rendered by the underlying consistency and the virtual background image to be rendered to obtain the rendered image.

3. The method according to claim 2, characterized in that Performing bottom-layer consistency rendering based on the virtual background image to be rendered to obtain a main image after bottom-layer consistency rendering, including: Inputting the low-frequency image of the virtual background image to be rendered and the image to be processed into a pre-trained first style transfer model to obtain the image to be processed after bottom-layer consistency rendering output by the first style transfer model; A main image is extracted from the image to be processed after the bottom layer is consistently rendered, to obtain the main image after the bottom layer is consistently rendered.

4. The method according to claim 3, characterized in that The training process of the style transfer model includes: Obtaining a training data set, wherein the training data set includes a first virtual background image and an original video image; Inputting the low-frequency image of the first virtual background image and the original video image into a pre-constructed second style transfer model to obtain a forward training result output by the second style transfer model; Calculating a first loss value between the forward training result and a low-frequency image of the first virtual background image; Inputting the forward training result and the low-frequency image of the original video image into the second style transfer model after the forward training, and obtaining the reverse training result output by the second style transfer model after the forward training; Calculating a second loss value between the reverse training result and the original video image; Calculating a third loss value between the reverse training result and the low-frequency image of the original video image; Adjusting the network parameters of the second style transfer model according to the first loss value, and adjusting the network parameters of the second style transfer model after forward training according to the second loss value and the third loss value; The training process is repeated, and when a predetermined condition is met, the trained first style transfer model is obtained.

5. The method according to claim 2, characterized in that: Performing bottom-layer consistency rendering based on the virtual background image to be rendered to obtain a main image after bottom-layer consistency rendering, including: Converting the virtual background image to be rendered into a LAB color space to obtain a first image; Calculate the first standard deviation and the first mean of the L channel, the A channel, and the B channel of the first image respectively; Convert the main image to LAB color space to obtain a second image; Correcting the second image according to the first standard deviation and the first mean to obtain a third image, wherein the difference between the second standard deviation of the L channel, the A channel, and the B channel of the third image and the first standard is within a first preset threshold interval, and the difference between the second mean and the first mean is within a second preset threshold interval; The third image is converted from the LAB color space to the RGB color space to obtain a fourth image, where the fourth image is the main image after the bottom layer is consistently rendered.

6. The method according to any one of claims 2 to 5, characterized in that: Obtaining the rendered image according to the underlying consistent rendered subject image and the to-be-rendered virtual background, including: Inputting the subject image after the bottom layer consistency rendering into a pre-trained first STN network to obtain a first change matrix output by the first STN network; Inputting the virtual background image to be rendered into a pre-trained second STN network to obtain a second change matrix output by the second STN network; Using the first change matrix, an image affine change is performed on the subject image after the underlying consistent rendering to obtain a first changed image; Using the second change matrix to perform an image affine change on the virtual background image to be rendered, to obtain a second changed image; The first changed image and the second changed image are synthesized to obtain the rendered image.

7. The method according to any one of claims 1 to 6, characterized in that: Placing the second virtual object to be rendered in the target virtual background image includes: Determining, according to the semantic segmentation result of the image to be processed, a first position of the interactive object corresponding to the preset action in the image to be processed; Using a second position in the target virtual background image corresponding to the first position as a rendering position of the first virtual object to be rendered; Determining that a depth value of the interactive object in the image to be processed is greater than a depth value of the target object; The second virtual object to be rendered is rendered at the rendering position of the target virtual background image.

8. The method according to claim 1, characterized in that Detecting that a target object in the image to be processed is blocked by a first object includes: Determining the category to which each pixel in the image to be processed belongs according to the semantic segmentation result of the image to be processed; Acquiring depth information of the image to be processed; When it is determined according to the depth information that there is a target pixel point with a depth value less than the depth value of the target object within the preset range of the target object, the category corresponding to the target pixel point is taken as the first object, and it is determined that the target object is blocked by the first object.

9. The method according to any one of claims 1 to 8, characterized in that: Before rendering the first virtual object to be rendered and / or the second virtual object to be rendered in the target virtual background image to obtain the virtual background image to be rendered, the method further includes: Determining a virtual background image to be recommended according to the similarity between the original background image of the image to be processed and each second virtual background image; The virtual background image to be recommended is displayed.

10. The method according to claim 9, characterized in that Determining a virtual background image to be recommended according to the similarity between the original background image of the image to be processed and each second virtual background image includes: Segmenting the image to be processed into foreground and background to obtain the original background image of the image to be processed; Performing multi-class semantic segmentation on the original background image to obtain a second semantic segmentation result; Performing multi-class semantic segmentation on each of the second virtual background images to obtain a third semantic segmentation result of each of the second virtual background images; Calculating an IOU value of the original background image and each of the second virtual background images according to the second semantic segmentation result and the third semantic segmentation result; respectively calculating a first color distribution curve of the original background image and a second color distribution curve of each of the second virtual background images; calculating a curve similarity between the first color distribution curve and each of the second color distribution curves; The to-be-recommended virtual background image is determined from the second virtual background images according to the curve similarity and the IOU value.

11. The method according to claim 1, characterized in that: The method further comprises: If the depth value of the interactive object corresponding to the second virtual object to be rendered is less than the depth value of the target object, rendering the first virtual object to be rendered in the target virtual background image to obtain the virtual background image to be rendered, or using the target virtual background image as the virtual background image to be rendered; After performing image rendering according to the virtual background image to be rendered and the main image to obtain a rendered image, the method further includes: The second virtual object to be rendered is rendered in the rendered image to obtain an output image.

12. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images