A computer-implemented method and computing device for relighting a video

Abstract

Implementations described herein relate to methods, systems, and computer-readable media for relighting a video. In some implementations, a computer-implemented method includes receiving a plurality of frames of a video. Each video frame includes depth data and color data for a plurality of pixels. The method also includes segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The method also includes setting a depth value of each background pixel to a fixed depth value and applying a Gaussian filter to smooth the depth values of the plurality of pixels. The method also includes computing surface normals based on the depth values of the plurality of pixels. The method also includes rendering a relit frame by adding virtual lights based on the surface normals and the color data.

Description

[0001] This application is a divisional application of the following application,

[0002] International Application No. US16 / 505,45

[0003] International Application Date of the Original Application: July 08, 2019

[0004] National Application No. of the Original Application: 202010644815.9

[0005] Title of the Invention of the Original Application: Video Illumination Using Depth and Virtual Light BACKGROUND

[0006] Users often use devices, such as smartphones, tablets, computers, wearable devices, etc., to capture video. When the lighting conditions under which the video is captured are not suitable, the video can be dark, overexposed, or have artifacts. In addition, when a participant captures video in a video call, other participants in the video call can perceive objects in the background that the participant can not want the other participants to see, e.g., clutter in a room in which the participant has joined the video call.

[0007] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The work of the inventors, to the extent the inventors were aware of it, and that of other individuals and SUMMARY

[0008] The implementations described herein relate to methods, systems, and computer-readable media for relighting a video. In some implementations, a computer-implemented method includes receiving a plurality of frames of a video. Each frame of the video can include depth data and color data for a plurality of pixels. The method also includes segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The method also includes setting a depth value for each background pixel to a fixed depth value. The method also includes applying a Gaussian filter to smooth the depth values for the plurality of pixels. The method also includes computing surface normals based on the depth values for the plurality of pixels. The method also includes rendering a relit frame by adding virtual light based on the surface normals and the color data. In some implementations, rendering the relit frame can be performed using a graphics processing unit (GPU).

[0009] In some implementations, segmenting the frame can include generating a segmentation mask based on the depth range. In the segmentation mask, each pixel having a depth value within the depth range is classified as a foreground pixel, and each pixel having a depth value outside the depth range is classified as a background pixel. Segmenting the frame further includes performing a morphological opening process to remove noise and performing a morphological closing process to fill one or more holes in the segmentation mask. In some implementations, segmenting the frame can further include applying a temporal low-pass filter that updates the segmentation mask based on a similarity between one or more previous frames and the frame.

[0010] In some implementations, the virtual light can be an ambient light, a directional light, or a point light. In some implementations, the virtual light can be a ring light based on a plurality of point lights.

[0011] In some implementations, the method can further include creating a three-dimensional (3D) mesh based on the depth values of the plurality of pixels and the surface normal, wherein the 3D mesh is used to render the relit frame. In some implementations, creating the 3D mesh can include obtaining an intrinsic matrix of a camera that captured the plurality of frames of the video, and calculating a position of each vertex of the 3D mesh based on the intrinsic matrix and the depth value of each pixel. In some implementations, calculating the position of each vertex can include calculating an x-coordinate and a y-coordinate in a world space based on the depth value and based on the intrinsic matrix of the camera. In some implementations, the method can further include calculating a texture displacement of each vertex of the 3D mesh based on the position of each vertex and a width and a height of the frame.

[0012] Some implementations can include a non-transitory computer-readable medium having instructions stored thereon. The instructions, when executed by one or more hardware processors, cause the processors to perform operations including receiving a plurality of frames of a video. Each frame of the video can include depth data and color data for a plurality of pixels. The operations further include segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The operations also include setting a depth value of each background pixel to a fixed depth value. The operations further include applying a Gaussian filter to smooth the depth values of the plurality of pixels. The operations also include calculating a surface normal based on the depth values of the plurality of pixels. The operations further include rendering a relit frame by adding a virtual light based on the surface normal and the color data.

[0013] In some implementations, the instructions to perform the segmenting each frame can include instructions to perform operations comprising generating a segmentation mask based on a depth range, where each pixel having a depth value within the depth range is classified as a foreground pixel and each pixel having a depth value outside the depth range is classified as a background pixel, performing a morphological opening process to remove noise, and performing a morphological closing process to fill one or more holes in the segmentation mask. In some implementations, the instructions to perform the segmenting each frame can further include instructions to apply a temporal low-pass filter that updates the segmentation mask based on a similarity between one or more previous frames and the frame.

[0014] In some implementations, the instructions can cause the one or more hardware processors to perform further operations to create a three-dimensional (3D) mesh based on the depth values of the plurality of pixels and the surface normal, where the 3D mesh is used to render the relit frame. In some implementations, creating the 3D mesh can include obtaining an intrinsic matrix of a camera that captured the plurality of frames of the video, and computing a position of each vertex of the 3D mesh based on the intrinsic matrix and the depth value of each pixel.

[0015] Some implementations can include a system that includes one or more hardware processors coupled to a memory. The memory can include instructions stored thereon. The instructions, when executed by the one or more hardware processors, cause the processors to perform operations including receiving a plurality of frames of a video. Each frame of the video can include depth data and color data for a plurality of pixels. The operations further include segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The operations also include setting a depth value of each background pixel to a fixed depth value. The operations further include applying a Gaussian filter to smooth the depth values of the plurality of pixels. The operations also include computing a surface normal based on the depth values of the plurality of pixels. The operations further include rendering a relit frame by adding virtual light based on the surface normal and the color data.

[0016] In some implementations, the instructions to perform the segmenting each frame can include instructions to perform operations comprising generating a segmentation mask based on a depth range, where each pixel having a depth value within the depth range is classified as a foreground pixel and each pixel having a depth value outside the depth range is classified as a background pixel, performing a morphological opening process to remove noise, and performing a morphological closing process to fill one or more holes in the segmentation mask. In some implementations, the instructions to perform the segmenting each frame can further include instructions to apply a temporal low-pass filter that updates the segmentation mask based on a similarity between one or more previous frames and the frame.

[0017] In some implementations, the instructions can cause the one or more hardware processors to perform further operations to create a three-dimensional (3D) mesh based on the depth values of the plurality of pixels and the surface normal, wherein the 3D mesh is used to render the relit frame. In some implementations, creating a 3D mesh can include obtaining an intrinsic parameter matrix of a camera that captured the plurality of frames of the video, and computing a position of each vertex of the 3D mesh based on the intrinsic parameter matrix and the depth value of each pixel. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a block diagram of an example network environment that can be used for one or more embodiments described herein.

[0019] Figure 2 is a flowchart illustrating an example method of relighting a video, according to some embodiments.

[0020] Figure 3 shows an example relit video frame generated using original color data and depth data included in a video.

[0021] Figure 4 shows an example video frame and a corresponding depth image.

[0022] Figure 5 shows an example relit video frame generated according to some embodiments.

[0023] Figure 6 is a block diagram of an example computing device that can be used for one or more embodiments described herein. DETAILED DESCRIPTION

[0024] Embodiments described herein generally relate to relighting a video. In particular, embodiments use virtual lights to relight a video. Using virtual lights to light a scene is based on depth data obtained as part of the video.

[0025] A technical problem in video capture is to provide a high quality, e.g., well-lit video without artifacts, in cases where the source video from a camera cannot be well-lit due to lighting conditions in the captured scene. Another technical problem is to provide a video that blurs a background portion of the video.

[0026] One or more embodiments described herein include methods, devices, and computer-readable media having instructions to relight a video. In some embodiments, a video call application can be implemented that implements code to relight a video.

[0027] Figure 1A block diagram of an example network environment 100 is shown that can be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, such as server system 102 in Figure 1 For example, the server system 102 can communicate with a network 130. The server system 102 can include a server device 104 and a database 106 or other storage device. In some embodiments, the server device 104 can provide a video call application 152b.

[0028] The network environment 100 can also include one or more client devices, such as client devices 120, 122, 124, and 126, which can communicate with each other and / or with the server system 102 via the network 130. The network 130 can be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc. In some embodiments, the network 130 can include peer-to-peer communication between devices, such as using a peer-to-peer wireless protocol (e.g., Bluetooth®, Wi-Fi Direct, etc.), etc. One example of peer-to-peer communication between two client devices 120 and 122 is shown by arrow 132.

[0029] For ease of illustration, Figure 1 One block is shown for the server system 102, the server device 104, the database 106, and four blocks are shown for the client devices 120, 122, 124, and 126. The server block can represent multiple systems, server devices, and network databases, and these blocks can be provided in different configurations than shown. For example, the server system 102 can represent multiple server systems that can communicate with other server systems via the network 130. In some embodiments, the server system 102 can include, for example, a cloud hosting server. In some examples, the database 106 and / or other storage device can be provided in a server system block that is separate from the server device 104 and can communicate with the server device 104 and other server systems via the network 130.

[0030] ​Further, there can be any number of client devices. Each client device can be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a cellular phone, a smart phone, a tablet computer, a television, a television set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices can also have a local database similar to database 106 or other storage. In some implementations, network environment 100 can not have all of the components shown and / or can have other elements including other types of elements instead of, or in addition to, those described herein.

[0031] In various implementations, end users U1, U2, U3, and U4 can use respective client devices 120, 122, 124, and 126 to communicate with server system 102 and / or to communicate with each other. In some examples, users U1, U2, U3, and U4 can interact with each other via applications running on respective client devices and / or server system 102 and / or via a network service (e.g., a social networking service or other type of network service) implemented on server system 102. For example, respective client devices 120, 122, 124, and 126 can transmit data to and from one or more server systems (e.g., system 102).

[0032] In some implementations, server system 102 can provide appropriate data to the client devices so that each client device can receive communication content or shared content uploaded to server system 102 and / or a network service. In some examples, users U1-U4 can interact via audio / video calls, audio, video, or text chats, or other communication modes or applications. A network service implemented by server system 102 can include a system that allows users to perform various communications, form links and associations, upload and post shared content (e.g., images, text, videos, audio, and other types of content), and / or perform other functions. For example, a client device can display received data, such as content posts sent or streamed to the client device and originated from a different client device through a server and / or a network service (or directly from a different client device), or originated from a server system and / or a network service. In some implementations, client devices can communicate directly with each other, e.g., using peer-to-peer communication between client devices as described above. In some implementations, a "user" can include one or more programs or virtual entities, as well as a human interfacing with the system or network.

[0033] In some implementations, any of the client devices 120, 122, 124, and / or 126 can provide one or more applications. For example, as shown in Figure 1 FIG. 1, the client device 120 can provide a video call application 152a and one or more other applications 154. The client devices 122-126 can also provide similar applications.

[0034] For example, the video call application 152 can provide users of the respective client devices (e.g., users U1-U4) with the ability to participate in video calls with one or more other users. In a video call, a client device can transmit locally captured video to other devices participating in the video call, with the permission of the user. For example, such video can include live video captured using a camera of the client device, such as a front-facing camera, a rear-facing camera, and / or one or more other cameras. In some implementations, the camera can be separate from the client device and can be coupled to the client device, e.g., via a network, via a hardware port of the client device, etc. The video call application 152 can be a software application executing on the client device 120. In some implementations, the video call application 152 can provide a user interface. For example, the user interface can enable a user to place a video call to one or more other users, receive a video call from other users, leave a video message for other users, watch a video message from other users, etc.

[0035] The video call application 152a can use hardware and / or software of the client device 120 to implement, as described with reference to Figure 6 FIG. 1. In different implementations, the video call application 152a can be a standalone client application executing, e.g., on any of the client devices 120-124, or can work in conjunction with a video call application 152b provided on the server system 102. The video call application 152a and the video call application 152b can provide video call (including video calls with two or more participants) functionality, audio or video messaging functionality, address book functionality, etc.

[0036] In some implementations, the client device 120 can include one or more other applications 154. For example, the other applications 154 can be applications that provide various types of functionality, such as a calendar, an address book, an email, a web browser, shopping, transportation (e.g., taxi, train, air reservations, etc.), entertainment (e.g., a music player, a video player, a gaming application, etc.), social networking (e.g., messaging or chatting, audio / video calling, sharing images / videos, etc.), image capture and editing (e.g., image or video capture, video editing, etc.), and the like. In some implementations, the one or more other applications 154 can be standalone applications that execute on the client device 120. In some implementations, the one or more other applications 154 can access server systems that provide data and / or functionality of the applications 154.

[0037] User interfaces on the client devices 120, 122, 124, and / or 126 can allow display of user content and other content, including images, videos, data and other content, and communications, privacy settings, notifications, and other data. Such user interfaces can be displayed using a combination of software on the client devices, software on server devices, and / or client and server software executing on the server devices 104 (e.g., application software or client software that communicates with the server systems 102). The user interfaces can be displayed by display devices of the client devices or server devices (e.g., touchscreens or other display screens, projectors, etc.). In some implementations, applications running on the server systems can communicate with the client devices to receive user input at the client devices and output data such as visual data, audio data, etc. at the client devices.

[0038] Other implementations of the features described herein can use any type of system and / or service. For example, other networking services (e.g., connected to the Internet) can be used instead of or in addition to social networking services. Any type of electronic device can utilize the features described herein. Some implementations can provide one or more features described herein on one or more client or server devices that are disconnected from a computer network or that are intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device can display content posts stored on a storage device local to the client device, e.g., content posts previously received over a communication network.

[0039] Figure 2 FIG. 2 is a flowchart showing one example of a method 200 of relighting a video, in accordance with some implementations. In some implementations, the method 200 can be implemented, for example, on the server system 102 as shown in Figure 1 In some implementations, some or all of the method 200 can be implemented on a server system, such as the server system 102 as shown in FIG. 1.Figure 1 The method 200 can be implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both server devices and client devices. In the described example, the implemented system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., the database 106 or other memory). In some implementations, different components of one or more servers and / or clients can perform different blocks or other portions of the method 200. In some examples, a first device is described as performing a block of the method 200. Some implementations can have one or more blocks of the method 200 performed by one or more other devices (e.g., other client devices or server devices), which can send results or data to the first device.

[0040] In some implementations, the method 200 or a portion of the method can be initiated automatically by the system. In some implementations, the implemented system is a first device. For example, the method (or portions thereof) can be performed periodically, or based on one or more particular events or conditions, such as a user initiating an application (e.g., a video call application), a camera of a user device being activated to capture video, a video editing application being launched, and / or one or more other conditions that can be specified in settings read by the method. In some implementations, such conditions can be specified by a user in stored user custom preferences.

[0041] In one example, the first device can be a camera, a cell phone, a smartphone, a tablet computer, a wearable device, or other client device capable of capturing video and capable of performing the method 200. In another example, a server device can perform the method 200 for video, such as a client device can capture video frames that are processed by the server device. Some implementations can initiate the method 200 based on user input. A user (e.g., an operator or an end user) can select to initiate the method 200, such as from a displayed user interface (e.g., an application user interface or other user interface). In some implementations, the method 200 can be implemented by a client device. In some implementations, the method 200 can be implemented by a server device.

[0042] A video referred to herein can include a sequence of image frames. Each image frame can include color data and depth data for a plurality of pixels. For example, the color data can include a color value for each pixel, while the depth data can include a depth value, such as a distance from a camera that captured the video. In some implementations, a video can also include audio data. The method 200 can begin at block 202.

[0043] In block 202, it is checked whether user consent (e.g., user permission) has been obtained to use user data in the implementation of the method 300. For example, user data can include video captured by a user using a client device, video stored or accessed by a user (e.g., using a client device), video metadata, user data related to use of a video call application, user preferences, etc. In some implementations, one or more blocks of the methods described herein can use such user data.

[0044] If user consent has been obtained from the relevant user for which user data can be used in the method 200, then in block 204 it is determined that blocks of the method here can be implemented with the ability to use user data as described for those blocks, and the method continues to block 210. If user consent has not been obtained, then in block 206 it is determined that the blocks are to be implemented without use of user data, and the method continues to block 210. In some implementations, if user consent has not been obtained, the blocks are implemented without use of user data and with synthetic data and / or general or publicly accessible and publicly usable data. In some implementations, if user consent has not been obtained, the method 200 is not performed.

[0045] In block 210 of the method 200, a plurality of video frames of a video are received. For example, the plurality of video frames can be captured by a client device. In some implementations, the plurality of video frames can be captured during a live video call, e.g., via a portion of a video call application, by the client device. In some implementations, the plurality of video frames can be pre-recorded, e.g., a portion of a recorded video. Block 210 can be followed by block 211.

[0046] In block 211, a first frame of the video is selected. Block 211 can be followed by block 212.

[0047] In block 212, the selected frame is segmented to classify each pixel as a foreground pixel or a background pixel. In some implementations, the depth data can be in a single color FLOAT 32 format. The depth data can be converted to an RGBA32 format prior to segmenting the selected frame. In some implementations, the segmentation is performed by generating a segmentation mask based on a depth range. The depth value of each pixel can be compared to the depth range to determine whether the depth value is within the depth range. If the depth value is within the depth range, the pixel is classified as a foreground pixel. If the depth value is outside the depth range, the pixel is classified as a background pixel. The resulting segmentation mask includes a value for each pixel of the selected frame indicating whether the pixel is a foreground pixel or a background pixel.

[0048] In some implementations, segmenting the image can further include performing a morphological opening process. The morphological opening process removes noise from the segmentation mask. In some implementations, segmenting the image can further include performing a morphological closing process. The morphological closing process fills one or more holes in the segmentation mask. In some implementations, the morphological opening-closing process can be implemented as a combination of 1D filters.

[0049] Noise and / or holes in the segmentation mask can arise for various reasons. For example, when a video frame is captured by a client device using a camera with depth capability, the depth value of one or more pixels can not be determined accurately. For example, such inaccuracy can arise due to lighting conditions in which the frame is captured, due to sensor errors, due to features in the captured scene, etc. For example, a hole can arise if the camera does not capture a depth value. For example, if the camera uses a reflection-based sensor to measure depth, one or more pixels can not have a depth value if no reflected light from those pixels is detected from the scene at the time of capture. Such pixels can result in holes in the segmentation mask.

[0050] In some implementations, segmenting the frame can further include applying a temporal low-pass filter. Even when a scene captured in a video is static, the depth value of a corresponding pixel can vary between consecutive frames. This can occur due to imperfect depth data captured by the sensor. The temporal low-pass filter updates the segmentation mask based on similarity between one or more previous frames and the current frame. For example, if a scene captured in multiple video frames is static, consecutive frames can include similar depth values for corresponding pixels. If there is a change in the depth value of a corresponding pixel while the scene is static, such depth value can be erroneous and is updated using the temporal low-pass filter. If the similarity between one or more previous frames and the current frame is high, applying the temporal low-pass filter results in segmentation of the current frame to be consistent with the segmentation of one or more previous frames. When the similarity is low, for example when the scene is not static, the consistency produced by the temporal low-pass filter is weaker. In some implementations, the frame similarity score can be computed by using a GPU to implement a multi-pass computation. Block 212 can be followed by block 214.

[0051] In block 214, the depth value of each background pixel is set to a fixed depth value. For example, the background depth captured by a camera, e.g., the background depth of a client device such as a smartphone, tablet, or computer, can often be inaccurate. For background points in a fixed scene captured in a sequence of frames, the depth values of corresponding pixels in two adjacent frames can have a difference of several meters. In this case, the relighting using virtual lights can produce flickering in the background portion of the video. Setting the depth value to a fixed value ensures that such flickering does not occur or is minimized. In some embodiments, the fixed depth value can be a value outside of the depth range. In some embodiments, e.g., when capturing the scene using a mobile phone camera (or other portable device camera), the depth value of each background pixel can be set to 5 meters. In various embodiments, the fixed depth value can be selected to be a value far from the depth values of the foreground pixels. Setting the depth value to a fixed value ensures that flickering in the video is reduced because the variation in the depth values of corresponding pixels in adjacent frames is eliminated. For example, such variation can occur due to inaccurate / inconsistent depth data from the camera. Block 214 can be followed by block 216.

[0052] In block 216, the depth values of the foreground pixels can be smoothed. For example, smoothing of the depth values of the foreground pixels can be performed by applying a Gaussian filter to the depth values of a plurality of pixels. Application of the Gaussian filter can smooth the foreground and the border between the foreground and the background. Application of the Gaussian filter ensures a smooth transition between the background portion and the foreground portion of the frame. In some embodiments, the Gaussian filter can be implemented as a combination of ID filters. In various embodiments, smoothing of the depth values can be performed using any suitable technique. Block 216 can be followed by block 218.

[0053] In block 218, surface normals are computed from the depth values. The surface normals can define the reflection direction on the surface. In some embodiments, a fixed forward direction can be set for all pixels. Block 218 can be followed by block 220.

[0054] In block 220, a three-dimensional (3D) mesh can be created based on the depth values and the surface normals. The 3D mesh can describe the 3D scene captured in the video frames. In some embodiments, each pixel of the frame can be used as a vertex of the 3D mesh. In some embodiments, creating the 3D mesh can include obtaining an intrinsic matrix of the camera that captured the frame, and computing the position of each vertex of the 3D mesh based on the intrinsic matrix and the depth value of each pixel. Using the 3D mesh allows the color change of each pixel to be computed based on one or more virtual lights in a realistic manner such that the resulting relit video frame looks natural.

[0055] In some embodiments, computing the position includes computing an x coordinate (xw) in the world space for each vertex based on the depth value of the pixel corresponding to the vertex (z) and the intrinsic matrix (K) of the camera that captured the frame (xw = K-1 * z). In some embodiments, computing the position includes computing a y coordinate (yw) in the world space for each vertex based on the depth value of the pixel corresponding to the vertex (z) and the intrinsic matrix (K) of the camera that captured the frame (yw = K-2 * z). In some embodiments, computing the position includes computing a z coordinate (zw) in the world space for each vertex based on the depth value of the pixel corresponding to the vertex (z) and the intrinsic matrix (K) of the camera that captured the frame (zw = z).c ) and y coordinate (y c ). This calculation can be performed by using the formulas x c = (u - c x )*z c / f x and y c = (v - c y )*z c / f y In the formulas, (u, v) represents coordinates in the camera pixel space, and z c is the depth value of the z

[0056]

[0057] where w is the camera pixel space, and M is the intrinsic matrix of the camera. The intrinsic matrix of the camera can be defined as:

[0058]

[0059] where f x and f y are focal lengths, c x and c y are principal point offsets of the camera.

[0060] In some implementations, the texture displacement of each vertex of the 3D mesh can be calculated based on the pixel coordinates in the camera pixel space using the formulas u t = u / width and v t = v / height, where width is the width of the frame, and height is the height of the frame. Block 220 can be followed by block 222.

[0061] In box 222, the relit frame is rendered by adding virtual light. If a real light source was present when the video was captured, the virtual light mimics the effect produced by the corresponding real light source. Adding virtual light involves calculating adjustments to the color values ​​of pixels in the frame based on the type of light and the location of the light source. For example, virtual light can include one or more of ambient light, directional light, or point light. A point light can be a light located at a single point in space that emits light equally in all directions. For example, point lights can be used to simulate lamps and other local light sources. Directional light can be used to simulate distant light sources existing at infinity, without any identifiable light source location. When illuminated with directional light, objects in the scene light up as if the light came from the same direction. Ambient light can be used to simulate light that exists around the entire scene and can contribute to the overall look and brightness of the scene. Furthermore, multiple point lights can be combined to provide a ring light. When relit in video with a ring light, objects in the foreground (e.g., a face) can be illuminated, while objects in the background appear dark.

[0062] The addition of virtual lights is based on the frame's surface normals and color data. Specifically, the frame's color data is used as a texture to render the relit frame. Virtual lights can be configurable; for example, the number of virtual lights, their type, intensity, color, and direction can be selected based on user preferences or scene characteristics. Adding virtual lights can improve lighting conditions in a 3D scene and can also provide different lighting effects. For example, adding directional lights can brighten the entire frame. In another example, using one or more point lights can brighten the foreground or a portion of the foreground while darkening the background. In yet another example, adding colored lights can change the frame's color. Box 222 can be followed by box 224.

[0063] In box 224, the next frame of the video is selected. Box 224 may be followed by box 212, which segments the selected frame to separate the background and foreground portions of that frame. In some implementations, boxes 212-224 may be repeated until a relit frame corresponding to each of the multiple frames of the video is rendered. In some implementations, for example, when the video is streaming, method 200 may be performed on each frame of the video stream, for example, until the video stream ends.

[0064] In some implementations, combinations are possible. Figure 2 One or more boxes are shown. For example, box 212 can be combined with box 214, or with box 214 and block 216. For example, boxes 218 and 220 can be combined.

[0065] In some implementations, the method 200 can be implemented as part of a video call application that provides functionality to enable two or more participants to participate in a video call using computing devices. In these implementations, video from a participant's client device can be transmitted to one or more of the other participants substantially in real time, e.g., so that the different participants can participate in a natural conversation via the video call application.

[0066] In some implementations, the method 200 can be implemented at a sender device. In these implementations, a video frame can be received at block 210 from a camera of the sender device. The re-illuminated frame rendered at block 222 can be transmitted over a network to one or more receiving devices associated with participants in a video call. The re-illuminated frame can represent a video source of the sender. The receiving devices can display the received frame, e.g., on a display screen of the receiving devices.

[0067] In some implementations, the method 200 can be implemented at a receiving device. In these implementations, a video frame can be received at block 210 from a sender device over a network. The frame received in these implementations is not re-illuminated. In these implementations, the receiving device can perform the method 200 to obtain a re-illuminated video frame, and can display the received frame, e.g., on a display screen of the receiving device. The method 200 can also be implemented on a server, with user permission, e.g., if a server intermediates communications between multiple client devices and if the video is not encrypted. In different implementations, different portions of the method 200 can be implemented using any combination of sender devices, receiver devices, or server devices, as permitted by the user.

[0068] The method 200 can be performed by a client device (e.g., any of the client devices 120-126) and / or a server device (e.g., the server device 104). For example, in some implementations, a client device can capture a video and perform the method 200 to relight the video locally. For example, when the client device has appropriate processing hardware (e.g., a dedicated graphics processing unit (GPU) or another image processing unit (e.g., ASIC, FPGA, etc.)), the method 200 can be performed locally. In another example, in some implementations, a client device can capture a video and send the video to a server device that performs the method 200 to relight the video. For example, when the client device lacks the processing capability to perform the method 200, or in other cases, such as when the battery power available on the client device is below a threshold, the method 200 can be performed by a server device. In some implementations, the relighting of a video can be performed by a client device other than the device that captured the video. For example, a sender device in a video call can capture video frames and send them to a receiver device. The receiver device can then perform the method 200 before displaying the video. Such an implementation can be advantageous when the sender device lacks the ability to perform the method 200 in real-time.

[0069] The method 200 provides several technical benefits. The use of a temporal low-pass filter that updates the segmentation mask based on the similarity between one or more previous frames and the frame can ensure that most pixels identified as background in a particular frame are also identified as background in adjacent frames, especially when there is a high degree of similarity between the particular frame and the adjacent frames. Thus, a video relit using the method 200 with virtual light can have lower flicker in the background portions. In addition, the lower flicker can also be a result of setting the depth values of the background pixels to a fixed value. Setting the depth values in this way can ensure that the addition of virtual light illuminates the background pixels in a consistent manner across multiple frames of the video.

[0070] Another technical benefit is that the video relit using method 200 can be smooth over multiple frames with lower visible artifacts that can be caused by errors in depth measurements and changes in depth measurements between adjacent frames, for example, that can occur due to the quality of the depth sensor performing the depth measurements. For example, depth data from a consumer device such as a smartphone can have significant changes between frames. Smoothing the depth values of multiple pixels using a Gaussian filter can smooth the foreground, for example, ensuring that foreground pixel depth values are consistent within a frame, and can smooth the boundaries between the foreground and the background. Using a temporal low-pass filter for frame coherence can ensure lower artifacts since the depth values of pixels corresponding to foreground objects can not change significantly between adjacent frames when the similarity between adjacent frames is high. For example, when there is less motion in the scene, adjacent frames can have a high degree of similarity.

[0071] A further technical effect of some embodiments is a reduction in the total energy usage of a display screen for displaying the relit video. For example, when displaying a video that has been relit using a ring light, the total energy is lower compared to the original video or a video that has been relit using a directional light because the ring light darkens the background portion of the video.

[0072] Further, method 200 is computationally efficient and can enable real-time video relighting on devices with relatively low computational power, for example, mobile devices such as phones or tablets, and can generate videos that can be sent in a video call. Setting the depth values of background pixels to a fixed value can reduce the computational cost of the relighting step.

[0073] Figure 3 Example relit video frames (302, 304, 306) generated using original color data and depth data included in a video are shown. In Figure 3 In the example, the image is relit to darken the background portion of the image and to light the foreground pixels that depict the face. As can be seen in each of frames 302, 304, and 306, the relit has artifacts in the face. For example, in frames 304 and 306, different areas appear to include ridges with different depths (for example, see the forehead area in frame 304) or inconsistent colors (as seen). Further, the background portion appears to not be uniformly dark.

[0074] Figure 4Example video frames and corresponding depth images are shown. The video frames include color data and depth data. Raw color images (402) and corresponding depth images (404) captured by a camera are shown. After performing segmentation of the video frames and setting background pixels to a fixed depth value, modified depth images (406) are obtained. In addition, smoothed depth images (408) are obtained when the images are smoothed.

[0075] Figure 5 Example relit video frames generated according to some embodiments are shown. Figure 5 The relit video frames shown correspond to raw video frames with color images (402) and depth images (404). In Figure 5 In the middle, images relit with a directional light (502) and images relit with a ring light (504) are shown. As Figure 5 As shown, the relit video frames do not have artifacts, unlike the video frames (302, 304, 306) shown in Figure 3 In the middle, images relit with a directional light (502) and images relit with a ring light (504) are shown. As Figure 5 While a sequence of frames is not shown, relighting video frames using the techniques described here results in video with low flicker.

[0076] As can be seen, the images relit with a directional light (502) have improved light on the foreground portion of the image depicting the person without any impact on the background portion. As can be seen, the images relit with a ring light (504) have improved light on the foreground portion of the image depicting the person while effectively darkening the background. While Figure 5 While a single image is shown, video frames relit using the techniques described here are consistent in time without artifacts due to errors in depth measurement or object motion between adjacent frames of the video.

[0077] The techniques can be applied in real-time and can produce relit video without lag. For example, relit video can be used in a video call application as a person's video stream. Relit video enables a user to participate in a video call from a poorly lit or poorly backgrounded place without negatively impacting the video.

[0078] Figure 6 is a block diagram of an example device 600 that can be used to implement one or more features described herein. In one example, device 600 can be used to implement a client device, e.g., a mobile device, a laptop computer, a desktop computer, a server, etc. Figure 1Alternatively, device 600 can implement a server device, such as server 101. In some embodiments, device 600 can be used to implement a client device, a server device, or both a client device and a server device. Device 600 can be any suitable computer system, server, or other electronic or hardware device as described above.

[0079] One or more of the methods described herein can be run in a standalone program that can be executed on any type of computing device, a program run on a web browser, a mobile application ("app") run on a mobile computing device, such as a cell phone, a smart phone, a tablet computer, a wearable device (watch, armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted display, etc.), a laptop computer, etc. In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device, and receives final output data for output (e.g., for display) from the server. In another example, all computations can be performed within a mobile application (and / or other application) on the mobile computing device. In another example, computations can be split between the mobile computing device and one or more server devices.

[0080] In some embodiments, device 600 includes a processor 602, a memory 604, an input / output (I / O) interface 606, and a camera 614. Processor 602 can be one or more processors and / or processing circuitry that execute program code and control the basic operations of device 600. A "processor" includes any suitable hardware system, mechanism or component that processes data, signals or other information. A processor can include a system on a chip (SoC) with a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a special-purpose circuit, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix computations (e.g., matrix multiplication), or other system.

[0081] In some embodiments, the processor 602 can include a CPU and a GPU (or other parallel processor). In implementations, the GPU or parallel processor can include a plurality of processing cores that can perform computations in parallel, e.g., 100 cores, 1000 cores, etc. Further, the GPU or parallel processor can include a GPU memory that is separate from the main memory 604. The GPU memory can be accessible by each of the GPU cores. An interface can be provided to enable data to be transferred between the main memory 604 and the GPU memory.

[0082] In some embodiments, a GPU can be used to implement the method 200 or portions thereof. In particular, a GPU can be used to render a video frame based on adding virtual lights, e.g., computing textures based on light and color data for a video frame. In some embodiments, color data and depth data can be stored in a GPU memory (also referred to as a GPU buffer). In these embodiments, the color data and depth data can be processed by the GPU, which can be faster than processing the data using a CPU. In some embodiments, where multiple virtual lights are added, each virtual light can be computed independently. In these embodiments, each pixel can be updated multiple times, e.g., once for each light.

[0083] In some embodiments, the processor 602 can include one or more co-processors that implement neural network processing. In some embodiments, the processor 602 can be a processor that processes data to produce a probabilistic output, e.g., the output produced by the processor 602 can be imprecise or can be precise within a range from an expected output. The processing need not be limited to a particular geographic location or have time limitations. For example, a processor can perform its functions in "real time," "offline," in a "batch mode," etc. Portions of processing can be performed at different times and by different (or the same) processing systems. A computer can be any processor in communication with a memory.

[0084] Memory 604 is typically provided in the device 600 to be accessed by the processor 602 and can be any suitable processor readable storage media, such as random access memory (RAM), read only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, or the like, suitable for storing instructions for execution by the processor 602, and located in the device 600 separately from or in conjunction with the processor 602. The memory 604 can store software that is executed by the processor 602 in the device 600, including an operating system 608, a video call application 610, and application data 612. One or more other applications can also be stored in the memory 604. For example, the other applications can include applications such as a data display engine, a web hosting engine, an image display engine, a notification engine, a social network engine, an image / video editing application, a media sharing application, and the like. In some embodiments, the video call application 610 and / or each of the other applications can include instructions that enable the processor 602 to perform some or all of the methods described herein (e.g., Figure 2 One or more of the methods disclosed herein can operate in several environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application ("app") that runs on a mobile computing device, and the like.

[0085] The application data 612 can include a video, such as a sequence of video frames. In particular, the application data 612 can include color data and depth data for each of a plurality of video frames of the video.

[0086] Any software in the memory 604 can alternatively be stored on any other suitable storage location or computer readable medium. Furthermore, the memory 604 (and / or other connected storage devices) can store one or more taxonomies, an electronic encyclopedia, a dictionary, a thesaurus, a knowledge base, message data, grammar, user preferences, and / or other instructions and data used in the features described herein. The memory 604 and any other type of storage (disk, optical disk, tape, or other tangible medium) can be considered a "storage" or "storage device."

[0087] I / O interface 606 can provide functionality for enabling the device 600 to interface to other systems and devices. Interface devices can be included as part of the device 600 or can be separate and in communication with the device 600. For example, network communication devices, storage devices (e.g., memory and / or databases 106), and input / output devices can communicate through I / O interface 606. In some embodiments, the I / O interface can connect to interface devices such as input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0088] Some examples of interface devices that can be connected to I / O interface 606 can include one or more display devices 630, which can be used to display content, such as images, video, and / or user interfaces of output applications described herein. Display devices 630 can be connected to device 600 via a local connection (e.g., a display bus) and / or via a networked connection, and can be any suitable display device. Display devices 630 can include any suitable display device, such as an LCD, LED (including OLED), or plasma display screen, CRT, television, monitor, touch screen, 3-D display screen, or other visual display device. For example, display devices 630 can be a flat display screen provided on a mobile device, multiple display screens provided in goggles or a head-mounted device, or a monitor screen for a computer device.

[0089] I / O interface 606 can interface to other input and output devices. Some examples include a camera 632 capable of capturing images and / or video. In particular, camera 632 can capture color data and depth data for each video frame of a video. Some embodiments can provide a microphone for capturing sound (e.g., as part of a captured image, voice commands, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0090] For ease of explanation, Figure 6A block is shown for each of the processor 602, the memory 604, the I / O interface 606, the operating system 608, and the video call application 610, as well as the application data 612. These blocks can represent one or more processors or processing circuitry, operating systems, memories, I / O interfaces, applications, and / or software modules. In other implementations, the device 600 can not have all of the components shown and / or can have other elements including other types of elements instead of, or in addition to, those shown herein. While some components are described as performing a block and operation as described in some implementations herein, any suitable components of the environment 100, the device 600, a similar system, or a combination of components or any suitable processor or processor-executable associated with such a system can perform the described blocks and operations.

[0091] The methods described herein can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer readable medium (e.g., a storage medium such as a magnetic, optical, electromagnetic, or semiconductor storage medium including semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read only memory (ROM), a flash memory, a hard disk, an optical disk, a solid-state memory drive, etc.). The program instructions can also be contained in, and provided as, an electronic signal, for example, as a software as a service (SaaS) delivered service from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or in a combination of hardware and software. Example hardware can be programmable processors (e.g., field programmable gate arrays (FPGAs), complex programmable logic devices), general purpose processors, graphics processors, application specific integrated circuits (ASICs), etc. One or more methods can be executed as part of or component of an application running on a system, or as an application or software running in conjunction with other applications and operating systems.

[0092] While the specification has been described in relation to certain implementations, these implementations are merely illustrative of the many possible implementations, and are not intended to limit the scope of the specification. Concepts illustrated in the examples are applicable to other examples and implementations.

[0093] In situations in which certain implementations discussed herein can collect or use personal information about users (e.g., user data, information about a user's social network, a user's location and time at a location, a user's biometric information, a user's activity and demographic information), users are provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information is collected about the user. That is, the systems and methods discussed herein collect, store and / or use user personal information only upon receiving explicit authorization from the relevant user. For example, a user is provided with control over whether programs or features collect user information about that particular user or other users relevant to the program or feature. Each user for which personal information is collected is presented with one or more options to allow control over the information collection relevant to that user, thereby providing permission or authorization regarding the collection of information and regarding what parts of the information to collect. For example, a user can be provided with one or more of these control options over a communications network. In addition, certain data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. As one example, a user's identity can be treated so that no personally identifiable information can be determined. As another example, a user's geographic location can be generalized to a larger region so that a specific location of a user cannot be determined.

[0094] Note that the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks as is known to those skilled in the art. Any suitable programming language and programming techniques can be used to implement the routines of particular implementations. Different programming techniques can be employed such as procedural or object-oriented. The routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations can be presented in a particular order, this order can be changed in different particular implementations. In some embodiments, the order of steps or operations presented can be executed concurrently.

Claims

1. A computer-implemented method for relighting a video, characterized in that, The method comprises: receiving frames of a video, wherein the frames comprise depth data and color data for a plurality of pixels; segmenting the frames based on the depth data to classify each pixel as a foreground pixel or a background pixel, wherein a depth value for the background pixels is set to a fixed value; smoothing the depth values for the foreground pixels; computing surface normals based on the smoothed depth values for the plurality of pixels; creating a three-dimensional (3D) mesh based on the depth values and the surface normals; and adding virtual lights to render relit frames based on the 3D mesh and the color data, wherein adding virtual lights comprises computing adjustments to color values for the plurality of pixels based on a type of the virtual light or a position of a light source of the virtual light.

2. The computer-implemented method of claim 1, wherein, The virtual light is a ring light, and adding virtual lights comprises relighting the frames so that objects in the foreground are lit up, while objects in the background appear dark.

3. The computer-implemented method of claim 1, wherein, The video is captured in a video call application at a sending device in a video call, and the method further comprises sending the relit frames from the sending device to a receiving device in the video call over a network.

4. The computer-implemented method of claim 1, wherein, The video is captured in a video call application at a sending device in a video call, and receiving frames of a video comprises receiving frames of the video at a receiving device of the video call.

5. The computer-implemented method of claim 4, wherein, Further comprising displaying the relit frames on a display of the receiving device.

6. The computer-implemented method of claim 1, wherein, The surface normals define a direction of reflection.

7. The computer-implemented method of claim 1, wherein, The method further comprises setting the depth value for each background pixel to a fixed depth value, the fixed depth value selected to be away from the depth values of the foreground pixels.

8. A non-transitory computer-readable medium having instructions stored thereon, the method comprising: The instructions, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising: receiving frames of a video, wherein the frames comprise depth data and color data for a plurality of pixels; segmenting the frames based on the depth data to classify each pixel as a foreground pixel or a background pixel, wherein a depth value for the background pixels is set to a fixed value; smoothing the depth values for the foreground pixels; computing surface normals based on the smoothed depth values for the plurality of pixels; creating a three-dimensional (3D) mesh based on the depth values and the surface normals; and adding virtual lights to render relit frames based on the 3D mesh and the color data, wherein adding virtual lights comprises computing adjustments to color values for the plurality of pixels based on a type of the virtual light or a position of a light source of the virtual light.

9. The non-transitory computer-readable medium of claim 8, wherein, The virtual light is a ring light, and adding virtual lights comprises relighting the frames so that objects in the foreground are lit up, while objects in the background appear dark.

10. The non-transitory computer-readable medium of claim 8, wherein, The video is captured in a video call application at a sending device in a video call, and the operations further comprise sending the relit frames from the sending device to a receiving device in the video call over a network.

11. The non-transitory computer-readable medium of claim 8, wherein, The video is captured in a video call application at a sending device in a video call, and the operations of receiving frames of a video comprise receiving frames of the video at a receiving device of the video call.

12. The non-transitory computer-readable medium of claim 8, wherein, The operations further include setting a depth value of each background pixel to a fixed depth value.

13. A computing device for relighting a video, the computing device comprising: The computing device includes: one or more hardware processors; and memory coupled to the one or more hardware processors, the memory having instructions thereon that when executed by the one or more hardware processors perform operations comprising: receiving a frame of a video, wherein the frame includes depth data and color data for a plurality of pixels; segmenting the frame based on the depth data to classify each pixel as a foreground pixel or a background pixel, wherein a depth value of the background pixel is set to a fixed value; smoothing the depth values of the foreground pixels; computing surface normals based on the smoothed depth values of the plurality of pixels; creating a three-dimensional (3D) mesh based on the depth values and the surface normals; and rendering a relit frame based on the 3D mesh and the color data by adding virtual lights, wherein adding virtual lights includes computing adjustments to color values of the plurality of pixels based on a type of the virtual light or a position of a light source of the virtual light.

14. The computing device of claim 13, wherein, The virtual light is a ring light, and adding virtual lights includes relighting the frame such that objects in the foreground are lit up, while objects in the background appear dark.

15. The computing device of claim 13, wherein, The computing device is a sending device participating in a video call, the video is captured by the sending device in the video call, and the operations further include sending the relit frame from the sending device to a receiving device in the video call over a network.

16. The computing device of claim 13, wherein, The computing device is a receiving device participating in a video call, the video is captured by a video call application at a sending device in the video call different from the receiving device, and receiving the frame of the video includes receiving the frame of the video from the sending device over a network.

17. The computing device of claim 16, wherein, The operations further include displaying the relit frame on a display screen of the receiving device.

Images