Method implemented during video call, user terminal and readable storage medium
By using a face tracking algorithm during a video call to map the drawing input to the face position in the video frame during a video call, the problem of difficulty for users to accurately draw is solved, and the three-dimensional movement of the drawn image is realized and the user experience is improved.
Patent Information
- Application Number
- CN202110009100.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2015-01-13
- Filing Date
- 2015-10-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2035-10-29
AI Technical Summary
During a video call, it is difficult for the user to accurately draw an image onto a video frame, especially when the faces in the video frame are moving, resulting in the drawing being out of control.
By performing a face tracking algorithm on the user terminal, the face position in the video frame is detected and the drawing input is mapped to the face position, ensuring that the drawn image moves along with the face in three-dimensional manner.
Accurate drawing during video calls is achieved, and users can accurately draw images on faces in video frames, improving user experience.
Smart Images

Figure CN112671994B_ABST
Abstract
Description
[0001] This application is a divisional application of the Chinese patent application with application number 201580058866.0. Background Art
[0002] Packet-based communication systems allow users of devices such as personal computers to communicate across computer networks using packet protocols such as the Internet Protocol (IP). Packet-based communication systems can be used for various types of communication events. Communication events that can be established include voice calls, video calls, instant messaging, voice mail, file transfers, etc. These systems are beneficial to users because they often have significantly lower costs compared to fixed lines or mobile networks. Long-distance communications are especially such a situation. In order to use a packet-based system, a user installs and executes client software on his or her device. The client software provides packet-based connections and other functions such as registration and authentication.
[0003] The communication system allows users of the device to communicate across a computer network such as the Internet. Communication events that can be established include voice calls, video calls, instant messaging, voice mail, file transfers, etc. With video calls, in addition to voice information, the callers can also view the video image of the other party. This allows for more natural communication between the parties, as facial expressions are also transmitted, making the video call more similar to a face-to-face conversation.
[0004] As is well known, the communication client application provides functionality that enables a user to modify a captured video at a user device by adding a two-dimensional clip art effect. For example, a user may choose to add a beard effect, the application detects the position of the user's face in the captured frame, and overlays the beard effect at the position in the frame according to the position of the user's face in the captured frame. Thus, the beard moves in a two-dimensional manner as the user's face changes position in the captured video. Summary of the invention
[0005] Embodiments of the present disclosure relate to drawing on a displayed video frame during a video call. A drawing drawn by a user is mapped to multiple points on the user's face in the video frame so that the drawing moves with the user's face in a three-dimensional manner.
[0006] The inventors have recognized that because the drawing is input by the user's freehand, it is difficult to accurately draw onto the displayed video frame, especially when the user's face in the video frame is moving. This problem is exacerbated when the user draws onto another user's face, i.e., cannot control the amount of movement of the face in the displayed video frame.
[0007] According to a first aspect of the present disclosure, there is provided a method implemented at a user terminal during a video call with at least one other user terminal via a communication network, the method comprising: displaying a user interface on a display of the user terminal for displaying a received video frame; detecting, when the received video frame is displayed in the user interface, a selection of a selectable button displayed in the user interface by a user using an input device of the user terminal; in response to the detection, disabling the display of video frames received after the received video frame; determining the position of the user's face in the received video frame by executing a face tracking algorithm on a processor of the user terminal; and when the selectable button is in a selected state. receiving a plurality of drawing inputs, each drawing input defining image data to be applied at face locations displayed on the face; modifying the displayed video frame in accordance with the received plurality of drawing inputs by applying the image data to the face locations on the face; detecting a condition and in response to detecting the condition, for each video frame received after the condition is detected, the method further comprises: determining the location of the face in the received video frame by executing the face tracker algorithm to locate the face locations in the video frame; modifying the video frame by applying the image data to each of the locations on the face; and displaying the modified video frame in the user interface.
[0008] According to another aspect of the present disclosure, a user terminal is provided, comprising: a display; an input device; a network interface, which is configured to send and receive video data between the user terminal and the communication network during a video call between the user terminal and at least one other user terminal; a processor, which is configured to run an application for performing the following operations during the video call: displaying a user interface on the display for displaying a received video frame; detecting a selection of a selectable button displayed in the user interface by the user using the input device when the received video frame is displayed on the user interface; disabling display of video frames received after the received video frame in response to the detection; determining the face of the user in the received video frame by executing a face tracker algorithm a position of a face on the face; receiving a plurality of drawing inputs when the selectable button is in a selected state, each drawing input defining image data to be applied at a face position on the face; modifying a displayed video frame in accordance with the received plurality of drawing inputs by applying the image data to each of the face positions on the face; and detecting a condition and in response to detecting the condition, for each video frame received after detecting the condition: determining the position of the face in the received video frame by executing the face tracker algorithm, thereby determining the location of the face position in the video frame; modifying the video frame by applying the image data to each of the positions on the face; and displaying the modified video frame in the user interface.
[0009] According to another aspect of the present disclosure, a computer program product is provided, which is implemented on a computer-readable medium and is configured to perform any of the methods described below in this document when executed on a processor of the user terminal during a video call between the user terminal and at least one other user terminal.
[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to better understand the present disclosure and to show how it may be implemented, reference will now be made by way of example to the following drawings, in which:
[0012] Figure 1 A communication system is shown;
[0013] Figure 2 A schematic diagram of a user terminal is shown;
[0014] Figure 3 shows the communication between a frame modification application executing on a user terminal and a communication client application;
[0015] Figure 4 shows a user interface provided by a frame modification application;
[0016] Figure 5a and Figure 5b A flow chart for a process of modifying video data is shown;
[0017] Figure 6 A three-dimensional triangulated mesh of a plurality of points identifying a face in a captured video frame is shown;
[0018] Figure 7 A two-bit texture map is shown; and
[0019] Figure 8 An application is shown drawing to a received video frame by a frame modification application. DETAILED DESCRIPTION
[0020] The embodiments of the present disclosure have been described by way of example only.
[0021] Figure 1 A communication system 100 is shown, which includes a first user 104 (User A) associated with a user terminal 102, a second user 110 (User B) associated with a user terminal 108. The user terminals 102 and 108 can communicate in the communication system 100 via a communication network 106, thereby allowing the users 104 and 110 to communicate with each other via the communication network 106. The communication network 106 can be any suitable network having the capability of providing a communication channel between the user terminal 102 and the second user terminal 108. For example, the communication network 106 can be the Internet or another type of network such as a high data rate mobile network, for example a third generation ("3G") mobile network.
[0022] It should be noted that, in an alternative embodiment, the user terminal may be connected to the Figure 1 For example, if the user terminal 102 is a mobile device, it may be connected to the communication network 106 via a cellular mobile network such as a GSM or UMTS network (not shown in FIG. Figure 1 ) and connected to the communication network 106.
[0023] The user terminal 102 may be, for example, a mobile phone, a personal digital assistant ("PDA"), a personal computer ("PC") (including, for example, Windows TM , Mac OS TM and LinuxTM The user terminal 102 may be a mobile phone, a laptop, a PC, a tablet computer, a gaming device, or other embedded device that can be connected to the communication network 106. The user terminal 102 is configured to receive information from the user A 104 or output information to the user A 104.
[0024] The user terminal 102 executes a communication client application 112 provided by a software provider associated with the communication system 100. The communication client application 112 is a software program that executes on a local processor in the user terminal 102. The communication client application 112 performs the required processing at the user terminal 102 to enable the user terminal 102 to send and receive data through the communication system 100. The communication client application 112 executed at the user terminal 102 can be authenticated to communicate through the communication system by the presentation of a digital certificate (e.g., proving that the user 104 is an authentic subscriber to the communication system - described in more detail in WO2005 / 009019).
[0025] The user terminal 108 may correspond to the user terminal 102. The user terminal 108 executes a communication client application 114 on a local processor that corresponds to the communication client application 112 executed at the user terminal 102. The communication client application 114 at the user terminal 108 performs the required processing to allow the user 110 to communicate over the network 106. Figure 1 The user terminal shown in is the end point in the communication system. For the sake of clarity, Figure 1 Only two users (104 and 110) and two user terminals (102 and 108) are shown, but more users and user devices may be included in the communication system 100 and may communicate through the communication system 100 using corresponding communication clients executed on the corresponding user devices, as is known in the art.
[0026] Figure 2A detailed view of a user terminal 102 on which a communication client application for communicating through the communication system 100 is executed is shown. The user terminal 102 includes a central processing unit ("CPU") 202 connected to a display 204 such as a screen or touch screen, an input device such as a keypad 206, a camera 208, and a touch screen 204. In the embodiments described below, the camera 208 can be a conventional web camera integrated into the user terminal 102 or coupled to the user device via a wired or wireless connection. Alternatively, the camera 208 can be a depth perception camera, such as a time of flight or structured light camera. An output audio device 210 (e.g., a speaker) and an input audio device 212 (e.g., a microphone) are connected to the CPU 202. The display 204, keypad 206, camera 208, output audio device 210, and input audio device 212 can be integrated into a user terminal 102 such as in FIG. Figure 2 . In an alternative user terminal, one or more of the display 204, keypad 206, camera 208, output audio device 210, and input audio device 212 may not be integrated into the user terminal 102 and may be connected to the CPU 202 via a corresponding interface. An example of such an interface is a USB interface. The CPU 202 is connected to a network interface 224, such as a modem for communicating with the communication network 106. The network interface 224 may be integrated into a Figure 2 . In an alternative user terminal, the network interface 224 is not integrated into the user terminal 102. The user terminal 102 also includes a memory 226 for storing data, as is known in the art. The memory 226 may be a permanent memory, such as a ROM. The memory 226 may alternatively be a temporary memory, such as a RAM.
[0027] The user terminal 102 is installed with a communication client application 112 , wherein the communication client application 112 is stored in the memory 226 and is arranged to be executed on the CPU 202 . Figure 2 Also shown is an operating system ("OS") 214 executing on CPU 202. Running on top of OS 214 is a software stack 216 of the communication client application 112 mentioned above. The software stack shows an I / O layer 218, a client engine layer 220, and a client user interface layer ("UI") 222. Each layer is responsible for a specific function. Because each layer typically communicates with two other layers, the layers are considered to be arranged in a stack, as shown in FIG. Figure 2. The operating system 214 manages the hardware resources of the computer and processes the data coming and going from the communication network 106 via the network interface 224. The I / O layer 218 includes an audio and / or video codec that receives the incoming encoded streams and decodes them to output appropriately to the speaker 210 and / or the display 204, and it also receives unencoded audio and / or video data from the microphone 212 and / or the camera 208 and encodes them to transmit to other end user terminals of the communication system 100 as a stream. The client engine layer 220 handles the connection management functions of the VoIP system as discussed above, such as establishing a call or other connection through server-based or P2P address lookup and authentication. The client engine can also be responsible for other auxiliary functions not discussed in this article. The client engine 220 also communicates with the client user interface layer 222. The client engine 220 may also be arranged to control the client user interface layer 222 to present information to a user of the user terminal 102 via a user interface of the communication client application 112 displayed on the display 204 , and to receive information from a user of the user terminal 102 via the user interface.
[0028] Also running on OS 214 is a frame modification application 230. Embodiments are described below with reference to frame modification application 230 and communication client application 112 as separate applications, however, the functionality of frame modification application 230 described in more detail below may be incorporated into communication client application 112.
[0029] Figure 3 It is shown how the frame modification application 230 interacts with the communication client application 112 .
[0030] As in Figure 3 As shown in FIG. 1 , the frame modification application 230 is arranged to receive the captured video and the drawing instructions. The frame modification application 230 is configured to modify the captured video frame according to the drawing instructions and provide the modified video frame to the communication client application 112 .
[0031] In the embodiment described below, the frame modification application 230 receives captured video frames from the camera 208, and the frame modification application 230 receives drawing instructions from the user A 104. The video frames that have been modified in response to the drawing instructions received from the user A 104 are supplied from the frame modification application 230 to the communication client application 112 for transmission to the user terminal 108 over the network 106.
[0032] The frame modification application 230 presents information to the user A 104 via a user interface displayed on the display 204 .
[0033] Figure 4The frame modification application 230 is shown displaying content in the available area 400 of the display 204. Figure 4 The available area of the display is the entire display, but the available area of the display (i.e., the area available to the frame modification application 230) may constitute only a portion of the display, for example where the remainder of the display is used by the operating system and / or (multiple) other applications executing thereon to display other content.
[0034] As in Figure 4 As shown in FIG. 1 , the user interface provided by the frame modification application 230 displays the video data captured by the camera 208 (in Figure 4 , user A 104 is looking at camera 208).
[0035] As in Figure 4 As shown in , the frame modification application 230 can overlay at least one selectable drawing tool button 402 on the displayed captured video for selection by the user A 104 using a suitable input device (e.g., a mouse, a stylus that interacts with a touch screen, or directly touching the touch screen). The selectable drawing tool buttons 402 may include a pen drawing tool button 404a, a lipstick drawing tool button 404b, and a brush drawing tool button 404c. In response to detecting a selection of one of the selectable drawing tool buttons 402, the frame modification application 230 is configured to overlay a color palette 406 on the displayed captured video. The color palette 406 overlaid on the displayed captured video is associated with the selected drawing tool button. Each color palette 406 includes at least one selectable color for the user A 104 to select using a suitable input device. Figure 4 A color palette 406 is shown including a first color 408a, a second color 408b, a third color 408c, and a fourth color 408d.
[0036] The selection of the drawing tool button 404 and color 408 enables the user A 104 to draw on the captured video data displayed in the user interface provided by the frame modification application 230. The frame modification application 230 includes a three-dimensional face tracker function that identifies multiple points of the user's face in the captured video data and tracks these points as the user moves. According to an embodiment described in more detail below, drawings drawn by the user A 104 onto the face of user A in the displayed captured video are mapped to the points of the user's face identified by the face tracker so that the drawings move with the user's face in three dimensions.
[0037] The frame modification application 230 may overlay a selectable frame rendering adjustment button 412 on the displayed captured video for selection by the user 104A using a suitable input device. In response to detecting the selection of the selectable frame rendering adjustment button 412, the frame modification application 230 adjusts the manner in which the received video frame is rendered in the user interface provided by the frame modification application 230.
[0038] Memory 226 includes a data buffer 228 for storing received video frames prior to being rendered in a user interface provided by frame modification application 230 .
[0039] In one embodiment, if the frame modification application 230 detects a selection of the selectable frame rendering adjustment button 412 (the selectable frame rendering adjustment button 412 is then in a selected state) while the received video frame is being displayed in the user interface provided by the frame modification application 230, the frame modification application 230 prevents any video frames in the data buffer 228 from being read from the data buffer 228 and rendered in the user interface provided by the frame modification application 230 until the selectable frame rendering adjustment button 412 is again selected. Thus, the selectable frame rendering adjustment button is used to pause the incoming real-time video data.
[0040] In response to detecting re-selection of the selectable frame rendering adjustment button 412 by user A 104 (the selectable frame rendering adjustment button 412 is then in an unselected state), the frame modification application 230 renders the first video frame received in the data buffer 228 after the selectable frame rendering adjustment button 412 is selected again, and continues to read subsequently received video frames from the data buffer 228 for rendering in the user interface provided by the frame modification application 230.
[0041] It should be appreciated that during the period between the first selection of the selectable frame rendering adjustment button 412 and the second selection of the selectable frame rendering adjustment button 412 , video frames are still being received by the frame modification application 230 and stored in the data buffer 228 .
[0042] In a second embodiment, if the frame modification application 230 detects a selection of the selectable frame rendering adjustment button 412 (the selectable frame rendering adjustment button 412 is then in a selected state) while the received video frame is being displayed in the user interface provided by the frame modification application 230, the frame modification application 230 prevents any video frames in the data buffer 228 from being read from the data buffer 228 and rendered in the user interface provided by the frame modification application 230 until a predetermined period of time has passed. Thus, the selectable frame rendering adjustment button is used to slow down the incoming real-time video data.
[0043] In response to detecting that a predetermined time period has passed since the selectable frame rendering adjustment button 412 was selected (the selectable frame rendering adjustment button 412 is then in an unselected state), the frame modification application 230 renders the first video frame received in the data buffer 228 from the time the selectable frame rendering adjustment button 412 was selected to the expiration of the predetermined time period, and continues to read subsequently received video frames from the data buffer 228 for rendering in the user interface provided by the frame modification application 230.
[0044] It should be appreciated that during the period between selection of the selectable frame rendering adjustment button 412 and expiration of the predetermined time period, video frames are still received by the frame modification application 230 and stored in the data buffer 228 .
[0045] Will refer to Figure 5a and Figure 5b The operation of frame modification application 230 in response to detecting selection of frame rendering adjustment button 412 is described in greater detail.
[0046] First reference Figure 5a , which shows a flow chart 500 of a method performed by the frame modification application 230 when a video call is ongoing between the first user terminal 102 and the second user terminal 108.
[0047] The process starts at step S502, whereby the frame modification application 230 displays a received video frame in a user interface provided by the frame modification application 230. In an exemplary embodiment, the frame modification application 230 receives a captured video frame from the camera 208 of the first user terminal 102.
[0048] At step S504 , the frame modification application 230 detects selection of the selectable frame rendering adjustment button 412 by the user A 104 .
[0049] At step 506, the frame modification application 230 detects the position of the user's face in the received video frame. The frame modification application 230 may perform step S506 by executing a face tracker algorithm. The face tracker algorithm may be based on a known active appearance model (AAM) method. The face tracker algorithm tracks facial feature points in two dimensions and fits a three-dimensional triangular mesh to the tracked facial feature points. The face tracker algorithm determines the three-dimensional pixel coordinates (x, y, z) of each vertex of the three-dimensional triangular mesh fitted to the user's face. The three-dimensional triangular mesh includes a plurality of vertices defining a plurality of triangles. In Figure 6An example three-dimensional triangular mesh fitted to a user's face (captured in a video frame) is shown in . The coordinates of each vertex of the three-dimensional triangular mesh define the position of the vertex in the frame, which can be defined in terms of a number of pixels. As described above, camera 208 can be a conventional camera or a depth-aware camera. When camera 208 is a depth-aware camera, the depth information output by the depth-aware camera is used as an additional input to the face tracker algorithm, but the output of the face tracker algorithm is the same regardless of whether camera 208 is a conventional camera or a depth-aware camera.
[0050] The frame modification application 230 is configured to compute a two-dimensional representation of the three-dimensional triangular mesh. This is referred to herein as a two-dimensional texture map 700, which is Figure 7 is shown in .
[0051] Various possible techniques for calculating a two-dimensional representation of a three-dimensional triangular mesh are well known to those skilled in the art and are therefore not discussed in detail herein. An exemplary method includes (i) projecting the three-dimensional triangular mesh onto an ellipsoid; and (ii) projecting the ellipsoid into two dimensions (e.g., using a Gall-Peters equal area projection).
[0052] The two-dimensional texture map 700 is overlaid on a pixel grid that is a plurality of pixels wide and a plurality of pixels high. The pixel width and pixel height define the resolution of the pixel grid. The resolution of the pixel grid may be, for example, 512×512. UV coordinates (expressed as pixel coordinates of the pixel grid) are used to describe the surface of the two-dimensional texture map 700. Each u, v coordinate in the two-dimensional texture map 700 has a corresponding x, y pixel coordinate in the video frame. The pixel resolution of the pixel grid is quite arbitrary. If the pixel resolution is too small, texture mapping (described below with reference to step S522) will introduce smoothness / blurring in the modified image. If the pixel resolution is too large, texture mapping and effect rendering will be unnecessarily slow. Therefore, it will be apparent to those skilled in the art that it is appropriate to select a pixel resolution of the pixel grid that corresponds approximately to the maximum size of the face in the received image. The u, v coordinates of the face grid are constant and invariant for the face position, expression, or posture. Before the process 500 begins, the u, v coordinates are calculated once in an offline pre-calculation step.
[0053] At step S508, the frame modification application 230 performs a texture mapping process, thereby copying the texture of each triangle in the three-dimensional triangular mesh calculated at step S506 to the corresponding triangle in the two-dimensional texture map 700. References to the "texture" of a triangle in a three-dimensional triangular mesh are used herein to refer to the color value of each pixel in a particular triangle in the three-dimensional triangular mesh. Pixel color values can be defined according to the RGB color model.
[0054] At step S510, the frame modification application 230 detects that drawing instructions (otherwise referred to herein as drawing input) have been received from the user A 104, the drawing instructions defining modifications to be applied to the video frame received at step S502. The frame modification application 230 receives the drawing instructions in response to the user A 104 drawing in an area provided by the frame modification application 230, wherein the received video frame is displayed as having the drawing tool button 404 and the color 408 selected. The drawing instructions define image data (e.g., a drawing effect) to be applied at a location on the user's face in the received video frame. At step 510, the frame modification application 230 determines a set of pixel coordinates (x, y) drawn by the user A 104.
[0055] At step S512, the frame modification application 230 determines the u,v coordinates corresponding to the coordinates (x,y) of the drawing instruction detected at step S510. At step S514, the frame modification application 230 stores the drawing command in the memory 226 in the form of the u,v coordinates (determined in step S510) and the associated effect information. The effect information associated with the u,v coordinates includes an indication of the drawing modification to be applied to the two-dimensional texture map 700 at the u,v coordinates based on the drawing tool button 404 and the color 408 selected by the user A 104. In some embodiments, the effect information does not include an absolute color value. Instead, the effect information includes a color modification factor that depends on the selected color 408. The color modification factor is defined in terms of a multiplication term and an addition term. Therefore, a pixel having an original color value (R,G,B) is modified to have a new color value (R',G',B') according to the following manner:
[0056] R'=α R R+β R
[0057] G'=α G G+β G
[0058] B'=α B B+β B
[0059] Thus, the three multiplication terms α R , α G , and α B And three additive terms β R , β G , and β BThe effect to be applied is described. The values of the three multiplication terms and the three addition terms are preconfigured and vary according to the selected color 408. By applying the color modification factor in this manner, the pixel is modified in a manner consistent with the unmodified pixel in the video frame. For example, if the light level in the room where the camera 208 is capturing the video changes, the color modification factor ensures that the pixel is modified based on its original color value. The embodiment also extends to the color 408 that defines the absolute color value. The stored effect information also includes information about the width of the drawing stroke, which gives an indication of the number of pixels around the pixel at the u,v coordinate to be modified. The width of the drawing stroke depends on the selected drawing tool button 404. For example, the width of the drawing stroke formed when the brush drawing tool button 404c is selected can be wider than the drawing stroke formed when the lipstick drawing tool button 404b is selected. Similarly, the width of the drawing stroke formed when the lipstick drawing tool button 404b is selected can be wider than the drawing stroke formed when the pen drawing tool button 404a is selected.
[0060] At step S516, the frame modification application 230 modifies the two-dimensional texture map 700 according to the drawing command stored at step S514. That is, for the u,v coordinates determined at step S512, the frame modification application 230 modifies the u,v coordinates of the two-dimensional texture map 700 according to its associated effect information.
[0061] At step 518, the frame modification application 230 performs a texture mapping process that is the inverse of the texture mapping performed at step S508. At step S518, the texture of each triangle in the two-dimensional texture map 700 is copied to the corresponding triangle of the three-dimensional triangle mesh determined at step S506. References to the "texture" of a triangle in the two-dimensional texture map 700 are used herein to refer to the color value of each pixel in a particular triangle in the two-dimensional texture map 700.
[0062] In response to step S518 being performed by frame modification application 230 , modifications to the video frame displayed at step S502 are rendered in a user interface provided by frame modification application 230 .
[0063] Step S518 may be implemented using a z-buffer. If a portion of a triangle in a three-dimensional triangular mesh must be rendered in the same x,y pixel in the video frame as a portion of another triangle of the three-dimensional triangular mesh, the depths (z coordinates) of these portions are compared, and the x,y pixel of the video frame obtains its color from the triangle closest to the viewer (e.g., user A) of the display 204 of the user terminal 102. Using a z-buffer to perform step S518 ensures that the x,y pixels belonging to more than one triangle in the three-dimensional triangular mesh obtain their colors from the triangle closest to the viewer (e.g., user A), i.e., triangles that are farther away are occluded by the closest one.
[0064] An alternative to the z-buffering technique is to determine the depth (z coordinate) value at the midpoint of each triangle in the three-dimensional triangle mesh, and sort all triangles in the three-dimensional triangle mesh into an ordered list according to their midpoint z value. This will also correctly handle occlusion by starting rendering with the triangle in the three-dimensional triangle mesh farthest from the viewer of the display 204, and completing the ordered list to render the triangle in the three-dimensional triangle mesh closest to the viewer of the display 204.
[0065] At step 520, the frame modification application 230 supplies the modified video frame to the communication client application 112 for transmission to the second user terminal 108 over the network 106 via the network interface 224. Those skilled in the art will appreciate that the modified video frame may be processed (e.g., encoded and packetized) by the communication client application 112 before transmission. It should be understood that in embodiments by which the functionality of the frame modification application 230 may be incorporated into the communication client application 112, at step S518, the communication renders the modified video frame in a user interface provided by the communication client application 112, and step S520 is not performed.
[0066] At step S522, the frame modification application 230 determines whether a condition is detected. In a first embodiment, the condition corresponds to the selectable frame rendering adjustment button 412 being selected again (after being selected at step S504), that is, the selectable frame rendering adjustment button 412 is in an unselected state. In a second embodiment, the condition corresponds to a predetermined period of time having passed since the selectable frame rendering adjustment button 412 was selected at step S504.
[0067] If the frame modification application 230 determines at step S522 that the condition is not detected, the process 500 proceeds back to step S510. While the process 500 loops between steps S510-S522, user A is able to draw on the user's face in the single video frame displayed at step S502. This enables user A 104 to accurately draw on the face in the video frame as compared to when the received video frames are continuously read from the data buffer for rendering in the user interface provided by the frame modification application 230 while the user is moving.
[0068] If frame modification application 230 determines at step 522 that the condition has been detected, process 500 proceeds to Figure 5b Step S524 shown in .
[0069] At step 524, the frame modification application 230 receives the video frame from the data buffer.
[0070] In a first embodiment, the video frame received at step S524 corresponds to a first video frame received in the data buffer after the selectable frame rendering adjustment button 412 is selected to be in an unselected state. In a second embodiment, the video frame received at step S524 corresponds to a first video frame received in the data buffer after a predetermined time period has expired since the selectable frame rendering adjustment button 412 is detected to be selected at S504.
[0071] At step S526, the frame modification application 230 detects the position of the user's face in the video frame received at step S522. This process has been described with reference to step S506 and will not be repeated for the sake of clarity.
[0072] At step S528 , the frame modification application 230 performs texture mapping, thereby copying the texture of each triangle in the three-dimensional triangle mesh calculated at step S526 to the corresponding triangle in the two-dimensional texture map 700 .
[0073] At step S530, all drawing commands that have been stored in memory 226 in response to step S514 being executed each time process 500 loops through steps S510-S522 are received by frame modification application 230 and used to modify two-dimensional texture map 700. That is, for each of the received u,v coordinates, frame modification application 230 modifies the u,v coordinates of two-dimensional texture map 700 according to the effect information associated with the u,v coordinates.
[0074] At step S532, the frame modification application 230 performs a texture mapping process which is the reverse of the texture mapping process performed at step 528. This process has already been described with reference to step S518 and will not be repeated for the sake of clarity.
[0075] In response to step S532 being performed by frame modification application 230 , the video frame received at step S524 is displayed in a modified form in a user interface provided by frame modification application 230 .
[0076] At step S534, the frame modification application 230 supplies the modified video frame to the communication client application 112 for transmission to the second user terminal 108 over the network 106 via the network interface 224. It should be appreciated that in embodiments where the functionality of the frame modification application 230 is incorporated into the communication client application 112, at step S532, the communication renders the modified video frame in a user interface provided by the communication client application 112 and step S534 is not performed.
[0077] Process 500 then proceeds to step S536, where frame modification application 230 determines whether selectable frame rendering adjustment button 412 is selected. If at step 536, frame modification application 230 determines that selectable frame rendering adjustment button 412 has been selected, process 500 returns to S524, where frame modification application 230 receives the next video frame from the data buffer and repeats the loop of steps S524-S536.
[0078] That is, all drawing performed on the video frame displayed at step S502 when the selectable frame rendering adjustment button 412 is in the selected state is applied to all video frames received in the data buffer when the selectable frame rendering adjustment button 412 is in the unselected state. The drawing drawn by user A 104 onto the face of user A in the video is mapped onto the points of the user's face identified by the face tracker so that the drawing moves with the user's face in a three-dimensional manner.
[0079] If at step S536, the frame modification application 230 determines that the selectable frame rendering adjustment button 412 has been selected, the process 500 proceeds to step S506.
[0080] In the embodiment described above, any modified frames displayed in the user interface provided by the frame modification application 230 on the display 204 are sent by the communication client application 112 to the second user terminal 108. Therefore, user B 110 sees what user A 104 sees. It should be understood that during the time period between the selection of the selectable frame rendering adjustment button 412 at step S504 and the detection of the condition at step S522, the video frames received by the frame modification application 230 and stored in the data buffer 228 are not sent to the second user terminal 108 and may be discarded from the data buffer 228. During this time period, the received video frames displayed at step S502 and in modified form are sent to the second user terminal 108.
[0081] In an alternative embodiment, the video frame received by the frame modification application 230 and stored in the data buffer 228 is sent to the second user terminal 108 during the time period between the selection of the selectable frame rendering adjustment button 412 at step S504 and the detection of the condition at step S522. That is, when the selectable frame rendering adjustment button 412 is in the selected state, the captured real-time video is sent to the second user terminal. In addition, drawing modification information is sent to the second user terminal each time it is determined. The drawing modification information is in the form of the pixel coordinates (x, y) determined at step S510 or the u, v coordinates determined at step S512. This allows the frame modification application executed on the second user terminal to modify the received video frame according to the received drawing modification information.
[0082] In another alternative embodiment, during the time period between the selectable frame rendering adjustment button 412 being selected at step S504 and the condition being detected at step S522, the video frame received by the frame modification application 230 and stored in the data buffer 228 is sent to the second user terminal 108. That is, when the selectable frame rendering adjustment button 412 is in the selected state, the captured real-time video is sent to the second user terminal. In this other alternative embodiment, once the condition is detected at step S522, the drawing modification information is sent to the second user terminal. The drawing modification information is in the form of a plurality of pixel coordinates (x, y) determined each time step S510 is performed or a plurality of u, v coordinates determined each time step S512 is performed. This allows the frame modification application executed on the second user terminal to modify the received video frame according to the received drawing modification information after receiving the drawing modification information.
[0083] The two-dimensional texture map 700 must be re-rendered for each video frame (at steps S508 and S526) because it must contain the latest updated texture from the captured image (in the video frame). On the other hand, the drawing effects can be pre-rendered into separate UV maps (referred to as "effect maps" in this article). The "effect map" remains unchanged until the user adds or deletes drawing strokes. The effect map is blended onto the two-dimensional texture map 700 to create a blended two-dimensional representation of the three-dimensional triangle mesh. By setting the resolution of the effect map equal to the resolution of the two-dimensional texture map 700, the blending can be completed with a simple pixel-by-pixel loop without the need for knowledge of the three-dimensional triangle mesh or occlusion. Each of the u,v coordinates of the blended two-dimensional representation of the three-dimensional triangle mesh obtains the texture from the corresponding u,v coordinates of the two-dimensional texture map 700, unless the effect map has been modified at a specific u,v coordinate, in which case the u,v coordinates of the blended two-dimensional representation of the three-dimensional triangle mesh obtain the texture from the effect map.
[0084] Figure 8 A user interface provided by the frame modification application 230 is shown after user A has drawn brush strokes 801a and 801b by moving a cursor 802 while the selectable frame rendering adjustment button 412 is in a selected state.
[0085] When the selectable frame rendering adjustment button 412 is in the unselected state, the drawings 801a and 801b are pasted onto the skin of the face in the captured video data and move with the face in three dimensions. Figure 8 208, but instead the user turns his head 90 degrees to face sideways to the camera 208, only one of the strokes 801a, 801b will be rendered in the user interface provided by the frame modification application 230 (depending on whether the user turns his head clockwise or counterclockwise).
[0086] Drawing is defined by multiple drawing instructions. The u, v coordinates of each drawing are stored in a queue in the memory 226, whereby the u, v coordinates of the most recent drawing are stored at the top of the queue and the u, v coordinates of the oldest drawing are stored at the bottom of the queue.
[0087] As in Figure 4 and 8As shown in , the frame modification application 230 may overlay a selectable undo button 414 on the displayed captured video for selection by the user A 104 using a suitable input device. In response to detecting the selection of the selectable undo button 414, the frame modification application 230 is arranged to remove the u,v coordinates at the top of the queue in the memory 226. Therefore, when step S528 is executed, the most recent drawing drawn by the user A 104 is no longer rendered (subsequent video frames received by the frame modification application 230 are not modified with the most recent drawing).
[0088] As in Figure 4 and 8 As shown in , the frame modification application 230 may overlay a selectable new drawing button 416 on the displayed captured video for selection by the user A 104 using a suitable input device. In response to detecting the selection of the selectable new drawing button 416, the frame modification application 230 is arranged to remove the u,v coordinates of all drawings stored in the queue in the memory 226. Therefore, all drawings drawn by the user A 104 up to that point in time are no longer rendered by the frame modification application 230.
[0089] As in Figure 4 and 8 As shown in , the frame modification application 230 may overlay a selectable save button 418 on the displayed captured video for selection by the user A 104 using a suitable input device. In response to detecting the selection of the selectable save button 418, the frame modification application 230 is arranged to save the drawing file to the memory 226. The saved drawing file includes all u,v coordinates (and their associated effect information) stored in the memory 226 when the save button 418 was selected. Therefore, the drawing file is saved in a format that is independent of the face shape and the resolution and type of the camera 208. This enables the saved drawing file to be shared between users of the communication system 100.
[0090] Although the exemplary embodiments have been described above with reference to a scenario in which the frame modification application 230 receives captured video frames from the camera 208 of the first user terminal 102 and the user A 104 draws on these captured video frames (drawing instructions received from the user A 104), the principles of the present disclosure extend to other scenarios.
[0091] In another scenario, the captured video (in Figure 3 ) is received from the second user terminal 108 via the network 106 (captured by the camera of the second user terminal 108) and displayed in the user interface provided by the frame modification application 230 - the drawing instructions (in Figure 3 ) is received from user A 104 in response to user A 104 drawing on these displayed video frames. Thus, in this embodiment, user A 104 draws on the face of user B displayed on the first user terminal 102, and the drawing is rendered on the display 204 of the first user terminal 102.
[0092] Although embodiments have been described with reference to drawings drawn by a user drawing brush strokes on frames of incoming real-time video, embodiments of the present disclosure extend to drawing instructions that include applying pre-computed three-dimensional graphics objects to frames of incoming real-time video. For example, a user may choose to place a hat graphic on a face, which, according to the embodiments described above, would be associated with vertices of a three-dimensional triangular mesh and would therefore move in three dimensions as the user (as captured in the video) moves.
[0093] The first aspect of the present disclosure has been described above.
[0094] The step of detecting the condition includes determining an additional selection of a selectable button displayed in the user interface.
[0095] The step of detecting the condition may include detecting that a predetermined period of time has elapsed since a selectable button displayed in the user interface was selected.
[0096] Determining the location of a user's face in the received video frame includes identifying facial landmarks in the face in two dimensions and fitting a three-dimensional triangular mesh to the tracked facial landmarks, wherein each drawing input defines image data to be applied at a location on the three-dimensional triangular mesh.
[0097] The step of modifying the displayed video frame includes mapping the texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; and for each drawing input received: determining a position in the two-dimensional representation of the three-dimensional triangular mesh that corresponds to the position on the three-dimensional triangular mesh; applying the image data to the determined position in the two-dimensional representation of the three-dimensional triangular mesh; and mapping the texture of each triangle in the two-dimensional representation of the three-dimensional triangular mesh onto the three-dimensional triangular mesh.
[0098] The step of modifying the displayed video frame includes mapping the texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; and for each drawing input received: determining a position in a second two-dimensional representation of the three-dimensional triangular mesh corresponding to the position on the three-dimensional triangular mesh; applying the image data to the determined position in the second two-dimensional representation of the three-dimensional triangular mesh; blending the second two-dimensional representation of the three-dimensional triangular mesh onto the first two-dimensional representation of the three-dimensional triangular mesh to create a blended two-dimensional representation of the three-dimensional triangular mesh; and mapping the texture of each triangle in the blended two-dimensional representation of the three-dimensional triangular mesh onto the three-dimensional triangular mesh.
[0099] Determining the location of a face of the user in each video frame received after the condition is detected includes identifying facial landmarks in the face in two dimensions and fitting a three-dimensional triangular mesh to the tracked facial landmarks, wherein each drawing input defines image data to be applied at a location on the three-dimensional triangular mesh.
[0100] The step of modifying each video frame received after the condition is detected includes: mapping the texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; and for each drawing input received: determining a position in the two-dimensional representation of the three-dimensional triangular mesh that corresponds to the position on the three-dimensional triangular mesh; applying the image data to the determined position in the two-dimensional representation of the three-dimensional triangular mesh; and mapping the texture of each triangle in the two-dimensional representation of the three-dimensional triangular mesh onto the three-dimensional triangular mesh.
[0101] The step of modifying each video frame received after the condition is detected includes: mapping the texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; and for each drawing input received: determining a position in a second two-dimensional representation of the three-dimensional triangular mesh that corresponds to the position on the three-dimensional triangular mesh; applying the image data to the determined position in the second two-dimensional representation of the three-dimensional triangular mesh; blending the second two-dimensional representation of the three-dimensional triangular mesh onto the first two-dimensional representation of the three-dimensional triangular mesh to create a blended two-dimensional representation of the three-dimensional triangular mesh; and mapping the texture of each triangle in the blended two-dimensional representation of the three-dimensional triangular mesh onto the three-dimensional triangular mesh.
[0102] The method may also include displaying at least one selectable drawing tool button in the user interface.
[0103] The at least one selectable drawing tool button may include one or any combination of the following: a pen drawing tool button, a lipstick drawing tool button, and a brush drawing tool button.
[0104] The method may further comprise, in response to detecting selection of the at least one selectable drawing tool button by a user using an input device of the user terminal, displaying a plurality of selectable colors associated with the selected drawing tool button.
[0105] The image data may be based on a selected drawing tool button and a selected color from the plurality of selectable colors associated with the selected drawing tool button.
[0106] The video frames may be received from a camera of the user terminal.
[0107] The method may further comprise sending each modified video frame to the at least one further user terminal over the communications network.
[0108] The method may further comprise preventing the video frame from being inhibited from being transmitted to the at least one further user terminal over the communications network while displaying the video frame.
[0109] The method further comprises sending the video frames received while display of the video frames is disabled to the at least one further user terminal over the communications network.
[0110] The method further comprises: when each drawing input is received, sending an indication of the drawing input to the at least one further user terminal.Alternatively, the method may further comprise sending indications of a plurality of drawing inputs in response to detecting the condition.
[0111] exist Figure 5a and Figure 5b Steps shown separately in the drawings may or may not be implemented as separate steps, and may or may not be implemented in the order shown.
[0112] Generally speaking, any function described in this article can be implemented using software, firmware, hardware (e.g., fixed logic circuits) or a combination of these implementations. As used in this article, the terms "controller", "function", "component" and "application" generally represent software, firmware, hardware or a combination thereof. In the case of software implementation, a controller, function, component or application represents a program code that performs a specified task when executed on a processor (e.g., one or more CPUs). The program code can be stored in one or more computer-readable memory devices. The feature of the technology described hereinafter is platform-independent, which means that the technology can be implemented on a variety of commercial computing platforms with a variety of processors.
[0113] For example, the user terminal may also include an entity (e.g., software) that causes the hardware of the user terminal to perform operations, such as a processor function block, etc. For example, the user device may include a computer-readable medium that may be configured to store instructions that cause the user terminal, and more particularly cause the operating system and the associated hardware of the user terminal to perform operations. Therefore, the instructions are used to configure the operating system and the associated hardware to perform operations, and in this way cause the conversion of the state of the operating system and cause the associated hardware to perform functions. Instructions may be provided to the user terminal by a variety of different configurations by a computer-readable medium.
[0114] One such configuration of a computer-readable medium is a signal bearing medium, and is thus configured to send instructions (e.g., as a carrier wave) to a computing device, e.g., via a network. A computer-readable medium may also be configured as a computer-readable storage medium, and is thus not a signal bearing medium. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that may use magnetic, optical, and other technologies to store instructions and other data.
[0115] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Instead, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims
1. A system comprising: one or more processors; as well as One or more computer-readable memory devices storing processor-executable instructions that, in response to being executed by the one or more processors, cause the system to perform operations including: one or more facial locations representing a face in the first image; determining a pre-computed graphical object to be applied at a corresponding one of the one or more face locations on the face in the first image; modifying the first image by applying the pre-computed graphical objects to corresponding facial locations on the face in the first image; as well as modifying one or more images captured after the first image by applying the pre-computed graphical object to corresponding facial positions of the face in the one or more images captured after the first image to generate a video having the images modified based on the pre-computed graphical object, The one or more facial positions representing the face in the first image include: identifying facial feature points in the face in two dimensions; and A three-dimensional triangular mesh is fitted to the identified facial feature points, wherein the applying the pre-calculated graphic object comprises applying the pre-calculated graphic object at a corresponding position on the three-dimensional triangular mesh.
2. The system according to claim 1, wherein: The characterizing the one or more facial locations in the first image comprises fitting a three-dimensional triangulated mesh to facial landmarks identified in two dimensions; And wherein, said modifying said first image comprises: Mapping the texture of triangles in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; determining corresponding positions in the two-dimensional representation of the three-dimensional triangular mesh that correspond to corresponding positions on the three-dimensional triangular mesh; and The pre-computed graphics objects are applied to corresponding locations in the two-dimensional representation of the three-dimensional triangular mesh.
3. The system according to claim 1, wherein: The operations also include determining the pre-calculated graphical object in response to a user selection of a drawing tool button.
4. The system according to claim 1, wherein: The determining the pre-computed graphical object further comprises determining the pre-computed graphical object in response to a user selection of a selectable representation of the pre-computed graphical object.
5. The system according to claim 1, wherein: The operations also include determining corresponding facial locations on the face in the one or more images captured after the first image by: identifying facial feature points in the face in two dimensions; and fitting a three-dimensional triangular mesh to the identified facial feature points, wherein the pre-computed graphics object includes data to be applied at corresponding locations on the three-dimensional triangular mesh.
6. The system of claim 1 further comprising a camera, and wherein: The first image and the one or more images captured after the first image are captured via the camera.
7. A method implemented during a video call, comprising: characterizing one or more facial locations of a face in a first image from a camera by using a face tracker algorithm; determining a corresponding face location among the one or more face locations on the face in the first image; receiving a selection of a graphics object; modifying the first image according to the selected graphical object by applying the graphical object to a corresponding facial location on the face in the first image; determining, for one or more images from the camera received after the first image, by using the face tracker algorithm, corresponding locations of corresponding face positions on the face in the one or more images from the camera received after the first image; as well as modifying the one or more images from the camera received after the first image by applying the graphical object to corresponding locations of corresponding facial positions on the face in the one or more images from the camera received after the first image to generate a video having the image modified based on the selected graphical object, The characterizing the one or more facial positions of the face in the first image comprises: identifying facial feature points in the face in a two-dimensional manner; and fitting a three-dimensional triangular mesh to the identified facial feature points, wherein applying the graphic object comprises applying the graphic object at a corresponding position on the three-dimensional triangular mesh.
8. The method according to claim 7, wherein: The characterizing the one or more facial locations in the first image comprises fitting a three-dimensional triangulated mesh to facial landmarks identified in two dimensions; And wherein, said modifying said first image comprises: mapping a texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; and for each corresponding image data associated with the graphics object: determining corresponding positions in the two-dimensional representation of the three-dimensional triangular mesh that correspond to corresponding positions on the three-dimensional triangular mesh; and Respective image data associated with the graphical objects is applied to respective locations in the two-dimensional representation of the three-dimensional triangular mesh.
9. The method according to claim 7, wherein: The receiving of the selection of the graphic object includes: a user selecting a drawing tool button.
10. The method according to claim 7, wherein: The receiving a selection of the graphical object includes: a user selecting a selectable representation of the graphical object.
11. The method according to claim 7, wherein: The receiving a selection of the graphic object includes: receiving a user drawing as the graphic object.
12. The method according to claim 7, wherein: The determining of the corresponding location of the corresponding facial position on the face in the one or more images received from the camera after the first image comprises: fitting a three-dimensional triangulated mesh to the facial landmarks identified in two dimensions, and wherein the modifying of the one or more images received from the camera after the first image comprises: Mapping the texture of each triangle in the three-dimensional triangular mesh onto a two-dimensional representation of the three-dimensional triangular mesh; determining corresponding positions in the two-dimensional representation of the three-dimensional triangular mesh that correspond to corresponding positions on the three-dimensional triangular mesh; applying image data associated with the graphics object to corresponding locations in the two-dimensional representation of the three-dimensional triangular mesh; and A texture of each triangle in the two-dimensional representation of the three-dimensional triangular mesh is mapped onto the three-dimensional triangular mesh.
13. A device comprising: camera; as well as A processing system configured to execute instructions to cause the device to perform operations including: one or more facial locations representing a face in a first image captured via the camera; receiving a selection of a graphical object to be applied at a corresponding one of the one or more face locations on the face in the first image; modifying the first image by applying the graphical object to corresponding facial locations on the face in the first image; as well as modifying one or more images captured by the camera after the first image by applying image data associated with the graphical object to corresponding facial locations on the face in the one or more images captured by the camera after the first image to generate a video having an image modified based on the graphical object, The one or more facial positions representing the face in the first image include: identifying facial feature points in the face in two dimensions; and A three-dimensional triangular mesh is fitted to the identified facial feature points, wherein the modifying each corresponding image of the one or more images comprises applying the graphical object at a corresponding position on the three-dimensional triangular mesh.
14. The device according to claim 13, wherein: The receiving of the selection of the graphic object includes: a user selecting a drawing tool button.
15. The apparatus according to claim 13, wherein: The receiving a selection of the graphical object includes: a user selecting a selectable representation of the graphical object.
16. The apparatus according to claim 13, wherein: The receiving of the selection of the graphic object comprises: a user's selection of a drawing tool button, and wherein the drawing tool button comprises one or more of the following: a pen drawing tool button, a lipstick drawing tool button, or a brush drawing tool button.
17. The apparatus according to claim 13, wherein: The receiving a selection of the graphic object includes: receiving a user drawing as the graphic object.
