Real-time virtual teleportation in the browser

Abstract

1. A method comprising: opening a web-based video call in a browser on a first device (145); receiving a request to join the web-based video call from a second device (150); the first device capturing (110) a video including frames (105); the first device segmenting (115) the frames; the first device selecting at least one segment (120) of the segmented frames; and streaming (125) the video including the at least one segment as a real-time virtual teleportation (140) directly from the first device to the second device.

Description

[Technical field]

[0001] Field SUMMARY OF THE DISCLOSURE Embodiments relate to streaming video in a video conferencing web-based environment. [Background technology]

[0002] background Video calling can make users feel distant from each other. In other words, social interaction can feel distant because two or more participants are in different locations and each participant is viewing the other's location or an artificial background on a viewing device (e.g., a mobile phone). Furthermore, to conduct a video conference with advanced features (e.g., background correction), a full-featured application needs to be installed on the user device. Summary of the Invention

[0003] overview In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method may perform a process using a method that includes opening a web-based video call in a browser on a first device, receiving a request to join the web-based video call from a second device, the first device capturing a video including frames, the first device segmenting the frames, the first device selecting at least one segment of the segmented frames, and streaming the video including the at least one segment directly from the first device to the second device as a real-time virtual teleportation.

[0004] Implementations may include one or more of the following features. For example, opening a web-based video call may include loading a webpage, the webpage including code configured to implement a trained machine-learned model, the trained machine-learned model may be configured to segment a frame and select at least one segment. The at least one segment may be an image of a participant in the web-based video call. The web-based video call may be implemented using a web-based communication standard. Segmenting the frame may include grouping pixels in the frame into semantic regions to identify locations of objects and boundaries, classifying the pixels of the frame into two categories: 1) pixels representing people and 2) pixels representing background, and segmenting the pixels representing people from the frame. Segmenting the frame may include identifying each object in the frame, and selecting the at least one segment includes selecting an object as the at least one segment, the object may be a participant in the web-based video call. The at least one segment may be an image of a participant in the web-based video call, the method may further include converting the image from a two-dimensional image to a three-dimensional image. At least one segment may be an image of a participant in a web-based video call, and the method may further include applying a filter to the image.The web-based video call may be implemented as a zero-install web application.

[0005] In another general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method may perform a process using a method that includes opening a web-based video calling webpage in a browser on a first device; the first device communicating a request to join the web-based video call from a second device; receiving at the first device a streamed video directly from the second device as a first video; the first device capturing the second video; the first device orienting the first video based on the second video; the first device projecting the first video onto the second video to generate a third video; and the first device rendering a webpage including the third video.

[0006] Implementations may include one or more of the following features. For example, the method may further include generating a plane, and orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector. The method may further include generating a plane and positioning the plane within the second video, and projecting the first video onto the second video includes adding the first video to the plane. The method may further include generating a plane and positioning the plane within the second video, and orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector, and projecting the first video onto the second video may include adding the first video to the plane. The first video may be a video of a first participant of a web-based video call, and the second video may be a real-world video. The plane may be a transparent two-dimensional virtual structure positioned within the second video. The plane may have a size proportional to a display of a device that renders the web-based video calling webpage. The web-based video calling webpage may include code configured to implement a trained machine-learned model, and the web-based video calling webpage may include code configured to implement a web-based augmented reality tool. The web-based video calling webpage and the web-based video calling may be implemented as zero-install web applications.

[0007] In yet another general aspect, a device, system, non-transitory computer-readable medium (having computer-executable program code stored thereon) and / or method includes opening a web-based video call in a browser on a first device, receiving a request to join the web-based video call from a second device, the first device capturing a first video including frames, the first device segmenting the frames, the first device selecting at least one segment of the segmented frames, and transmitting the at least one segment directly from the first device to the second device as a first real-time video. The process may be performed using a method including: streaming a first video including the second real-time virtual teleportation image to a second device; the first device receiving the streamed video directly from the second device as a second video, the second video including the second real-time virtual teleportation image; the first device capturing a third video; the first device orienting the second video based on the third video; the first device projecting the second video onto the third video to generate a fourth video including the second real-time virtual teleportation image; and the first device rendering a webpage including the fourth video.

[0008] Implementations may include one or more of the following features. For example, opening a web-based video call may include loading a webpage, the webpage may include code configured to implement a trained machine-learned model, the trained machine-learned model may be configured to segment a frame and select at least one segment, and the webpage including the code may be configured to implement a web-based augmented reality tool. The web-based video call may be implemented as a zero-install web application. Segmenting the frame may include identifying each object in the frame, and selecting the at least one segment may include selecting an object as the at least one segment, the object may be a participant in the web-based video call. The method may further include generating a plane and positioning the plane within the second video, orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector, and projecting the first video onto the second video may include adding the first video to the plane.

[0009] BRIEF DESCRIPTION OF THE DRAWINGS The example embodiments will become more fully understood from the detailed description given below and the accompanying drawings, in which like elements are indicated with like reference numerals, which are provided for purposes of illustration only and are not intended to be limiting of the example embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a block diagram of a signal flow in accordance with at least one exemplary embodiment. [Figure 2A] FIG. 1 illustrates a block diagram of an image processing module in accordance with at least one example embodiment. [Figure 2B]1 illustrates an encoder system in accordance with at least one example embodiment. [Figure 3A] 1 illustrates a decoder system in accordance with at least one exemplary embodiment. [Figure 3B] FIG. 1 illustrates a block diagram of a projector module in accordance with at least one exemplary embodiment. [Figure 4] 1 illustrates a block diagram of a method for implementing portions of a web-based video call in accordance with at least one exemplary embodiment. [Diagram 5] 1 illustrates a block diagram of a method for conducting another portion of a web-based video call in accordance with at least one exemplary embodiment. [Figure 6] 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] It should be noted that these figures are intended to illustrate the general characteristics of methods, structures and / or materials utilized in certain exemplary embodiments and to supplement the following description. However, these figures are not to scale, may not precisely reflect the exact structure or performance characteristics of any given embodiment, and should not be interpreted as defining or limiting the range of values ​​or properties encompassed by the exemplary embodiments. For example, the relative thicknesses and positioning of molecules, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0012] Detailed Description of the Preferred Embodiments The user experience in a video call may not be as desired because participants are presented in their own environment and are limited to a view (e.g., a rectangular view). Participants in different environments may make participants feel socially distant and / or may result in undesirable social interaction between users.

[0013] To solve the above problem, an image of a first participant in a video call may be extracted from a first environment and projected (or reprojected) into a second environment of a second participant. In other words, an exemplary embodiment may generate and communicate an image of one or more participants to another environment (e.g., another participant's environment) as seen on a device (e.g., a mobile device). Implementations may enable a virtual teleportation video call that includes generating and communicating an image of a first video call participant, projecting the image of the first video call participant into a field of view on a device of a second video call participant, and enabling the second video call participant to walk to and / or around the first video call participant as if the first video call participant were in the second video call participant's space.

[0014] An exemplary implementation may include segmenting a streaming video of at least one participant and projecting the segmented portion of the streaming video including the participant on a display of the device. Furthermore, the generation and communication of the participant's image may be implemented in real-time (e.g., live, minimal delay, etc.) in a web page. The implementation in a web page may not involve the installation of an application on a local device. In other words, the implementation may be in a zero-install computing environment (e.g., a user does not download a file or a memory insert of a file to install an application). A zero-install computing environment may have the advantage of being flexible in that changes to a web-based video calling application may affect all users of the web-based video calling application, and users may not have to take any action other than opening a web page to use it.

[0015] In addition, the video calling application may be web-based and / or may use an application installed on the device. In either case, a server configured to control the streaming communication is used. In an exemplary implementation, the video call may stream video directly from a first device to a second device. Thus, the exemplary implementation discloses a novel video calling feature including at least providing a tool for virtual teleportation video calling in a browser without using a server (e.g., a third-party server) configured to control the streaming communication. In an exemplary implementation, "web-based" refers to functionality implemented in a browser using a web server to communicate (over the Internet using HTTP(S)), for example, images, videos, text, etc., that may be displayed in the browser as a web page on the display of the computing device. Furthermore, the web server may communicate software code (e.g., JavaScript, c++, Visual Basic, etc.) that may be executed by the computing device in association with the web page. The server (e.g., a third-party server) configured to control the streaming communication is independent (or operates independently) from (e.g., is not the same as) the web server.

[0016] FIG. 1 illustrates a block diagram of a signal flow according to at least one exemplary embodiment. As shown in FIG. 1, signal flow 100 includes a capture 110 block, a segment 115 block, a communication 125 block, a projector 130 block, and a rendering 135 block. In the capture 110 block, an image 105 is captured using a camera of a computing device (e.g., a desktop, a laptop, a mobile device, a standalone image capture system, etc.). Through the embodiments illustrated herein, images of one or more participants may be projected (or reprojected) from at least one environment to at least one other environment as viewed on the device. Thus, signal flow 100 may illustrate an exemplary implementation of a virtual teleportation video call.

[0017] Image 105 may be a frame of video corresponding to video of a video call on a first device 145 that includes a camera. Image 105 may include pixels corresponding to at least the participants in the video call and the environment in which the participants in the video call are located (sometimes referred to as a background). Data representing image 105 is communicated to a segment 115 block. The segment 115 block may segment image 105 into at least two segments, one of which may be pixels corresponding to the participants in the video call. The segmented image including the participants in the video call is shown as image 120.

[0018] The image 120 is communicated from a first device 145 that captures the image (e.g., a device that includes a capture 110 block and a segment 115 block) to a second device 150 via a communication 125 block. The communication 125 block may stream video corresponding to a video call in which the image 120 represents a frame of the video. The image 120 may be a portion of a captured frame (e.g., image 105). Thus, the image 120 may include less data than a complete frame. As a result, the exemplary implementation may use fewer resources (e.g., bandwidth) than a video call that streams a complete frame. The streaming may use web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.).

[0019] The second device may receive image 120 and project the image (projector 130) onto an image being captured by the second device that is not yet displayed on the second device (e.g., a real-world image). Projecting image 120 may include selecting a position and orientation of image 120 relative to the image captured by the second device. The resulting image may be rendered (render 135 block) and displayed on the second device as shown in image 140.

[0020] FIG. 2A illustrates a block diagram of an image processing module according to at least one exemplary embodiment. As illustrated in FIG. 2A, the image processing 230 module includes an object identification unit 235, a segmentation unit 240, an image correction 245 module, and a segment 115. As described above, the video call may be implemented in a web page. Thus, the image processing 230 module may be an element of the web page. For example, the image processing 230 module may be implemented in JavaScript. The image processing 230 module may include a machine-learned element, for example, the image processing 230 module may include a machine-learned model (e.g., a convolutional neural network (CNN)). Thus, the image processing 230 module may include a trained machine-learned (ML) model implemented in JavaScript (e.g., TensorFlow.js). Furthermore, the image processing 230 module may be loaded onto a computing device together with the loading of a web page. Thus, a trained ML model implemented in JavaScript may be loaded onto a computing device together with the loading of a web page. Thus, the image processing 230 module can enable (or help enable) virtual teleportation video calls to be performed in a browser without installing an application on the device.

[0021] The object identifier 235 may be configured to identify each object in an image or frame of a video call. The object identifier 235 may be configured to identify an object as a participant in a video call. The object identifier 235 may identify the object using a trained ML model (e.g., a convolutional neural network (CNN)). Thus, the object identifier 235 may identify the object using a trained ML model implemented in JavaScript (e.g., TensorFlow.js).

[0022] An image or frame of a video (e.g., image 105) may contain multiple objects. A trained ML model associated with the object identifier 235 may place multiple boxes (sometimes called bounding boxes) on the image. The object identifier may associate data (e.g., features associated with pixels of the image) with each of the boxes. The data may indicate an object within the box (the object may be no object or a portion of an object). An object may be identified by its features. The data, cumulatively, may be referred to as a class or classifier. A class or classifier may be associated with an object. The data (e.g., bounding boxes) may also include a confidence score (e.g., a number between 0 and 1).

[0023] After the trained ML model processes an image or frame of the video, the trained ML model may process an object or multiple classifiers that indicate the same object. In other words, an object (or part of an object) may be in multiple overlapping bounding boxes. However, the confidence scores for each of the classifiers may be different. For example, a classifier that identifies a part of an object may have a lower confidence score than a classifier that identifies a complete (or substantially complete) object. The trained ML model may further be configured to discard bounding boxes that do not have an associated classifier. In other words, the trained ML model may discard bounding boxes that do not have an object in them. The trained ML model may identify the object using the classifier with the highest confidence score. One of the objects may be identified as a participant in the video call (e.g., as a human or part of a human).

[0024] The segmenter 240 may be configured to generate an image that includes the participant (and does not include other objects or background pixels). For example, the object identifier 235 may pass the coordinates of a box that includes the participant. The segmenter 240 may remove pixels in the image that are not within that box. The segmenter 240 may copy the contents of the box to a new image. Additionally, the segmenter 240 may be configured to remove any undesirable pixels, smooth transitions, and modify object boundaries to improve the participant image (e.g., image 120). The segmented image is stored as a segment 115.

[0025] In some implementations, the object identifier 235 and the segmenter 240 may be combined into one operation. For example, image segmentation of body parts may be part of an ML tool or model. The ML tool may be configured to group pixels in an image into semantic regions to identify the location of objects and boundaries. For example, the ML tool or model may be configured to classify pixels of an image into two categories: 1) pixels representing people and 2) pixels representing background. The pixels representing people may then be segmented from the image.

[0026] The image correction 245 module may be configured to correct the segmented image and / or generate a new image (e.g., as segment 115) based on the segmented image. The image correction 245 module may be configured to generate a three-dimensional (3D) image from a two-dimensional image. The 2D-3D conversion tool may be an element of a web page (e.g., as JavaScript). The conversion tool may be implemented with the segmented image as input. For example, the conversion tool may use a depth map (e.g., associated with the participant), a 3D mesh, a warping algorithm, etc. The conversion tool may be implemented as a trained ML model. In an example implementation, the 2D to 3D conversion may be a partial conversion (e.g., adding depth to a portion of the segmented image).

[0027] The image modification 245 module may be configured to apply image filters to the segmented image. For example, the image modification 245 module may apply ghost filters, color filters, enhancement filters, holographic filters, overlay filters, etc. The image modification 245 module may be configured to enhance (e.g., improve quality, resolution, etc.) the segmented image. The image modification 245 module may be configured to complete the segmented image. For example, the segmented image may be a portion of a participant (e.g., a head) and the image modification 245 module may add to (e.g., add a body) to the segmented image. The image modification 245 module may be configured to modify the segmented image using other techniques not described herein for brevity.

[0028] In the example of Figure 2B, the encoder system 200 may be or may include at least one computing device, and should be understood to represent virtually any computing device configured to perform the techniques described herein. The encoder system 200 may therefore be understood to include various components that may be utilized to implement the techniques described herein or different or future versions thereof. By way of example, the encoder system 200 is shown as including at least one processor 205 and at least one memory 210 (e.g., a non-transitory computer-readable storage medium).

[0029] FIG. 2B illustrates an encoder system according to at least one exemplary embodiment. As shown in FIG. 2B, the encoder system 200 includes at least one processor 205, at least one memory 210, a controller 220, and an encoder 225. The at least one processor 205, the at least one memory 210, the controller 220, and the encoder 225 are communicatively coupled via a bus 215. The encoder system may be an element of a video call implemented via a web page. In an exemplary implementation, the encoder 225 and the controller 220 are loaded onto a computer when a web page configured to implement a video call is loaded. The encoder 225 and the controller 220 may use (or be an element of) a web-based communication standard (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.). The encoder system 200 may use the segment 115 as an input.

[0030] The at least one processor 205 may be utilized to execute instructions stored in the at least one memory 210. Thus, the at least one processor 205 may implement various features and functions described herein, or additional or alternative features and functions. For example, the processor 205 may execute code associated with a web page configured to implement a video call, stored in the at least one memory 210. The at least one processor 205 and the at least one memory 210 may be utilized for various other purposes. For example, the at least one memory 210 may represent examples of various types of memory and associated hardware and software that may be used to implement any of the modules described herein.

[0031] The at least one memory 210 may be configured to store data and / or information associated with the encoder system 200 (e.g., for implementing web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.)). The at least one memory 210 may be a shared resource. For example, the encoder system 200 may be an element of a larger system (e.g., a server, a personal computer, a mobile device, etc.). Thus, the at least one memory 210 may be configured to store data and / or information associated with other elements (e.g., image / video services, web browsing, or wired / wireless communication) in the larger system.

[0032] The controller 220 may be configured to generate various control signals and communicate the control signals to various blocks within the encoder system 200 and / or image processing 230 module. The controller 220 may be configured to generate the control signals to implement the techniques described herein. According to an example implementation, the controller 220 may be configured to control the encoder 225 to encode an image, a sequence of images, a video frame, a sequence of video frames, etc. For example, the controller 220 may generate control signals corresponding to encoding and communicating an image (or frame) associated with a web-based communication standard (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.).

[0033] The encoder 225 may be configured to receive an input image 5 (and / or video stream) and output compressed (e.g., encoded) bits 10. The encoder 225 may convert the video input (e.g., as an image) into discrete video frames. The input image 5 may be compressed (e.g., encoded) as compressed image bits. The encoder 225 may further convert each image (or discrete video frame) into a matrix of blocks or macroblocks (hereafter referred to as blocks). For example, an image may be converted into a 32x32, 32x16, 16x16, 16x8, 8x8, 4x8, 4x4, or 2x2 matrix of blocks, each having a number of pixels. Eight exemplary matrices are listed, but exemplary implementations are not limited thereto.

[0034] The compressed bits 10 may represent an output of the encoder system 200. For example, the compressed bits 10 may represent an encoded image (or video frame). For example, the compressed bits 10 may be stored in a memory (e.g., at least one memory 210). For example, the compressed bits 10 may be ready for transmission to a receiving device (not shown). For example, the compressed bits 10 may be transmitted to a system transceiver (not shown) for transmission to the receiving device.

[0035] The at least one processor 205 may be configured to execute computer instructions associated with the image processing 230 module, the controller 220, and / or the encoder 225. The at least one processor 205 may be a shared resource. For example, the encoder system 200 may be an element of a larger system (e.g., a mobile device, a desktop, a laptop, etc.). Thus, the at least one processor 205 may be configured to execute computer instructions associated with other elements (e.g., image / video capture, web browsing, and / or wired / wireless communication) in the larger system.

[0036] In an example implementation, the image processing 230 module may be an element of the encoder 225. For example, the image 5 may be multiple frames of a streaming video. The encoder 225 may be configured to process each frame individually. Thus, the encoder may select a frame of the streaming video and communicate the selected frame to the image processing module 230 as an input to the image processing 230 module. After processing the frame, the image processing module 230 may generate a segment 115, which may then be processed (e.g., compressed) by the encoder 225.

[0037] 3A illustrates a decoder system according to at least one exemplary embodiment. As shown in FIG. 3A, the decoder system 300 includes at least one processor 305, at least one memory 310, a controller 320, and a decoder 325. The at least one processor 305, the at least one memory 310, the controller 320, and the decoder 325 are communicatively coupled via a bus 315.

[0038] In the example of Figure 3A, the decoder system 300 may be at least one computing device and should be understood to represent virtually any computing device configured to execute the techniques described herein. Thus, the decoder system 300 may be understood to include various components that may be utilized to implement the techniques described herein or different or future versions thereof. For example, the decoder system 300 is shown as including at least one processor 305 and at least one memory 310 (e.g., a computer-readable storage medium).

[0039] Thus, the at least one processor 305 may be utilized to execute instructions stored in the at least one memory 310. Thus, the at least one processor 305 may implement various features and functions described herein (e.g., to implement web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.), or additional or alternative features and functions. The at least one processor 305 and the at least one memory 310 may be utilized for various other purposes. For example, the at least one memory 310 may be understood to represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein. According to an exemplary implementation, the decoder system 300 may be included in a larger system (e.g., a personal computer, a laptop computer, a mobile device, etc.).

[0040] The at least one memory 310 may be configured to store data and / or information associated with the projector 130 and / or the decoder system 300. The at least one memory 310 may be a shared resource. For example, the decoder system 300 may be an element of a larger system (e.g., a personal computer, a mobile device, etc.). Thus, the at least one memory 310 may be configured to store data and / or information associated with other elements (e.g., web browsing or wireless communication) in the larger system (e.g., to implement web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.)).

[0041] The controller 320 may be configured to generate various control signals and communicate the control signals to various blocks in the projector and / or decoder system 300. The controller 320 may be configured to generate the control signals to implement the video encoding / decoding techniques described herein. The controller 320 may be configured to control the decoder 325 to decode video frames, according to an example implementation.

[0042] The decoder 325 may be configured to receive as input the compressed (e.g., encoded) bits 10 and output the image 5. The compressed (e.g., encoded) bits 10 may also represent compressed video bits (e.g., video frames). Thus, the decoder 325 may convert discrete video frames of the compressed bits 10 into a video stream.

[0043] The at least one processor 305 may be configured to execute computer instructions associated with the projector 130, the controller 320, and / or the decoder 325. The at least one processor 305 may be a shared resource. For example, the decoder system 300 may be an element of a larger system (e.g., a personal computer, a mobile device, etc.). Thus, the at least one processor 305 may be configured to execute computer instructions associated with other elements (e.g., web browsing or wireless communications) in the larger system.

[0044] FIG. 3B illustrates a block diagram of a projector 130 according to at least one exemplary embodiment. As illustrated in FIG. 3B, the projector 130 includes a plane generation 330 module, a plane positioning 335 module, a normal determination 340 module, and a projection 345 module. In an exemplary implementation, a web-based video call includes at least two computing devices. A first computing device may be utilized by a first participant, and a second computing device may be used by a second participant. In a web-based virtual teleportation video call, the first device may include elements described with respect to FIG. 2A and FIG. 2B to generate an image of the first participant, and the second device may include elements described with respect to FIG. 3A and FIG. 3B to receive an image of the first participant and project the first participant into the environment of the second participant. Thus, the projector 130 may have the advantage of being able to enable (or help enable) a virtual teleportation video call to be performed in a browser without installing an application on the device. Additionally, example implementations may include communicating images of the first participant, the second participant, and / or both the first and second participants.

[0045] Thus, the plane generation 330 module may be configured to generate a plane to project an image of the first participant onto the second device. The plane positioning 335 module may be configured to select a position on the display of the second device to place the plane in the physical world environment of the second device. In an example implementation, a user of the second device may refer to a real-world location to be rendered on the display of the second device. This is sometimes referred to as mixed reality. The normal determination 340 module may be configured to orient the plane on the display of the second device. The projection 345 module may be configured to project the first participant (e.g., as segment 115) into the plane.

[0046] The Plane Generation 330 module may generate the plane by generating a 2D virtual structure (e.g., a rectangle). The 2D structure may have a size based on the proximity to a plane in the real world on which the plane is placed. In some cases, if the user is nearby, the rectangle may be larger than the device itself and only a subsection of the first participant may be displayed. The 2D structure may have a size based on the display of the second device. For example, the 2D structure may have a size proportional to and smaller than the display. The 2D structure may be transparent (e.g., so that the background is visible). The 2D structure may be realized via function calls (e.g., via a web-based viewing tool and / or a web browser). An exemplary code snippet is shown below:

[0047]

number

[0048] The plane positioning 335 module may be configured to select a position on the display of the second device to place the plane. This position may be based on an image (e.g., a preview image) captured by the second device and displayed on the device using a web-based application (e.g., a browser). For example, referring to FIG. 1, image 140 is an image of a hallway. The position of the plane may be placed approximately in the middle of the hallway at a comfortable depth for viewing. Thus, this position may be based on pixel position (e.g., X, Y position) and depth. Depth may be determined using a depth sensor or a camera on the device and / or calculated using a depth algorithm (e.g., using a web-based tool, a web-based augmented reality tool, and / or a JavaScript tool (e.g., WebXR)). In other words, depth may be determined using a function call in a web-based (e.g., JavaScript) augmented reality tool (e.g., WebXR) that may return a depth map. In an example implementation, the location may be determined using a web-based tool configured to render a virtual object (e.g., a plane, or a communicated image of a participant) in the real world (e.g., as an image captured using a device's camera and displayed via a web browser). An example code snippet is shown below:

[0049]

number

[0050] The normal determination 340 module may be configured to orient the plane on the display of the second device. For example, a normal associated with the real world (e.g., as an image captured using the device's camera and displayed via a web browser) and with the plane position may be determined. The plane may then be oriented (e.g., translated, rotated, etc.) with respect to the real world such that the plane approximates a perpendicular direction to the normal. In an example implementation, a normal direction associated with each pixel in the real world may be determined (e.g., estimated). Once there is a normal vector associated with each pixel, the normal may be associated with the pixel in world coordinates. The normal vectors and pixels may be generated using web-based augmented reality and / or JavaScript tools (e.g., WebXR). For example, the normal vector may be projected from the plane to the pixel in the real world. The orientation may be achieved and / or confirmed by projecting the normal vector from the oriented plane to the real world. The normal vectors should be approximately equal and opposite. An example code snippet is shown below:

[0051]

number

[0052] The projection 345 module may be configured to project the first participant (e.g., as segment 115) into a plane. For example, pixels representing the image of the first participant (e.g., the uncompressed segment 115) may be added to the plane. The projection may include adding look and feel features to improve the user experience. For example, a shadow may be added to the resulting image. The projection 345 module may be implemented as a function that renders a modified real-world image (e.g., with the communicated image of the first participant) on a display of the second device. The rendering may be a function that displays a web page in a browser that implements video calling and / or web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.). An exemplary code snippet is shown below:

[0053]

number

[0054] 4 and 5 show block diagrams of the method. The steps described with respect to FIG. 4 and FIG. 5 may be performed by execution of software code stored in a memory (e.g., at least one memory 210, 310) associated with the device (e.g., as shown in FIG. 2B and FIG. 3A) and executed by at least one processor (e.g., at least one processor 205, 305) associated with the device. However, alternative embodiments such as a system embodied as a dedicated processor are contemplated. Although the steps described below are described as being performed by a processor, the steps are not necessarily performed by the same processor. In other words, at least one processor may perform the steps described below with respect to FIG. 4 and FIG. 5.

[0055] FIG. 4 illustrates a block diagram of a method for implementing a portion of a web-based video call according to at least one exemplary embodiment. As shown in FIG. 4, in step S405, a web-based video conference is established. The web-based video call may be a virtual teleportation video call realized in a browser. For example, a first participant on a first device may open a web page including a video calling web application and invite a second participant on a second device using the video calling web application (or other communication tools (e.g., email, messaging, etc.)). The second participant may join the video call by opening a web page including the video calling web application on the second device and requesting to join the video call. The first participant may accept the second participant into the video call. For example, the first participant on the first device may open a web page including the video calling web application and call the second participant on the second device using the video calling web application. The second participant may join the video call by opening a web page including the video calling web application and answering the video call.

[0056] The video calling application may be web-based and / or may use an application installed on the device. In either case, a server configured to control the streaming communication is used. In an exemplary implementation, the video call may stream video directly from the first device to the second device. In other words, the video call may be streamed (e.g., peer-to-peer) from the first device to the second device without using a server (e.g., a third-party server) configured to control the streaming communication. Thus, the exemplary implementation enables a new video calling feature (e.g., virtual teleportation video calling) in the browser without using a server (e.g., a third-party server) configured to control the streaming communication.

[0057] By conducting video calling in a browser, the exemplary implementations can enable (or help enable) zero-install (e.g., without any application or plug-in installed by the user) video calling. By eliminating the server, the exemplary implementations can enable (or help enable) communications that are low latency, device / platform independent (e.g., works in any browser), more secure (e.g., a server may add a layer of security risk, no third party services may be added), adaptable to network conditions, and do not require dedicated tools (e.g., plug-ins), etc. Web pages that include the video calling web application may use web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.).

[0058] In step S410, a video is captured. For example, the first participant may capture the video using a computing device (e.g., a desktop computer, a laptop computer, a mobile device, etc.). The video may include multiple frames. Each frame may be used in a real-time virtual teleportation video call via a web page running in a browser. Each frame may represent the first participant as an image of the person being communicated.

[0059] In step S415, the video is segmented. For example, each frame of the streaming video may be segmented. Image segmentation of body parts may be part of an ML tool or model. The ML tool may be configured to group pixels in an image into semantic regions to identify the location of objects and boundaries. For example, the ML tool or model may be configured to classify pixels of an image into two categories: 1) pixels representing people and 2) pixels representing background. The pixels representing people may then be segmented from the image. Alternatively, segmenting a frame may include identifying each object in the frame and selecting a first participant as a segment. For example, as described in more detail above, the trained ML model may place a number of boxes (sometimes called bounding boxes) on the image. An object identifier may associate data (e.g., features associated with pixels of the image) with each of the boxes. The data may indicate an object within the box (the object may be no object or a portion of an object). The object may be identified by its features. The features may be classified as a person. The pixels representing the person can then be segmented from the image.

[0060] A web page (e.g., a video calling web application running in a browser) may include a trained machine learning (ML) model implemented in JavaScript (e.g., TensorFlow.js). The trained machine learning (ML) model implemented in JavaScript may be configured to segment a frame (e.g., identify an object) and select a first participant (e.g., a human) as an image segment (e.g., segment 115).

[0061] In an exemplary implementation, there may be more than one participant (e.g., human). A first participant may be standing closer to the camera of a first device and may not be the full view participant who wishes to be projected. Thus, 1 to n participants may be selected from a scene to project 1 to n humans simultaneously as desired.

[0062] In step S420, at least one segment is processed. In an exemplary implementation, processing at least one segment may be optional. That is, processing may proceed to step S425 without performing step S420. Processing at least one segment may include image modification of at least one segment (e.g., segment 115). For example, the image modification may include image enhancement (e.g., quality improvement), image transformation (e.g., 2D to 3D conversion), image warping (making a 2D image appear 3D without 3D conversion), etc.

[0063] At least one segment is encoded in step S425. For example, the at least one segment may be encoded using a standard used to make a video call. The standard used to make a video call may include loading an encoder when a web-based video conference is established. For example, a video call may be made using WebRTC. A WebRTC compatible browser may or may not use or support at least VP8 and / or AVC encoder / decoder standards. At least one encoded segment is streamed in step S430. For example, the at least one encoded segment may be communicated from a first device to a second device, for example, over the Internet using the WebRTC standard. In another implementation, data (e.g., raw binary data) may be communicated without using a standard.

[0064] FIG. 5 shows a block diagram of a method for implementing another portion of a web-based video call according to at least one exemplary embodiment. As shown in FIG. 5, in step S505, a web-based video conference is established. For example, a first participant on a first device may open a web page including a video calling web application and invite a second participant on a second device using the video calling web application (or other communication tools (e.g., email, messaging, etc.)). The second participant may join the video call by opening a web page including the video calling web application on the second device and requesting to join the video call. The first participant may accept the second participant into the video call. For example, the first participant on the first device may open a web page including the video calling web application and call the second participant on the second device using the video calling web application. The second participant may join the video call by opening a web page including the video calling web application and answering the video call. The video call may stream video directly from the first device to the second device. In other words, a video call may be streamed from a first device to a second device without the use of a server configured to control the streaming communication.

[0065] The video calling application may be web-based and / or may use an application installed on the device. In either case, a server configured to control the streaming communication is used. In an exemplary implementation, the video call may stream video directly from the first device to the second device. In other words, the video call may be streamed (e.g., peer-to-peer) from the first device to the second device without using a server (e.g., a third-party server) configured to control the streaming communication. Thus, the exemplary implementation enables a new video calling feature (e.g., virtual teleportation video calling) in the browser without using a server (e.g., a third-party server) configured to control the streaming communication.

[0066] By conducting video calling in a browser, the exemplary implementations can enable (or help enable) zero-install (e.g., without any application or plug-in installed by the user) video calling. By eliminating the server, the exemplary implementations can enable (or help enable) communications that are low latency, device / platform independent (e.g., works in any browser), more secure (e.g., a server may add a layer of security risk, no third party services may be added), adaptable to network conditions, and do not require dedicated tools (e.g., plug-ins), etc. Web pages that include the video calling web application may use web-based communication standards (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.).

[0067] In step S510, a video stream is received. For example, encoded frames of a video corresponding to a video call may be communicated from a first device to a second device over the Internet, for example, using the WebRTC standard. The second device may receive the video stream frame by frame and / or in groups of frames. In step S515, the video stream is decoded as the first video. For example, each frame of the streaming video may be decoded. The standard used to make the video call may include loading a decoder when a web-based video conference is established. For example, a video call may be made using WebRTC. A WebRTC compatible browser may use or support at least the VP8 and / or AVC encoder / decoder standards. In another implementation, data (e.g., raw binary data) may be communicated without using a standard.

[0068] In step S520, a video is captured as a second video. For example, the second participant may capture the video using a computing device (e.g., a desktop computer, a laptop computer, a mobile device, etc.). The video may include multiple frames. Each frame may be used in a real-time virtual teleportation video call via a web page running in a browser. Each frame may represent a real world into which an image of the first participant may be projected.

[0069] In step S525, a normal vector associated with the second video is determined. For example, a normal vector associated with the real world (e.g., as an image captured using a device's camera and displayed via a web browser) and with the plane position may be determined. In an example implementation, a normal vector associated with each pixel in the real world may be determined (e.g., estimated). Once there is a normal vector associated with each pixel, the normal vector may be associated with the pixel in world coordinates. The normal vector and pixel may be generated using a web-based augmented reality tool and / or JavaScript tool (e.g., WebXR). For example, the normal vector may be projected from the plane to the pixel in the real world.

[0070] In step S530, a plane is generated. For example, the plane may be generated by generating a 2D virtual structure (e.g., a rectangle). The 2D structure may have a size based on the display of the second device. For example, the 2D structure may have a size proportional to and smaller than the display. The 2D structure may be transparent (e.g., so that the background is visible). The 2D structure may be realized via a function call (e.g., via a web-based display tool and / or a web browser).

[0071] In step S535, the plane is oriented based on the normal vector. For example, the plane may be oriented (e.g., translated, rotated, etc.) with respect to the real world such that the plane approximates a perpendicular orientation with the normal vector. The orientation may be achieved and / or confirmed by projecting the normal vector from the plane onto the real world. The normal vectors associated with the real world and the plane should be approximately equal and opposite.

[0072] In step S540, the first video is projected onto a plane of the second video. For example, pixels representing the image of the first participant (e.g., the decompressed segment 115) may be added to the plane. In step S545, the first video and the second video are rendered. The projection may be realized as a function of rendering the second (or real-world) video modified (e.g., with the first video (or the communicated image of the first participant)) on a display of the second device. The rendering may be a function of displaying a web page in a browser implementing a video call and / or a web-based communication standard (e.g., webRTC, VoIP, RTP, PTP Telephony, etc.).

[0073] 6 illustrates examples of a computing device 600 and a mobile computing device 650 that may be used with the techniques described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The illustrated components, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention as described and / or claimed herein.

[0074] The computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 connecting to the memory 604 and a high-speed expansion port 610, and a low-speed interface 612 connecting to a low-speed bus 614 and the storage device 606. Each of the components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. The processor 602 is capable of processing instructions that are executed within the computing device 600, including instructions stored in the memory 604 or on the storage device 606 for displaying graphical information for a GUI on an external I / O device, such as a display 616 coupled to the high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, as appropriate. Also, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0075] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0076] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a number of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. A computer program product can be tangibly embodied in an information medium. Also, the computer program product can include instructions that, when executed, perform one or more methods as described above. The information medium is a computer-readable or machine-readable medium, such as the memory 604, the storage device 606, or a memory on the processor 602.

[0077] The high-speed controller 608 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 612 manages less bandwidth-intensive operations. Such an allocation of functionality is merely exemplary. In one implementation, the high-speed controller 608 is coupled to the memory 604, the display 616 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 610 that may accept various expansion cards (not shown). In such an implementation, the low-speed controller 612 is coupled to the storage device 606 and the low-speed expansion port 614. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., via a network adapter.

[0078] Computing device 600 may be implemented in many different forms as shown in the figure. For example, computing device 600 may be implemented as a standard server 620 or multiple times within a group of such servers. Computing device 600 may also be implemented as part of a rack server system 624. Additionally, computing device 600 may be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of computing devices 600, 650, and the entire system may be made up of multiple computing devices 600, 650 in communication with each other.

[0079] Computing device 650 includes, among other components, a processor 652, memory 664, input / output devices such as a display 654, a communications interface 666, and a transceiver 668. Device 650 may also be provided with a storage device such as a microdrive or other device to provide additional storage. Each of components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as appropriate.

[0080] The processor 652 can execute instructions within the computing device 650, including instructions stored in the memory 664. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination of other components of the device 650, such as, for example, a user interface, applications executed by the device 650, and control of wireless communications by the device 650.

[0081] The processor 652 may communicate with a user via a control interface 658 and a display interface 656 coupled to a display 654. The display 654 may be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 656 may include appropriate circuitry for driving the display 654 to present graphical and other information to the user. The control interface 658 may receive commands from a user and convert the commands for submission to the processor 652. Additionally, an external interface 662 may be provided in communication with the processor 652 to enable adjacent communication between the device 650 and other devices. The external interface 662 may, for example, provide for wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may be used.

[0082] The memory 664 stores information within the computing device 650. The memory 664 may be realized as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. Additionally, an expansion memory 674 may be provided and connected to the device 650 via an expansion interface 672, which may include, for example, a Single In Line Memory Module (SIMM) card interface. Such expansion memory 674 may provide extra storage space to the device 650 or may further store applications or other information for the device 650. In particular, the expansion memory 674 may include instructions for performing or supplementing the above-mentioned processes, and may further include secure information. Thus, for example, the expansion memory 674 may be provided as a security module for the device 650 and may be programmed with instructions that allow secure use of the device 650. Additionally, secure applications may be provided via a SIMM card along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0083] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information medium. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information medium is a computer-readable or machine-readable medium, such as memory 664, expansion memory 674, or memory on processor 652, which may be received, for example, on transceiver 668 or external interface 662.

[0084] Device 650 may communicate wirelessly via communication interface 666, which may include digital signal processing circuitry as necessary. Communication interface 666 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via radio frequency transceiver 668. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 670 may provide additional navigation-related and location-related wireless data to device 650, which may be used as appropriate by applications executing on device 650.

[0085] Device 650 may also communicate audibly using a voice codec 660, which may receive verbal information from a user and convert that information into usable digital information. Voice codec 660 may also generate sounds audible to the user, such as through a speaker in a handset of device 650. Such sounds may include sounds from a voice telephone call, recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications executing on device 650.

[0086] Computing device 650 may be implemented in many different forms as shown in the figure. For example, computing device 650 may be implemented as a cellular telephone 680. Computing device 650 may also be implemented as part of a smartphone 682, personal digital assistant, or other similar mobile device.

[0087] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method may perform a process using a method that includes opening a web-based video call in a browser on a first device, receiving a request to join the web-based video call from a second device, the first device capturing a video including frames, the first device segmenting the frames, the first device selecting at least one segment of the segmented frames, and streaming the video including the at least one segment directly from the first device to the second device as a real-time virtual teleportation.

[0088] Implementations may include one or more of the following features. For example, opening a web-based video call may include loading a webpage, the webpage including code configured to implement a trained machine-learned model, the trained machine-learned model may be configured to segment a frame and select at least one segment. The at least one segment may be an image of a participant in the web-based video call. The web-based video call may be implemented using a web-based communication standard. Segmenting the frame may include grouping pixels in the frame into semantic regions to identify locations of objects and boundaries, classifying the pixels of the frame into two categories: 1) pixels representing people and 2) pixels representing background, and segmenting the pixels representing people from the frame. Segmenting the frame may include identifying each object in the frame, and selecting the at least one segment includes selecting an object as the at least one segment, the object may be a participant in the web-based video call. The at least one segment may be an image of a participant in the web-based video call, the method may further include converting the image from a two-dimensional image to a three-dimensional image. At least one segment may be an image of a participant in a web-based video call, and the method may further include applying a filter to the image.The web-based video call may be implemented as a zero-install web application.

[0089] In another general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method may perform a process using a method that includes opening a web-based video calling webpage in a browser on a first device; the first device communicating a request to join the web-based video call from a second device; receiving at the first device a streamed video directly from the second device as a first video; the first device capturing the second video; the first device orienting the first video based on the second video; the first device projecting the first video onto the second video to generate a third video; and the first device rendering a webpage including the third video.

[0090] Implementations may include one or more of the following features. For example, the method may further include generating a plane, and orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector. The method may further include generating a plane and positioning the plane within the second video, and projecting the first video onto the second video includes adding the first video to the plane. The method may further include generating a plane and positioning the plane within the second video, and orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector, and projecting the first video onto the second video may include adding the first video to the plane. The first video may be a video of a first participant of a web-based video call, and the second video may be a real-world video. The plane may be a transparent two-dimensional virtual structure positioned within the second video. The plane may have a size proportional to a display of a device that renders the web-based video calling webpage. The web-based video calling webpage may include code configured to implement a trained machine-learned model, and the web-based video calling webpage may include code configured to implement a web-based augmented reality tool. The web-based video calling webpage and the web-based video calling may be implemented as zero-install web applications.

[0091] In yet another general aspect, a device, system, non-transitory computer-readable medium (having computer-executable program code stored thereon) and / or method includes opening a web-based video call in a browser on a first device, receiving a request to join the web-based video call from a second device, the first device capturing a first video including frames, the first device segmenting the frames, the first device selecting at least one segment of the segmented frames, and transmitting the at least one segment directly from the first device to the second device as a first real-time video. The process may be performed using a method including: streaming a first video including the second real-time virtual teleportation image to a second device; the first device receiving the streamed video directly from the second device as a second video, the second video including the second real-time virtual teleportation image; the first device capturing a third video; the first device orienting the second video based on the third video; the first device projecting the second video onto the third video to generate a fourth video including the second real-time virtual teleportation image; and the first device rendering a webpage including the fourth video.

[0092] Implementations may include one or more of the following features. For example, opening a web-based video call may include loading a webpage, the webpage may include code configured to implement a trained machine-learned model, the trained machine-learned model may be configured to segment a frame and select at least one segment, and the webpage including the code may be configured to implement a web-based augmented reality tool. The web-based video call may be implemented as a zero-install web application. Segmenting the frame may include identifying each object in the frame, and selecting the at least one segment may include selecting an object as the at least one segment, the object may be a participant in the web-based video call. The method may further include generating a plane and positioning the plane within the second video, orienting the first video may include determining a normal vector associated with the second video and at least one of rotating and translating the first video based on the normal vector, and projecting the first video onto the second video may include adding the first video to the plane.

[0093] Exemplary embodiments may include various modifications and alternative forms, which are shown by way of example in the drawings and described in detail herein. It is to be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but rather, the exemplary embodiments are intended to encompass all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the drawings.

[0094] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Various implementations of the systems and techniques described herein may be realized and / or generally referred to herein as circuits, modules, blocks, or systems that may combine aspects of software and hardware. For example, a module may include functions / operations / computer program instructions that execute on a processor (e.g., a processor formed on a silicon substrate, GaAs substrate, etc.) or other programmable data processing apparatus.

[0095] Some of the above exemplary embodiments are described as processes or methods that are depicted as flow charts. Although the flow charts describe operations as sequential processes, many of the operations may occur in parallel, simultaneously, or simultaneously. Additionally, the order of operations may be rearranged. Processes may be terminated upon the completion of their operations, but may have additional steps not included in the figures. These processes may correspond to methods, functions, procedures, subroutines, subprograms, and the like.

[0096] The methods discussed above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments that perform the necessary tasks may be stored in a machine-readable or computer-readable medium, such as a storage medium. A processor may perform the necessary tasks.

[0097] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

[0098] Terms such as first, second, etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0099] When an element is referred to as being connected or coupled to another element, it will be understood that the element may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements. Other words used to describe relationships between elements should be construed in a similar manner (e.g., "between" and "directly between," "adjacent" and "directly adjacent," etc.).

[0100] The terms used herein are merely for describing specific embodiments and are not intended to be limited to exemplary embodiments. As used herein, the singular forms "a, an" and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that the terms "comprises, comprising" and / or "includes, including", when used herein, specify the presence of a referenced feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0101] Moreover, in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be performed concurrently or may sometimes be performed in the reverse order, depending on the functions / acts involved.

[0102] Unless otherwise specified, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Furthermore, it will be understood that terms, such as, for example, terms defined in commonly used dictionaries, should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and not to be interpreted in an idealized or overly formal sense, unless expressly defined as such in the present specification.

[0103] Portions of the above exemplary embodiments and corresponding detailed description are presented in terms of software or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones by which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps require physical manipulations of physical quantities. Usually, but not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has been recognized that it is sometimes convenient, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0104] In the above exemplary embodiments, references to symbolic representations of acts and operations (e.g., in the form of flowcharts) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types and that may be written and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, or field programmable gate array (FPGA) computers, etc.

[0105] It should be borne in mind, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, or as will be apparent from the discussion, the terms processing, computing, calculating, or determining of the representations refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical electronic quantities in the computer system's registers and memory, and convert that data into other data similarly represented as physical quantities in the computer system's memory, registers, or other such information storage, transmission, or display devices.

[0106] Also, the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a read-only compact disk or CD-ROM) and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0107] Finally, although the appended claims recite particular combinations of features described herein, the scope of the disclosure is not limited to the particular combinations claimed, but instead is extended to encompass any combination of features or embodiments disclosed herein, regardless of whether that particular combination is currently specifically recited in the appended claims.

Claims

1. 1. A method comprising: Opening a web-based video call in a browser on the first device; receiving, by the first device, a request to join the web-based video call from a second device; the first device capturing a first video including frames; the first device segmenting the frame; the first device selecting at least one segment of the segmented frame; streaming the first video including the at least one segment directly from the first device to the second device as a first real-time virtual teleportation; and receiving, by the first device, a streamed video directly from the second device as a second video, the second video including a second real-time virtual teleportation image, the method further comprising: generating a plane; and positioning the plane within the second video, the plane having a size proportional to a display of a device rendering the web-based video call, the method further comprising: the first device orienting the second video based on an environment of the first device; projecting, by the first device, the second video onto a background representing the environment of the first device to generate a third video including a second real-time virtual teleportation image; and the first device rendering the third video.

2. establishing the web-based video call includes loading a web page; the web page includes code configured to implement a trained machine learning model; The method of claim 1 , wherein the trained machine-learned model is configured to segment the frame and select the at least one segment.

3. The method of claim 1 or 2, wherein the at least one segment is an image of a participant in the web-based video call.

4. The method according to any one of claims 1 to 3, wherein the web-based video call is implemented using a web-based communication standard.

5. The segmentation of the frame comprises: grouping pixels in the frames into semantic regions to locate objects and boundaries; classifying pixels of said frame into two categories: 1) pixels representing people and 2) pixels representing background; and segmenting the pixels representing the person from the frame.

6. said segmenting said frame including identifying each object within said frame; Selecting the at least one segment includes selecting an object as the at least one segment; The method of any one of claims 1 to 4, wherein the objects are participants in the web-based video call.

7. The at least one segment is an image of a participant in the web-based video call, the method further comprising: A method according to any preceding claim, comprising converting the image from a two-dimensional image to a three-dimensional image.

8. The at least one segment is an image of a participant in the web-based video call, the method further comprising: A method according to any preceding claim, comprising applying a filter to the image.

9. The method according to any one of claims 1 to 8, wherein said web-based video calling is implemented as a zero-installation web application.

10. The method of claim 2 , wherein the web page including the code is configured to implement a web-based augmented reality tool.

11. Determining the orientation of the second video includes: determining a normal vector associated with the second video; and at least one of rotating and translating the first video based on the normal vector; The method of claim 1 , wherein projecting the first video onto the second video comprises adding the first video to the plane.

12. 1. A method comprising: Opening a web-based video call in a browser on the first device; Communicating a request to join the web-based video call from a second device; receiving, at the first device, a streamed video directly from the second device as a first video; generating a plane, the plane having a size proportional to a display of a device on which the web-based video call is rendered, the method further comprising: the first device capturing a second video; the first device orienting the first video based on the second video; the first device projects the first video onto the second video to generate a third video; and the first device rendering the third video.

13. Determining the orientation of the first video includes: determining a normal vector associated with the second video; and at least one of rotating and translating the first video based on the normal vector.

14. 14. The method of claim 12 or 13, further comprising generating a plane and positioning the plane within the second video, and wherein projecting the first video onto the second video comprises adding the first video to the plane.

15. generating a plane and positioning the plane within the second video; Orienting the first video includes: determining a normal vector associated with the second video; and at least one of rotating and translating the first video based on the normal vector; The method of claim 12 , wherein projecting the first video onto the second video comprises adding the first video to the plane.

16. the first video being a video of a first participant in the web-based video call; The method of any one of claims 12 to 15, wherein the second video is a real-world video.

17. The web-based video call includes code configured to implement a trained machine learning model; The method of any one of claims 12 to 16, wherein the web-based video call comprises code configured to implement a web-based augmented reality tool.

18. The method according to any one of claims 12 to 17, wherein said web-based video calling and said web-based video calling are implemented as a zero-installation web application.

19. 1. A method comprising: Opening a web-based video call in a browser on the first device; receiving a request to join the web-based video call from a second device; the first device capturing a first video including frames; the first device segmenting the frame; the first device selecting at least one segment of the segmented frame; streaming the first video including the at least one segment as a first real-time virtual teleportation image directly from the first device to the second device; and receiving, by the first device, a streamed video directly from the second device as a second video, the second video including a second real-time virtual teleportation image, the method further comprising: generating a plane; and positioning the plane within the second video, the plane having a size proportional to a display of a device rendering the web-based video call, the method further comprising: the first device capturing a third video; and the first device orienting the second video based on the third video; the first device projects the second video onto the third video to generate a fourth video including a second real-time virtual teleportation image; and the first device rendering a web page including the fourth video.

20. establishing the web-based video call includes loading a web page; the web page includes code configured to implement a trained machine learning model; the trained machine learning model is configured to segment the frames and select the at least one segment; The method of claim 19 , wherein the web page including the code is configured to implement a web-based augmented reality tool.

21. The method according to claim 19 or 20, wherein the web-based video calling is implemented as a zero-installation web application.

22. The segmentation of the frame comprises: grouping pixels in the frames into semantic regions to locate objects and boundaries; classifying pixels of said frame into two categories: 1) pixels representing people and 2) pixels representing background; Segmenting the pixels representing the person from the frame.

23. said segmenting said frame including identifying each object within said frame; Selecting the at least one segment includes selecting an object as the at least one segment; The method according to any one of claims 19 to 21, wherein said objects are participants in said web-based video call.

24. The method of claim 23, further comprising: determining a normal vector associated with the second video; and at least one of rotating and translating the first video based on the normal vector; The method of any one of claims 19 to 23, wherein projecting the first video onto the second video comprises adding the first video to the plane.

25. An apparatus comprising one or more processors and a memory storing instructions that, when executed by said one or more processors, cause said one or more processors to perform a method according to any one of claims 1 to 24.

26. A program comprising instructions which, when executed by one or more processors of a computer, cause the computer to carry out a method according to any one of claims 1 to 24.

Images