Systems and methods for providing interactivity with pre‑recorded media

WO2024229140A1PCT designated stage expired Publication Date: 2024-11-07MIXCORD INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/US2024/027279
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-03
Filing Date
2024-05-01
Publication Date
2024-11-07

Smart Images

  • Figure US2024027279_07112024_PF_FP_ABST
    Figure US2024027279_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Systems and methods for providing interactivity with pre-recorded media are provided. In one aspect, an electronic device incudes a display, a camera configured to capture video, and a processor configured to receive a selection of pre-recorded media and display the pre-recorded media on a first portion of the display and the video captured by the camera on a second portion of the display. The processor is also configured to segment the subject and overlay a portion of the segmented subject that crossed the boundary onto the pre-recorded media.
Need to check novelty before this filing date? Find Prior Art

Description

MXCRD.001WO PATENT SYSTEMS AND METHODS FOR PROVIDING INTERACTIVITY WITH PRE-RECORDED MEDIA INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

[0001] This application is based upon and claims the benefit of priority from United States Provisional Patent Application No. 63 / 499,839 filed on May 3, 2023. Moreover, any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57. The entire contents of each of the above-listed items is hereby incorporated into this document by reference and made a part of this specification for all purposes, for all that each contains. BACKGROUND Technological Field

[0002] This disclosure relates to electronic devices that can be used to provide interactivity with pre-recorded media. Description of the Related Technology

[0003] Electronic devices are configured to record and share multimedia including images and video. One way in which this type of media is commonly shared is over social media. Social networks are often more engaging for users when users are able to interact with each other. SUMMARY OF CERTAIN INVENTIVE ASPECTS

[0004] The innovations described in the claims each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of the claims, some prominent features of this disclosure will now be briefly described.

[0005] One aspect is an electronic device, comprising: a display; a camera configured to capture video; a processor; and a non-transitory computer readable medium having stored thereon instructions which, when executed by the processor, cause the processorto: receive a selection of pre-recorded media; display the pre-recorded media on a first portion of the display and the video captured by the camera on a second portion of the display, the first and second portions of the display being separated by a boundary, and the video captured by the camera including a subject; detect the subject in the video received from the camera; segment the subject; overlay a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generate a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the video of the subject received from the camera.

[0006] In some embodiments, the instructions further cause the processor to: determine whether the portion of the segmented subject includes one or more predetermined features, wherein the overlaying of the portion of the segmented subject onto the pre-recorded media is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

[0007] In some embodiments, the predetermined features include at least one of a head, a hand, and a foot of the subject.

[0008] In some embodiments, the instructions further cause the processor to: receive a selection of a side-by-side template; receive a selection to mirror the video received from the camera; and arrange the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display in response to the selection of the side-by-side template and the selection to mirror the video received from the camera.

[0009] In some embodiments, the instructions further cause the processor to: in response to receiving the selection of the side-by-side template, determine that the electronic device is oriented in a portrait orientation; and in response to determining that the electronic device is oriented in a portrait orientation, display instructions to rotate the electronic device to a landscape orientation.

[0010] In some embodiments, the instructions further cause the processor to: receive a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template, arrange the second portion on a bottom of the display.

[0011] In some embodiments, the instructions further cause the processor to: receive a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template: segment the portion of the subject that cross the boundary to the left, right, and above the second portion of the display, and mask any portion of the subject that crosses the boundary below the second portion of display.

[0012] In some embodiments, the instructions further cause the processor to: detect a second subject in the pre-recorded media; segment the second subject; and overlay a portion of the segmented second subject onto the video captured by the camera.

[0013] Another aspect is a method comprising: receiving a selection of pre- recorded media; displaying the pre-recorded media on a first portion of a display and an image of a subject received from a camera on a second portion of the display, the first and second portions being separated by a boundary; detecting the subject in the image received from the camera; segmenting the subject; overlaying a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generating a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the image of the subject received from the camera.

[0014] In some embodiments, the method further comprises: determining whether the portion of the segmented subject includes one or more predetermined features, wherein overlaying the portion of the segmented subject onto the pre-recorded media is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

[0015] In some embodiments, the predetermined features include at least one of a head, a hand, and a foot of the subject.

[0016] In some embodiments, the method further comprises: receiving a selection of a side-by-side template; receiving a selection to mirror the image received from the camera; and arranging the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display in response to the selection of the side-by-side template and the selection to mirror the image received from the camera.

[0017] In some embodiments, the method further comprises: in response to receiving the selection of the side-by-side template, determining that the display is oriented ina portrait orientation; and in response to determining that the display is oriented in a portrait orientation, displaying instructions to rotate the display to a landscape orientation.

[0018] In some embodiments, the method further comprises: receiving a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template, arranging the second portion on a bottom of the display.

[0019] In some embodiments, the method further comprises: receiving a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template: segment the portion of the subject that cross the boundary to the left, right, and above the second portion of the display, and masking any portion of the subject that crosses the boundary below the second portion of display.

[0020] In some embodiments, the method further comprises: detect a second subject in the pre-recorded media; segment the second subject; and overlay a portion of the segmented second subject onto the video captured by the camera.

[0021] Yet another aspect is a non-transitory computer-readable medium having stored thereon instructions which, when executed by a processor, cause the processor to: receive a selection of pre-recorded media; display the pre-recorded media on a first portion of a display and an image of a subject received from a camera on a second portion of the display, the first and second portions being separated by a boundary; detect the subject in the image received from the camera; segment the subject; overlay a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generate a video including the pre- recorded media with the portion of the segmented subject overlayed thereon and a portion of the image of the subject received from the camera.

[0022] In some embodiments, the instruction further cause the processor to: determining whether the portion of the segmented subject includes one or more predetermined features, wherein overlaying the portion of the segmented subject onto the pre-recorded media or image is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

[0023] In some embodiments, the predetermined features include at least one of a head, a hand, and a foot of the subject.

[0024] In some embodiments, the instruction further cause the processor to: receive a selection of a side-by-side template; receive a selection to mirror the image received from the camera; and arrange the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display in response to the selection of the side-by-side template and the selection to mirror the image received from the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG.1A is a block diagram of an electronic device (such as a mobile device, smartphone, etc.) that can implement the techniques described herein in accordance with aspects of this disclosure.

[0026] FIG.1B illustrates an image that can be displayed on the user interface of a mobile device in accordance with aspects of this disclosure.

[0027] FIG. 2 illustrates a method for providing interactivity with pre-recorded media in accordance with aspects of this disclosure.

[0028] FIGs. 3A and 3B illustrate another method for providing interactivity with pre-recorded media in accordance with aspects of this disclosure.

[0029] FIGs.4A and 4B illustrate a method for displaying video captured from the camera and the pre-recorded media in the selected template in accordance with aspects of this disclosure.

[0030] FIG.5 illustrates a method for adding a frame of pre-recorded media into a template in accordance with aspects of this disclosure.

[0031] FIG. 6 illustrates a method for adding a segmented object to pre-recorded media in accordance with aspects of this disclosure.

[0032] FIGs.7A-7C are example images that may be involved in implementing the method in accordance with aspects of this disclosure.

[0033] FIGs. 8A and 8B are further example images that may be involved in implementing the method of FIG.6 in accordance with aspects of this disclosure.

[0034] FIGs. 9A-9D are example images that may be involved in implementing blocks of the method of FIGs.4A and 4B in accordance with aspects of this disclosure.

[0035] FIGs. 10A and 10B are example images that may be involved in implementing blocks of the method of FIGs. 3A and 3B in accordance with aspects of this disclosure.

[0036] FIGs. 11A-11C are images that illustrate exemplary orientations of the template in accordance with aspects of this disclosure.

[0037] FIG.12 is an example image illustrating another embodiment of a template that can be used to record an interactive video in accordance with aspects of this disclosure.

[0038] FIG. 13 is an example image illustrating yet another embodiment of a template that can be used to record an interactive video in accordance with aspects of this disclosure. DETAILED DESCRIPTION Introduction to Media Interactivity

[0039] Many social networks (such as TikTok, Instagram, YouTube, etc.) allow users to interact by recording and sharing short videos. One way in which users can interact with each other is to record a video that is played back on a portion of the screen while pre- recorded media (e.g., a video or photo) is played on the remaining portion of the screen.

[0040] FIG. 1A is a block diagram of an electronic device 100 (such as a mobile device, smartphone, etc.) that can implement the techniques described herein in accordance with aspects of this disclosure. FIG. 1A is merely an example block diagram of an electronic device 100, and certain elements may be removed, other elements added, two or more elements combined or one element can be separated into multiple elements depending on the specification and requirements. The electronic device 100 includes a processor 102, a memory 104, a user interface 106, a camera 108, a microphone 110, and a transceiver 111. In some embodiments, at least one of the processor 102, the memory 104, the user interface 106, the camera 108, and the microphone 110 can be implemented with corresponding elements (e.g., processor, memory, user interface or transceiver circuit) used in Android or iOS based smartphones or tablets, iPhone, iPad, a personal computer or laptop (e.g., Windows or Mac based), etc. The processor 102 may communicate data and signals with and control the operations of the memory 104, the user interface 106, the camera 108, and the microphone 110.

[0041] The memory 104 may communicate data with the processor 102. The memory 104 may store instructions which, when executed by the processor 102, can cause the processor 102 to perform actions to implement the techniques and methods described herein.

[0042] The user interface 106 may display pre-recorded media, images, and / or audio captured from the camera 108 and / or the microphone 110. The user interface 106 is also configured to receive inputs from a user, for example, via virtual buttons, sliders, and / or other inputs included in the user interface 106. The camera 108 is configured to capture images / video, which can be displayed on the user interface 106 and / or saved to the memory 104. In some embodiments, the user interface 106 can include one or more displays 112 (e.g., a touchscreen display), and / or one or more speakers 114. Further, the user interface 106 can also display pre-recorded media which can be stored on the memory 104 and / or received via the transceiver 111. The transceiver 111 can be configured to communicate with other electronic device(s) 100 including viewing and / or uploading interactive videos.

[0043] Traditional methods for recording interactive videos in which a new video is recorded for display along with pre-recorded media involve splitting the screen into two portions and displaying the pre-recorded media on one side of a display and the newly recorded video on the other side of the display.

[0044] FIG.1B illustrates an image 150 that can be displayed on the user interface 106 of a mobile device, such as the electronic device 100 in FIG. 1A, in accordance with aspects of this disclosure. The image 150 includes a grid having a first portion 152 in which the pre-recorded media is displayed and a second portion 154 in which images (e.g., forming a video, such as an interactive video) captured via a camera 158 can be displayed. The first portion 152 and the second portion 154 can be separated by a boundary 156. The image 150 can also include a record input 160 (e.g., a virtual button) configured to receive a user input indicating whether to start and / or stop recording a video from the camera 158.

[0045] When using a traditional method for recording an interactive video, the video recorded using the camera 158 can be displayed in the second portion 154 next to the pre-recorded media displayed in the first portion 152. However, such traditional methods limit the variety of interactions available to the user between the interactive video in the second portion 154 and the pre-recorded media (such as a video or image) in the first portion 152. For example, the video recorded via the camera 158 may be confined to the second portion 154,limiting the user’s ability to use their hands to point to or otherwise interact with the pre- recorded media. In some cases, users may use software to edit the recorded video (including the pre-recorded media and the newly recorded video) by adding pointers or other graphics to the pre-recorded media to interact with portions of the pre-recorded media. Providing Interactivity with Pre-Recorded Media

[0046] To overcome the limitations described above, aspects of this disclosure enable a user (i.e., the video being recorded by the user) to cross the boundary between the pre-recorded media and the video being recorded. Aspects of this disclosure relate to a method for producing an interactive video in which a user can cross the boundary between the pre- recorded media and the video being recorded. This method can involve detecting a subject in the video being recorded and applying image segmentation to portion(s) of the subject that cross into the pre-recorded media to mask out the background behind the segmented portion(s) of the subject. This allows a user to record a video while enabling the subject of the video to cross-into the pre-recorded media while continuing the playback of the pre-recorded media as a background for portion(s) of the subject that have crossed into the area of the pre-recorded media.

[0047] As described herein, aspects of this disclosure can provide systems and methods that enable a user to record a video that interacts with pre-recorded media, such as a video or photo. In some embodiments, this can enable the user to interact with the pre-recorded media in a manner similar to a whiteboard (e.g., the user can point to items within the pre- recorded media similar to pointing to items on a whiteboard). In certain embodiments, the user can freely move between the recording space and the pre-recorded space. In still further embodiment, when the pre-recorded media is a photo, the display can be split into two different spaces, enabling the user(s) to move between their physical location and a virtual space using the photo as a background. The following systems and methods can be implemented with components from FIGs. 1A and 1B, but any reference to components from FIGs. 1A and 1B are provided for illustration purposes only:

[0048] FIG.2 illustrates a method 200 for providing interactivity with pre-recorded media in accordance with aspects of this disclosure. Depending on the implementation, one or more blocks of the method 200 of FIG. 2 can be performed in a different order, concurrently, and / or omitted while other blocks can also be included. The method 200 will be described asperformed by a processor 102 of an electronic device 100, although one or more blocks can be performed by other components (e.g., a remote processor, a server, etc.).

[0049] The method 200 begins at block 201. At block 202, the processor 102 receives a selection of pre-recorded media. At block 204, the processor 102 displays the pre- recorded media on a first portion of a display (e.g., of a user interface 106) and an image of a subject received from a camera (e.g., the camera 108) on a second portion of the display. The first and second portions of the display may be separated by a boundary (e.g., the boundary 156).

[0050] At block 206, the processor 102 detects the subject in the image received from the camera 108. At block 208, the processor 102 segments the subject. The processor 102 may store both the segmented subject and the original image received from the camera 108 for further processing.

[0051] At block 210, the processor 102 overlays a portion of the subject that crossed the boundary onto the pre-recorded media. In some embodiments, the processor 102 may determine that the portion of the subject has crossed the boundary into the first portion of the display and overlay the identified portion onto the pre-recorded media. In other embodiments, the processor 102 may segment substantially the entire subject and overlay the segmented subject over the first and second portions of the display as well as over the boundary. Segmenting and overlaying substantially the entire subject may be more efficient than determining which portion of the subject has crossed the boundary, and thus, this embodiment may have the advantage of processing efficiency in certain cases. At block 212, the processor 102 generates a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the image of the subject received from the camera, which may also have the segmented subject overlaid thereon. The method 200 ends at block 214.

[0052] FIGs. 3A and 3B illustrate another method 300 for providing interactivity with pre-recorded media in accordance with aspects of this disclosure. Depending on the implementation, one or more blocks of the method 300 of FIGs.3A and 3B can be performed in a different order, concurrently, and / or omitted while other blocks can also be included. The method 300 will be described as performed by a processor 102 of an electronic device 100,although one or more blocks can be performed by other components (e.g., a remote processor, a server, etc.).

[0053] The method 300 begins at block 301. At block 302, the processor 102 receives a selection of a layout template from a user via the user interface 106. For example, the user interface 106 may be configured to display a list of layout templates to the user and receive a selection of one of the displayed layout templates. The layout templates can include, for example, side-by-side, top and bottom, multi-grid (also referred to as “Acapella”), picture- in-picture (PiP), etc. The grid may include a default location for the video to be recorded within the template. Depending on the implementation, the default location for the video to be recorded may be on the side of the display farthest away from the camera 108 for a side-by- side template and on the bottom of the display for other templates. The display of the video capture by the camera 108 on the bottom of the display can help simulate a pop-out visual effect. In certain implementations, the user may be able to change the default location of the video to be recorded within the template.

[0054] At block 304, the processor 102 initializes the camera 108 and / or microphone 110 of the electronic device 100. However, in some cases, the camera 108 and / or microphone 110 may already be initialized, and thus, the processor 102 may skip block 304 in some situations. At block 306, the processor 102 receives a selection of pre-recorded media via the user interface 106. The pre-recorded media can include a pre-recorded video (also referred to as video media) and / or an image (also referred to as a photo).

[0055] At block 308, the processor 102 determines whether the selected layout template is a landscape layout template (e.g., a side-by-side layout template). When the selected layout template is not a landscape layout template, the method 300 continues at block 314. When the selected layout template is a landscape layout template, the method 300 continues at block 310 where the processor 102 rotates the pre-recorded media. In some embodiments, the processor 102 may rotate the pre-recorded media if the pre-recorded media is not already in landscape orientation. At block 312, the processor 102 prompts the user to rotate the electronic device 100. For example, the processor 102 may display instructions to the user to rotate the electronic device (see, for example, FIG.10A).

[0056] At block 314, the processor 102 displays the video captured from the camera 108 and the pre-recorded media in the selected template on the display of the user interface106. At block 316, the processor 102 determines whether an input from the user with instructions to start recording has been received via the user interface 106. If no instructions to start recording have been received, the method 300 returns to block 314. In response to receiving instructions to start recording, the method 300 proceeds to block 318. If video is already recording when the method 300 arrives at block 316, the processor 102 may confirm that the video is still recording and proceed to block 318. At block 318, the processor 102 starts recording video received from the camera 108, and in some implementations, also with audio received from the microphone 110 (e.g., concurrent recording of audio and video). In some embodiments, the audio may be added from another source, such as audio associated with the pre-recorded media, an audio file stored on the memory 104, etc. which can be used alone and / or mixed with audio received from the microphone 110.

[0057] At block 320, the processor 102 determines whether an input from the user with instructions to stop recording has been received via the user interface 106. If no instructions to stop recording have been received, the method 300 returns to block 314. In response to receiving instructions to stop recording, the method 300 proceeds to block 322 at which the processor 102 displays the recorded video (e.g., as a preview before finalizing the interactive video). For example, the processor 102 may display the pre-recorded media, the newly recorded video, as well as one or more inputs for adjusting the pre-recorded media, the newly recorded video, and / or the audio associated with the pre-recorded media, the newly recorded video. In some embodiments, the processor 102 may display one or more sliders configured to enable the user to adjust the volume of the audio associated with each of the pre- recorded media and the newly recorded video. At block 324, the processor 102 mixes the recorded audio and / or audio received from another source with the recorded video and the pre- recorded media and saves the mixed file to the memory 104. The method 300 ends at block 326.

[0058] FIGs.4A and 4B illustrate a method 400 for displaying video captured from the camera 108 and the pre-recorded media in the selected template in accordance with aspects of this disclosure. For example, the method 400 may be one embodiment for implementing block 314 of method 300. Depending on the implementation, one or more blocks of the method 400 of FIGs. 4A and 4B can be performed in a different order, concurrently, and / or omitted while other blocks can also be included. The method 400 will be described as performed by aprocessor 102 of an electronic device 100, although one or more blocks can be performed by other components (e.g., a remote processor, a server, etc.).

[0059] The method 400 begins at block 401. At block 402, the processor 102 obtains an image from the camera 108. At block 404, the processor 102 determines whether the video obtained from the camera 108 is being mirrored when displayed on the display of the user interface 106. In response to determining that the video is not mirrored, the method 400 continues at block 406. Alternatively, in response to determining that the video is mirrored, the method 400 continues at block 408.

[0060] At block 406, the processor 102 swaps the template grid positions. For example, in some implementations the template grid positions may have a default in which the grid positions are arranged for mirrored video. In alternative implementations in which the default is for non-mirrored video, the processor 102 may swap the template grid positions in response to the video being mirrored.

[0061] At block 408, the processor 102 adjusts the pre-recorded media based on whether the video received from the camera 108 is mirrored. In some embodiments, the processor 102 may mirror the pre-recorded media when the video received from the camera 108 is mirrored and the processor 102 may not adjust the pre-recorded media. In other embodiments, the processor 102 may receive an input from the user with instructions regarding whether to mirror the pre-recorded media. Thus, the processor 102 may mirror the pre-recorded media in response to an input received from the user via the user interface 106.

[0062] At block 410, the processor 102 adds a next frame of the pre-recorded media in the selected template. At block 412, the processor 102 adds grid separation (e.g., a boundary) between the images within the selected template. At block 414, the processor 102 scales each of the videos within the template to fill their respective regions within the template.

[0063] At block 416, the processor 102 applies image segmentation to mask a desired object. For example, the processor 102 may identify an object within a current frame (e.g., image) received from the camera 108 as a desired object and apply image segmentation to the desired object. The image segmentation and masking can be performed, for example, using “green screening” techniques in which the processor 102 can classify the pixels of an image as belonging to the subject or the background. Pixels classified as belonging to thebackground can be masked such that they are not displayed over the pre-recorded media, while pixels classified as belonging to the subject can be overlaid over the pre-recorded media.

[0064] At block 418, the processor 102 adds the segmented object to the pre- recorded media. For example, the processor 102 may overlay the portion of the segmented object that crosses the boundary into the portion of the template occupied pre-recorded media over the pre-recorded media. In other embodiments, the processor 102 may overlay substantially the entire segmented object onto the pre-recorded media, the boundary, and / or the image received from the camera 108. At block 420, the processor 102 displays the pre- recorded media and the image received from the camera 108 along with the overlaid segmented object. The method 400 ends at block 422.

[0065] FIG. 5 illustrates a method 500 for adding a frame of pre-recorded media into a template in accordance with aspects of this disclosure. For example, the method 500 may be one embodiment for implementing block 410 of method 400. Depending on the implementation, one or more blocks of the method 500 of FIG. 5 can be performed in a different order, concurrently, and / or omitted while other blocks can also be included. The method 500 will be described as performed by a processor 102 of an electronic device 100, although one or more blocks can be performed by other components (e.g., a remote processor, a server, etc.).

[0066] The method 500 begins at block 501. At block 502, the processor 102 determines whether the pre-recorded media is video media or an image. In response to determining that the pre-recorded media is video media, the method 500 proceeds to block 504 and in response to determining that the pre-recorded media is an image, the method 500 proceeds to block 512.

[0067] At block 504, the processor 102 obtains the duration of the recorded video. For example, the duration may be the length of time elapsed since the start of recording the video. At block 506, the processor 102 obtains the pre-recorded video at the recorded duration (e.g., the frame of the pre-recorded video that corresponds to the time elapsed since the start of recording the video) in order to maintain synchronization. At block 508, the processor 102 scales the pre-recorded video to fill the corresponding portion of the template. At block 510, the processor 102 adds the scaled pre-recorded video into the corresponding portion of the template.

[0068] At block 512, the processor 102 scales the image to fill the corresponding portion of the template. At block 514, the processor 102 adds the scaled image into the corresponding portion of the template. The method 500 ends at block 516.

[0069] FIG. 6 illustrates a method 600 for adding a segmented object to pre- recorded media in accordance with aspects of this disclosure. For example, the method 600 may be one embodiment for implementing block 418 of method 400. Depending on the implementation, one or more blocks of the method 600 of FIG. 6 can be performed in a different order, concurrently, and / or omitted while other blocks can also be included. The method 600 will be described as performed by a processor 102 of an electronic device 100, although one or more blocks can be performed by other components (e.g., a remote processor, a server, etc.).

[0070] The method 600 begins at block 601. At block 602, the processor 102 removes portions of the pre-recorded media that are outside of corresponding portion the template. For example, the processor 102 may crop the pre-recorded media such that the pre- recorded media fills substantially the entire corresponding portion of the template. At block 604, the processor 102 overlays the segmented object over the pre-recorded media. In some implementations, the processor 102 may overlay segmented objects to the left, right, and top of the portion of the template occupied by the video captured from the camera 108. In certain embodiments, the processor 102 may also overlay segmented object(s) to the bottom of the portion of the template occupied by the video captured from the camera 108 in response to the processor determining that the segment object(s) including a head, hand, and / or foot of the subject. The method 600 ends at block 606.

[0071] FIGs.7A-7C are example images that may be involved in implementing the method 600 in accordance with aspects of this disclosure. In this example, the selected template (e.g., selected at block 302) may be a vertical template in which the pre-recorded media is displayed above the video captured from the camera 108.

[0072] FIG. 7A illustrates a pre-recorded media image 702 that may have been selected by a user, for example, at block 202. Here, the pre-recorded media image 702 may have been cropped to fit within the corresponding portion of the template. FIG. 7B illustrates an image 704 of the video captured from the camera 108 including a plurality of segments objects that extend into the portion of the template occupied by the pre-recorded media. In thisembodiment, the segmented objects include the arms and hair of the subject. FIG.7C illustrates a combined image 706 in which the pre-recorded media image 702 and the image 704 are combined into the template with the segmented objects overlaid onto the pre-recorded media image 702 and the border such that the segmented objects extend into the pre-recorded media image’s 702 portion of the template.

[0073] FIGs. 8A and 8B are further example images that may be involved in implementing the method 600 of FIG. 6 in accordance with aspects of this disclosure. In this example, the selected template (e.g., selected at block 302) may be a vertical template in which the pre-recorded media is displayed below the video captured from the camera 108.

[0074] FIG. 8A illustrates an example of an interactive video 802 in which substantially the entire subject of the video captured from the camera 108 is segmented and overlaid onto the pre-recorded media. As shown in FIG.8A, when the video captured from the camera 108 is oriented above the pre-recorded media, the segmented subject may obstruct large portions of the pre-recorded media. Thus, it may be advantageous to segment only certain portions of the subject to be overlaid onto the pre-recorded media, thereby allowing interactivity while also obstructing less of the pre-recorded media.

[0075] FIG.8B illustrates another example of an interactive video 804 in which the same image captured from the camera 108 of FIG.8A is segmented and overlaid onto the pre- recorded media. In particular, the FIG. 8B embodiment segments different portions of the subject such that less of the pre-recorded media is obstructed. For example, the processor 102 can determine whether the portion(s) of the subject that cross the border into the portion of the template occupied by the pre-recorded media include one or more predetermined features. In the embodiment of FIG.8B, the predetermined features include: a head, a hand, and / or a foot, but other predetermined features can be implemented within the scope of the present disclosure. In some embodiments, the predetermined features can also include certain recognized objects, such as cars, guitars, tables, chairs, etc. In certain implementations, the processor 102 is configured to segment any objects being held by the subject together with the subject. For example, this would allow the subject and the held object (e.g., a guitar) to freely cross the boundary together.

[0076] Depending on the implementation, the processor 102 may also determine whether the segmented object is missing the one or more predetermined features. For example,in embodiments in which the processor 102 segments substantially the entire subject, the processor 102 can determine whether any part of the segmented subject is missing. In some examples, the processor 102 can determine that one or more of the predetermined features (e.g., a head, a hand, a foot, etc.) are missing from the segmented subject. In response to determining that the one or more predetermined features are missing from the segmented subject, the processor 102 can mask corresponding portions of the subject such that the identified portions missing the one or more features are not overlaid onto the pre-recorded media.

[0077] For the portion(s) of the subject that cross the border and include one or more of the predetermined features, the processor 102 may treat these portion(s) of the subject as the desired object for which image segmentation is applied, for example, at block 416 of the method 400. For the portion(s) of the subject that cross the border and do not include one or more of the predetermined features, the processor 102 may treat these portion(s) of the subject similar to the background and refrain from overlaying these portion(s) of the subject onto the pre-recorded media. Thus, in the embodiment of FIG. 8B, the partial leg of the subject is not overlaid onto the pre-recorded media, thereby allowing more of the pre-recorded media to be visible in the interactive video 804.

[0078] FIGs. 9A-9D are example images that may be involved in implementing block 406 of the method 400 of FIGs.4A and 4B in accordance with aspects of this disclosure. In this example, the selected template (e.g., selected at block 302) may be a horizontal template (e.g., side-by-side) in which the pre-recorded media is displayed to the side of the video captured from the camera 108.

[0079] FIG. 9A illustrates an image 902 displayed on the user interface 106 when recording an interactive video prior to swapping the positions of the pre-recorded media and the video captured from the camera 108 within the template. In particular, the pre-recorded image is displayed on the left side of the display, closer to the camera, while the video captured from the camera 108 is displayed on the right side of the screen, closer to the record input (e.g., the user input displayed on the user interface 106 enabling the user to start or stop recording).

[0080] FIG.9B illustrates an image 904 of the video captured from the camera 108 including the portion of the segmented subject that crossed the boundary. The image 904 may be a mirrored image of the video received from the camera 108. The processor 102 can flip thevideo received from the camera 108 horizontally and swap the position of the video in the template in order to remove the mirroring of the image, resulting in the image 906 of FIG.9C. By swapping the position of the video received from the camera 108, the portion of the segmented subject that crosses the boundary will still be overlaid onto the pre-recorded image as shown in the image 908 of FIG. 9D. FIG. 9D illustrates the final image 908 displayed on the user interface 106 when recording the interactive video after swapping the positions of the pre-recorded media and the video captured from the camera 108 within the template.

[0081] After the swapping is complete, the video captured from the camera 108 within the template is located closer to the camera 108 on the left side of the display, while the pre-recorded media is displayed on the right side of the screen, closer to the record input. While the mirroring of video captured from the camera 108 may be reversed in certain implementations of block 406, in certain embodiments the pre-recorded media may not be altered other than changing the position of the pre-recorded media within the template. However, this disclosure is not limited thereto and the mirroring of pre-recorded media may be reversed in other embodiments when implementing block 406.

[0082] FIGs. 10A and 10B are example images that may be involved in implementing blocks 310 and 312 of the method 300 of FIGs. 3A and 3B in accordance with aspects of this disclosure. As shown in FIG.10A, in response to determining that the selected template is a landscape template and that the electronic device 100 is currently in a portrait orientation, the processor 102 may display an image 1002 with instructions to the user to rotate the electronic device 100. After determining that the electronic device 100 has been rotated into a landscape orientation, the processor 102 may display an image 1004 with the pre- recorded media within the selected template along with the record input.

[0083] FIGs. 11A-11C are images that illustrate exemplary orientations of the template in accordance with aspects of this disclosure. For example, FIG. 11A is an image 1102 that illustrates a user holding the electronic device 100 with one of the user’s hands, leaving the opposite hand free for interacting with the pre-recorded media. FIG. 11B is an image 1104 showing the user’s perspective of the electronic device 100. With reference to FIGs. 11A and 11B, the user’s left hand may be holding the left side of the electronic device 100 such that the user’s right hand is free. FIG. 11C is an image 1106 showing the display of the electronic device 100 during recording of an interactive video. The video captured fromthe camera 108 is mirrored and displayed on the left side of the screen close to the record input, allowing the user’s right hand to cross the boundary with the pre-recorded media displayed on the right side of the screen. Advantageously, this allows the user to easily interact with the pre- recorded media since the user’s free hand is located closer to the pre-recorded media in the mirrored image. Another advantage is that the user’s hand holding the electronic device 100 is located away from the camera 108, which can reduce or minimize any obstruction of the camera 108 by the user’s hand holding the electronic device 100.

[0084] As shown in FIGS. 11A-11C, when a side-by-side template is selected, the processor 102 may default to displaying the video captured by the camera 108 on the side of the screen that allows the user’s free hand to more easily interact with the pre-recorded media on the opposite side of the screen. Depending on the implementation, the processor 102 may allow the user to override the default which may be desirable, for example, when the electronic device 100 is secured to a hands-free device (e.g., a tripod).

[0085] FIG. 12 is an example image 1202 illustrating another embodiment of a template that can be used to record an interactive video in accordance with aspects of this disclosure. In this image 1202, the template provides a picture-in-picture template in which the pre-recorded media occupies substantially the entire screen with the video recorded from the camera 108 occupying a smaller area within the screen. The video recorded from the camera 108 includes borders separating the video recorded the camera 108 on all sides from the pre- recorded media.

[0086] FIG.13 is an example image 1302 illustrating yet another embodiment of a template that can be used to record an interactive video in accordance with aspects of this disclosure. In this image 1302, the template provides a multi-grid template in which the pre- recorded media includes a plurality of different pre-recorded medias playing together, with the video recorded from the camera 108 occupying one of a plurality of areas within the template. Depending on the position of the video recorded from the camera 108 within the template, the video recorded from the camera 108 may include borders separating the video recorded the camera 108 on one or more sides from the pre-recorded media.

[0087] In some embodiments, when implementing a picture-in-picture and / or a multi-grid template, the processor 102 can be configured to apply image segments to the portion(s) of the subject crossing the boundaries differently. For example, in oneimplementation, the processor 102 is configured to identify whether the portion(s) of the subject are the arms, legs, or head of the subject, and apply segmentation to the portion(s) of the subject identified as the arms, legs, or head of the subject. The processor 102 may treat other portion(s) of the subject that cross the boundaries the same as the background of the video recorded the camera 108. In other embodiments, the processor 102 can be configured to apply image segments to the portion(s) of the subject crossing the boundaries differently depending on which of the boundaries the portion(s) of the subject is crossing. For example, the processor 102 may apply segmentation to the portion(s) of the subject that cross the boundaries to the left, right, and above the portion of the template occupied by the video recorded the camera 108, while treating any portion(s) of the subject that cross the boundary below the portion of the template occupied by the video recorded the camera 108 the same as the background (e.g., masking these portion(s) of the subject).

[0088] In some embodiments, the processor 102 can also be configured to apply segmentation and masking to the pre-recorded media in addition to the video received from the camera 108. For example, the processor 102 may detect a second subject in the pre- recorded media, segment the second subject, and overlay the segmented second subject onto the video captured by the camera 108 as well as the video received from the camera 108. In some embodiments, the processor 102 may also determine that a portion of the second subject has crossed the boundary into the portion of the display occupied by the video received from the camera 108 and segment the identified portion of the second subject. These implementations can allow both the subject of the video received from the camera 108 as well as the subject of the pre-recorded media to cross the boundary therebetween. In one example, both subjects can move their hands and arms across the boundary.

[0089] Although aspects of this disclosure have been described in a connection with an interactive video including pre-recorded media and a recorded video having a segmented subject, this disclosure is not limited thereto. For example, in some embodiments, one or more subjects from a pre-recorded video can also be segments and move between different portions of the template in addition to one or more subjects from the recorded video. In addition, aspects of this disclosure can also include an interactive video including two or more videos recorded at substantially the same time (e.g., from two or more cameras 108 on the electronic device 100 such as a front facing camera 108 and a rear facing camera 108 ortwo or more cameras from separate camera 108 recording at the same time). These embodiments can employ any template (e.g., side-by-side, multi-grid, picture-in-picture, etc.). Conclusion

[0090] Information and signals disclosed herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0091] The various illustrative logical blocks, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0092] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as devices or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer- readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0093] Processor(s) in communication with (e.g., operating in collaboration with) the computer-readable medium (e.g., memory or other data storage device) may execute instructions of program code, and may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software or hardware configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC). Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0094] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of inter-operativehardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0095] Although the foregoing has been described in connection with various different embodiments, features or elements from one embodiment may be combined with other embodiments without departing from the teachings of this disclosure. However, the combinations of features between the respective embodiments are not necessarily limited thereto. Various embodiments of the disclosure have been described. These and other embodiments are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. An electronic device, comprising: a display; a camera configured to capture video; a processor; and a non-transitory computer readable medium having stored thereon instructions which, when executed by the processor, cause the processor to: receive a selection of pre-recorded media; display the pre-recorded media on a first portion of the display and the video captured by the camera on a second portion of the display, the first and second portions of the display being separated by a boundary, and the video captured by the camera including a subject; detect the subject in the video received from the camera; segment the subject; overlay a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generate a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the video of the subject received from the camera.

2. The electronic device of Claim 1, wherein the instructions further cause the processor to: determine whether the portion of the segmented subject includes one or more predetermined features, wherein the overlaying of the portion of the segmented subject onto the pre- recorded media is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

3. The electronic device of Claim 2, wherein the predetermined features include at least one of a head, a hand, and a foot of the subject.

4. The electronic device of Claim 1, wherein the instructions further cause the processor to: receive a selection of a side-by-side template; receive a selection to mirror the video received from the camera; and arrange the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display in response to the selection of the side-by-side template and the selection to mirror the video received from the camera.

5. The electronic device of Claim 4, wherein the instructions further cause the processor to: in response to receiving the selection of the side-by-side template, determine that the electronic device is oriented in a portrait orientation; and in response to determining that the electronic device is oriented in a portrait orientation, display instructions to rotate the electronic device to a landscape orientation.

6. The electronic device of Claim 1, wherein the instructions further cause the processor to: receive a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template, arrange the second portion on a bottom of the display.

7. The electronic device of Claim 1, wherein the instructions further cause the processor to: receive a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template: segment the portion of the subject that cross the boundary to the left, right, and above the second portion of the display, and mask any portion of the subject that crosses the boundary below the second portion of display.

8. The electronic device of Claim 1, wherein the instructions further cause the processor to: detect a second subject in the pre-recorded media; segment the second subject; and overlay a portion of the segmented second subject onto the video captured by the camera.

9. A method comprising: receiving a selection of pre-recorded media; displaying the pre-recorded media on a first portion of a display and an image of a subject received from a camera on a second portion of the display, the first and second portions being separated by a boundary; detecting the subject in the image received from the camera; segmenting the subject; overlaying a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generating a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the image of the subject received from the camera.

10. The method of Claim 9, further comprising: determining whether the portion of the segmented subject includes one or more predetermined features, wherein overlaying the portion of the segmented subject onto the pre-recorded media is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

11. The method of Claim 10, wherein the predetermined features include at least one of a head, a hand, and a foot of the subject.

12. The method of Claim 9, further comprising: receiving a selection of a side-by-side template; receiving a selection to mirror the image received from the camera; andarranging the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display in response to the selection of the side-by-side template and the selection to mirror the image received from the camera.

13. The method of Claim 12, further comprising: in response to receiving the selection of the side-by-side template, determining that the display is oriented in a portrait orientation; and in response to determining that the display is oriented in a portrait orientation, displaying instructions to rotate the display to a landscape orientation.

14. The method of Claim 9, further comprising: receiving a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template, arranging the second portion on a bottom of the display.

15. The method of Claim 9, further comprising: receiving a selection of a picture-in-picture template or a multi-grid template; and in response to receiving the selection of the picture-in-picture template or the multi-grid template: segment the portion of the subject that cross the boundary to the left, right, and above the second portion of the display, and masking any portion of the subject that crosses the boundary below the second portion of display.

16. The method of Claim 9, further comprising: detect a second subject in the pre-recorded media; segment the second subject; and overlay a portion of the segmented second subject onto the video captured by the camera.

17. A non-transitory computer-readable medium having stored thereon instructions which, when executed by a processor, cause the processor to: receive a selection of pre-recorded media; display the pre-recorded media on a first portion of a display and an image of a subject received from a camera on a second portion of the display, the first and second portions being separated by a boundary; detect the subject in the image received from the camera; segment the subject; overlay a portion of the segmented subject that crossed the boundary onto the pre-recorded media; and generate a video including the pre-recorded media with the portion of the segmented subject overlayed thereon and a portion of the image of the subject received from the camera.

18. The non-transitory computer-readable medium of Claim 17, wherein the instruction further cause the processor to: determining whether the portion of the segmented subject includes one or more predetermined features, wherein overlaying the portion of the segmented subject onto the pre-recorded media or image is performed in response to determining that the portion of the segmented subject includes the one or more predetermined features.

19. The non-transitory computer-readable medium of Claim 18, wherein the predetermined features include at least one of a head, a hand, and a foot of the subject.

20. The non-transitory computer-readable medium of Claim 17, wherein the instruction further cause the processor to: receive a selection of a side-by-side template; receive a selection to mirror the image received from the camera; and arrange the first portion of the display on a side of the display closer to the camera and the second portion of the display on an opposite side of the display inresponse to the selection of the side-by-side template and the selection to mirror the image received from the camera.

Citation Information

Patent Citations

  • Method and apparatus for beam link indication

    US62634998P0

  • Target object display method and device and electronic equipment

    CN111669502A

  • Video processing method, apparatus and storage medium

    US11310443B2

  • Video processing method and apparatus, electronic device, and non-transitory computer readable storage medium

    US20220377259A1

  • Video processing method, terminal device and storage medium

    US20220394192A1