Method and apparatus for providing media

Abstract

An embodiment of the present disclosure may provide a method for synthesizing media by using an electronic apparatus. The method may include a step of receiving, from a plurality of terminals, a plurality of media including one or more frames. The method may include a step of identifying, from among the plurality of media, a number of media that require synthesis. The method may include a step of setting a layout of a synthesis frame on the basis of the number of media. The method may include a step of generating a synthesis frame set comprising one or more synthesis frames on the basis of the plurality of media and the layout. The method may include a step of transmitting the synthesis frame set to the plurality of terminals.

Description

Method and device for providing media

[0001] The present disclosure relates to a method and apparatus for providing media, and a method and apparatus for receiving a plurality of media and providing synthesized media can be provided.

[0002] In multi-party video calls and conferences, hardware performance or network bandwidth may be limited. In these cases, it may be difficult to transmit or play all the video from multiple participants. A video synthesis device (e.g., an MCU server) can synthesize the media from multiple participants into a single media for transmission.

[0003] According to an embodiment of the present disclosure, a method of an electronic device for synthesizing media may be provided. The method may include receiving a plurality of media, each including one or more frames, from a plurality of terminals. The method may include identifying a number of media requiring synthesis among the plurality of media. The method may include setting a layout of a synthesized frame based on the number of media. The method may include generating a set of synthesized frames, each set including one or more synthesized frames, based on the plurality of media and the layout. The method may include transmitting the set of synthesized frames to the plurality of terminals.

[0004] According to an embodiment of the present disclosure, a method of a terminal receiving synthesized media may be provided. The method may include transmitting media to an external electronic device that synthesizes the media. The method may include receiving, from the external electronic device, a set of synthesized frames including one or more synthesized frames, wherein the synthesized frames include a plurality of media regions.

[0005] According to an embodiment of the present disclosure, an electronic device for synthesizing media may be provided. The electronic device for synthesizing media may include a transceiver; a memory storing at least one instruction; and at least one processor connected to the transceiver. The electronic device for synthesizing media may correspond to the image synthesizing device of the present disclosure. The at least one processor may execute the at least one instruction to receive a plurality of media, each including one or more frames, from a plurality of terminals. The at least one processor may execute the at least one instruction to identify the number of media that require synthesizing. The at least one processor may execute the at least one instruction to set a layout of a synthesized frame based on the number of media. The at least one processor may execute the at least one instruction to generate a single synthesized frame set including one or more synthesized frames based on the plurality of media and the layout. The synthesized frame may include a plurality of media areas corresponding to the number of media. The at least one processor may execute the at least one instruction to transmit the composite frame set to the plurality of terminals.

[0006] According to an embodiment of the present disclosure, a terminal for receiving synthesized media may be provided. The terminal for receiving synthesized media may include a transceiver; a memory storing at least one instruction; and at least one processor connected to the transceiver. The at least one processor may execute the at least one instruction to transmit media to an external electronic device that synthesizes media. The at least one processor may execute the at least one instruction to receive, from the external electronic device, a set of synthesized frames including one or more synthesized frames. The synthesized frame may include a step of including a plurality of media regions. The at least one processor may execute the at least one instruction to identify a media region corresponding to a terminal among the plurality of media regions. The at least one processor may execute the at least one instruction to obtain output media based on the media region corresponding to the terminal. The at least one processor may execute the at least one instruction to display the obtained output media.

[0007] FIG. 1A and FIG. 1B are drawings for explaining how an image synthesis device according to an embodiment of the present disclosure transmits synthetic media.

[0008] FIG. 2 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0009] FIG. 3 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0010] FIG. 4 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0011] FIG. 5 is a diagram illustrating a video synthesis device and a terminal transmitting and receiving media according to an embodiment of the present disclosure.

[0012] Figures 6, 7, and 8 are diagrams illustrating a terminal processing synthetic media.

[0013] FIGS. 9, 10, and 11 are diagrams illustrating synthetic media according to embodiments of the present disclosure.

[0014] FIG. 12 is a flowchart illustrating a method for transmitting synthetic media by an image synthesis device according to an embodiment of the present disclosure.

[0015] FIG. 13 is a flowchart of a method for a terminal to receive synthetic media according to an embodiment of the present disclosure.

[0016] FIG. 14 is a block diagram illustrating the configuration of an image synthesis device according to an embodiment of the present disclosure.

[0017] FIG. 15 is a block diagram illustrating the configuration of a terminal according to an embodiment of the present disclosure.

[0018] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” can refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0019] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0020] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0021] Additionally, terms including ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

[0022] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0023] The term "couple" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not they are in physical contact with each other. The terms "transmit," "receive," and "communicate," as well as their derivatives, encompass both direct and indirect communication. The terms "include," "comprising," and "including," as well as their derivatives, encompass without limitation.

[0024] The term "or" is inclusive and means "and / or." The phrase "associated with" as well as its derivatives means includes, is included within, interconnects with, contains, is contained within, connects to or with, couples to or with, is communicable with, cooperates with, interleaves with, juxtaposes with, is proximate to, is associated with or is connected to, has, has the characteristics of, is related to or is related to, etc.

[0025] Additionally, in the present disclosure, when a component is referred to as being “connected” or “connected” to another component, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0026] At this time, it will be understood that each block of the processing flow diagrams and combinations of the flow diagrams can be performed by computer program instructions. These computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions, when executed by the processor of the computer or other programmable data processing equipment, create a means for performing the functions described in the flow diagram block(s).

[0027] These computer program instructions may also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing device to implement a function in a particular manner, so that the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the function described in the flowchart block(s).

[0028] Since the computer program instructions may be installed on a computer or other programmable data processing device, a series of operational steps may be performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for performing the functions described in the flowchart block(s) may also provide steps for performing the functions described in the flowchart block(s).

[0029] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative implementations, the functions referenced in a block may occur out of order.

[0030] For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on their respective functions.

[0031] For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on their respective functions.

[0032] Here, the term '~ unit' used in this embodiment refers to software or hardware components such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and '~ unit' performs certain roles. However, '~ unit' is not limited to software or hardware. '~ unit' may be configured to be on an addressable storage medium or may be configured to play one or more processors.

[0033] As an example, '~bu' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0034] The functionality provided within the components and '~sub-units' may be combined into a smaller number of components and '~sub-units' or further separated into additional components and '~sub-units'. Furthermore, the components and '~sub-units' may be implemented to activate one or more CPUs within the device or secure multimedia card. Furthermore, in an embodiment, the '~sub-unit' may include one or more processors.

[0035] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, portions irrelevant to the description have been omitted for clarity of explanation, and similar reference numerals have been used throughout the specification to designate similar parts.

[0036] The present disclosure will be described in detail with reference to the attached drawings below.

[0037] The present disclosure is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail herein. However, the various embodiments discussed below, as well as those used to illustrate the principles of the present disclosure in this specification, are merely illustrative and should not be construed as limiting the scope of the present disclosure in any way.

[0038] The video synthesis device (100) may include an MCU (multipoint control unit) server, another device for synthesizing video, at least one terminal among a plurality of terminals, or another device for synthesizing media, but this is only an example and is not limited to the examples mentioned.

[0039] In embodiments of the present disclosure, the terminal may include a display or various types of devices connectable to the display.

[0040] For example, the terminal may include a TV, smart monitor, tablet PC, laptop, digital signage, large display, 360-degree projector, MS (Mobile Station), vehicular, satellite, airborne, cellular phone, smart phone, computer, etc. that includes a display. Alternatively, the terminal may include, but is not limited to, a set-top box or desktop PC that can be connected to a display.

[0041] In the present disclosure, media may include a wide variety of content forms, including, but not limited to, video, audio, images, text, social media, games, virtual reality, web content, and the like, which are merely examples and are not limited to the examples mentioned.

[0042] For convenience of explanation, the present disclosure may be described using video or images as examples, but the present disclosure can also be applied to other forms of media. The images or images of the present disclosure may include a video stream in which video data is continuously transmitted.

[0043] The contents of receiving a video of the present disclosure, obtaining a synthesized video, and transmitting the obtained synthesized video can also be applied to the contents of receiving a video stream, obtaining synthesized media using the stream, and transmitting the synthesized media.

[0044] In this disclosure, "media" may refer to data including at least one of audio or video generated by a user device participating in an online conference during the online conference. Here, the data may include data divided into units such as packets, frames, and segments.

[0045] Here, 'voice' may include data obtained by converting an analog voice signal into a digital signal through a sampling process, and 'image' may include digital data obtained by processing and encoding an analog image as an image signal. In addition, 'voice' and 'image' may be data in a compressed form according to standards such as H.264, MPEG, VP9, ​​and AV1.

[0046] Embodiments of the present disclosure may be applied to at least one service, such as an online meeting or video call. An online meeting or video call involves a technology that enables two or more hardware or software sets to interact by transmitting and receiving media (e.g., at least one of voice and video data) from two or more geographically separated locations. Users can participate in an online meeting or video call using various terminals.

[0047] Users participating in an online meeting can hold a realistic, real-time meeting from geographically distant locations without having to meet in person. An online meeting typically involves two or more user devices exchanging media (e.g., at least one of audio and video data) in real time via a server device and a network. "Online meetings" can be initiated using various platforms, including Zoom, Google Meet, MS Teams, and Cisco Webex.

[0048] For convenience of explanation, the following example assumes a media type requiring synthesis of four. However, this is merely an example and is not limiting. For example, the same approach can be applied to multiple media types requiring synthesis, other than four.

[0049] FIG. 1A and FIG. 1B are drawings for explaining how an image synthesis device according to an embodiment of the present disclosure transmits synthetic media.

[0050] Referring to FIG. 1A, when the image synthesis device (100a) generates composite media corresponding to multiple terminals and transmits the composite media corresponding to each terminal to each terminal, a large amount of hardware resources, software resources, or network resources may be required. When the image synthesis device (100a) generates composite media for each terminal, a large amount of resources may be required.

[0051] When synthesizing a video that includes only the video of a participant corresponding to a specific terminal, individual video synthesis and encoding may be performed for each participant, which may require a lot of HW resources.

[0052] For example, the image synthesis device (100a) can receive multiple media (media 1, media 2, media 3, media 4) from multiple terminals. For example, it can receive media 1 from terminal 1, and media 2 from terminal 2. It can receive media 3 from terminal 3, and media 4 from terminal 4.

[0053] The media corresponding to the terminal may include at least one of media received from the terminal or media requiring overlay on the terminal.

[0054] The video synthesis device (100a) can generate a first composite media by performing synthesis on media other than media 1 corresponding to terminal 1 for terminal 1. The first composite media can include (media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4). The video synthesis device (100a) can transmit the first composite media to terminal 1.

[0055] The image synthesis device (100a) can obtain a second synthesis media by performing synthesis on the media excluding the media 2 corresponding to the terminal 2 for the terminal 2. The second synthesis media can include (media area 3 corresponding to the media 3, media area 4 corresponding to the media 4, and media area 1 corresponding to the media 1). The image synthesis device (100a) can transmit the second synthesis media to the terminal 2.

[0056] The image synthesis device (100a) can obtain a third composite media by performing synthesis on the media other than the media 3 corresponding to the terminal 3. The third composite media can include (media area 4 corresponding to the media 4, media area 1 corresponding to the media 1, and media area 2 corresponding to the media 2). The image synthesis device (100a) can transmit the third composite media to the terminal 3.

[0057] The video synthesis device (100a) can obtain a fourth composite media by performing synthesis on the media other than the media 4 corresponding to the terminal 4 for the terminal 4. The fourth composite media can include (media area 1 corresponding to the media 1, media area 2 corresponding to the media 2, and media area 3 corresponding to the media 3). The video synthesis device (100a) can transmit the fourth composite media to the terminal 3.

[0058] When the video synthesis device (100a) generates a first synthetic media corresponding to terminal 1 and transmits it to terminal 1, generates a second synthetic media corresponding to terminal 2 and transmits it to terminal 2, generates a third synthetic media corresponding to terminal 3 and transmits it to terminal 3, and generates a fourth synthetic media corresponding to terminal 4 and transmits it to terminal 4, hardware resource usage may increase.

[0059] The video synthesis device (100a) may synthesize, render, and encode video streams in a layout optimized for individual users, which may require a lot of hardware resources.

[0060] Referring to FIG. 1b, the image synthesis device (100b) can receive multiple media (media 1, media 2, media 3, media 4) from multiple terminals.

[0061] The video synthesis device (100b) can obtain composite media using multiple media. The video synthesis device (100b) can identify media requiring synthesis. The media requiring synthesis may refer to media received from a terminal participating in an online conference or media received from a terminal participating in a video call.

[0062] For example, if the media received from terminals participating in the conference are media 1, media 2, media 3, and media 4, the media requiring synthesis may correspond to media 1, media 2, media 3, and media 4.

[0063] For example, if the media received from terminals participating in the conference are media 1, media 2, and media 4, the media requiring synthesis may correspond to media 1, media 2, and media 4.

[0064] A composite media may include media areas corresponding to multiple media. The media areas corresponding to multiple media may be arranged, allocated, or configured in the composite media based on a layout.

[0065] In cases where the media that require synthesis in online meetings and video calls are media 1, media 2, media 3, and media 4, the synthesized media may include (media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4).

[0066] A video synthesis device (100b) can generate a single composite media (e.g., a common video stream) for all participants and transmit it to multiple terminals. The video synthesis device (100b) can transmit the composite media not only to terminals that have transmitted media to the video synthesis device (100b), but also to terminals that have not transmitted media to the video synthesis device (100b).

[0067] A terminal receiving synthetic media can adjust the synthetic media (e.g., viewport coordinates). The terminal can display only a portion of the media area corresponding to multiple media in the synthetic media on the screen.

[0068] For example, if the composite media includes (media area 1, media area 2, media area 3, media area 4), terminal 1 can display (media area 2, media area 3, media area 4) on the display.

[0069] According to an embodiment of the present disclosure, it is possible to reduce the hardware resource usage of the image synthesis device (100b) and reduce the cost required for the image synthesis device (100b).

[0070] Meanwhile, when an image synthesized by an image synthesizer (100b) is displayed on a specific terminal, the service quality may be degraded due to a difference between the image synthesized by the image synthesizer (100b) and the image perceived by the user of the specific terminal.

[0071] When receiving synthesized media through a video synthesis device (100b), a difference may occur between the media of the terminal recognized by the user (e.g., media of the terminal currently acquired) and the media of the terminal located in the media area of ​​the synthesized media (e.g., media transmitted by the terminal).

[0072] For example, when terminal 1 transmits media 1 to a video synthesis device (100b), and the video synthesis device (100b) generates synthetic media using media 1 and transmits the synthetic media to the terminal, a time difference may occur between media 1 transmitted to the video synthesis device (100b) located in the synthetic media and media 1 acquired by the terminal from the time of transmission to the video synthesis device (100b) to the present.

[0073] According to an embodiment of the present disclosure, a terminal can obtain output media based on a media area corresponding to the terminal. The terminal can display the output media on a display or transmit the output media to a device including a display.

[0074] The output media may include at least a portion of the composite media. The terminal may adjust the composite media so that the terminal's media included in the composite media is not included or displayed in the output media. The terminal may obtain the output media by performing clipping, cropping, viewport adjustment, or window adjustment on a portion of the composite media to make at least a portion of the composite media visible on the display.

[0075] For example, referring to FIG. 1B, if the composite media includes (media area 1, media area 2, media area 3, media area 4, media area 1, media area 2), terminal 1 can perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media such that the output media includes (media area 2, media area 3, media area 4) of the composite media. Terminal 1 can perform clipping, cropping, viewport adjustment, window adjustment, etc. on the composite media such that the output media does not include (media area 1) of the composite media.

[0076] A terminal can obtain output media by synthesizing or overlaying its media on at least a portion of a composite media. For example, referring to FIG. 1B, terminal 1 can synthesize or overlay terminal 1's media on a region (450a) other than the media region corresponding to terminal 1. Terminal 2 can synthesize or overlay terminal 2's media on a region (450b) other than the media region corresponding to terminal 2. Terminal 3 can synthesize or overlay terminal 3's media on a region (450c) other than the media region corresponding to terminal 3. Terminal 4 can synthesize or overlay terminal 4's media on a region (450d) other than the media region corresponding to terminal 4.

[0077] The terminal may synthesize or overlay the terminal's media in an area other than the media area corresponding to the terminal, but this is only an example, and it is also possible to synthesize or overlay in an area corresponding to the media area corresponding to the terminal, or at least a portion of the synthesized media.

[0078] For example, a terminal receiving synthetic media can synthesize or overlay its own media into a media area corresponding to the terminal.

[0079] After the terminal transmits media to the video synthesis device, the terminal synthesizes or overlays the media acquired by the terminal onto the synthesized media received, thereby reducing the sense of incongruity or the time difference felt by the user, thereby improving the quality of the video call perceived by the user.

[0080] When the time range of media transmitted by a terminal to a video synthesis device (100b) is t1, the media of the terminal that is synthesized or overlaid may represent the media of the terminal corresponding to the time range after t1. For example, the media of the terminal that is synthesized or overlaid may include media captured by the terminal, media generated by the terminal, media received by the terminal from another device, etc. for the time range after t1.

[0081] When the media transmitted by the terminal to the video synthesis device (100b) includes frames up to the time point (t-10), the media of the terminal that is synthesized or overlaid may include frames after the time point (t-10).

[0082] The terminal can mask the media area corresponding to the terminal with a background color or background image.

[0083] FIG. 2 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0084] The video synthesis device (100) can receive multiple media from multiple terminals. For example, the video synthesis device (100) can receive media 1 from terminal 1, media 2 from terminal 2, media 3 from terminal 3, and media 4 from terminal 4.

[0085] The video synthesis device (100) may include an MCU (multipoint control unit) server, another device for synthesizing video, at least one terminal among a plurality of terminals, or another device for synthesizing media, but this is only an example and is not limited to the examples mentioned.

[0086] In embodiments of the present disclosure, the terminal may include a display or may include various types of devices connectable to the display, which are merely examples and are not limited to the examples mentioned.

[0087] For example, the terminal may include a TV, smart monitor, tablet PC, laptop, digital signage, large display, 360-degree projector, MS (Mobile Station), vehicular, satellite, airborne, cellular phone, smart phone, computer, etc. that includes a display. Alternatively, the terminal may include, but is not limited to, a set-top box or desktop PC that can be connected to a display.

[0088] For example, terminal 1 may correspond to a TV, terminal 2 to a vehicle, terminal 3 to a smartphone, and terminal 4 to a set-top box.

[0089] Media can include a wide variety of content forms, including, but not limited to, video, audio, images, text, social media, games, virtual reality, and web content, to name a few.

[0090] For convenience of explanation below, the media may be described as an example of a video containing one or more frames, but the same may be applied to other forms of media. The video of the present disclosure may include a video stream in which video data is continuously transmitted.

[0091] The mixer unit of the video synthesis device (100) can identify the number of media that require synthesis among a plurality of media.

[0092] The number of media that needs to be synthesized may correspond to the number of terminals, accounts, or participants participating in the online meeting.

[0093] Media requiring synthesis may include media received from a terminal participating in an online meeting or media received from a terminal participating in a video call.

[0094] For example, if the media received from terminals participating in the conference are media 1, media 2, media 3, and media 4, the media requiring synthesis correspond to media 1, media 2, media 3, and media 4, and the number of media requiring synthesis can be identified as 4.

[0095] For example, if the media received from terminals participating in the conference are media 1, media 2, and media 4, the media requiring synthesis correspond to media 1, media 2, and media 4, and the number of media requiring synthesis can be identified as 3.

[0096] The video synthesis device (100) (e.g., mixer unit) can set the layout of the synthesis frame based on the number of media.

[0097] The layout may include settings regarding at least one of the resolution of the composite frame, the arrangement of the media area, the position of each media, or the background of the composite frame.

[0098] Settings regarding the background of the composite frame may include settings for the background color of the composite frame, or settings for using a predetermined image as the background, which are just examples and are not limited to the examples mentioned.

[0099] The layout of the media area may include at least one of a single-row layout, a single-column layout, a multiple-row and multiple-column layout, or a rotational layout, but this is only an example and is not limited to the examples mentioned. Some of the content that may correspond to and overlap with the content of FIG. 1b and FIGS. 3 to 11 may be omitted.

[0100] Depending on the arrangement of the media areas in the composite frame, the positions of media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4 can be determined.

[0101] The video synthesis device (100) can set the layout so that the synthesis frame includes a number of media areas greater than the number of media that require synthesis.

[0102] When the number of media requiring synthesis is four, the composite frame may include media areas corresponding to four or more media. For example, the composite frame may include four or more media areas, each of which is composed of (media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4). Each media area may be included one or more times.

[0103] The mixer unit can generate a set of composite frames including one or more composite frames based on multiple media and layouts.

[0104] A composite frame may include media areas corresponding to multiple media. The media areas corresponding to multiple media may be arranged, allocated, or set in the composite media based on a layout.

[0105] For example, referring to FIG. 2, the composite frame may include (media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, media area 4 corresponding to media 4). Referring to FIG. 1B, the composite frame may include (media area 1, media area 2, media area 3, media area 4, media area 1, media area 2).

[0106] The mixer unit can composite, for example, render, copy, place, overlay, or paint, media (e.g., frames) corresponding to each terminal into each media area according to a layout set in a frame buffer for video synthesis.

[0107] For example, Media Area 1 may render Media 1. Media Area 2 may render Media 2. Media Area 3 may render Media 3. Media Area 4 may render Media 4.

[0108] The encoder unit can obtain synthetic media by performing encoding on a set of synthetic frames. The synthetic frames included in the set of synthetic frames can be encoded into data in a video format at a set resolution and frame rate.

[0109] The video synthesis device (100) can transmit synthetic media to a terminal. The video synthesis device (100) can transmit encoded data and synthetic media to the terminal via a stream transmitter. The video synthesis device (100) can perform encryption on the synthetic media and transmit the encrypted synthetic media to the terminal.

[0110] The video synthesis device (100) may transmit metadata of the synthesized media to the terminal. The metadata may include at least one of information indicating an area to be displayed on the display for each terminal, location information of multiple media areas in a synthesis frame, information indicating a media area requiring masking, information indicating a media area to be overlaid, or location information of a media area requiring synthesis in the terminal.

[0111] The video synthesis device (100) can transmit at least one of the synthesis media and metadata not only to the terminal that transmitted the media to the video synthesis device (100) but also to the terminal that did not transmit the media to the video synthesis device (100).

[0112] The terminal can receive synthetic media from the image synthesis device (100). The terminal can receive synthetic media from the image synthesis device (100).

[0113] The terminal can perform decoding on synthetic media and obtain decoded synthetic media and a set of synthetic frames. In this case, if the synthetic media is encrypted, decryption can be performed to obtain the decrypted synthetic media, and decoding can be performed.

[0114] For example, a terminal may receive a synthesized video from a stream receiver and decode it into an image via a video decoder. The terminal may display at least a portion of the image on a screen.

[0115] A terminal can identify a media area corresponding to the terminal among multiple media areas of a composite frame. The media area corresponding to the terminal can be identified using metadata.

[0116] For example, terminal 1 may identify media area 1 (250a) as the media area corresponding to the terminal. Terminal 2 may identify media area 2 (250b) as the media area corresponding to the terminal. Terminal 3 may identify media area 3 (250c) as the media area corresponding to the terminal. Terminal 4 may identify media area 4 (250d) as the media area corresponding to the terminal.

[0117] The terminal can obtain output media based on a media area corresponding to the terminal.

[0118] A terminal may mask a media area corresponding to the terminal with a background color or a background image. For example, terminal 1 may mask media area 1 (250a) of the synthetic media with a background color or a background image. Terminal 2 may mask media area 2 (250b) of the synthetic media with a background color or a background image. Terminal 3 may mask media area 3 (250c) of the synthetic media with a background color or a background image. Terminal 4 may mask media area 4 (250d) of the synthetic media with a background color or a background image.

[0119] A terminal can obtain output media by synthesizing or overlaying its media onto at least a portion of a composite media. While the terminal can synthesize or overlay its media onto a media area corresponding to the terminal, this is merely an example; it is also possible to synthesize or overlay onto an area other than the media area corresponding to the terminal, or onto an area corresponding to at least a portion of the composite media.

[0120] Referring to FIG. 2, terminal 1 can overlay media 1 on media area 1 (250a) corresponding to terminal 1. Terminal 2 can overlay media 2 on media area 2 (250b) corresponding to terminal 2. Terminal 3 can overlay media 3 on media area 3 (250c) corresponding to terminal 3. Terminal 4 can overlay media 4 on media area 4 (250d) corresponding to terminal 4.

[0121] After the terminal transmits media to the video synthesis device, the terminal synthesizes or overlays the media acquired by the terminal onto the synthesized media received, thereby reducing the time difference felt by the user and improving the video call quality perceived by the user.

[0122] When the time range of media transmitted by a terminal to a video synthesis device (100) is t1, the media of the terminal that is synthesized or overlaid may represent the media of the terminal corresponding to the time range after t1. For example, the media of the terminal that is synthesized or overlaid may include media filmed by the terminal, media generated by the terminal, media received by the terminal from another device, etc. for the time range after t1.

[0123] When the media transmitted by the terminal to the video synthesis device (100) includes frames up to the point (t-10), the media of the terminal that is synthesized or overlaid may include frames after the point (t-10).

[0124] The output media displayed on the display may include at least a portion of the composite media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media to set at least a portion of the output media to be displayed. The terminal may display the output media on the display or transmit it to a device including a display.

[0125] FIG. 3 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0126] The contents of Fig. 3 may correspond to the contents of Fig. 2, and some overlapping parts may be omitted for convenience of explanation.

[0127] The video synthesis device (100) receives multiple media from multiple terminals. For example, the video synthesis device (100) may receive media 1 from terminal 1, media 2 from terminal 2, media 3 from terminal 3, and media 4 from terminal 4.

[0128] The media identification unit of the video synthesis device (100) can identify a main media among multiple media. For example, the media identification unit can identify any one of media 1, media 2, media 3, and media 4 as the main media. For example, the main media may include the media of the main speaker.

[0129] The media identification unit can identify the main media based on the media's speech information. For example, if speech occurs in media 1 for a certain period of time or if media 1 has a longer speech duration or greater speech weight than other media, the media identification unit can identify media 1 as the main media.

[0130] When the video synthesis device (100) receives a main media request from a terminal, the media identification unit can identify the media corresponding to the terminal as the main media. For example, when terminal 1 transmits information requesting to be the main speaker, the media identification unit can identify media 1 as the main media.

[0131] The video synthesis device (100) (e.g., mixer unit) can set the layout of the synthesis frame based on the identified main media and the number of media. The video synthesis device (100) can set the layout so that the synthesis frame includes a number of media areas corresponding to media greater than the number of media requiring synthesis.

[0132] A composite frame may include media areas corresponding to multiple media. The media areas corresponding to multiple media may be arranged, allocated, or set based on a layout in the composite media.

[0133] The mixer unit can composite, for example, render, copy, arrange, overlay, or paint, each media (e.g., frame) corresponding to each terminal into each media area according to the layout set in the frame buffer for video synthesis.

[0134] The mixer unit can generate a composite frame so that the main media area corresponding to the main media is larger than other media areas, or so that the border of the main media area is emphasized.

[0135] For example, referring to FIG. 3, a composite frame can be generated such that the main media area corresponding to the main media (media 1) is larger than other media areas 2, 3, or 4.

[0136] The encoder unit can obtain synthetic media by performing encoding on a synthetic frame set. For example, the synthetic frames included in the synthetic frame set can be encoded into video format data at a set resolution and frame rate. The video synthesis device (100) can transmit the synthetic media to multiple terminals.

[0137] The video synthesis device (100) can transmit composite media to multiple terminals. The video synthesis device (100) can transmit encoded data and composite media to the terminals via a stream transmitter. The video synthesis device (100) can encrypt the composite media and transmit the encrypted composite media to multiple terminals.

[0138] The video synthesis device (100) may transmit metadata of the synthesized media to the terminal. The metadata may include at least one of information indicating an area to be displayed on the display for each terminal, location information of multiple local media areas in the synthesis frame, information indicating a media area requiring masking, information indicating a media area to be overlaid, or location information of a local media area requiring synthesis for each terminal.

[0139] The video synthesis device (100) can transmit at least one of the synthesis media and metadata not only to the terminal that transmitted the media to the video synthesis device (100) but also to the terminal that did not transmit the media to the video synthesis device (100).

[0140] The terminal can receive synthetic media from an image synthesis device (100). The terminal can receive encoded synthetic media from the image synthesis device (100). The terminal can perform decoding on the synthetic media and obtain decoded synthetic media and a synthetic frame set. The synthetic frame set can include one or more synthetic frames. The synthetic frame can include media areas corresponding to multiple media.

[0141] The terminal can identify a media region corresponding to the terminal among multiple media regions of the composite frame. The media region corresponding to the first local terminal can be identified using metadata. The terminal can obtain output media based on the media region corresponding to the terminal.

[0142] The terminal may mask the media area corresponding to the terminal with a background color or background image. The terminal may obtain output media by synthesizing or overlaying the terminal's media onto at least a portion of the synthesized media.

[0143] The terminal can improve the quality of video calls perceived by the user by reducing the time difference felt by the user by overlaying the acquired media on the received synthesized media after transmitting it to the video synthesis device.

[0144] When the time range of media transmitted by a terminal to a video synthesis device (100) is t1, the media of the terminal that is synthesized or overlaid may represent the media of the terminal corresponding to the time range after t1. For example, the media of the terminal that is synthesized or overlaid may include media filmed by the terminal, media generated by the terminal, media received by the terminal from another device, etc. for the time range after t1.

[0145] When the media transmitted by the terminal to the video synthesis device (100) includes frames up to the point (t-10), the media of the terminal that is synthesized or overlaid may include frames after the point (t-10).

[0146] The output media displayed on the display may include at least a portion of the composite media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media to set at least a portion of the output media to be displayed.

[0147] The terminal can display the output media on a display or transmit it to a device that includes a display.

[0148] FIG. 4 is a diagram illustrating an image synthesis device according to an embodiment of the present disclosure obtaining a synthetic media.

[0149] The contents of Fig. 4 may correspond to the contents of Figs. 2 and 3, and some overlapping parts may be omitted for convenience of explanation.

[0150] The video synthesis device (100) can receive multiple media from multiple terminals. For example, the video synthesis device (100) can receive media 1 from terminal 1, media 2 from terminal 2, media 3 from terminal 3, and media 4 from terminal 4.

[0151] The mixer unit of the video synthesis device (100) can identify the number of media that require synthesis among a plurality of media. The video synthesis device (100) can set the layout so that the synthesis frame includes a number of media areas corresponding to a number greater than the number of media that require synthesis.

[0152] The video synthesis device (100) (e.g., media identification unit) can identify a main media among a plurality of media. The video synthesis device (100) (e.g., media identification unit) can identify the main media based on speech information of the media.

[0153] When the video synthesis device (100) receives a main media request from a terminal, the video synthesis device (100) (e.g., media identification unit) can identify the media corresponding to the terminal as the main media.

[0154] The video synthesis device (100) (e.g., mixer unit) can set the layout of the composite frame based on at least one of the identified main media and the number of media. The composite frame can include media areas corresponding to multiple media. The media areas corresponding to multiple media can be arranged, allocated, or set based on the layout in the composite media.

[0155] The mixer unit may generate a composite frame such that the main media area corresponding to the main media is larger in size than the media areas corresponding to other media, or such that the border of the main media area corresponding to the main media is emphasized. For example, the mixer unit may generate a composite frame such that media area 1 corresponding to media 1 is larger than the areas corresponding to other media.

[0156] The mixer unit can composite, for example, render, copy, arrange, overlay, or paint, each media (e.g., frame) corresponding to each terminal into each media area according to a layout set in a frame buffer for video synthesis.

[0157] Referring to FIG. 4, a composite frame can be generated to include (media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, media area 4 corresponding to media 4, media area 1 corresponding to media 1, media area 2 corresponding to media 2).

[0158] The encoder unit can obtain synthetic media by performing encoding on a synthetic frame set. For example, the synthetic frames included in the synthetic frame set can be encoded into video format data at a set resolution and frame rate. The video synthesis device (100) can transmit the synthetic media to multiple terminals.

[0159] The video synthesis device (100) can obtain multiple synthetic media by performing encoding on at least a portion of a plurality of media areas. The device can transmit synthetic media corresponding to at least one of the plurality of synthetic media to at least one of the plurality of terminals.

[0160] The encoder unit performs encoding on an area including (media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4) of the composite frame, and the image synthesis device (100) can transmit the encoded data to terminal 1.

[0161] The encoder unit performs encoding on an area including (media area 3 corresponding to media 3, media area 4 corresponding to media 4, and media area 1 corresponding to media 1) of the composite frame, and the image synthesis device (100) can transmit the encoded data to terminal 2.

[0162] The encoder unit performs encoding on an area including (media area 4 corresponding to media 4, media area 1 corresponding to media 1, and media area 2 corresponding to media 2) of the composite frame, and the image synthesis device (100) can transmit the encoded data to terminal 3.

[0163] The encoder unit performs encoding on an area including (media area 1 corresponding to media 1, media area 2 corresponding to media 2, and media area 3 corresponding to media 3) of the composite frame, and the image synthesis device (100) can transmit the encoded data to terminal 4.

[0164] The video synthesis device (100) can transmit composite media to multiple terminals. The video synthesis device (100) can transmit encoded data and composite media to the terminals via a stream transmitter. The video synthesis device (100) can encrypt the composite media and transmit the encrypted composite media to multiple terminals.

[0165] The video synthesis device (100) may transmit metadata of the synthesized media to the terminal. The metadata may include at least one of information indicating an area to be displayed on the display for each terminal, location information of multiple local media areas in the synthesis frame, information indicating a media area requiring masking, information indicating a media area to be overlaid, or location information of a local media area requiring synthesis for each terminal.

[0166] The video synthesis device (100) can transmit synthesized media and metadata not only to a terminal that transmitted media to the video synthesis device (100) but also to a terminal that did not transmit media to the video synthesis device (100).

[0167] The terminal can receive synthetic media from an image synthesis device (100). The terminal can receive encoded synthetic media from the image synthesis device (100). The terminal can perform decoding on the synthetic media and obtain decoded synthetic media and a synthetic frame set. The synthetic frame set can include one or more synthetic frames. The synthetic frame can include media areas corresponding to multiple media.

[0168] The terminal can obtain output media by overlaying the terminal's media on at least a portion of the composite media.

[0169] When the media transmitted by the terminal to the video synthesis device (100) includes frames up to the point (t-10), the media of the terminal that is synthesized or overlaid may include frames after the point (t-10).

[0170] When the time range of media transmitted by a terminal to a video synthesis device (100) is t1, the media of the terminal that is synthesized or overlaid may represent the media of the terminal corresponding to the time range after t1. For example, the media of the terminal that is synthesized or overlaid may include media filmed by the terminal, media generated by the terminal, media received by the terminal from another device, etc. for the time range after t1.

[0171] After the terminal transmits media to the video synthesis device, the terminal synthesizes or overlays the media acquired by the terminal onto the synthesized media received, thereby reducing the time difference felt by the user and improving the video call quality perceived by the user.

[0172] The output media displayed on the display may include at least a portion of the composite media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media to set at least a portion of the output media to be displayed.

[0173] The terminal can display the output media on a display or transmit it to a device that includes a display.

[0174] FIG. 5 is a diagram illustrating a video synthesis device and a terminal transmitting and receiving media according to an embodiment of the present disclosure.

[0175] The terminal may transmit media to the video synthesis device (100) using a transceiver. For example, the terminal may transmit media to the video synthesis device (100) using a technology for peer-to-peer communication (e.g., WebRTC) or other communication technology, but this is merely an example and is not limited to the examples mentioned.

[0176] The terminal can encode media at a resolution and frame rate set by the encoder. For example, when transmitting video, the terminal can convert (encode) multiple images acquired using a camera into video format data by the encoder and transmit them to the video synthesis device (100) via the stream transmitter.

[0177] The terminal may perform encryption and transmit the encrypted media to the video synthesis device (100), but this is only an example, and the encryption may be performed in a device other than the terminal.

[0178] The video synthesis device (100) can receive multiple media from multiple terminals via a network. For example, the video synthesis device (100) can receive media 1 from terminal 1 and media 2 from terminal 2, respectively.

[0179] The video synthesis device (100) can perform decoding on media. For example, the video synthesis device (100) can receive a video from a terminal as a stream from a stream receiver and decode it into image frames through a video decoder.

[0180] If the media is encrypted, the video synthesis device (100) can perform decryption to obtain media from the encrypted media. The obtained media can be stored or managed in a frame buffer.

[0181] The video synthesis device (100) can obtain information on the resolution of images supported by the terminal. The video synthesis device (100) can identify the number of media that require synthesis. The number of media may include the number of video sources.

[0182] A layout for synthesizing an image can be set based on the number of media that require synthesizing. For example, the image synthesizing device (100) can set at least one of the number of media, background color, background image, border size, output resolution, position of a media area corresponding to each media, or position of a fixed media area.

[0183] The frame buffer size of the synthesized image may correspond to the resolution of the synthesized image. If the size of the image frame of the terminal does not match the size of the output area of ​​the synthesized image, the image synthesis device (100) may adjust the image size to match the size of the output area.

[0184] The video synthesis device (100) can generate a single composite frame set including one or more composite frames based on a plurality of media and layouts. Media areas corresponding to the plurality of media can be arranged, allocated, or set in the composite media based on the layout.

[0185] A composite frame may include media areas corresponding to multiple media. The video compositing device (100) may composite, for example, render, copy, batch overlay, or paint, each media in the media areas corresponding to the multiple media. For example, media 1 may be rendered in media area 1 corresponding to media 1, and media 3 may be rendered in media area 3 corresponding to media 2.

[0186] The video synthesis device (100) can obtain synthetic media by performing encoding on a set of synthetic frames. The video synthesis device (100) can transmit the synthetic media to a terminal.

[0187] For example, a composite frame included in a composite frame set can be encoded into video format data with a resolution and frame rate set through a video encoder, and the video synthesis device (100) can transmit the encoded data to a terminal through a stream transmitter. In this case, encryption can be performed on the encoded data.

[0188] The video synthesis device (100) can transmit synthesized media not only to a terminal that transmitted media to the video synthesis device (100), but also to a terminal that did not transmit media to the video synthesis device (100).

[0189] The video synthesis device (100) may transmit metadata of the synthesized media to the terminal. The metadata may include at least one of information indicating an area to be displayed on the display for each terminal, location information of multiple local media areas in the synthesis frame, information indicating a media area requiring masking, information indicating a media area to be overlaid, or location information of a local media area requiring synthesis for each terminal.

[0190] The terminal can receive synthetic media from the image synthesis device (100). The terminal can additionally receive metadata of the synthetic media. The terminal can obtain decoded data by performing decoding on the synthetic media. In this case, if the synthetic media is encrypted, decryption can be performed to obtain decrypted synthetic media, and then decoding can be performed.

[0191] The terminal may display the composite media on a display or transmit the composite media to a device including a display. The output media displayed on the display may include at least a portion of the composite media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media to display at least a portion of the composite media.

[0192] Figures 6, 7, and 8 are diagrams illustrating a terminal processing synthetic media.

[0193] The terminal can receive synthetic media from an image synthesis device (100).

[0194] The terminal can perform decoding on synthetic media and obtain decoded synthetic media and a set of synthetic frames. The set of synthetic frames can include one or more synthetic frames. The synthetic frame can include media regions corresponding to multiple media.

[0195] For example, a terminal may receive a synthesized video from a stream receiver and decode it into an image via a video decoder. The terminal may display at least a portion of the image on a screen. The terminal may obtain decoded data by performing decoding on the synthesized media. In this case, if the synthesized media is encrypted, decryption may be performed to obtain decrypted synthesized media, and decoding may be performed.

[0196] A terminal can identify a media region corresponding to the terminal among multiple media regions of a composite frame. For example, terminal 1 can identify media region 1 (250a) corresponding to media (media 1) of terminal 1 as a media region corresponding to terminal 1. Terminal 2 can identify media region 2 (250b) corresponding to media (media 2) of terminal 2 as a media region corresponding to terminal 2.

[0197] Terminal 3 can identify media area 3 (250c) corresponding to media (media 3) of terminal 3 as a media area corresponding to terminal 3. Terminal 4 can identify media area 4 (250d) corresponding to media (media 4) of terminal 4 as a media area corresponding to terminal 4.

[0198] The terminal can obtain output media based on a media area corresponding to the terminal.

[0199] The terminal can obtain output media by overlaying its own media on at least a portion of the composite media. After the terminal transmits media to the video synthesis device, the terminal synthesizes or overlays the media obtained by the terminal onto the received composite media. This reduces the time difference perceived by the user, thereby improving the user's perceived video call quality.

[0200] A terminal can overlay its media on a media area corresponding to the terminal of the composite media. For example, referring to FIG. 6, terminal 1 can overlay terminal 1's media 1 on media area 1 (250a). Terminal 2 can overlay terminal 2's media 2 on media area 2 (250b). Terminal 3 can overlay terminal 3's media 3 on media area 3 (250c). Terminal 4 can overlay terminal 4's media 4 on media area 3 (250d).

[0201] A terminal can overlay its media in a region other than the media region corresponding to the terminal. For example, referring to FIG. 7, terminal 1 can overlay terminal 1's media 1 in a region other than the media region corresponding to the terminal (450a). Terminal 2 can overlay terminal 2's media 2 in a region other than the media region corresponding to the terminal (450b). Terminal 3 can overlay terminal 3's media 3 in a region other than the media region corresponding to the terminal (450c). Terminal 4 can overlay terminal 4's media 4 in a region other than the media region corresponding to the terminal (450d).

[0202] The terminal can mask the media area corresponding to the terminal with a background color or background image.

[0203] For example, referring to FIG. 7, terminal 1 can mask media area 1 (250a) corresponding to terminal 1 of the synthetic media with a background color or a background image. Terminal 2 can mask media area 2 (250b) corresponding to terminal 2 of the synthetic media with a background color or a background image. Terminal 3 can mask media area 3 (250c) corresponding to terminal 3 of the synthetic media with a background color or a background image. Terminal 4 can mask media area 4 (250d) corresponding to terminal 4 of the synthetic media with a background color or a background image.

[0204] The output media displayed on the display may include at least a portion of the composite media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the composite media to display at least a portion of the composite media.

[0205] For example, referring to FIG. 8, if the media corresponding to terminal 1 is media 1, a window can be set in at least a portion of the composite media (550) to include (media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4). The media of terminal 1 can be overlaid on a portion of the composite media.

[0206] If the media corresponding to terminal 2 is media 2, terminal 3 can set a window in at least a portion (550) of the composite media so that the output media includes (media area 1 corresponding to media 1, media area 3 corresponding to media 3, and media area 4 corresponding to media 4). The media of terminal 2 can be overlaid on a portion of the composite media.

[0207] If the media corresponding to terminal 3 is media 3, terminal 3 can set a window in at least a portion (550) of the composite media so that the output media includes (media area 4 corresponding to media 4, media area 1 corresponding to media 1, and media area 2 corresponding to media 2). The media of terminal 3 can be overlaid on a portion of the composite media.

[0208] If the media corresponding to terminal 4 is media 4, terminal 4 can set a window in at least a portion (550) of the composite media so that the output media includes (media area 1 corresponding to media 1, media area 2 corresponding to media 2, and media area 3 corresponding to media 3). The media of terminal 4 can be overlaid on a portion of the composite media.

[0209] The terminal can display the output media on a display or transmit it to a device that includes a display.

[0210] FIGS. 9, 10, and 11 are diagrams illustrating synthetic media according to embodiments of the present disclosure.

[0211] The video synthesis device (100) can obtain synthetic media by performing encoding on a synthetic frame set. The synthetic frame set can include one or more synthetic frames.

[0212] The composite frames included in the composite frame set can be encoded as data in a video format with a set resolution and frame rate. The video synthesis device (100) can transmit the composite media to a terminal.

[0213] A composite frame may include media areas corresponding to multiple media. The media areas corresponding to multiple media may be arranged, allocated, or set based on a layout in the composite media.

[0214] The video synthesis device (100) (e.g., mixer unit) can set the layout of the synthesis frame based on at least one of the identified main media and the number of media. The video synthesis device (100) can set the layout so that the synthesis frame includes a number of media areas corresponding to media greater than the number of media requiring synthesis.

[0215] The video synthesis device (100) can generate a synthesis frame in which the main media area corresponding to the main media is larger in size than the media area corresponding to other media, or the border of the main media area corresponding to the main media is emphasized.

[0216] The layout may include settings regarding at least one of the resolution of the composite frame, the arrangement of the media area, the position of each media, or the background of the composite frame.

[0217] Settings regarding the background of the composite frame may include settings for the background color of the composite frame, or settings for using a predetermined image as the background, which are just examples and are not limited to the examples mentioned.

[0218] The layout of the media area may include at least one of a single-row layout, a single-column layout, a multiple-row and multiple-column layout, or a rotational layout, but these are only examples and are not limited to the examples mentioned. For example, the layout may correspond to FIGS. 1B to 11, and some overlapping content may be omitted.

[0219] The video synthesis device (100) can synthesize, for example, render, copy, arrange, overlay, or paint each media in a predetermined area according to the layout set in the frame buffer for video synthesis, for each image frame corresponding to each terminal.

[0220] The video synthesis device (100) can generate composite media according to the arrangement of multiple rows and multiple columns of media of the terminal. For example, referring to FIG. 9, the media of the terminal can be repeatedly arranged so that there is no overlapping media in a 2X2 area of ​​the composite media.

[0221] The output media may be displayed to include a portion (550) of the composite media. Referring to FIG. 9, the terminal may perform cropping or adjust the window for a portion (550) of the composite media to display it on the display. In this case, portions of the composite media other than the portion (550) may not be displayed or may be masked.

[0222] The video synthesis device (100) can generate composite media according to the rotational arrangement of the media of the terminal. For example, referring to FIG. 10, the video synthesis device (100) can generate composite media by arranging media 3 at the 12 o'clock direction, arranging media 2 at the 3 o'clock direction by rotating it 90 degrees clockwise with respect to media 3, arranging media 1 at the 6 o'clock direction by rotating it 180 degrees with respect to media 3, and arranging media 4 at the 9 o'clock direction by rotating it 270 degrees with respect to media 3.

[0223] The terminal can identify a media area corresponding to the terminal. By rotating the synthetic media and overlaying the terminal's media on the media area corresponding to the terminal, the terminal can obtain the output media.

[0224] For example, terminal 1 can obtain output media by overlaying the media (media 1) of terminal 1 on the media area 1 (250a) corresponding to terminal 1. Terminal 2 can obtain output media by rotating the composite media by 90 degrees and overlaying the media (media 2) of terminal 2 on the media area 2 (250b) corresponding to terminal 2.

[0225] Terminal 3 can obtain the output media by rotating the synthetic media by 180 degrees and overlaying the media (media 3) of terminal 3 on the media area 3 (250c) corresponding to terminal 3. Terminal 4 can obtain the output media by rotating the synthetic media by 270 degrees and overlaying the media (media 4) of terminal 4 on the media area 4 (250d) corresponding to terminal 4.

[0226] The video synthesis device (100) can generate composite media according to a single-row arrangement of media. For example, referring to FIG. 11, media can be arranged sequentially in a single row without overlapping. For example, referring to FIG. 8, media can be repeatedly arranged in three adjacent columns in a single row without overlapping.

[0227] FIG. 12 is a flowchart illustrating a method for transmitting synthetic media by an image synthesis device according to an embodiment of the present disclosure.

[0228] In step S1210, the image synthesis device receives multiple media from multiple terminals.

[0229] The video synthesis device (100) may include an MCU (multipoint control unit) server, another device for synthesizing video, at least one terminal among a plurality of terminals, or another device for synthesizing media, but this is only an example and is not limited to the examples mentioned.

[0230] In embodiments of the present disclosure, the terminal may include a display or may include various types of devices connectable to the display, which are merely examples and are not limited to the examples mentioned.

[0231] For example, the terminal may include a TV, smart monitor, tablet PC, laptop, digital signage, large display, 360-degree projector, MS (Mobile Station), vehicular, satellite, airborne, cellular phone, smart phone, computer, etc. that includes a display. Alternatively, the terminal may include, but is not limited to, a set-top box or desktop PC that can be connected to a display.

[0232] Media can include a wide variety of content forms, including, but not limited to, video, audio, images, text, social media, games, virtual reality, and web content, to name a few.

[0233] For convenience of explanation, the media may be described as a video containing one or more frames, but the same may be applied to other forms of media. For example, the media or video of the present disclosure may include a video stream in which video data is continuously transmitted.

[0234] A video synthesis device can receive media (e.g., video streams) from multiple terminals (e.g., terminal 1, terminal 2, ..., terminal 3). The video synthesis device can perform decoding on the media. The video synthesis device can receive video from the terminals as a stream from a stream receiver and decode it into image frames through a video decoder.

[0235] In step S1220, the image synthesis device can identify the number of media that require synthesis among a plurality of media.

[0236] The number of media that needs to be synthesized may correspond to the number of terminals, accounts, or participants participating in the online meeting.

[0237] A video synthesis device can identify media requiring synthesis. Media requiring synthesis may refer to media received from a terminal participating in an online conference or media received from a terminal participating in a video call.

[0238] For example, if the media received from terminals participating in the conference are media 1, media 2, media 3, and media 4, the media requiring synthesis correspond to media 1, media 2, media 3, and media 4, and the number of media requiring synthesis can be identified as 4.

[0239] For example, if the media received from terminals participating in the conference are media 1, media 2, and media 4, the media requiring synthesis correspond to media 1, media 2, and media 4, and the number of media requiring synthesis can be identified as 3.

[0240] In step S1230, the image synthesis device can set the layout of the synthesis frame based on the number of media.

[0241] The number of media required for synthesis may correspond to the number of terminals, accounts, or participants participating in the online conference. A synthesis frame may include media areas corresponding to multiple media.

[0242] The video synthesis device can set the layout of the composite frame based on the identified main media and the number of media. The composite frame can include media areas corresponding to multiple media.

[0243] The video synthesis device can configure the layout so that the composite frame includes a number of media areas corresponding to a number greater than or equal to the number of media to be synthesized. For example, if the number of media to be synthesized is four, the composite frame can include media areas corresponding to four or more media.

[0244] Media areas corresponding to multiple media can be arranged, allocated or set in the composite media based on the layout.

[0245] The layout may include settings regarding at least one of the resolution of the composite frame, the arrangement of the media area, the position of each media, or the background of the composite frame.

[0246] Settings regarding the background of the composite frame may include settings for the background color of the composite frame, or settings for using a predetermined image as the background, which are just examples and are not limited to the examples mentioned.

[0247] The layout of the media area may include at least one of a single row layout, a single column layout, a multiple row and multiple column layout, or a rotational layout, but this is only an example and is not limited to the examples mentioned. It may correspond to the layout of FIGS. 1b to 11.

[0248] The video synthesis device can identify a main media among multiple media. For example, it can identify any one of Media 1, Media 2, Media 3, or Media 4 as the main media. The main media can refer to the video of the main speaker.

[0249] The video synthesis device can identify the main media based on the speech information of the media. For example, if speech occurs in Media 1 or has a longer speech duration or greater proportion than other media, Media 1 can be identified as the main media.

[0250] When a video synthesis device receives a main media request from a terminal, the media corresponding to the terminal can be identified as the main media.

[0251] The main media area corresponding to the main media may be larger in size than the media area corresponding to other media, or the video synthesis device may generate a composite frame to emphasize the border of the main media area corresponding to the main media.

[0252] In step S1240, the video synthesis device can generate one set of synthesis frames including one or more synthesis frames based on a plurality of media and layouts.

[0253] A composite frame may include media areas corresponding to multiple media. The media areas corresponding to multiple media may be arranged, allocated, or set in the composite media based on a layout.

[0254] The video synthesis device can synthesize, for example, render, copy, arrange, overlay, or paint, each media (e.g., image frame) corresponding to each terminal into each media area according to a layout set in a frame buffer for video synthesis. For example, media 1 can be rendered in media area 1 corresponding to media 1, and media 2 can be rendered in media area 2 corresponding to media 2.

[0255] The frame buffer size of a composite image can correspond to the resolution of the composite image. If the size of the terminal's image frame does not match the size of the output area of ​​the composite image, the image synthesis device can adjust the image size to match the size of the output area.

[0256] In step S1250, the video synthesis device can transmit a set of synthesis frames to multiple terminals.

[0257] A video synthesis device can obtain synthetic media by performing encoding on a set of synthetic frames. The video synthesis device can transmit the synthetic media to multiple terminals. The video synthesis device can perform encryption on the synthetic media and transmit the encrypted synthetic media to the multiple terminals.

[0258] For example, a composite frame included in a composite frame set can be encoded into data in a video format with a resolution and frame rate set through a video encoder, and a video synthesis device can transmit the encoded data, the composite media, to a terminal through a stream transmitter.

[0259] A video synthesis device can obtain a plurality of composite media by performing encoding on at least a portion of a plurality of media areas. The video synthesis device can transmit, to terminal 1 among the plurality of terminals, a composite media corresponding to terminal 1 among the plurality of composite media.

[0260] For example, if the media corresponding to terminal 1 is media 1, and the composite frame includes (media area 1 corresponding to media 1, media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4), the video synthesis device can perform encoding on an area including (media area 2 corresponding to media 2, media area 3 corresponding to media 3, and media area 4 corresponding to media 4).

[0261] The video synthesis device can transmit encoded data to terminal 1. Terminal 1 can obtain output media by performing decoding on the synthetic media and overlaying the media 1 of terminal 1 on the synthetic media, and display the output media on a display.

[0262] The video synthesis device may transmit metadata of the synthesized media to the terminal. The metadata may include at least one of: information indicating an area to be displayed on the display for each terminal; location information of multiple local media areas in the synthesized frame; information indicating a media area requiring masking; information indicating a media area to be overlaid; or location information of a local media area requiring synthesis for each terminal.

[0263] The video synthesis device may transmit at least one of the synthesized media and metadata to a terminal that did not transmit the media to the video synthesis device, as well as to a terminal that transmitted the media to the video synthesis device.

[0264] The contents of Fig. 12 may correspond to the contents of Figs. 2 to 11, and some overlapping parts may be omitted.

[0265] FIG. 13 is a flowchart of a method for a terminal to receive synthetic media according to an embodiment of the present disclosure.

[0266] In step S1310, the terminal can transmit media to an external electronic device that synthesizes media.

[0267] The external electronic device may include a video synthesis device. The video synthesis device may include a multipoint control unit (MCU) server, at least one terminal, another device that synthesizes video, or a device that synthesizes media, but these are only examples and are not limited to the examples mentioned.

[0268] The terminal may include a display or may include various types of devices that can be connected to a display, and these are only examples and are not limited to the examples mentioned.

[0269] For example, the terminal may include a TV, smart monitor, tablet PC, laptop, digital signage, large display, 360-degree projector, MS (Mobile Station), vehicular, satellite, airborne, cellular phone, smart phone, computer, etc. that includes a display. Alternatively, the terminal may include, but is not limited to, a set-top box or desktop PC that can be connected to a display.

[0270] Media can include a wide variety of content forms, including, but not limited to, video, audio, images, text, social media, games, virtual reality, and web content, to name a few.

[0271] For convenience of explanation below, the media may be described as an example of a video containing one or more frames, but the same may be applied to other forms of media. The video of the present disclosure may include a video stream in which video data is continuously transmitted.

[0272] Multiple terminals (e.g., terminal 1, terminal 2, ..., terminal 3) can perform encoding and transmit media to a video synthesis device. The terminals can transmit media to the video synthesis device using a technology for peer-to-peer communication (e.g., WebRTC) or other communication technology.

[0273] For example, when a terminal transmits a video, multiple images acquired using a camera can be converted (encoded) into video format data by an encoder and transmitted to a video synthesis device via a stream transmitter.

[0274] In step S1320, the terminal may receive, from an external electronic device, one set of composite frames including one or more composite frames.

[0275] A terminal may receive synthetic media from an image synthesis device. The terminal may receive encoded synthetic media from the image synthesis device. The terminal may decode the synthetic media and obtain decoded synthetic media and a synthetic frame set. The synthetic frame set may include one or more synthetic frames. The synthetic frame may include media regions corresponding to multiple media.

[0276] For example, a terminal may receive a synthesized video from a stream receiver and decode it into an image via a video decoder. The terminal may display at least a portion of the image on a screen.

[0277] The terminal can additionally receive metadata for the synthetic media. The terminal can obtain decoded data by decoding the synthetic media. In this case, if the synthetic media is encrypted, decryption can be performed to obtain the decrypted synthetic media, and then decoding can be performed.

[0278] In step S1330, the terminal can identify a media area corresponding to the terminal among multiple media areas of the composite frame.

[0279] Using metadata, a media area corresponding to a terminal can be identified. For example, terminal 1 can identify media area 1 corresponding to terminal 1's media (Media 1) as the media area corresponding to the terminal. Terminal 2 can identify media area 2 corresponding to terminal 2's media (Media 2) as the media area corresponding to the terminal.

[0280] In step S1330, output media can be acquired based on a media area corresponding to the terminal.

[0281] The terminal may mask a media area corresponding to the terminal with a background color or a background image. The terminal may obtain output media by synthesizing or overlaying the terminal's media on at least a portion of the synthesized media. When the media transmitted by the terminal to the image synthesis device includes frames up to time (t-10), the synthesized or overlaid media of the terminal may include frames after time (t-10).

[0282] When the time domain of media transmitted by a terminal to a video synthesis device is t1, the media of the terminal that is synthesized or overlaid may represent the media of the terminal corresponding to the time domain after t1. For example, the media of the terminal that is synthesized or overlaid may include media filmed by the terminal, media generated by the terminal, media received by the terminal from another device, etc. for the time domain after t1. The output media displayed on the display may include at least a portion of the synthesized media. For example, the terminal may perform clipping, cropping, viewport adjustment, window adjustment, etc. on a portion of the synthesized media to set at least a portion of the output media to be displayed.

[0283] In step S1340, the terminal can display the acquired output media on the display.

[0284] The terminal can display the output media on a display or transmit it to a device that includes a display.

[0285] The contents of Fig. 13 may correspond to the contents of Figs. 2 to 11, and some overlapping parts may be omitted.

[0286] According to an embodiment of the present disclosure, a method of an electronic device for synthesizing media may be provided. The method may include receiving a plurality of media, each including one or more frames, from a plurality of terminals. The method may include identifying a number of media requiring synthesis among the plurality of media. The method may include setting a layout of a synthesized frame based on the number of media. The method may include generating a set of synthesized frames, each set including one or more synthesized frames, based on the plurality of media and the layout. The method may include transmitting the set of synthesized frames to the plurality of terminals.

[0287] The step of setting the layout may include the step of identifying a main media among a plurality of media and the step of setting the layout of the composite frame based on the identified main media and the number of media.

[0288] The step of transmitting the composite frame set may include the step of obtaining composite media by performing encoding on the composite frame set and the step of transmitting the composite media to a plurality of terminals.

[0289] The step of transmitting a set of composite frames may include the step of obtaining a plurality of composite media by performing encoding on at least a portion of a plurality of media areas, and the step of transmitting, to a first terminal among the plurality of terminals, a composite media corresponding to the first terminal among the plurality of composite media.

[0290] The method may include a step of transmitting at least one of location information of a plurality of media areas, or location information of a media area required for each terminal.

[0291] The layout may include settings regarding at least one of the resolution of the composite frame, the layout of the media area, or the background of the composite frame.

[0292] The layout form of the media area may include at least one of a single row layout form, a single column layout form, a multiple row and multiple column layout form, or a rotational layout form.

[0293] The step of transmitting the synthetic media may include the step of performing encryption on the synthetic media and transmitting the encrypted synthetic media to a plurality of terminals.

[0294] According to an embodiment of the present disclosure, a method of a terminal receiving synthesized media may be provided. The method may include transmitting media to an external electronic device that synthesizes the media. The method may include receiving, from the external electronic device, a set of synthesized frames including one or more synthesized frames, wherein the synthesized frames include a plurality of media regions.

[0295] The method may include a step of identifying a media region corresponding to a terminal among a plurality of media regions. The method may include a step of obtaining output media based on the media region corresponding to the terminal. The method may include a step of displaying the obtained output media.

[0296] The step of obtaining the output media may include the step of obtaining the output media by placing the media of the terminal in a media area corresponding to the terminal of the composite frame set.

[0297] The step of obtaining the output media may include the step of masking a media area corresponding to a terminal of the composite frame set and the step of obtaining the output media by placing the media of the terminal in at least a portion of the composite frame set.

[0298] The step of receiving one set of synthetic frames may include the step of receiving, from an external electronic device, synthetic media obtained by performing encoding on the set of synthetic frames, and the step of obtaining one set of synthetic frames by performing decoding on the synthetic media.

[0299] The step of receiving the synthetic media may include the step of receiving the encrypted synthetic media from an external electronic device and the step of performing decryption on the encrypted synthetic media.

[0300] It may further include a step of receiving at least one of location information of multiple media areas or location information of a media area required for each terminal.

[0301] A media area corresponding to a terminal can be identified based on at least one of location information of multiple media areas or location information of a media area required for each terminal.

[0302] According to an embodiment of the present disclosure, an electronic device for synthesizing media may be provided. The electronic device for synthesizing media may include a transceiver; a memory storing at least one instruction; and at least one processor connected to the transceiver. The electronic device for synthesizing media may correspond to the image synthesizing device of the present disclosure.

[0303] The at least one processor may execute the at least one instruction to receive a plurality of media including one or more frames from a plurality of terminals.

[0304] The at least one processor may execute the at least one instruction to identify the number of media that require synthesis.

[0305] The at least one processor may execute the at least one instruction to set a layout of the composite frame based on the number of media.

[0306] The at least one processor may execute the at least one instruction to generate a composite frame set including one or more composite frames based on the plurality of media and the layout. The composite frame may include a plurality of media areas corresponding to the number of media.

[0307] The at least one processor may execute the at least one instruction to transmit the composite frame set to the plurality of terminals.

[0308] According to an embodiment of the present disclosure, an electronic device for synthesizing media may be provided. The electronic device for synthesizing media may include a transceiver; a memory storing at least one instruction; and at least one processor connected to the transceiver.

[0309] According to an embodiment of the present disclosure, a terminal for receiving synthetic media may be provided. The terminal for receiving synthetic media may include a transceiver; a memory storing at least one instruction; and at least one processor connected to the transceiver.

[0310] The at least one processor may transmit media to an external electronic device that synthesizes media by executing the at least one instruction. The at least one processor may receive, from the external electronic device, a set of synthesized frames including one or more synthesized frames by executing the at least one instruction. The synthesized frame may include a step of including a plurality of media regions.

[0311] The at least one processor can execute the at least one instruction to identify a media area corresponding to a terminal among a plurality of media areas.

[0312] The at least one processor may execute the at least one instruction to obtain output media based on a media area corresponding to the terminal.

[0313] The at least one processor may include a step of executing the at least one instruction to display the acquired output media.

[0314] FIG. 14 is a block diagram illustrating the configuration of an image synthesis device according to an embodiment of the present disclosure.

[0315] In an embodiment, the image synthesis device (100) may include a transceiver (1410), a memory (1420), and a processor (1430). However, the components of the image synthesis device (100) are not limited to the examples described above. The image synthesis device (100) may include more or fewer components than the components described above. The image synthesis device (100) of the present disclosure may correspond to the image synthesis devices of FIGS. 1B to 13 described above, and an external device for synthesizing images.

[0316] The transceiver (1410) may include a communication circuit. The transceiver (1410) may use at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication. The transceiver (1410) may be implemented to include a plurality of modules (for example, a Wi-Fi module, a Bluetooth module, etc.) for implementing the above-described communication methods.

[0317] The transceiver (1410) can transmit and receive data for performing the operation of the image synthesis device (100) to and from devices. For example, the image synthesis device (100) can receive media from a terminal and transmit synthesized media to the terminal through the transceiver (1410).

[0318] The memory (1420) may store instructions, data structures, program codes, and data that can be read by the processor (1430). There may be one or more memories (1420). In the disclosed embodiments, operations performed by the processor (1430) may be implemented by executing instructions or codes of a program stored in the memory (1420).

[0319] The memory (1420) may include non-volatile memory such as read-only memory (ROM) (e.g., programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory (e.g., memory card, solid-state drive (SSD)), and analog recording type (e.g., hard disk drive (HDD), magnetic tape, optical disk), and volatile memory such as random-access memory (RAM) (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)).

[0320] The memory (1420) may store data, instructions, and programs that cause the image synthesis device (2000) to operate to provide synthesized media to a terminal. For example, the memory (1420) may include at least one of a mixer unit, a media identification unit, an encoder unit, or a decoder unit.

[0321] The processor (1430) can control the overall operations of the image synthesis device (100). For example, the processor (1430) can control the overall operations of the image synthesis device (100) to provide synthetic media to the terminal by executing one or more instructions of a program stored in the memory (1420). There can be one or more processors (1430).

[0322] The processor (1430) can cause the aforementioned operations of the image synthesis device (100) to be performed. The operations of the image synthesis device (100) have been described in detail in the previous drawings, so a repeated description will be omitted.

[0323] Meanwhile, the modules stored in the aforementioned memory (1420) are for convenience of explanation and are not necessarily limited thereto. Other modules may be added to implement the aforementioned embodiments, some of the aforementioned modules may be implemented as a single module, or a single model of the aforementioned modules may be implemented by separating them into multiple modules.

[0324] In an embodiment of the present disclosure, one or more processors (1430) may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), and a Neural Processing Unit (NPU). The one or more processors (2300) may be implemented in the form of an integrated system-on-chip (SoC) including one or more electronic components. Each of the one or more processors (2300) may also be implemented as separate hardware (H / W).

[0325] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor (1430) or may be performed by multiple processors (1430).

[0326] For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-only processor).

[0327] Here, the second processor may be an AI-dedicated processor, and the AI-dedicated processor may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0328] One or more processors (1430) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0329] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors (1430).

[0330] Meanwhile, although not illustrated in FIG. 14, the image synthesis device (100) may further include additional components to perform the operations described in the aforementioned embodiments. For example, the image synthesis device (100) may further include a display, a camera, a microphone, an input / output interface, and the like.

[0331] FIG. 15 is a block diagram illustrating a configuration of a terminal according to one embodiment of the present disclosure.

[0332] In one embodiment, the terminal (1500) may include a transceiver (1510), a memory (1520), and a processor (1530). However, the components of the terminal (1500) are not limited to the examples described above. The terminal (1500) may include more or fewer components than the components described above. The terminal (1500) of the present disclosure may correspond to the terminals of FIGS. 1 to 15 described above.

[0333] The transceiver (1510) may include a communication circuit. The transceiver (1510) may use at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication. The transceiver (1510) may be implemented to include a plurality of modules (for example, a Wi-Fi module, a Bluetooth module, etc.) for implementing the above-described communication methods.

[0334] The transceiver (1510) can transmit and receive data for performing operations of the terminal (1500) with devices. For example, the terminal (1500) can transmit media to the image synthesis device (100) and receive synthesized media from the image synthesis device (100) through the transceiver (1510).

[0335] The memory (1520) may store instructions, data structures, program codes, and data that can be read by the processor (1530). There may be one or more memories (1520). In the disclosed embodiments, operations performed by the processor (1530) may be implemented by executing instructions or codes of a program stored in the memory (1520).

[0336] The memory (1520) may include non-volatile memory such as read-only memory (ROM) (e.g., programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory (e.g., memory card, solid-state drive (SSD)), and analog recording type (e.g., hard disk drive (HDD), magnetic tape, optical disk), and volatile memory such as random-access memory (RAM) (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)).

[0337] The memory (1520) may store data, instructions, and programs that cause the image synthesis device (2000) to operate to provide synthesized media to a terminal. For example, the memory (1520) may include at least one of an encoder unit, a decoder unit, or a synthesis unit.

[0338] The processor (1530) can control the overall operations of the terminal (1500). For example, the processor (1530) can control the overall operations for the terminal (1500) to receive synthetic media from an image synthesis device by executing one or more instructions of a program stored in the memory (1520). There may be one or more processors (1530).

[0339] The processor (1530) can cause the aforementioned operations of the terminal (1500) to be performed. The operations of the terminal (1500) have been described in detail in the previous drawings, so a repeated description will be omitted.

[0340] Meanwhile, the modules stored in the aforementioned memory (1520) are for convenience of explanation and are not necessarily limited thereto. Other modules may be added to implement the aforementioned embodiments, some of the aforementioned modules may be implemented as a single module, or a single model of the aforementioned modules may be implemented by separating them into multiple modules.

[0341] In one embodiment, the one or more processors (1530) may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), and a Neural Processing Unit (NPU). The one or more processors (2300) may be implemented in the form of an integrated system on a chip (SoC) including one or more electronic components. Each of the one or more processors (2300) may also be implemented as separate hardware (H / W).

[0342] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor (1530) or may be performed by multiple processors (1530).

[0343] For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-only processor).

[0344] Here, the second processor may be an AI-dedicated processor, and the AI-dedicated processor may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0345] One or more processors (1530) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0346] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors (1530).

[0347] Meanwhile, although not illustrated in FIG. 15, the image synthesis device (100) may further include additional components to perform the operations described in the above-described embodiment. For example, the image synthesis device (100) may further include at least one of a display, a camera, a video processing module, an audio processing module, a power module, a microphone, and an input / output interface.

[0348] The display can output a video signal to the screen of the terminal (1500) under the control of the processor (1530). For example, the terminal (100) can output a composite media or output media through the display.

[0349] The display can convert image signals, data signals, OSD signals, control signals, etc. processed by the processor (1530) to generate driving signals and display images according to the driving signals.

[0350] Cameras can capture video and / or images by capturing images of a space and / or objects or users. There may be one or more cameras. Cameras may include, but are not limited to, RGB cameras, depth cameras, infrared cameras, and the like.

[0351] The terminal (1500) can acquire images of a user using the terminal using a camera. The terminal (1500) can also acquire images of devices positioned within the space using a camera. The specific types and detailed functions of the cameras can be readily inferred by those skilled in the art, and therefore, a detailed description thereof will be omitted.

[0352] The video processing module processes video data played by the terminal. The video processing module can perform various image processing operations, such as decoding, scaling, noise filtering, frame rate conversion, resolution conversion, and rendering.

[0353] The audio processing module performs processing on audio data. The audio processing module can perform various processing operations on audio data, such as decoding, amplification, and noise filtering.

[0354] The power module supplies power input from an external power source to components inside the terminal (1500) under the control of the processor (1530). In addition, the power module can supply power output from one or more batteries (not shown) located inside the terminal (1500) to the internal components under the control of the processor (1530).

[0355] The input / output interface processes input / output from the outside of the terminal (1500). It receives video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.). The input / output interface may include any one of a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Display Port (DP), a Thunderbolt, a Video Graphics Array (VGA) port, an RGB port, a D-subminiature (D-SUB), a Digital Visual Interface (DVI), a component jack, a PC port, and an audio jack. That is, the input / output interface (2900) may be implemented to include a plurality of modules (e.g., a USB port, an HDMI port, etc.) for implementing the above-described input / output methods. The electronic device (2000) can be connected to external devices such as a display, camera, microphone, speaker, touch pad, etc. through an input / output interface (2900).

[0356] The technical problems to be achieved in the present disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from the description of this specification.

[0357] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, and removable and non-removable media.

[0358] Additionally, computer-readable media may include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically contain computer-readable instructions, data structures, or other data in the form of a modulated data signal, such as a program module.

[0359] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0360] According to one embodiment, the methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store or directly between two user devices (e.g., smartphones).

[0361] In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0362] These programs (software modules, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage, a magnetic cassette, or a memory formed by a combination of some or all of these. In addition, each configuration memory may include multiple copies.

[0363] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0364] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0365] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different form than the described method, or are replaced or substituted with other components or equivalents.

[0366] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0367] Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner. For example, some of the steps in FIGS. 1b through 13 may be implemented in a combined manner or in a distributed manner.

[0368] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method of an electronic device for synthesizing media, A step of receiving a plurality of media including one or more frames from a plurality of terminals; A step of identifying the number of media that require synthesis among the above multiple media; A step of setting the layout of a composite frame based on the number of the above media; A step of generating a set of composite frames including one or more composite frames based on the plurality of media and the layout, wherein the composite frame includes a plurality of media areas corresponding to the number of the media; and A method comprising the step of transmitting the composite frame set to the plurality of terminals.

2. In paragraph 1, The steps to set the above layout are: A step of identifying a main media among the above multiple media; and A method comprising the step of setting a layout of a composite frame based on the identified main media and the number of said media.

3. In any one of paragraphs 1 to 2, The step of transmitting the above synthetic frame set is: A step of obtaining synthetic media by performing encoding on the above synthetic frame set; A method comprising the step of transmitting the above synthetic media to the plurality of terminals.

4. In any one of paragraphs 1 to 3, The step of transmitting the above synthetic frame set is: A step of obtaining a plurality of synthetic media by performing encoding on at least some of the plurality of media areas; and A method comprising the step of transmitting, to a first terminal among the plurality of terminals, a synthetic media corresponding to the first terminal among the plurality of synthetic media.

5. In any one of paragraphs 1 to 4, A method further comprising the step of transmitting at least one of location information of the plurality of media areas, or location information of a media area required for each terminal.

6. In any one of paragraphs 1 to 5, A method wherein the layout includes settings regarding at least one of the resolution of the composite frame, the arrangement shape of the media area, or the background of the composite frame.

7. In paragraph 6, A method wherein the layout form of the above media area includes at least one of a single row layout form, a single column layout form, a multiple row and multiple column layout form, or a rotational layout form.

8. In paragraph 3, The step of transmitting the above synthetic media is: A method comprising the steps of performing encryption on the synthetic media and transmitting the encrypted synthetic media to the plurality of terminals.

9. In the method of operation in a terminal receiving synthetic media, A step of transmitting media to an external electronic device that synthesizes the media; A step of receiving, from said external electronic device, a set of composite frames including one or more composite frames, wherein said composite frames include a plurality of media areas; A step of identifying a media area corresponding to the terminal among the plurality of media areas; A step of obtaining output media based on a media area corresponding to the terminal; and A method comprising the step of displaying the obtained output media.

10. In paragraph 9, The step of obtaining the above output media is A method comprising the step of obtaining the output media by placing the media of the terminal in a media area corresponding to the terminal of the synthetic frame set.

11. In any one of paragraphs 9 to 10, The step of obtaining the above output media is: A step of masking a media area corresponding to the terminal of the above synthetic frame set; and A method comprising the step of obtaining said output media by placing media of said terminal in at least a portion of said composite frame set.

12. In any one of paragraphs 9 to 11, The step of receiving the above one set of synthetic frames comprises: A step of receiving, from the external electronic device, a synthetic media obtained by performing encoding on the synthetic frame set; and A method comprising the step of obtaining said one set of synthetic frames by performing decoding on said synthetic media.

13. In paragraph 12, The step of receiving the above synthetic media comprises: A step of receiving encrypted synthetic media for said synthetic media from said external electronic device; and A method comprising the step of performing decryption on the encrypted synthetic media.

14. In paragraph 9, Further comprising a step of receiving at least one of the location information of the plurality of media areas, or the location information of the media area required for each terminal, A method in which a media area corresponding to the terminal is identified based on at least one of location information of the plurality of media areas or location information of a media area required for each terminal.

15. In an electronic device that synthesizes media, Transmitter and receiver; Memory in which at least one instruction is stored; and At least one processor connected to the transceiver, The at least one processor executes the at least one instruction, Receiving multiple media containing one or more frames from multiple terminals, Identify the number of media that need to be synthesized, Set the layout of the composite frame based on the number of the above media, Generating a set of composite frames including one or more composite frames based on the plurality of media and the layout, wherein the composite frame includes a plurality of media areas corresponding to the number of the media, An electronic device that transmits the composite frame set to the plurality of terminals.

Images