Method for synchronizing alpha and video of cloud streaming, and apparatus therefor

Abstract

A method for synchronizing alpha and video of cloud streaming, and an apparatus therefor are disclosed. In the method for synchronizing alpha and video of cloud streaming, according to the present invention, a terminal device using a cloud streaming service receives video data and alpha data from a cloud streaming server, decodes the alpha data on the basis of a time point corresponding to a preset checkpoint while processing the video data, and applies a synchronization time set in consideration of the preset checkpoint so as to synchronize the decoded video data and the decoded alpha data.

Description

Method for synchronizing alpha and video in cloud streaming and device for the same

[0001] The present invention relates to a method for synchronizing alpha and video in cloud streaming and an apparatus therefor, and more particularly, to a technique for preventing synchronization errors between video and alpha that may occur due to video buffering in a terminal device during video cloud streaming, and for efficiently synchronizing video and alpha regardless of the type or performance of the terminal.

[0002] This invention claims the benefit of Korean Patent Application No. 10-2023-0100963, filed August 2, 2023, the entire contents of which are incorporated herein by reference.

[0003] The rapid development of the Internet has resulted in a dramatic increase in personal communication speeds. This improvement in communication speeds has enabled users to access remote computers, download or upload large amounts of data, or use remote computer control programs to access remote computers as if logged in locally. Furthermore, with the development of diverse applications running on mobile devices like smartphones, numerous virtualization technologies have been proposed to enable high-performance applications to run on relatively low-performance user devices.

[0004] Cloud streaming technology is a solution that runs applications or services requiring high-end system resources on a cloud streaming server and delivers only the rendered results to the user's device. This allows high-end video playback or 3D games even on low-end devices.

[0005] Meanwhile, cloud streaming for user interfaces (UIs), such as menus or posters displayed on live channels, is primarily achieved through image encoding, as it facilitates transparency processing. However, image encoding has the disadvantage of incurring significant overhead when processing fast-changing screens, such as animations.

[0006] Therefore, the need for technology utilizing video encoding based on video cloud streaming is being emphasized. However, the H.264 codec, which is widely used for video encoding, cannot process alpha data, which is transparency information. Furthermore, while the H.265 codec has some specifications related to alpha data, it is not a basic function, so it is not implemented in user terminals or is not supported in hardware, making it difficult to use.

[0007] Ultimately, video reception and alpha reception were created according to the characteristics of the user terminal, and alpha data implemented separately from video data causes synchronization errors due to video buffering. In other words, since the timing of playback and the timing of transmission are different, a problem occurs where the video data and alpha data are not displayed normally on the screen. Generally, synchronization is achieved using a timestamp, but in reality, the timestamp value does not accurately determine the delay due to video buffering, so synchronization with alpha becomes off.

[0008] The purpose of the present invention is to provide a technology for efficiently synchronizing video and alpha regardless of the type or performance of a terminal device by utilizing the time at which an I-frame is output from a video buffer.

[0009] In addition, it is an object of the present invention to prevent synchronization errors between video and alpha from occurring due to decoding time and buffering time of video data.

[0010] In addition, an object of the present invention is to efficiently provide an application execution screen with transparency applied during video cloud streaming.

[0011] In order to achieve the above-described object, the present invention provides a method for synchronizing alpha and video in cloud streaming, comprising: a step in which a terminal device using a cloud streaming service receives video data and alpha data from a cloud streaming server; a step in which the alpha data is decoded based on a time corresponding to a preset check point during the processing of the video data; and a step in which the decoded video data and the decoded alpha data are synchronized by applying a synchronization time set in consideration of the preset check point.

[0012] At this time, the preset check points may correspond to a first check point corresponding to a time at which the first frame of the video data is input into the video buffer, a second check point corresponding to a time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to a time at which the first frame decoded from the video decoder is output.

[0013] At this time, decoding of alpha data can be performed at a point in time corresponding to the second check point.

[0014] At this time, the synchronization time may correspond to a time corresponding to the first check point to a time corresponding to the third check point.

[0015] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0016] At this time, the terminal device may further include a step of displaying video data to which alpha data has been applied through the synchronization.

[0017] In addition, a terminal device according to the present invention includes a processor that receives video data and alpha data from a cloud streaming server, performs decoding of the alpha data based on a time corresponding to a preset check point during the processing of the video data, and synchronizes the decoded video data and the decoded alpha data by applying a synchronization time set in consideration of the preset check point; and a memory that stores the video data and the alpha data.

[0018] At this time, the preset check points may correspond to a first check point corresponding to a time at which the first frame of the video data is input into the video buffer, a second check point corresponding to a time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to a time at which the first frame decoded from the video decoder is output.

[0019] At this time, decoding of alpha data can be performed at a point in time corresponding to the second check point.

[0020] At this time, the synchronization time may correspond to a time corresponding to the first check point to a time corresponding to the third check point.

[0021] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0022] At this time, the processor can display video data with alpha data applied through the above synchronization.

[0023] According to the present invention, a technology for efficiently synchronizing video and alpha regardless of the type or performance of a terminal device can be provided by utilizing the time at which an I-frame is output from a video buffer.

[0024] In addition, the present invention can prevent synchronization errors between video and alpha from occurring due to decoding time and buffering time of video data.

[0025] In addition, the present invention can efficiently provide an application execution screen with transparency applied during video cloud streaming.

[0026] FIG. 1 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0027] FIG. 2 is a flowchart illustrating a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention.

[0028] FIG. 3 is a diagram showing a synchronization process of alpha and video according to the present invention.

[0029] FIG. 4 is a diagram showing an example of a check point and synchronization time according to the present invention.

[0030] FIG. 5 is a detailed flowchart illustrating a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention.

[0031] Figure 6 is a block diagram showing a terminal device according to one embodiment of the present invention.

[0032] The present invention will be described in detail with reference to the attached drawings. Herein, repetitive descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. The embodiments of the present invention are provided to more fully explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

[0033] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0034] FIG. 1 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0035] Referring to FIG. 1, a cloud streaming service system according to one embodiment of the present invention includes a cloud streaming server (110), user terminals (120-1 to 120-N), and a network (130).

[0036] Cloud streaming service or cloud computing service is a service that allows users to store content in cloud storage on the Internet rather than in local storage, and access the cloud storage and use the content anytime, anywhere using an Internet-enabled device.

[0037] The cloud streaming server (110) may include a video cloud streaming engine for processing a cloud streaming service requested from a user terminal (120-1 to 120-N).

[0038] At this time, the cloud streaming server (110) can create a predefined number of browsers at the time the cloud streaming engine is executed, and the browser can download and execute HTML, CSS, JavaScript, etc. and render them on the screen.

[0039] At this time, HTML (Hyper Text Markup Language) provides the large framework of a web page, CSS (Cascading Style Sheets) manages design elements such as color and font, and JavaScript is a cross-platform, object-oriented scripting language that can be responsible for the operation of a web page.

[0040] User terminals (120-1 to 120-N) receive video data and alpha data from a cloud streaming server (110).

[0041] At this time, the cloud streaming server (110) can transmit video data and alpha data to the terminal device through the process of rendering, capturing, encoding, and streaming the web application execution screen.

[0042] That is, the cloud streaming server (110) according to the present invention can encode video data and alpha data respectively and transmit them to a terminal device.

[0043] At this time, video data and alpha data can be encoded and transmitted using each encoding method.

[0044] Additionally, the user terminal (120-1 to 120-N) performs decoding of alpha data based on a point in time corresponding to a preset check point during the video data processing process.

[0045] Typically, timestamps are used to synchronize video data and alpha data. However, timestamp values ​​cannot accurately determine the delay time that occurs during the buffering and decoding of video data received by user terminals (120-1 to 120-N), which can lead to synchronization errors with alpha data.

[0046] Therefore, in order to solve this problem, the present invention proposes a method for preventing synchronization errors between video data and alpha data by setting separate check points on the user terminal (120-1 to 120-N) side where video data and alpha data are received, and determining the time point for decoding alpha data based on the time point corresponding to the check point.

[0047] At this time, the preset check points may correspond to a first check point corresponding to the time at which the first frame of video data is input into the video buffer, a second check point corresponding to the time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to the time at which the first frame decoded from the video decoder is output.

[0048] At this time, the preset check point can be set based on the time at which the user terminal (120-1 to 120-N) receives video data and alpha data from the cloud streaming server (110).

[0049] Therefore, the synchronization method according to the present invention can be applied to any user terminal (120-1 to 120-N) regardless of the type or performance of the user terminal (120-1 to 120-N).

[0050] At this time, decoding of alpha data can be performed at a point corresponding to the second checkpoint.

[0051] That is, the user terminal (120-1 to 120-N) goes through a process of buffering video data. Therefore, if alpha data is decoded immediately after receiving it, a synchronization time difference with the alpha data may occur corresponding to the time taken to buffer the video data. To prevent this time difference from occurring, a second checkpoint can be set to correspond to the point in time when decoding of the video data begins, and decoding of the alpha data can be performed at that point in time.

[0052] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0053] That is, synchronization can be performed by continuously updating preset checkpoints according to the received data whenever video data and alpha data are received from the cloud streaming server (110).

[0054] Additionally, the user terminal (120-1 to 120-N) synchronizes the decoded video data and the decoded alpha data by applying the set synchronization time considering the preset check point.

[0055] At this time, the synchronization time may correspond to a time corresponding to the first checkpoint to a time corresponding to the third checkpoint.

[0056] That is, alpha data can be applied to video data for a period of time corresponding to the synchronization time.

[0057] If you synchronize video data and alpha data using the existing synchronization method using timestamps, the decoded video data and decoded alpha data may not be in sync when performing synchronization due to errors caused by the video buffering process. If the synchronization is not in sync like this, the screen may be distorted when the application execution screen is actually displayed on the user terminal (120-1 to 120-N).

[0058] Additionally, the user terminal (120-1 to 120-N) displays video data with alpha data applied through synchronization.

[0059] That is, the user terminal (120-1 to 120-N) can request a cloud streaming service from the cloud streaming server (110), and receive an execution screen according to the request from the cloud streaming server (110) and provide it to the user. Accordingly, the user can feel as if a video or application corresponding to the requested service is running on the user terminal (120-1 to 120-N).

[0060] At this time, the user terminals (120-1 to 120-N) may correspond to devices that are each connected to a communication network and can communicate with the cloud streaming server (110). For example, the user terminals (120-1 to 120-N) may correspond to various terminals such as all information and communication devices, multimedia terminals, wired terminals, fixed terminals, mobile communication terminals, and IP (Internet Protocol) terminals. In addition, the user terminals (120-1 to 120-N) may be mobile terminals having various mobile communication specifications such as cell phones, mobile phones, portable multimedia players (PMPs), mobile Internet devices (MIDs), smart phones, desktops, tablet computers (Tablet PCs), notebooks, netbooks, personal digital assistants (PDAs), smart TVs, and information and communication devices.

[0061] In addition, the user terminals (120-1 to 120-N) can input various information such as number and character information, set various functions, and transmit signals input in relation to the control of the functions of the user terminals (120-1 to 120-N) to the control unit through the input unit. In addition, the input unit of the user terminals (120-1 to 120-N) can be configured to include at least one of a keypad and a touchpad that generate an input signal according to the user's touch or operation. In this case, the input unit of the user terminals (120-1 to 120-N) is configured in the form of a single touch panel (or touch screen) together with the display unit of the user terminals (120-1 to 120-N) so that input and display functions can be performed simultaneously. In addition to input devices such as a keyboard, keypad, mouse, joystick, etc., the input unit of the user terminals (120-1 to 120-N) can use any form of input means that may be developed in the future.

[0062] In addition, the display unit of the user terminal (120-1 to 120-N) can display information about a series of operation states and operation results that occur during the performance of the functions of the user terminal (120-1 to 120-N). In addition, the display unit of the user terminal (120-1 to 120-N) can display menus of the user terminal (120-1 to 120-N) and user data input by the user. Here, the display unit of the user terminal (120-1 to 120-N) can be composed of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), a light emitting diode (LED), an organic light emitting diode (OLED), an active matrix OLED (AMOLED), a retina display, a flexible display, a three-dimensional (3 Dimension) display, etc. At this time, if the display unit of the user terminal (120-1 to 120-N) is configured in the form of a touch screen, the display unit of the user terminal (120-1 to 120-N) can perform some or all of the functions of the input unit of the user terminal (120-1 to 120-N).

[0063] In addition, the storage unit of the user terminal (120-1 to 120-N) is a device for storing data, and includes a main memory device and an auxiliary memory device, and can store application programs required for the functional operation of the user terminal (120-1 to 120-N). The storage unit of the user terminal (120-1 to 120-N) can largely include a program area and a data area. Here, when the user terminal (120-1 to 120-N) activates each function in response to a user request, it executes the corresponding application programs under the control of the control unit to provide each function.

[0064] In addition, the communication unit of the user terminal (120-1 to 120-N) may perform a function for transmitting and receiving data through a network with the cloud streaming server (110). Here, the communication unit of the user terminal (120-1 to 120-N) may include an RF transmitting means for up-converting and amplifying the frequency of a transmitted signal, an RF receiving means for low-noise amplifying a received signal and down-converting the frequency, etc. The communication unit of the user terminal (120-1 to 120-N) may include a wireless communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the user terminal (120-1 to 120-N) uses wireless communication, any one of a wireless network communication module, a wireless LAN communication module, and a wireless fan communication module may be used to transmit and receive data to and from the cloud streaming server (110). That is, the user terminal (120-1 to 120-N) connects to the network using a wireless communication module and can transmit and receive data with the cloud streaming server (110) through the network.

[0065] In addition, the control unit of the user terminal (120-1 to 120-N) may be a process device that drives an operating system (OS) and each component. For example, the control unit may control the entire process of connecting to the cloud streaming server (110). At this time, when the user terminal (120-1 to 120-N) connects to the cloud streaming server (110), the entire process of executing content such as a video or an application according to the user's request may be controlled, and a service use request may be transmitted to the cloud streaming server (110) simultaneously with the execution. At this time, information of the user terminal (120-1 to 120-N) required for user authentication may also be controlled to be transmitted together.

[0066] The network (130) provides a path for transmitting data between the cloud streaming server (110) and the user terminals (120-1 to 120-N), and is a concept that encompasses both existing networks and networks that can be developed in the future. For example, the network (130) may be a wired or wireless local area network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobiles and between mobiles and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired or wireless communication network, or a combination of two or more of them. Meanwhile, the transmission method standard of the network is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future.

[0067] A system like this can provide a technology to efficiently synchronize video and alpha regardless of the type or performance of the terminal device by utilizing the time at which the I-frame is output from the video buffer.

[0068] Additionally, it is possible to prevent synchronization errors between video and alpha caused by decoding time and buffering time of video data.

[0069] Additionally, it can efficiently provide an application execution screen with transparency applied when streaming video clouds.

[0070] FIG. 2 is a flowchart illustrating a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention.

[0071] Referring to FIG. 2, in a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention, a terminal device using a cloud streaming service receives video data and alpha data from a cloud streaming server (S210).

[0072] At this time, the cloud streaming server can transmit video data and alpha data to the terminal device through the process of rendering, capturing, encoding, and streaming the web application execution screen.

[0073] That is, the cloud streaming server according to the present invention can encode video data and alpha data respectively and transmit them to a terminal device.

[0074] At this time, video data and alpha data can be encoded and transmitted using each encoding method.

[0075] In addition, in a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention, a terminal device using a cloud streaming service performs decoding of alpha data based on a point in time corresponding to a preset check point during the processing of video data (S220).

[0076] Typically, timestamps are used to synchronize video data and alpha data. However, timestamp values ​​cannot accurately reflect the delays that occur during buffering and decoding of video data received by the terminal device, which can lead to synchronization errors with alpha data.

[0077] Therefore, in order to solve this problem, the present invention proposes a method for preventing synchronization errors between video data and alpha data by setting separate check points on the terminal device side where video data and alpha data are received and determining the time point for decoding alpha data based on the time point corresponding to the check point.

[0078] At this time, the preset check points may correspond to a first check point corresponding to the time at which the first frame of the video data is input into the video buffer, a second check point corresponding to the time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to the time at which the first frame decoded from the video decoder is output.

[0079] At this time, the preset checkpoint can be set based on the time at which the terminal device receives video data and alpha data from the cloud streaming server.

[0080] Therefore, the synchronization method according to the present invention can be applied to any terminal device regardless of the type or performance of the terminal device.

[0081] At this time, decoding of alpha data can be performed at a point corresponding to the second checkpoint.

[0082] That is, the terminal device goes through a buffering process for video data. Therefore, if alpha data is decoded immediately after receiving it, a synchronization time difference with the alpha data may occur equal to the time it takes to buffer the video data. To prevent this time difference, a second checkpoint can be set to correspond to the point in time when video data decoding begins, and alpha data decoding can be performed at that point.

[0083] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0084] That is, synchronization can be performed by continuously updating preset checkpoints according to the received data whenever video data and alpha data are received from a cloud streaming server.

[0085] In addition, in a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention, a terminal device using a cloud streaming service synchronizes decoded video data and decoded alpha data by applying a set synchronization time in consideration of a preset check point (S230).

[0086] At this time, the synchronization time may be from a time corresponding to the first checkpoint to a time corresponding to the third checkpoint.

[0087] That is, alpha data can be applied to video data for a period of time corresponding to the synchronization time.

[0088] If you synchronize video and alpha data using the existing synchronization method using timestamps, errors introduced during the video buffering process can cause the decoded video data and decoded alpha data to become out of sync during synchronization. This mismatch can result in the screen being distorted when the application execution screen is actually displayed on the terminal device.

[0089] In addition, although not shown in FIG. 2, a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention allows a terminal device using a cloud streaming service to display video data to which alpha data has been applied through synchronization.

[0090] In other words, a terminal device can request a cloud streaming service from a cloud streaming server, receive a requested execution screen from the cloud streaming server, and provide it to the user. Therefore, the user can feel as if the video or application corresponding to the requested service is running on the terminal device.

[0091] At this time, the terminal devices may be devices that are connected to a communication network and can communicate with the cloud streaming server. For example, the terminal devices may correspond to various terminals such as all information and communication devices, multimedia terminals, wired terminals, fixed terminals, mobile communication terminals, and IP (Internet Protocol) terminals. In addition, the terminal devices may be mobile terminals with various mobile communication specifications such as cell phones, mobile phones, portable multimedia players (PMPs), mobile Internet devices (MIDs), smart phones, desktops, tablet computers (Tablet PCs), notebooks, netbooks, personal digital assistants (PDAs), smart TVs, and information and communication devices.

[0092] This method of synchronizing alpha and video in cloud streaming can provide a technology for efficiently synchronizing video and alpha regardless of the type or performance of the terminal device by utilizing the time at which I-frames are output from the video buffer.

[0093] Additionally, it is possible to prevent synchronization errors between video and alpha caused by decoding time and buffering time of video data.

[0094] Additionally, it can efficiently provide an application execution screen with transparency applied when streaming video clouds.

[0095] FIG. 3 is a diagram showing a synchronization process of alpha and video according to the present invention.

[0096] Referring to FIG. 3, a terminal device using a cloud streaming service according to the present invention can receive video data and alpha data from a cloud streaming server (S302, S312).

[0097] At this time, the cloud streaming server can transmit video data and alpha data to the terminal device through the process of rendering, capturing, encoding, and streaming the web application execution screen.

[0098] That is, the cloud streaming server according to the present invention can encode video data and alpha data respectively and transmit them to a terminal device.

[0099] At this time, video data and alpha data can be encoded and transmitted using each encoding method.

[0100] After this, a terminal device using a cloud streaming service according to the present invention must decode video data and alpha data through each process and then synchronize them. To this end, a checkpoint can be set first.

[0101] Typically, timestamps are used to synchronize video data and alpha data. However, timestamp values ​​cannot accurately reflect the delays that occur during buffering and decoding of video data received by the terminal device, which can lead to synchronization errors with alpha data.

[0102] Therefore, in order to solve this problem, the present invention proposes a method for preventing synchronization errors between video data and alpha data by setting separate check points on the terminal device side where video data and alpha data are received and determining the time point for decoding alpha data based on the time point corresponding to the check point.

[0103] At this time, the preset check points may correspond to a first check point corresponding to the time at which the first frame of the video data is input into the video buffer, a second check point corresponding to the time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to the time at which the first frame decoded from the video decoder is output.

[0104] For example, referring to FIG. 4, a first check point (CHECK 1) may be set corresponding to the time when video buffering (S304) starts, i.e., the time when the first frame of video data is input into the video buffer. In addition, a second check point (CHECK 2) may be set corresponding to the time when video decoding (S306) starts, i.e., the time when the first frame is input into the video decoder. In addition, a third check point (CHECK 3) may be set corresponding to the time when the first frame by video decoding is output.

[0105] At this time, the preset checkpoint can be set based on the time at which the terminal device receives video data and alpha data from the cloud streaming server.

[0106] Therefore, the synchronization method according to the present invention can be applied to any terminal device regardless of the type or performance of the terminal device.

[0107] After this, a terminal device using a cloud streaming service according to the present invention can perform decoding of alpha data based on a point in time corresponding to a preset check point during the video data processing process (S314).

[0108] At this time, decoding of alpha data can be performed at a point corresponding to the second checkpoint.

[0109] That is, the terminal device goes through a process of buffering video data (S304). Therefore, if alpha data is decoded immediately after receiving it, a synchronization time difference with the alpha data may occur corresponding to the time taken to buffer the video data. To prevent this time difference from occurring, a second checkpoint can be set to correspond to the point in time when decoding of the video data begins, and decoding of the alpha data can be performed at that point in time.

[0110] At this time, the preset checkpoints (CHECK 1, CHECK 2, CHECK 3) can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0111] That is, synchronization can be performed by continuously updating preset checkpoints according to the received data whenever video data and alpha data are received from a cloud streaming server.

[0112] After this, a terminal device using a cloud streaming service according to the present invention can synchronize decoded video data and decoded alpha data by applying a set synchronization time considering a preset check point (S320).

[0113] At this time, the synchronization time may correspond to a time corresponding to the first check point (CHECK 1) to a time corresponding to the third check point (CHECK 3), as illustrated in FIG. 4.

[0114] That is, alpha data can be applied to video data for a period of time corresponding to the synchronization time.

[0115] If video data and alpha data are synchronized using the existing synchronization method using a timestamp, a problem may occur in which the decoded video data and the decoded alpha data are out of sync when synchronization is performed in step (S320) due to an error caused by the video buffering process (S304). If the synchronization is not synchronized in this way, a problem may occur in which the screen is distorted when the application execution screen is actually output on the terminal device.

[0116] After this, a terminal device using a cloud streaming service according to the present invention can display video data to which alpha data has been applied through synchronization (S330).

[0117] Through this process, a technology can be provided to efficiently synchronize video and alpha regardless of the type or performance of the terminal device by utilizing the time at which the I-frame is output from the video buffer.

[0118] Additionally, it is possible to prevent synchronization errors between video and alpha caused by decoding time and buffering time of video data.

[0119] Additionally, it can efficiently provide an application execution screen with transparency applied when streaming video clouds.

[0120] FIG. 5 is a detailed flowchart illustrating a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention.

[0121] Referring to FIG. 5, in a method for synchronizing alpha and video of cloud streaming according to an embodiment of the present invention, first, a terminal device using a cloud streaming service receives video data and alpha data from a cloud streaming server (S510).

[0122] At this time, the cloud streaming server can transmit video data and alpha data to the terminal device through the process of rendering, capturing, encoding, and streaming the web application execution screen.

[0123] That is, the cloud streaming server according to the present invention can encode video data and alpha data respectively and transmit them to a terminal device.

[0124] At this time, video data and alpha data can be encoded and transmitted using each encoding method.

[0125] After this, a terminal device using a cloud streaming service can store each point in time according to a preset check point during the video data processing process, and first stores the first point in time corresponding to the first check point where video buffering begins (S520).

[0126] At this time, the first checkpoint can be set to correspond to the time at which the first frame of video data is input into the video buffer.

[0127] After this, the terminal device using the cloud streaming service stores a second point in time corresponding to the second check point where video decoding begins (S530).

[0128] At this time, the second checkpoint can be set to correspond to the time when the first frame output from the video buffer is input to the video decoder.

[0129] After this, the terminal device using the cloud streaming service starts decoding alpha data corresponding to the second time point (S540).

[0130] That is, the terminal device goes through a buffering process for video data. Therefore, if alpha data is decoded immediately after receiving it, a synchronization time difference with the alpha data may occur equal to the time it takes to buffer the video data. To prevent this time difference, a second checkpoint can be set to correspond to the point in time when video data decoding begins, and alpha data decoding can be performed at that point.

[0131] After this, the terminal device using the cloud streaming service stores the third point in time corresponding to the third check point corresponding to the time at which the first frame decoded by the video decoder is output (S550).

[0132] At this time, the preset checkpoint can be set based on the time at which the terminal device receives video data and alpha data from the cloud streaming server.

[0133] Therefore, the synchronization method according to the present invention can be applied to any terminal device regardless of the type or performance of the terminal device.

[0134] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0135] That is, synchronization can be performed by continuously updating preset checkpoints according to the received data whenever video data and alpha data are received from a cloud streaming server.

[0136] After this, the terminal device using the cloud streaming service sets the synchronization time from the first point in time to the third point in time (S560).

[0137] After this, the terminal device using the cloud streaming service performs synchronization so that alpha data can be applied to the video data for a time corresponding to the synchronization time (S570).

[0138] In addition, although not shown in FIG. 5, a terminal device using a cloud streaming service displays video data to which alpha data is applied through synchronization.

[0139] In other words, a terminal device can request a cloud streaming service from a cloud streaming server, receive a requested execution screen from the cloud streaming server, and provide it to the user. Therefore, the user can feel as if the video or application corresponding to the requested service is running on the terminal device.

[0140] This method of synchronizing alpha and video in cloud streaming can provide a technology for efficiently synchronizing video and alpha regardless of the type or performance of the terminal device by utilizing the time at which I-frames are output from the video buffer.

[0141] Additionally, it is possible to prevent synchronization errors between video and alpha caused by decoding time and buffering time of video data.

[0142] Additionally, it can efficiently provide an application execution screen with transparency applied when streaming video clouds.

[0143] Figure 6 is a block diagram showing a terminal device according to one embodiment of the present invention.

[0144] Referring to FIG. 6, a terminal device according to one embodiment of the present invention includes a communication unit (610), a processor (620), and a memory (630).

[0145] The processor (620) receives video data and alpha data from a cloud streaming server.

[0146] At this time, the cloud streaming server can transmit video data and alpha data to the terminal device through the process of rendering, capturing, encoding, and streaming the web application execution screen.

[0147] That is, the cloud streaming server according to the present invention can encode video data and alpha data respectively and transmit them to a terminal device.

[0148] At this time, video data and alpha data can be encoded and transmitted using each encoding method.

[0149] Additionally, the processor (620) performs decoding of alpha data based on a point in time corresponding to a preset check point during the video data processing process.

[0150] Typically, timestamps are used to synchronize video data and alpha data. However, timestamp values ​​cannot accurately reflect the delays that occur during buffering and decoding of video data received by the terminal device, which can lead to synchronization errors with alpha data.

[0151] Therefore, in order to solve this problem, the present invention proposes a method for preventing synchronization errors between video data and alpha data by setting separate check points on the terminal device side where video data and alpha data are received and determining the time point for decoding alpha data based on the time point corresponding to the check point.

[0152] At this time, the preset check points may correspond to a first check point corresponding to the time at which the first frame of the video data is input into the video buffer, a second check point corresponding to the time at which the first frame output from the video buffer is input into the video decoder, and a third check point corresponding to the time at which the first frame decoded from the video decoder is output.

[0153] At this time, the preset checkpoint can be set based on the time at which the terminal device receives video data and alpha data from the cloud streaming server.

[0154] Therefore, the synchronization method according to the present invention can be applied to any terminal device regardless of the type or performance of the terminal device.

[0155] At this time, decoding of alpha data can be performed at a point corresponding to the second checkpoint.

[0156] That is, the terminal device goes through a process of buffering video data.

[0157] Therefore, if alpha data is decoded immediately after it is received, a synchronization time difference with the alpha data may occur due to the time required to buffer the video data. To prevent this timing difference, a second checkpoint can be set to correspond to the point in time when video data decoding begins, and alpha data decoding can be performed at that point.

[0158] At this time, the preset checkpoint can be updated when the next video data and next alpha data are received from the cloud streaming server.

[0159] That is, synchronization can be performed by continuously updating preset checkpoints according to the received data whenever video data and alpha data are received from a cloud streaming server.

[0160] Additionally, the processor (620) synchronizes the decoded video data and the decoded alpha data by applying a set synchronization time considering a preset check point.

[0161] At this time, the synchronization time may be from a time corresponding to the first checkpoint to a time corresponding to the third checkpoint.

[0162] If you synchronize video and alpha data using the existing synchronization method using timestamps, errors introduced during the video buffering process can cause the decoded video data and decoded alpha data to become out of sync during synchronization. This mismatch can result in the screen being distorted when the application execution screen is actually displayed on the terminal device.

[0163] That is, alpha data can be applied to video data for a period of time corresponding to the synchronization time.

[0164] Additionally, the processor (620) displays video data with alpha data applied through synchronization.

[0165] In other words, a terminal device can request a cloud streaming service from a cloud streaming server, receive a requested execution screen from the cloud streaming server, and provide it to the user. Therefore, the user can feel as if the video or application corresponding to the requested service is running on the terminal device.

[0166] Additionally, the memory (630) of the terminal device stores video data and alpha data.

[0167] In addition, the memory (630) of the terminal device is a device for storing data, including a main memory and an auxiliary memory, and can store application programs required for the functional operation of the terminal device. The storage section of the terminal device can largely include a program area and a data area. Here, when the terminal device activates each function in response to a user request, it executes the corresponding application programs under the control of the control unit to provide each function.

[0168] In addition, the communication unit (610) of the terminal device may perform a function for transmitting and receiving data with the cloud streaming server through a network. Here, the communication unit of the terminal device may include an RF transmitting means for up-converting and amplifying the frequency of a transmitted signal, an RF receiving means for low-noise amplifying a received signal and down-converting the frequency, etc. The communication unit of the terminal device may include a wireless communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the terminal device uses wireless communication, the data may be transmitted and received with the cloud streaming server using any one of a wireless network communication module, a wireless LAN communication module, and a wireless fan communication module. That is, the terminal device connects to a network using the wireless communication module, and can transmit and receive data with the cloud streaming server through the network.

[0169] By utilizing the time at which an I-frame is output from a video buffer through a terminal device such as this, a technology can be provided to efficiently synchronize video and alpha regardless of the type or performance of the terminal device.

[0170] Additionally, it is possible to prevent synchronization errors between video and alpha caused by decoding time and buffering time of video data.

[0171] Additionally, it can efficiently provide an application execution screen with transparency applied when streaming video clouds.

[0172] The video synchronization method according to the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and all types of hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands may include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. These hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0173] As described above, the alpha and video synchronization method and device for cloud streaming according to the present invention are not limited to the configuration and method of the embodiments described above, but all or part of the embodiments may be selectively combined to allow for various modifications.

[0174] According to the present invention, a terminal device utilizing a cloud streaming service can receive video data and alpha data from a cloud streaming server, decode the alpha data based on a time corresponding to a preset checkpoint during the video data processing process, and synchronize the decoded video data with the decoded alpha data by applying a synchronization time set in consideration of the preset checkpoint. Accordingly, video data and alpha data can be efficiently synchronized regardless of the type or performance of the terminal device.

Claims

1. Terminal devices that use cloud streaming services, A step of receiving video data and alpha data from a cloud streaming server; A step of performing decoding of the alpha data based on a point in time corresponding to a preset check point during the processing of the video data; and A step of synchronizing the decoded video data and the decoded alpha data by applying the set synchronization time considering the above-described check points. A method for synchronizing alpha and video in cloud streaming, characterized in that it includes.

2. In claim 1, The above preset check points are A method for synchronizing alpha and video in cloud streaming, characterized in that a first check point corresponds to a time at which a first frame of the video data is input into a video buffer, a second check point corresponds to a time at which the first frame output from the video buffer is input into a video decoder, and a third check point corresponds to a time at which the first frame decoded from the video decoder is output.

3. In claim 2, A method for synchronizing alpha and video in cloud streaming, characterized in that the decoding of the above alpha data is performed at a point in time corresponding to the second check point.

4. In claim 2, The above synchronization time is A method for synchronizing alpha and video in cloud streaming, characterized in that it corresponds to a time corresponding to the first check point and a time corresponding to the third check point.

5. In claim 1, The above preset check points are A method for synchronizing alpha and video in cloud streaming, characterized in that the next video data and next alpha data are updated when received from the cloud streaming server.

6. In claim 1, The above terminal device, A method for synchronizing alpha and video in cloud streaming, characterized in that it further comprises a step of displaying video data to which alpha data is applied through the above synchronization.

7. A processor that receives video data and alpha data from a cloud streaming server, performs decoding of the alpha data based on a time corresponding to a preset check point during the processing of the video data, and synchronizes the decoded video data and the decoded alpha data by applying a synchronization time set in consideration of the preset check point; and Memory for storing the above video data and the above alpha data A terminal device characterized by including:

8. In claim 7, The above preset check points are A terminal device characterized in that it corresponds to a first check point corresponding to a time at which a first frame of the video data is input into a video buffer, a second check point corresponding to a time at which the first frame output from the video buffer is input into a video decoder, and a third check point corresponding to a time at which the first frame decoded from the video decoder is output.

9. In claim 8, A terminal device characterized in that the decoding of the above alpha data is performed at a point in time corresponding to the second check point.

10. In claim 8, The above synchronization time is A terminal device characterized in that it corresponds to a time corresponding to the first check point and a time corresponding to the third check point.

Images