Method for providing video cloud streaming service based on efficient transparency processing and apparatus therefor

Abstract

Disclosed are a method for providing a video cloud streaming service based on efficient transparency processing and an apparatus therefor. In the method for providing a video cloud streaming service according to the present invention, a cloud streaming server may: display a current frame on a GPU memory by using alpha data obtained from a renderer process; determine whether the alpha data is changed between frames on the GPU memory; when the alpha data is changed, transfer the alpha data to a CPU memory and encode same; and stream, to a user terminal, the encoded alpha data from the CPU memory and encoded RGB data from the GPU memory.

Description

Method for providing video cloud streaming service based on efficient transparency processing and device therefor

[0001] The present invention relates to a method for providing a video cloud streaming service based on efficient transparency processing and an apparatus therefor, and more particularly, to a technology capable of reducing the amount of data transferred between GPU memory and CPU memory by processing transparency changes on GPU memory, a technology capable of reducing the amount of data transferred between GPU memory and CPU memory by processing transparency changes by dividing frames on GPU memory, and a technology capable of reducing the amount of data transferred between GPU memory and CPU memory by performing screen change detection by dividing frames on GPU memory.

[0002] The present invention claims the benefit of the filing dates of Korean Patent Application No. 10-2023-0093561, filed July 19, 2023, Korean Patent Application No. 10-2023-0093562, filed July 19, 2023, and Korean Patent Application No. 10-2023-0111383, filed August 24, 2023, the entire contents of which are incorporated herein by reference.

[0003] The rapid development of the Internet has resulted in a dramatic increase in personal communication speeds. This improvement has enabled users to access remote computers, download or upload large amounts of data, or use remote computer control programs to access remote computers as if logged in locally. Furthermore, with the development of diverse applications running on mobile devices like smartphones, virtualization technologies are increasingly being proposed to enable high-performance applications to run on relatively low-performance user devices.

[0004] Cloud streaming technology is a solution that runs applications or services requiring high-performance system resources on a cloud streaming server and delivers only the rendered results to the user's device. This allows high-performance video playback or 3D games even on low-performance devices.

[0005] Meanwhile, cloud streaming for user interfaces (UIs), such as menus or posters displayed on live channels, is primarily achieved through image encoding, as it facilitates transparency processing. However, image encoding has the disadvantage of incurring significant overhead when processing fast-changing screens, such as animations.

[0006] Therefore, the need for technology using video encoding based on video cloud streaming is being emphasized, but there are issues related to processing alpha data, which is transparency information, in the browser of video cloud streaming, such as synchronization with the GPU process frame buffer or additional costs due to resolution adjustment.

[0007] The purpose of the present invention is to process transparency information to match the final screen composition on the GPU process while solving the synchronization problem between the renderer process and the GPU process in the browser of video cloud streaming.

[0008] In addition, an object of the present invention is to efficiently obtain alpha values, which are transparency information, while simultaneously using the Native Windows method, which is a high-performance method, in video cloud streaming.

[0009] In addition, an object of the present invention is to minimize performance degradation due to data transfer between memories by minimizing the amount of data transfer between GPU memories for transparency processing.

[0010] In addition, an object of the present invention is to quickly and efficiently detect a screen change area in an image cloud streaming service based on Native Browser Mode.

[0011] In addition, an object of the present invention is to minimize system performance degradation by minimizing the amount of data transferred between GPU memories for detecting screen changes in an image cloud.

[0012] In addition, an object of the present invention is to minimize latency in image cloud streaming by detecting pixel change areas on a tile-by-tile basis in GPU memory.

[0013] In order to achieve the above-described purpose, a video cloud streaming service providing method according to the present invention comprises the steps of: a cloud streaming server displaying a current frame on a GPU memory using alpha data obtained through a renderer process; a step of determining whether the alpha data on the GPU memory changes between frames; a step of transmitting the alpha data to a CPU memory and encoding it when the alpha data has changed; and a step of streaming the alpha data encoded in the CPU memory and the RGB data encoded in the GPU memory to a user terminal.

[0014] At this time, the judging step further includes a step of copying the frame buffer (FRAME BUFFER) in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the alpha data changes between frames.

[0015] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in the GPU memory with the alpha data of the current frame copied from the frame buffer.

[0016] At this time, the CPU memory can examine the alpha data and encode the alpha data change area.

[0017] At this time, the encoded RGB data may correspond to encoded video element stream data.

[0018] At this time, the browser of the cloud streaming server can be operated in the native Windows mode.

[0019] At this time, the user terminal can decode the encoded alpha data and the encoded RGB data, respectively, and apply the decoded alpha data to the decoded RGB data to display a screen with transparency applied to the user.

[0020] In addition, the cloud streaming server according to the present invention includes a processor that displays a current frame on a GPU memory using alpha data obtained from a renderer process, determines whether the alpha data changes between frames on the GPU memory, and, if the alpha data changes, transmits the alpha data to a CPU memory to encode it, and streams the alpha data encoded in the CPU memory and the RGB data encoded in the GPU memory to a user terminal; and a memory that stores the alpha data.

[0021] At this time, the processor can copy the frame buffer (FRAME BUFFER) in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the alpha data changes between frames.

[0022] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in the GPU memory with the alpha data of the current frame copied from the frame buffer.

[0023] At this time, the CPU memory can examine the alpha data and encode the alpha data change area.

[0024] At this time, the encoded RGB data may correspond to encoded video element stream data.

[0025] At this time, the browser of the cloud streaming server can be operated in the native Windows mode.

[0026] At this time, the user terminal can decode the encoded alpha data and the encoded RGB data, respectively, and apply the decoded alpha data to the decoded RGB data to display a screen with transparency applied to the user.

[0027] In addition, a video cloud streaming service providing method according to the present invention for achieving the above-described purpose includes a step of a cloud streaming server displaying a current frame on a GPU memory using alpha data obtained from a renderer process; a step of determining whether the alpha data on the GPU memory changes between frames on a tile basis; a step of transmitting and encoding alpha data corresponding to the number of changed tiles to a CPU memory when the alpha data has changed; and a step of streaming the encoded alpha data in the CPU memory and the encoded RGB data in the GPU memory to a user terminal.

[0028] At this time, the judging step may include a step of dividing the current frame into a plurality of tiles; and a step of comparing an area corresponding to each of the plurality of tiles with a previous frame to detect whether there is a change in the alpha data.

[0029] At this time, the encoding step can transfer alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among the plurality of tiles to the CPU memory.

[0030] At this time, the size of the plurality of tiles can be determined by considering the service resolution and the data transfer speed from the GPU memory to the CPU memory.

[0031] At this time, the judging step further includes a step of copying the frame buffer (FRAME BUFFER) in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can judge whether the alpha data changes between frames on a tile basis.

[0032] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in the GPU memory with the alpha data of the current frame copied from the frame buffer.

[0033] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0034] In addition, the cloud streaming server according to the present invention includes a processor that displays a current frame on a GPU memory using alpha data obtained from a renderer process, determines whether the alpha data changes between frames on the GPU memory on a tile-by-tile basis, and, if the alpha data changes, transmits and encodes the alpha data corresponding to the number of changed tiles to a CPU memory, and streams the alpha data encoded in the CPU memory and the RGB data encoded in the GPU memory to a user terminal; and a memory that stores the alpha data.

[0035] At this time, the processor can divide the current frame into a plurality of tiles and compare the area corresponding to each of the plurality of tiles with the previous frame to detect whether there is a change in the alpha data.

[0036] At this time, the processor can transmit alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among the plurality of tiles to the CPU memory.

[0037] At this time, the size of the plurality of tiles can be determined by considering the service resolution and the data transfer speed from the GPU memory to the CPU memory.

[0038] At this time, the processor copies the frame buffer in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the alpha data changes between frames on a tile basis.

[0039] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in the GPU memory with the alpha data of the current frame copied from the frame buffer.

[0040] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0041] In addition, the method for providing an image cloud streaming service according to the present invention for achieving the above-described purpose includes the steps of: a cloud streaming server determining whether a screen change has occurred between a current frame and a previous frame on a tile basis on a GPU memory; a step of transmitting a tile in which a screen change has occurred to a CPU memory; a step of calculating a change area for the tile in which the screen change has occurred on the CPU memory; and a step of encoding the change area on the CPU memory and transmitting the same to a user terminal.

[0042] At this time, the judging step may include a step of dividing the current frame into a plurality of tiles; and a step of comparing an area corresponding to each of the plurality of tiles with the previous frame to detect whether the screen has changed.

[0043] At this time, the size of the plurality of tiles can be determined by considering the service resolution and the data transfer speed from the GPU memory to the CPU memory.

[0044] At this time, the judging step further includes a step of copying the frame buffer (FRAME BUFFER) in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the screen has changed on a tile-by-tile basis.

[0045] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0046] At this time, the step of calculating the change area can calculate the change area by performing a loop for the number of tiles in which the screen change occurred on the CPU memory.

[0047] At this time, the transmitting step can transmit pixel values ​​corresponding to the tiles where the screen change occurred to the CPU memory.

[0048] In addition, the cloud streaming server according to the present invention includes a processor that determines whether there is a screen change between a current frame and a previous frame on a tile basis on a GPU memory, transmits a tile where a screen change has occurred to a CPU memory, calculates a change area for the tile where the screen change has occurred on the CPU memory, encodes the change area on the CPU memory, and transmits the change area to a user terminal; and a memory that stores the current frame, the previous frame, and the change area.

[0049] At this time, the processor can divide the current frame into a plurality of tiles and compare the area corresponding to each of the plurality of tiles with the previous frame to detect whether the screen has changed.

[0050] At this time, the size of the plurality of tiles can be determined by considering the service resolution and the data transfer speed from the GPU memory to the CPU memory.

[0051] At this time, the processor copies the frame buffer in the GPU memory to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the screen has changed on a tile-by-tile basis.

[0052] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0053] At this time, the processor can calculate the change area by performing a loop for the number of tiles in which the screen change occurred on the CPU memory.

[0054] At this time, the processor can transfer pixel values ​​corresponding to the tiles where the screen change occurred to the CPU memory.

[0055] According to the present invention, transparency information can be processed to match the final screen composition on the GPU process while resolving the synchronization problem between the renderer process and the GPU process in the browser of video cloud streaming.

[0056] In addition, the present invention can efficiently obtain alpha values, which are transparency information, while simultaneously using the Native Windows method, which is a high-performance method, in video cloud streaming.

[0057] In addition, the present invention can minimize performance degradation due to data transfer between memories by minimizing the amount of data transfer between GPU memories for transparency processing.

[0058] In addition, the present invention can quickly and efficiently detect a screen change area in an image cloud streaming service based on Native Browser Mode.

[0059] In addition, the present invention can minimize system performance degradation by minimizing the amount of data transferred between GPU memories for detecting screen changes in an image cloud.

[0060] In addition, the present invention can minimize latency during image cloud streaming by detecting pixel change areas based on tile units in GPU memory.

[0061] FIG. 1 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0062] FIG. 2 is a flowchart illustrating a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention.

[0063] FIG. 3 is a drawing showing an example of a process for transmitting alpha data according to the present invention.

[0064] Figure 4 is a drawing showing in more detail the process of transmitting alpha data in Figure 3.

[0065] FIG. 5 is a detailed flowchart illustrating a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention.

[0066] FIG. 6 is a block diagram illustrating a cloud streaming server according to one embodiment of the present invention.

[0067] FIG. 7 is a block diagram illustrating a cloud streaming service system according to another embodiment of the present invention.

[0068] FIG. 8 is a flowchart illustrating a method for providing a video cloud streaming service based on efficient segmentation transparency processing according to one embodiment of the present invention.

[0069] Figure 9 is a diagram showing an example of a process for transmitting alpha data without dividing frames.

[0070] FIG. 10 is a drawing showing an example of dividing a frame into a plurality of tiles according to the present invention.

[0071] FIG. 11 is a diagram showing an example of a data transfer process between memories according to the present invention.

[0072] FIG. 12 is a detailed flowchart illustrating a method for providing a video cloud streaming service based on efficient segmentation transparency processing according to one embodiment of the present invention.

[0073] FIG. 13 is a block diagram illustrating a cloud streaming server according to another embodiment of the present invention.

[0074] FIG. 14 is a block diagram illustrating a cloud streaming service system according to another embodiment of the present invention.

[0075] FIG. 15 is a flowchart illustrating a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention.

[0076] Figure 16 is a diagram showing an example of a process for transmitting data without dividing frames.

[0077] Figure 17 is a diagram showing an example of a data transfer process between memories according to the present invention.

[0078] FIG. 18 is a drawing showing an example of dividing a frame into a plurality of tiles according to the present invention.

[0079] FIG. 19 is a detailed flowchart illustrating a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention.

[0080] FIG. 20 is a block diagram illustrating a cloud streaming server according to another embodiment of the present invention.

[0081] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or components that may obscure the gist of the present invention will be omitted in the following description and the accompanying drawings. It should also be noted that, where possible, identical components are indicated by the same reference numerals throughout the drawings.

[0082] The terms and words used in this specification and claims described below should not be interpreted as limited to their conventional or dictionary meanings, but should be interpreted with meanings and concepts that conform to the technical idea of ​​the present invention based on the principle that the inventor can appropriately define the concept of the term to best describe his or her invention. Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are only the most preferred embodiments of the present invention and do not represent all of the technical idea of ​​the present invention. Therefore, it should be understood that there may be various equivalents and modified examples that can replace them at the time of this application. In addition, terms such as “first” and “second” are used to describe various components, and are used only for the purpose of distinguishing one component from another, and are not used to limit the components.

[0083] FIG. 1 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0084] Referring to FIG. 1, a cloud streaming service system according to one embodiment of the present invention includes a cloud streaming server (110), user terminals (120-1 to 120-N), and a network (130).

[0085] Cloud streaming service or cloud computing service is a service that allows users to store content in cloud storage on the Internet rather than in local storage, and access the cloud storage and use the content anytime, anywhere using an Internet-enabled device.

[0086] The cloud streaming server (110) may include a video cloud streaming engine for processing a cloud streaming service requested from a user terminal (120-1 to 120-N).

[0087] At this time, the cloud streaming server (110) can create a predefined number of browsers at the time the cloud streaming engine is executed, and the browser can download and execute HTML, CSS, JavaScript, etc. and render them on the screen.

[0088] At this time, HTML (Hyper Text Markup Language) provides the large framework of a web page, CSS (Cascading Style Sheets) manages design elements such as color and font, and JavaScript is a cross-platform, object-oriented scripting language that can be responsible for the operation of a web page.

[0089] The cloud streaming server (110) displays the current frame on the GPU memory using the alpha data obtained through the renderer process.

[0090] At this time, the browser of the cloud streaming server can be operated in the native Windows mode.

[0091] Additionally, the cloud streaming server (110) determines whether there is a change between frames in alpha data on the GPU memory.

[0092] At this time, the frame buffer in GPU memory can be copied to the GPU memory control framework.

[0093] At this time, the kernel of the GPU memory control framework can determine whether alpha data changes between frames.

[0094] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0095] Additionally, when alpha data changes, the cloud streaming server (110) transmits the alpha data to the CPU memory and encodes it.

[0096] At this time, the CPU memory can examine the alpha data and encode the alpha data change area.

[0097] Additionally, the cloud streaming server (110) streams alpha data encoded in CPU memory and RGB data encoded in GPU memory to user terminals (120-1 to 120-N).

[0098] At this time, the encoded RGB data may correspond to encoded video element stream data.

[0099] At this time, the user terminal (120-1 to 120-N) can decode the coded alpha data and the encoded RGB data, respectively, and apply the decoded alpha data to the decoded RGB data to display a screen with transparency applied to the user.

[0100] User terminals (120-1 to 120-N) can request a cloud streaming service from a cloud streaming server (110), and receive an execution screen according to the request from the cloud streaming server (110) and provide it to the user. Accordingly, the user can feel as if a video or application corresponding to the requested service is running on the user terminal (120-1 to 120-N).

[0101] At this time, the user terminals (120-1 to 120-N) may correspond to devices that are each connected to a communication network and can communicate with the cloud streaming server (110). For example, the user terminals (120-1 to 120-N) may correspond to various terminals such as all information and communication devices, multimedia terminals, wired terminals, fixed terminals, mobile communication terminals, and IP (Internet Protocol) terminals. In addition, the user terminals (120-1 to 120-N) may be mobile terminals having various mobile communication specifications such as cell phones, mobile phones, portable multimedia players (PMPs), mobile Internet devices (MIDs), smart phones, desktops, tablet computers (Tablet PCs), notebooks, netbooks, personal digital assistants (PDAs), smart TVs, and information and communication devices.

[0102] In addition, the user terminals (120-1 to 120-N) can input various information such as number and character information, set various functions, and transmit signals input in relation to the control of the functions of the user terminals (120-1 to 120-N) to the control unit through the input unit. In addition, the input unit of the user terminals (120-1 to 120-N) can be configured to include at least one of a keypad and a touchpad that generate an input signal according to the user's touch or operation. In this case, the input unit of the user terminals (120-1 to 120-N) is configured in the form of a single touch panel (or touch screen) together with the display unit of the user terminals (120-1 to 120-N) so that input and display functions can be performed simultaneously. In addition to input devices such as a keyboard, keypad, mouse, joystick, etc., the input unit of the user terminals (120-1 to 120-N) can use any form of input means that may be developed in the future.

[0103] In addition, the display unit of the user terminal (120-1 to 120-N) can display information about a series of operation states and operation results that occur during the performance of the functions of the user terminal (120-1 to 120-N). In addition, the display unit of the user terminal (120-1 to 120-N) can display menus of the user terminal (120-1 to 120-N) and user data input by the user. Here, the display unit of the user terminal (120-1 to 120-N) can be composed of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), a light emitting diode (LED), an organic light emitting diode (OLED), an active matrix OLED (AMOLED), a retina display, a flexible display, a three-dimensional (3 Dimension) display, etc. At this time, if the display unit of the user terminal (120-1 to 120-N) is configured in the form of a touch screen, the display unit of the user terminal (120-1 to 120-N) can perform some or all of the functions of the input unit of the user terminal (120-1 to 120-N).

[0104] In addition, the storage unit of the user terminal (120-1 to 120-N) is a device for storing data, and includes a main memory device and an auxiliary memory device, and can store application programs required for the functional operation of the user terminal (120-1 to 120-N). The storage unit of the user terminal (120-1 to 120-N) can largely include a program area and a data area. Here, when the user terminal (120-1 to 120-N) activates each function in response to a user request, it executes the corresponding application programs under the control of the control unit to provide each function.

[0105] In addition, the communication unit of the user terminal (120-1 to 120-N) may perform a function for transmitting and receiving data through a network with the cloud streaming server (110). Here, the communication unit of the user terminal (120-1 to 120-N) may include an RF transmitting means for up-converting and amplifying the frequency of a transmitted signal, an RF receiving means for low-noise amplifying a received signal and down-converting the frequency, etc. The communication unit of the user terminal (120-1 to 120-N) may include a wireless communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the user terminal (120-1 to 120-N) uses wireless communication, any one of a wireless network communication module, a wireless LAN communication module, and a wireless fan communication module may be used to transmit and receive data to and from the cloud streaming server (110). That is, the user terminal (120-1 to 120-N) connects to the network using a wireless communication module and can transmit and receive data with the cloud streaming server (110) through the network.

[0106] In addition, the control unit of the user terminal (120-1 to 120-N) may be a process device that drives an operating system (OS) and each component. For example, the control unit may control the entire process of connecting to the cloud streaming server (110). At this time, when the user terminal (120-1 to 120-N) connects to the cloud streaming server (110), the entire process of executing content such as a video or an application according to the user's request may be controlled, and a service use request may be transmitted to the cloud streaming server (110) simultaneously with the execution. At this time, information of the user terminal (120-1 to 120-N) required for user authentication may also be controlled to be transmitted together.

[0107] The network (130) provides a path for transmitting data between the cloud streaming server (110) and the user terminals (120-1 to 120-N), and is a concept that encompasses both existing networks and networks that can be developed in the future. For example, the network (130) may be a wired or wireless local area network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobiles and between mobiles and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired or wireless communication network, or a combination of two or more of them. Meanwhile, the transmission method standard of the network is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future.

[0108] Such a system would allow for the synchronization issue between the renderer process and the GPU process in the browser for video cloud streaming, while allowing transparency information to be processed on the GPU process to match the final screen composition.

[0109] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0110] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0111] FIG. 2 is a flowchart illustrating a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention.

[0112] Referring to FIG. 2, a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention causes a cloud streaming server to display a current frame on a GPU memory using alpha data obtained through a renderer process (S210).

[0113] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0114] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0115] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0116] At this time, the browser of the cloud streaming server can be operated in the native Windows mode.

[0117] Typically, obtaining alpha values ​​for transparency data from a video cloud streaming server requires the browser to operate in Off-Screen Rendering (OSR) mode. However, OSR mode may not be suitable for seamlessly delivering video cloud streaming services, as it has lower performance than native Windows mode.

[0118] However, when using the Native Windows method, which is higher-performance than the OSR method, the APIs for GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen. Furthermore, even if transparency information is acquired during the renderer process, the screen ultimately displayed to the user is processed through the GPU process, which may cause additional related issues such as additional costs for synchronization and resolution adjustment.

[0119] To solve these problems, the present invention proposes a method for efficiently processing transparency information on a GPU process while operating a browser in a native Windows manner when providing a video cloud streaming service.

[0120] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0121] In addition, in a method for providing a video cloud streaming service based on efficient transparency processing according to one embodiment of the present invention, a cloud streaming server determines whether there is a change between frames of alpha data on the GPU memory (S220).

[0122] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0123] Therefore, in the present invention, it is possible to minimize the amount of data transferred between memories by determining whether there is a change between frames for alpha data and transmitting alpha data only when a change occurs.

[0124] At this time, the frame buffer in GPU memory can be copied to the GPU memory control framework.

[0125] At this time, the kernel of the GPU memory control framework can determine whether alpha data changes between frames.

[0126] For example, in the present invention, data to be transmitted to CPU memory can be selected by determining alpha data with changes using a GPU memory control framework such as CUDA.

[0127] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0128] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0129] In addition, in a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention, a cloud streaming server transmits alpha data to CPU memory and encodes the alpha data when the alpha data changes (S230).

[0130] At this time, the CPU memory can examine the alpha data and encode the alpha data change area.

[0131] That is, in order to encode alpha data even in CPU memory, a changed alpha area can be calculated. According to the present invention, an alpha data change area can be quickly obtained by performing a calculation on alpha data transmitted from GPU memory.

[0132] In addition, in a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention, a cloud streaming server streams alpha data encoded in CPU memory and RGB data encoded in GPU memory to a user terminal (S240).

[0133] At this time, the encoded RGB data may correspond to encoded video element stream data.

[0134] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0135] At this time, the user terminal can decode the encoded alpha data and the encoded RGB data, respectively, and apply the decoded alpha data to the decoded RGB data to display a screen with transparency applied to the user.

[0136] For example, a user terminal may receive encoded alpha data and encoded video element stream data, and then decode each piece of data. Thereafter, the decoded video element stream data and alpha data may be combined to display a video screen with transparency changes applied to the user.

[0137] This method of providing a video cloud streaming service based on efficient transparency processing can solve the synchronization problem between the renderer process and the GPU process in the browser of video cloud streaming, while processing transparency information to match the final screen composition on the GPU process.

[0138] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0139] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0140] FIG. 3 is a drawing showing an example of a process for transmitting alpha data according to the present invention.

[0141] Referring to FIG. 3, the process of comparing the current frame alpha data copied from the frame buffer and the previous frame alpha data in the GPU memory (321) of the browser (300) of the cloud streaming server and transferring the entire frame alpha data to the CPU memory (330) when the alpha data has changed is shown.

[0142] To explain in more detail, the alpha data of the current frame obtained through the renderer process (310) in the browser (300) of the video cloud streaming server can be shared with the GPU process (320).

[0143] At this time, the renderer process (310) may correspond to a rendering process for composing a screen to be output to a browser. For example, when an application execution screen is received from a web application server executing a web application, the renderer processor (310) may compute the received application execution screen to compose a screen so that it can be output to a browser of a video cloud streaming server.

[0144] At this time, the GPU process (320) corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0145] At this time, the browser (300) can be driven in a native window mode.

[0146] Typically, obtaining alpha values ​​for transparency data from a video cloud streaming server requires the browser to operate in Off-Screen Rendering (OSR) mode. However, OSR mode may not be suitable for seamlessly delivering video cloud streaming services, as it has lower performance than native Windows mode.

[0147] However, when using the Native Windows method, which is higher-performance than the OSR method, the APIs for GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen. Furthermore, even if transparency information is acquired during the renderer process, the screen ultimately displayed to the user is processed through the GPU process, which may cause additional related issues such as additional costs for synchronization and resolution adjustment.

[0148] To solve these problems, the present invention proposes a method for efficiently processing transparency information on a GPU process while operating a browser in a native Windows manner when providing a video cloud streaming service.

[0149] Referring again to FIG. 3, the GPU process (320), which has received alpha data of the current frame from the renderer process (310), determines whether there is a change in the alpha data between frames on the GPU memory (321).

[0150] At this time, the frame buffer (FRAME BUFFER) in the GPU memory (321) can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether there is a change in alpha data between frames.

[0151] For example, as illustrated in Figure 3, a GPU memory control framework such as CUDA can be used to select data to be transferred to CPU memory by determining alpha data that has changed.

[0152] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0153] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0154] After this, if the alpha data changes, the alpha data is transmitted to the CPU memory (330) and encoded.

[0155] That is, since data is transferred from the GPU memory (321) to the CPU memory (330) only when a change in alpha data occurs between frames, the amount of data transferred between memories can be reduced.

[0156] After this, the encoded alpha data is transmitted to the CPU memory (330) and can be streamed to the user terminal together with the encoded RGB data in the GPU memory (321).

[0157] Figure 4 is a drawing showing in more detail the process of transmitting alpha data in Figure 3.

[0158] Referring to FIG. 4, the GPU process that has received alpha data of the current frame from the renderer process determines whether there is a change in alpha data between frames in the GPU memory (410).

[0159] At this time, the frame buffer (FRAME BUFFER) in the GPU memory (410) can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether there is a change in alpha data between frames.

[0160] For example, as illustrated in FIG. 4, data to be transmitted to CPU memory (420) can be selected by determining alpha data with changes using a GPU memory control framework such as CUDA.

[0161] At this time, the GPU memory control framework can compare the alpha data (412) of the previous frame stored in the GPU memory (410) with the alpha data (411) of the current frame copied from the frame buffer.

[0162] That is, while processing the previous frame, a comparison between the alpha data (412) of the previous frame stored in the GPU memory and the alpha data (411) of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0163] After this, if the alpha data has changed, the alpha data (430) is transmitted to the CPU memory (420) and encoded.

[0164] That is, data is transferred from GPU memory (410) to CPU memory (420) only when alpha data changes between frames, thereby reducing the amount of data transferred between memories.

[0165] At this time, the CPU memory (420) can search for alpha data (FIND CHANGE ALPHA) based on a separate alpha compression module (421) and encode the alpha data change area (COMPRESS CHANGE ALPHA).

[0166] That is, in order to encode alpha data even in CPU memory, a changed alpha area can be calculated. According to the present invention, an alpha data change area can be quickly obtained by performing a calculation on alpha data transmitted from GPU memory.

[0167] In this way, by significantly reducing the amount of data transferred from the GPU memory (410) to the CPU memory (420), performance degradation due to data transfer can be prevented.

[0168] After this, the encoded alpha data is transmitted to the CPU memory (420) and can be streamed to the user terminal together with the encoded RGB data in the GPU memory (410).

[0169] This process allows us to handle transparency information on the GPU process to match the final screen composition, while solving synchronization issues between the renderer process and the GPU process in the browser for video cloud streaming.

[0170] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0171] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0172] FIG. 5 is a detailed flowchart illustrating a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention.

[0173] Referring to FIG. 5, a method for providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention can first output an RGBA screen corresponding to the current frame to a frame buffer in GPU memory using alpha data of the current frame obtained through a renderer process (S510).

[0174] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0175] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0176] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0177] At this time, the browser can be run in NATIVE WINDOWS mode.

[0178] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0179] After this, the frame buffer in GPU memory can be copied to the GPU memory control framework (CUDA) (S520).

[0180] After this, the CUDA kernel can detect whether there is a change in alpha data by comparing the previous frame and the current frame (S530).

[0181] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0182] Therefore, in the present invention, it is possible to minimize the amount of data transferred between memories by determining whether there is a change between frames for alpha data and transmitting alpha data only when a change occurs.

[0183] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0184] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0185] After this, it is determined whether a change in alpha data has been detected (S535), and if a change in alpha data has been detected, the alpha data in which a change has been detected can be transmitted to the CPU memory and encoded (S540).

[0186] At this time, the CPU memory can examine the changed alpha data and encode the alpha data change area.

[0187] That is, in order to encode alpha data even in CPU memory, a changed alpha area can be calculated. According to the present invention, an alpha data change area can be quickly obtained by performing a calculation on alpha data transmitted from GPU memory.

[0188] After this, RGB data encoding can be performed in GPU memory (S550).

[0189] At this time, the encoded RGB data may correspond to video element stream data.

[0190] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0191] After this, the encoded data can be streamed to the user terminal (S560).

[0192] At this time, the user terminal can display a screen with transparency changes applied by combining encoded alpha data and encoded RGB data.

[0193] For example, a user terminal may receive encoded alpha data and encoded video element stream data, and then decode each piece of data. Thereafter, the decoded video element stream data and alpha data may be combined to display a video screen with transparency changes applied to the user.

[0194] In addition, if no change in alpha data is detected as a result of the judgment in step (S535), the encoding process for alpha data may be omitted, RGB data encoding may be performed through step (S550), and only the encoded RGB data may be streamed to the user terminal (S560).

[0195] This method allows us to handle transparency information on the GPU process to match the final screen composition, while solving synchronization issues between the renderer process and the GPU process in the browser for video cloud streaming.

[0196] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0197] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0198] FIG. 6 is a block diagram illustrating a cloud streaming server according to one embodiment of the present invention.

[0199] Referring to FIG. 6, a cloud streaming server according to one embodiment of the present invention includes a communication unit (610), a processor (620), and a memory (630).

[0200] The communication unit (610) serves to transmit and receive information related to multiple user terminals via a communication network, such as the network illustrated in FIG. 1. In particular, the communication unit (610) according to one embodiment of the present invention may receive a request for a cloud streaming service from a user terminal and provide a data packet corresponding to the cloud streaming service requested by the user terminal to the user terminal.

[0201] Here, the network provides a path for transmitting data between the cloud streaming server and the user terminal, and is a concept encompassing both existing networks and networks that may be developed in the future. For example, the network may be a wired or wireless local area network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobile devices and between mobile devices and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired or wireless communication network, or a combination of two or more of these. Meanwhile, the network transmission method standard is not limited to existing transmission method standards and may include all transmission method standards that will be developed in the future.

[0202] The processor (620) displays the current frame on the GPU memory using the alpha data obtained through the renderer process.

[0203] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0204] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0205] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0206] At this time, the browser of the cloud streaming server can be operated in the native Windows mode.

[0207] Typically, obtaining alpha values ​​for transparency data from a video cloud streaming server requires the browser to operate in Off-Screen Rendering (OSR) mode. However, OSR mode may not be suitable for seamlessly delivering video cloud streaming services, as it has lower performance than native Windows mode.

[0208] However, when using the Native Windows method, which is higher-performance than the OSR method, the APIs for GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen. Furthermore, even if transparency information is acquired during the renderer process, the screen ultimately displayed to the user is processed through the GPU process, which may cause additional related issues such as additional costs for synchronization and resolution adjustment.

[0209] To solve these problems, the present invention proposes a method for efficiently processing transparency information on a GPU process while operating a browser in a native Windows manner when providing a video cloud streaming service.

[0210] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0211] Additionally, the processor (620) determines whether there is a change between frames in alpha data on the GPU memory.

[0212] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0213] Therefore, in the present invention, it is possible to minimize the amount of data transferred between memories by determining whether there is a change between frames for alpha data and transmitting alpha data only when a change occurs.

[0214] At this time, the frame buffer in GPU memory can be copied to the GPU memory control framework.

[0215] At this time, the kernel of the GPU memory control framework can determine whether alpha data changes between frames.

[0216] For example, in the present invention, data to be transmitted to CPU memory can be selected by determining alpha data with changes using a GPU memory control framework such as CUDA.

[0217] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0218] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0219] Additionally, the processor (620) encodes the alpha data by transmitting it to the CPU memory when the alpha data changes.

[0220] At this time, the CPU memory can examine the alpha data and encode the alpha data change area.

[0221] That is, in order to encode alpha data even in CPU memory, a changed alpha area can be calculated. According to the present invention, an alpha data change area can be quickly obtained by performing a calculation on alpha data transmitted from GPU memory.

[0222] Additionally, the processor (620) streams alpha data encoded in CPU memory and RGB data encoded in GPU memory to the user terminal.

[0223] At this time, the encoded RGB data may correspond to encoded video element stream data.

[0224] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0225] At this time, the user terminal can decode the encoded alpha data and the encoded RGB data, respectively, and apply the decoded alpha data to the decoded RGB data to display a screen with transparency applied to the user.

[0226] Memory (630) stores alpha data.

[0227] In addition, the memory (630) stores various information generated in the process of providing a video cloud streaming service based on efficient transparency processing according to an embodiment of the present invention as described above.

[0228] In some embodiments, the memory (630) may be configured independently from the cloud streaming server to support functions for providing a video cloud streaming service based on efficient transparency processing. In this case, the memory (630) may function as a separate large-capacity storage and may include control functions for performing operations.

[0229] Additionally, the cloud streaming server configured as described above may be implemented with one or more servers.

[0230] Meanwhile, the cloud streaming server is equipped with memory and can store information within the device. In one embodiment, the memory is a computer-readable medium. In one embodiment, the memory may be a volatile memory unit, and in another embodiment, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various other embodiments, the storage device may include, for example, a hard disk drive, an optical disk drive, or any other mass storage device.

[0231] Such cloud streaming servers allow for the synchronization between the renderer process and the GPU process in the browser of video cloud streaming, while allowing transparency information to be processed on the GPU process to match the final screen composition.

[0232] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0233] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0234] FIG. 7 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0235] Referring to FIG. 7, a cloud streaming service system according to one embodiment of the present invention includes a cloud streaming server (710), user terminals (720-1 to 720-N), and a network (730).

[0236] The cloud streaming server (710) displays the current frame on the GPU memory using the alpha data obtained from the renderer process.

[0237] Additionally, the cloud streaming server (710) determines whether there is a change between frames of alpha data on a tile-by-tile basis on the GPU memory.

[0238] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether there is a change in alpha data.

[0239] At this time, the size of multiple tiles can be determined by considering the service resolution and the data transfer speed from GPU memory to CPU memory.

[0240] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether alpha data changes between frames on a tile-by-tile basis.

[0241] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0242] Additionally, when alpha data changes, the cloud streaming server (710) transmits alpha data corresponding to the number of changed tiles to the CPU memory and encodes it.

[0243] At this time, alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among multiple tiles can be transferred to the CPU memory.

[0244] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0245] Additionally, the cloud streaming server (710) streams alpha data encoded in CPU memory and RGB data encoded in GPU memory to user terminals (720-1 to 720-N).

[0246] User terminals (720-1 to 720-N) can request a cloud streaming service from a cloud streaming server (710), and receive an execution screen according to the request from the cloud streaming server (710) and provide it to the user. Accordingly, the user can feel as if a video or application corresponding to the requested service is running on the user terminal (720-1 to 720-N).

[0247] The network (730) provides a path for transmitting data between the cloud streaming server (710) and the user terminals (720-1 to 720-N), and is a concept that encompasses both existing networks and networks that can be developed in the future. For example, the network (730) may be a wired or wireless local area network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobiles and between mobiles and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired or wireless communication network, or a combination of two or more of them. Meanwhile, the transmission method standard of the network is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future.

[0248] Such a system would allow for the synchronization issue between the renderer process and the GPU process in the browser for video cloud streaming, while allowing transparency information to be processed on the GPU process to match the final screen composition.

[0249] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0250] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0251] FIG. 8 is a flowchart illustrating a method for providing a video cloud streaming service based on efficient segmentation transparency processing according to one embodiment of the present invention.

[0252] Referring to FIG. 8, a video cloud streaming service providing method based on efficient split transparency processing according to one embodiment of the present invention is such that a cloud streaming server displays a current frame on GPU memory using alpha data obtained from a renderer process (S810).

[0253] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0254] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0255] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0256] At this time, the browser can be run in NATIVE WINDOWS mode.

[0257] Typically, obtaining alpha values ​​for transparency data from a video cloud streaming server requires the browser to operate in Off-Screen Rendering (OSR) mode. However, OSR mode may not be suitable for seamlessly delivering video cloud streaming services, as it has lower performance than native Windows mode.

[0258] However, when using the Native Windows method, which is higher-performance than the OSR method, the APIs for GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen. Furthermore, even if transparency information is acquired during the renderer process, the screen ultimately displayed to the user is processed through the GPU process, which may cause additional related issues such as additional costs for synchronization and resolution adjustment.

[0259] To solve these problems, the present invention proposes a method for efficiently processing transparency information on a GPU process while operating a browser in a native Windows manner when providing a video cloud streaming service.

[0260] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0261] In addition, in a method for providing a video cloud streaming service based on efficient segmentation transparency processing according to one embodiment of the present invention, a cloud streaming server determines whether there is a change between frames of the alpha data on a tile basis in GPU memory (S820).

[0262] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether there is a change in alpha data.

[0263] For example, the GPU memory area that stores the current frame can be divided into NxN sized tiles, and then the previous frame and the current frame can be compared for each divided tile area to detect changes in alpha data.

[0264] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0265] Additionally, even if alpha data is transferred to CPU memory only when a change in alpha data occurs, if the resolution of the frame is large, the value of alpha data transferred from GPU memory to CPU memory may be turned on, which may cause performance degradation.

[0266] Therefore, in the present invention, when determining whether there is a change between frames for alpha data, the change can be determined in units of tiles rather than the entire frame in order to minimize the amount of data transferred between memories.

[0267] At this time, the size of multiple tiles can be determined by considering the service resolution and the data transfer speed from GPU memory to CPU memory.

[0268] For example, if the service resolution is large, the tile size can be set small so that the alpha data size of the area corresponding to one tile is small.

[0269] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0270] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether alpha data changes between frames on a tile-by-tile basis.

[0271] For example, in the present invention, a GPU memory control framework such as CUDA can be used to select the minimum data to be transferred to CPU memory by judging the alpha data with changes in units of tiles.

[0272] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0273] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0274] In addition, in a video cloud streaming service providing method based on efficient segmentation transparency processing according to an embodiment of the present invention, when alpha data has changed, the cloud streaming server transmits and encodes alpha data corresponding to the number of changed tiles to CPU memory (S830).

[0275] At this time, alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among multiple tiles can be transferred to the CPU memory.

[0276] That is, instead of transferring the entire area of ​​the current frame to CPU memory, alpha data can be transferred from GPU memory to CPU memory only for the number of tiles where the alpha data has changed.

[0277] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0278] That is, in order to encode alpha data even in CPU memory, it is possible to calculate the changed area among the entire frame area, but according to the present invention, the alpha data change area can be quickly obtained by performing calculations only for the area corresponding to the tile transferred from GPU memory, not the entire frame area.

[0279] In addition, in a method for providing a video cloud streaming service based on efficient split transparency processing according to one embodiment of the present invention, a cloud streaming server streams alpha data encoded in CPU memory and RGB data encoded in GPU memory to a user terminal (S840).

[0280] At this time, the encoded RGB data may correspond to video element stream data.

[0281] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0282] At this time, the user terminal can display a screen with transparency changes applied by combining encoded alpha data and encoded RGB data.

[0283] For example, a user terminal may receive encoded alpha data and encoded video element stream data, and then decode each piece of data. Thereafter, the decoded video element stream data and alpha data may be combined to display a video screen with transparency changes applied to the user.

[0284] This method of providing a video cloud streaming service based on efficient split transparency processing solves the synchronization problem between the renderer process and the GPU process in the browser of video cloud streaming, while processing transparency information to match the final screen composition on the GPU process.

[0285] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0286] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0287] Figure 9 is a diagram showing an example of a process for transmitting alpha data without dividing frames.

[0288] Referring to FIG. 9, the process of comparing the current frame alpha data (911) copied from the frame buffer in the GPU memory (910) of the browser of the cloud streaming server with the previous frame alpha data (912) and transferring the entire frame alpha data (930) to the CPU memory (920) when the alpha data has changed is illustrated.

[0289] That is, the method illustrated in FIG. 9 can reduce the amount of data transferred between memories because data is transferred from the GPU memory (910) to the CPU memory (920) only when alpha data changes occur between frames. However, if the resolution of the frame being served is large, the amount of data transferred between memories is bound to increase even if data is transferred only when alpha data changes occur.

[0290] Therefore, in order to solve this problem, the present invention proposes a method of dividing a GPU memory area for storing frames into multiple tiles and processing them, as illustrated in FIG. 10.

[0291] FIG. 10 is a drawing showing an example of dividing a frame into a plurality of tiles according to the present invention.

[0292] Referring to FIG. 10, the present invention uses a GPU memory control framework (1000) such as CUDA to determine alpha data with changes in units of tiles (1010-1 to 1010-(NxN)), thereby selecting the minimum data to be transferred to the CPU memory.

[0293] That is, as illustrated in Fig. 10, the GPU memory where the current frame is stored is divided into NxN-sized tiles (1010-1 to 1010-(NxN)), and the CUDA kernel can detect changes in alpha data for each tile. If a change in alpha data is detected only in tile (1010-1), the amount of data transferred to the CPU memory can be minimized by transferring only the alpha data for tile (1010-1) to the CPU memory.

[0294] In addition, by only examining the alpha data for the tile (1010-1) received from the GPU memory in the CPU memory and calculating and encoding the alpha data change area, the operation for transparency change can be minimized.

[0295] FIG. 11 is a diagram showing an example of a data transfer process between memories according to the present invention.

[0296] Referring to FIG. 11, the data transfer process between memories according to the present invention can compare the current frame alpha data (1111) copied from the frame buffer in the GPU memory (1110) of the browser of the cloud streaming server with the previous frame alpha data (1112).

[0297] At this time, the present invention can compare alpha data on a tile-by-tile basis as illustrated in Fig. 10. If, as a result of the comparison, there is a tile whose alpha data has changed, the tile (1131) whose alpha data has changed and the alpha data (1132) of the area corresponding to the changed tile can be transmitted to the CPU memory (1120).

[0298] That is, compared to the process illustrated in FIG. 9, in the process according to the present invention illustrated in FIG. 11, rather than transmitting alpha data of the entire frame area to the CPU memory (1120), only the tiles (1131) in which alpha data has changed and the alpha data (1132) of the area corresponding to the changed tiles are transmitted to the CPU memory (1120), thereby significantly reducing the amount of data transferred between memories.

[0299] In this way, by significantly reducing the amount of data transferred from the GPU memory (1110) to the CPU memory (1120), performance degradation due to data transfer can be prevented.

[0300] FIG. 12 is a detailed flowchart illustrating a method for providing a video cloud streaming service based on efficient segmentation transparency processing according to one embodiment of the present invention.

[0301] Referring to FIG. 12, a method for providing a video cloud streaming service based on efficient split transparency processing according to an embodiment of the present invention can first output an RGBA screen corresponding to the current frame to a frame buffer in GPU memory using alpha data of the current frame obtained through a renderer process (S1210).

[0302] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0303] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0304] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0305] At this time, the browser can be run in NATIVE WINDOWS mode.

[0306] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0307] After this, the frame buffer in GPU memory can be copied to the GPU memory control framework (CUDA) (S1220).

[0308] After this, the GPU memory area storing the current frame is divided into NxN sized tiles (S1230), and the CUDA kernel can compare the previous frame and the current frame for each divided tile area to detect whether there is a change in alpha data (S1240).

[0309] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0310] Additionally, even if alpha data is transferred to CPU memory only when a change in alpha data occurs, if the resolution of the frame is large, the value of alpha data transferred from GPU memory to CPU memory may be turned on, which may cause performance degradation.

[0311] Therefore, in the present invention, when determining whether there is a change between frames for alpha data, the change can be determined in units of tiles rather than the entire frame in order to minimize the amount of data transferred between memories.

[0312] At this time, the size of multiple tiles can be determined by considering the service resolution and the data transfer speed from GPU memory to CPU memory.

[0313] For example, if the service resolution is large, the tile size can be set small so that the alpha data size of the area corresponding to one tile is small.

[0314] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0315] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0316] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0317] After this, it is determined whether a change in alpha data has been detected (S1245), and if a change in alpha data has been detected, the alpha data of the tile in which the change has been detected can be transmitted to the CPU memory (S1250).

[0318] At this time, alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among multiple tiles can be transferred to the CPU memory.

[0319] That is, instead of transferring the entire area of ​​the current frame to CPU memory, alpha data can be transferred from GPU memory to CPU memory only for the number of tiles where the alpha data has changed.

[0320] Additionally, if no change in alpha data is detected as a result of the judgment in step (S1245), it can be determined whether a change in alpha data has been detected for all segmented tiles (S1255).

[0321] As a result of the judgment in step (S1255), if changes in alpha data have not yet been detected for all tiles, the steps can be performed sequentially from step (S1245).

[0322] In addition, if it is determined in step (S1255) that alpha data has changed for all tiles, alpha data encoding can be performed in CPU memory and RGB data encoding can be performed in GPU memory (S1260).

[0323] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0324] That is, in order to encode alpha data even in CPU memory, it is possible to calculate the changed area among the entire frame area, but according to the present invention, the alpha data change area can be quickly obtained by performing calculations only for the area corresponding to the tile transferred from GPU memory, not the entire frame area.

[0325] At this time, the encoded RGB data may correspond to video element stream data.

[0326] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0327] After this, the encoded data can be streamed to the user terminal (S1270).

[0328] At this time, the user terminal can display a screen with transparency changes applied by combining encoded alpha data and encoded RGB data.

[0329] For example, a user terminal may receive encoded alpha data and encoded video element stream data, and then decode each piece of data. Thereafter, the decoded video element stream data and alpha data may be combined to display a video screen with transparency changes applied to the user.

[0330] FIG. 13 is a block diagram illustrating a cloud streaming server according to one embodiment of the present invention.

[0331] Referring to FIG. 13, a cloud streaming server according to one embodiment of the present invention includes a communication unit (1310), a processor (1320), and a memory (1330).

[0332] The communication unit (1310) transmits and receives information related to multiple user terminals via a communication network, such as the network illustrated in FIG. 7. In particular, the communication unit (1310) according to one embodiment of the present invention may receive a request for a cloud streaming service from a user terminal and provide a data packet corresponding to the cloud streaming service requested by the user terminal to the user terminal.

[0333] The processor (1320) displays the current frame on the GPU memory using the alpha data obtained from the renderer process.

[0334] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0335] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0336] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0337] At this time, the browser can be run in NATIVE WINDOWS mode.

[0338] Typically, obtaining alpha values ​​for transparency data from a video cloud streaming server requires the browser to operate in Off-Screen Rendering (OSR) mode. However, OSR mode may not be suitable for seamlessly delivering video cloud streaming services, as it has lower performance than native Windows mode.

[0339] However, when using the Native Windows method, which is higher-performance than the OSR method, the APIs for GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen. Furthermore, even if transparency information is acquired during the renderer process, the screen ultimately displayed to the user is processed through the GPU process, which may cause additional related issues such as additional costs for synchronization and resolution adjustment.

[0340] To solve these problems, the present invention proposes a method for efficiently processing transparency information on a GPU process while operating a browser in a native Windows manner when providing a video cloud streaming service.

[0341] At this time, the RGBA values ​​of the current frame including alpha data can be written to the Frame Buffer in GPU memory.

[0342] Additionally, the processor (1320) determines whether there is a change between frames of the alpha data on a tile-by-tile basis in the GPU memory.

[0343] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether there is a change in alpha data.

[0344] For example, the GPU memory area that stores the current frame can be divided into NxN sized tiles, and then the previous frame and the current frame can be compared for each divided tile area to detect changes in alpha data.

[0345] At this time, the actual user viewing the application execution screen through the user terminal will not see the screen with the alpha data applied, but the alpha data can be transmitted to the user terminal. To compress and transmit the alpha data to the user terminal, the frame buffer in GPU memory must be brought into CPU memory. However, because the cost of transmitting data from GPU memory to CPU memory is high, the transmission of minimal data is necessary.

[0346] Additionally, even if alpha data is transferred to CPU memory only when a change in alpha data occurs, if the resolution of the frame is large, the value of alpha data transferred from GPU memory to CPU memory may be turned on, which may cause performance degradation.

[0347] Therefore, in the present invention, when determining whether there is a change between frames for alpha data, the change can be determined in units of tiles rather than the entire frame in order to minimize the amount of data transferred between memories.

[0348] At this time, the size of multiple tiles can be determined by considering the service resolution and the data transfer speed from GPU memory to CPU memory.

[0349] For example, if the service resolution is large, the tile size can be set small so that the alpha data size of the area corresponding to one tile is small.

[0350] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0351] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether alpha data changes between frames on a tile-by-tile basis.

[0352] For example, in the present invention, a GPU memory control framework such as CUDA can be used to select the minimum data to be transferred to CPU memory by judging the alpha data with changes in units of tiles.

[0353] At this time, the GPU memory control framework can compare the alpha data of the previous frame stored in GPU memory with the alpha data of the current frame copied from the frame buffer.

[0354] That is, while processing the previous frame, a comparison between the alpha data of the previous frame stored in GPU memory and the alpha data of the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0355] Additionally, when alpha data changes, the processor (1320) transmits alpha data corresponding to the number of changed tiles to the CPU memory and encodes it.

[0356] At this time, alpha data of an area corresponding to a tile whose alpha data has changed based on the previous frame among multiple tiles can be transferred to the CPU memory.

[0357] That is, instead of transferring the entire area of ​​the current frame to CPU memory, alpha data can be transferred from GPU memory to CPU memory only for the number of tiles where the alpha data has changed.

[0358] At this time, the CPU memory can encode the alpha data change area by examining the alpha data of the area corresponding to the changed tile.

[0359] That is, in order to encode alpha data even in CPU memory, it is possible to calculate the changed area among the entire frame area, but according to the present invention, the alpha data change area can be quickly obtained by performing calculations only for the area corresponding to the tile transferred from GPU memory, not the entire frame area.

[0360] Additionally, the processor (1320) streams alpha data encoded in the CPU memory and RGB data encoded in the GPU memory to the user terminal.

[0361] At this time, the encoded RGB data may correspond to video element stream data.

[0362] For example, a video cloud streaming server can generate video element stream data by capturing and encoding based on texture values ​​transmitted corresponding to screen change detection through a GPU process. In other words, the GPU process that processes the screen displayed to the user can detect changes in the application execution screen received through the web application server. Therefore, when a screen change is detected, the texture values ​​corresponding to the changed screen can be captured and encoded to generate video element stream data.

[0363] At this time, the user terminal can display a screen with transparency changes applied by combining encoded alpha data and encoded RGB data.

[0364] For example, a user terminal may receive encoded alpha data and encoded video element stream data, and then decode each piece of data. Thereafter, the decoded video element stream data and alpha data may be combined to display a video screen with transparency changes applied to the user.

[0365] Memory (1330) stores alpha data.

[0366] In addition, the memory (1330) stores various information generated in the process of providing a video cloud streaming service based on efficient segmentation transparency processing according to an embodiment of the present invention as described above.

[0367] In some embodiments, the memory (1330) may be configured independently from the cloud streaming server to support functions for providing a video cloud streaming service based on efficient segmentation transparency processing. In this case, the memory (1330) may function as a separate large-capacity storage device and may include control functions for performing operations.

[0368] Additionally, the cloud streaming server configured as described above may be implemented with one or more servers.

[0369] Meanwhile, the cloud streaming server is equipped with memory and can store information within the device. In one embodiment, the memory is a computer-readable medium. In one embodiment, the memory may be a volatile memory unit, and in another embodiment, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various other embodiments, the storage device may include, for example, a hard disk drive, an optical disk drive, or any other mass storage device.

[0370] Such cloud streaming servers allow for the synchronization between the renderer process and the GPU process in the browser of video cloud streaming, while allowing transparency information to be processed on the GPU process to match the final screen composition.

[0371] Additionally, it is possible to efficiently obtain alpha values, which are transparency information, while using the Native Windows method, which is a high-performance method for video cloud streaming.

[0372] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for transparency processing, performance degradation due to data transfer between memories can be minimized.

[0373] FIG. 14 is a block diagram illustrating a cloud streaming service system according to one embodiment of the present invention.

[0374] Referring to FIG. 14, a cloud streaming service system according to one embodiment of the present invention includes a cloud streaming server (1410), user terminals (1420-1 to 1420-N), and a network (1430).

[0375] The cloud streaming server (1410) determines whether there is a screen change between the current frame and the previous frame on a tile-by-tile basis in the GPU memory.

[0376] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether the screen has changed.

[0377] At this time, the size of the multiple tiles can be determined by considering the service resolution and the data transfer speed from CPU memory to CPU memory.

[0378] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the screen has changed on a tile-by-tile basis.

[0379] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0380] Additionally, the cloud streaming server (1410) transmits tiles in which screen changes have occurred to the CPU memory.

[0381] At this time, the pixel values ​​corresponding to the tiles where the screen change occurred can be transferred to the CPU memory.

[0382] Additionally, the cloud streaming server (1410) calculates a change area for tiles where a screen change has occurred on the CPU memory.

[0383] At this time, the change area can be calculated by performing a loop for the number of tiles where the screen change occurred on the CPU memory.

[0384] Additionally, the cloud streaming server (1410) encodes the change area on the CPU memory and transmits it to the user terminal (1420-1 to 120-N).

[0385] User terminals (1420-1 to 1420-N) can request a cloud streaming service from a cloud streaming server (1410), and receive an execution screen according to the request from the cloud streaming server (1410) and provide it to the user. Accordingly, the user can feel as if a video or application corresponding to the requested service is running on the user terminal (1420-1 to 1420-N).

[0386] The network (1430) provides a path for transmitting data between the cloud streaming server (110) and the user terminals (1420-1 to 1420-N), and is a concept that encompasses both existing networks and networks that can be developed in the future. For example, the network (1430) may be a wired or wireless local area network that provides communication between various information devices within a limited area, a mobile communication network that provides communication between mobiles and between mobiles and the outside world, a satellite communication network that provides communication between earth stations using satellites, or a wired or wireless communication network, or a combination of two or more of them. Meanwhile, the transmission method standard of the network is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future.

[0387] Through this system, screen change areas can be detected quickly and efficiently in image cloud streaming services based on Native Browser Mode.

[0388] In addition, the present invention can minimize system performance degradation by minimizing the amount of data transferred between GPU memories for detecting screen changes in an image cloud.

[0389] In addition, the present invention can minimize latency during image cloud streaming by detecting pixel change areas based on tile units in GPU memory.

[0390] FIG. 15 is a flowchart illustrating a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention.

[0391] Referring to FIG. 15, in a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention, a cloud streaming server determines whether there is a screen change between a current frame and a previous frame on a tile basis in GPU memory (S1510).

[0392] At this time, the browser can be run in NATIVE WINDOWS mode.

[0393] Typically, achieving transparency on cloud streaming servers requires the browser to operate in Off-Screen Rendering (OSR). However, OSR has lower performance than native Windows, making it unsuitable for seamlessly delivering services with fast screen changes, such as video cloud streaming.

[0394] However, when using the Native Windows method, which is higher performing than the OSR method, the APIs for receiving GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen.

[0395] In this regard, the present invention can obtain alpha data (transparency information) with minimal load by acquiring alpha data through a renderer processor. Therefore, the transparency information required for image cloud streaming can be used even in native browser mode using the native Windows method.

[0396] At this time, the cloud streaming server can obtain alpha data through the renderer process and use the obtained alpha data to display the current frame in a form corresponding to RGBA on the GPU memory.

[0397] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0398] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0399] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0400] At this time, the RGBA values ​​of the current frame can be written to the frame buffer in the GPU memory.

[0401] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether the screen has changed.

[0402] For example, the GPU memory area that stores the current frame can be divided into NxN sized tiles, and then the screen can be detected by comparing the previous frame and the current frame for each divided tile area.

[0403] At this time, image cloud streaming requires rapid detection of areas where pixel differences occur between the previous and current frames. Conventionally, this process was handled via the CPU. This required retrieving the frame buffer from GPU memory to CPU memory. However, because transferring data from GPU memory to CPU memory is expensive, transferring minimal data is necessary.

[0404] Additionally, even if the data corresponding to the current frame is transferred to the CPU memory only when a screen change occurs, if the resolution of the frame is large, the pixel value transferred from the GPU memory to the CPU memory may become large, which may cause a decrease in performance.

[0405] Therefore, in the present invention, whether or not the screen changes on the GPU memory can be determined, but whether or not the change is determined on a tile basis rather than on the entire frame in order to minimize the amount of data transferred between memories.

[0406] At this time, the size of the multiple tiles can be determined by considering the service resolution and the data transfer speed from CPU memory to CPU memory.

[0407] For example, if the service resolution is large, the tile size can be set small so that the data size of the area corresponding to one tile is small.

[0408] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0409] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the screen has changed on a tile-by-tile basis.

[0410] For example, in the present invention, a GPU memory control framework such as CUDA can be used to determine screen changes on a tile-by-tile basis, thereby selecting the minimum amount of data to be transferred to CPU memory.

[0411] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0412] That is, while processing the previous frame, a comparison between the previous frame stored in GPU memory and the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0413] In addition, in a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention, a cloud streaming server transmits a tile in which a screen change has occurred to CPU memory (S1520).

[0414] At this time, the pixel values ​​corresponding to the tiles where the screen change occurred can be transferred to the CPU memory.

[0415] That is, instead of transferring the entire area of ​​the current frame to the CPU memory, pixel values ​​can be transferred from GPU memory to CPU memory only for the tiles where the pixel difference from the previous frame occurs.

[0416] In addition, in a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention, a cloud streaming server calculates a change area for a tile in which a screen change has occurred on a CPU memory (S1530).

[0417] At this time, the CPU memory can encode the changed area by examining the pixel values ​​of the area corresponding to the changed tile.

[0418] That is, even in CPU memory, the changed area among the entire frame area can be calculated for image encoding, but according to the present invention, the changed area can be quickly obtained by performing calculations only on the area corresponding to the tile transferred from GPU memory, not the entire frame area.

[0419] At this time, the change area can be calculated by performing a loop for the number of tiles where the screen change occurred on the CPU memory.

[0420] That is, instead of performing CPU loops for the number of pixels contained in the frame, we can calculate the change area by performing CPU loops only for the number of tiles where screen changes have occurred.

[0421] In addition, in a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention, a cloud streaming server encodes a change area in CPU memory and transmits it to a user terminal (S1540).

[0422] Through this method of providing an image cloud streaming service based on efficient split screen processing, a screen change area can be detected quickly and efficiently in an image cloud streaming service based on Native Browser Mode.

[0423] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for screen change detection in the image cloud, system performance degradation can be minimized.

[0424] Additionally, latency can be minimized when streaming images to the cloud by detecting pixel change areas on a tile-by-tile basis in GPU memory.

[0425] Figure 16 is a diagram showing an example of a process for transmitting data without dividing frames.

[0426] Referring to FIG. 16, it illustrates a process of transferring the entire pixels of the current frame copied from the frame buffer in the GPU memory (1610) of the browser of the cloud streaming server to the CPU memory (1620).

[0427] In the method of Fig. 16, first, all pixels can be copied from the frame buffer (1611) of the GPU memory (1610) to the cur frame buffer (1621-1) of the CPU memory (1620) in the size of the resolution of the current frame.

[0428] After this, the entire pixels of the previous frame stored in the prev frame buffer (1621-2) on the CPU memory (1620) are compared with the entire pixels of the current frame copied to the cur frame buffer (1621-1), and if there is a difference in the pixels, it can be determined that a screen change has occurred.

[0429] After this, you can find the change area corresponding to the changed pixels by looping the size of the frame resolution.

[0430] After this, the change area is encoded through an image encoder (1622) and transmitted to a streamer (1623), thereby transmitting an encoded image corresponding to the change area to a user terminal.

[0431] That is, the method illustrated in Fig. 16 can reduce the amount of data transmitted to the user terminal because the CPU processes the change area only when a screen change occurs between frames.

[0432] However, as the screen resolution increases, this method requires more data to be transferred from GPU memory (1610) to CPU memory (1620), and the number of loops to find a change area in the CPU memory (1620) also increases. In other words, as the screen resolution increases, the required resources and costs may increase, and the latency rate may increase.

[0433] Therefore, in order to solve this problem, the present invention proposes a method for detecting screen changes on GPU memory (1610) and simultaneously dividing frames to transmit only the pixels of the changed area to CPU memory (1620) for processing.

[0434] Figure 17 is a diagram showing an example of a data transfer process between memories according to the present invention.

[0435] Referring to FIG. 17, the data transfer process between memories according to the present invention can compare the pixels of the current frame copied from the frame buffer (1711) in the GPU memory (1710) of the browser of the cloud streaming server with the pixels of the previous frame.

[0436] For example, the current frame copied from the frame buffer (1711) can be stored in GPGPU-capable CUDA MEMORY (1712-1) and compared with the previous frame stored in CUDA MEMORY (PREV FRAME) (1712-2).

[0437] At this time, in the present invention, it is possible to compare whether the pixels of the current frame are different from the pixels of the previous frame in tile units as shown in FIG. 5.

[0438] After this, tiles with differences in pixels and the pixels contained in the tiles can be transferred to the CPU memory (1720).

[0439] After this, the CPU memory (1720) can calculate the change area where the screen change occurred by performing a loop only for the number of tiles received.

[0440] After this, the pixels corresponding to the change area can be image encoded and streamed to the user terminal.

[0441] That is, compared to the existing process illustrated in FIG. 16, in the process according to the present invention illustrated in FIG. 17, whether or not the screen has changed can be determined on the GPU memory (1710). In addition, rather than transmitting pixels of the entire frame area to the CPU memory (1720), only the pixels of the area corresponding to the tile where the screen change occurred and the changed tile are transmitted to the CPU memory (1720), thereby significantly reducing the amount of data transferred between the memories.

[0442] Typically, GPUs consist of thousands of small cores, each specialized for processing threads. The present invention leverages this GPU characteristic by allocating threads equal to the number of tiles and comparing pixel differences between the previous and current frames, enabling faster detection of screen changes than CPUs.

[0443] At this time, [Table 1] compares the existing process illustrated in Fig. 16 with the process according to the present invention illustrated in Fig. 17.

[0444] Data transfer amount from GPU memory to CPU memory Number of CPU loops CPU change area processing (Figure 16 method) Total area of ​​frame Number of pixels (resolution size) GPU tile-based change area processing (Figure 17 method) Changed tile area Number of changed tiles

[0445] For example, assuming that the resolution of the frame is 1920x1080 and the size of the tile is 8x8, the number of tiles is 32,400, and the number of pixels corresponding to the resolution size is 2,073,600. That is, according to the existing process according to Fig. 16, the entire screen data having 2,073,600 pixels must be transferred from the GPU memory to the CPU memory for each frame, and the CPU loop must be performed as many times as the number of 2,073,600 pixels in order to calculate the change area on the CPU memory.

[0446] However, according to a process such as FIG. 17 proposed in the present invention, only the data corresponding to the changed tile area where the screen change is detected through GPGPU is transferred from the GPU memory to the CPU memory, and the changed area can be calculated by performing a CPU loop only for the number of changed tiles in the CPU memory.

[0447] In this way, by efficiently performing detection of screen changes on the GPU memory (1710) and significantly reducing the amount of data transferred from the GPU memory (1710) to the CPU memory (1720), the cost and latency incurred when providing an image cloud streaming service can be reduced.

[0448] FIG. 18 is a drawing showing an example of dividing a frame into a plurality of tiles according to the present invention.

[0449] Referring to FIG. 18, the present invention uses a GPU memory control framework (1800) such as CUDA to determine screen changes in units of tiles (1810-1 to 1810-(NxN)), thereby selecting the minimum data to be transferred to the CPU memory.

[0450] That is, as shown in Fig. 18, the GPU memory where the current frame is stored is divided into NxN sized tiles (1810-1 to 1810-(NxN)), and the CUDA kernel can detect screen changes for each tile.

[0451] At this time, the pixels of the current frame corresponding to the tile can be compared with the pixels of the previous frame, and if there are changed pixels, it can be detected that a screen change has occurred.

[0452] If a screen change is detected only in tile (1810-1), the amount of data transferred to CPU memory can be minimized by transferring only the pixels for tile (1810-1) to CPU memory.

[0453] In addition, the computation for image cloud streaming can be minimized by examining only the pixels for the tile (1810-1) received from the GPU memory in the CPU memory and computing and encoding the screen change area.

[0454] FIG. 19 is a detailed flowchart illustrating a method for providing an image cloud streaming service based on efficient split screen processing according to one embodiment of the present invention.

[0455] Referring to FIG. 19, a method for providing an image cloud streaming service based on efficient split screen processing according to an embodiment of the present invention can first output an RGBA screen corresponding to the current frame to a frame buffer in GPU memory using alpha data of the current frame obtained through a renderer process (S1910).

[0456] At this time, the browser can be run in NATIVE WINDOWS mode.

[0457] Typically, achieving transparency on cloud streaming servers requires the browser to operate in Off-Screen Rendering (OSR). However, OSR has lower performance than native Windows, making it unsuitable for seamlessly delivering services with fast screen changes, such as video cloud streaming.

[0458] However, when using the Native Windows method, which is higher performing than the OSR method, the APIs for receiving GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen.

[0459] In this regard, the present invention can obtain alpha data (transparency information) with minimal load by acquiring alpha data through a renderer processor. Therefore, the transparency information required for image cloud streaming can be used even in native browser mode using the native Windows method.

[0460] At this time, the cloud streaming server can obtain alpha data through the renderer process and use the obtained alpha data to display the current frame in a form corresponding to RGBA on the GPU memory.

[0461] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0462] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0463] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0464] At this time, the RGBA values ​​of the current frame can be written to the frame buffer in the GPU memory.

[0465] After this, the frame buffer in GPU memory can be copied to the GPU memory control framework (CUDA) (S1920).

[0466] After this, the GPU memory area storing the current frame can be divided into NxN sized tiles (S1930), and the CUDA kernel can compare the previous frame and the current frame for each divided tile area to detect whether there is a pixel change (S1940).

[0467] At this time, image cloud streaming requires rapid detection of areas where pixel differences occur between the previous and current frames. Conventionally, this process was handled via the CPU. This required retrieving the frame buffer from GPU memory to CPU memory. However, because transferring data from GPU memory to CPU memory is expensive, transferring minimal data is necessary.

[0468] Additionally, even if the data corresponding to the current frame is transferred to the CPU memory only when a screen change occurs, if the resolution of the frame is large, the pixel value transferred from the GPU memory to the CPU memory may become large, which may cause a decrease in performance.

[0469] Therefore, in the present invention, whether or not the screen changes on the GPU memory can be determined, but whether or not the change is determined on a tile basis rather than on the entire frame in order to minimize the amount of data transferred between memories.

[0470] At this time, the size of the multiple tiles can be determined by considering the service resolution and the data transfer speed from CPU memory to CPU memory.

[0471] For example, if the service resolution is large, the tile size can be set small so that the data size of the area corresponding to one tile is small.

[0472] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0473] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0474] That is, while processing the previous frame, a comparison between the previous frame stored in GPU memory and the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0475] After this, it is determined whether a pixel change has been detected (S1945), and if a pixel change has been detected, the pixel of the tile in which the change has been detected can be transferred to the CPU memory (S1950).

[0476] At this time, data corresponding to pixels in an area corresponding to a tile whose pixels have changed based on the previous frame among multiple tiles can be transmitted to the CPU memory.

[0477] That is, instead of transferring the entire area of ​​the current frame to CPU memory, only the number of tiles where the pixels have changed can be transferred from GPU memory to CPU memory.

[0478] Additionally, if no pixel change is detected as a result of the judgment in step (S1945), it can be determined whether pixel changes have been detected for all segmented tiles (S1955).

[0479] As a result of the judgment at step (S1955), if pixel changes have not yet been detected for all tiles, the steps can be performed sequentially from step (S1945).

[0480] In addition, if it is determined in step (S1955) that there is a change in pixels for all tiles, the CPU memory can perform a CPU loop for the number of changed tiles to calculate the change area (S1960).

[0481] That is, instead of performing CPU loops for the number of pixels contained in the frame, we can calculate the change area by performing CPU loops only for the number of tiles where screen changes have occurred.

[0482] After this, the calculated change area can be image encoded and streamed to the user terminal (S1970).

[0483] Through this method of providing an image cloud streaming service based on efficient split screen processing, a screen change area can be detected quickly and efficiently in an image cloud streaming service based on Native Browser Mode.

[0484] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for screen change detection in the image cloud, system performance degradation can be minimized.

[0485] Additionally, latency can be minimized when streaming images to the cloud by detecting pixel change areas on a tile-by-tile basis in GPU memory.

[0486] FIG. 20 is a block diagram illustrating a cloud streaming server according to one embodiment of the present invention.

[0487] Referring to FIG. 20, a cloud streaming server according to one embodiment of the present invention includes a communication unit (2010), a processor (2020), and a memory (2030).

[0488] The communication unit (2010) transmits and receives information related to multiple user terminals via a communication network, such as the network illustrated in FIG. 14. In particular, the communication unit (2010) according to one embodiment of the present invention may receive a request for a cloud streaming service from a user terminal and provide a data packet corresponding to the cloud streaming service requested by the user terminal to the user terminal.

[0489] The processor (2020) determines on a tile-by-tile basis whether the screen has changed between the current frame and the previous frame in GPU memory.

[0490] At this time, the browser can be run in NATIVE WINDOWS mode.

[0491] Typically, achieving transparency on cloud streaming servers requires the browser to operate in Off-Screen Rendering (OSR). However, OSR has lower performance than native Windows, making it unsuitable for seamlessly delivering services with fast screen changes, such as video cloud streaming.

[0492] However, when using the Native Windows method, which is higher performing than the OSR method, the APIs for receiving GPU acceleration in each operating system such as Windows / Linux do not support transparency, so there is no way to provide transparency data when composing the final screen.

[0493] In this regard, the present invention can obtain alpha data (transparency information) with minimal load by acquiring alpha data through a renderer processor. Therefore, the transparency information required for image cloud streaming can be used even in native browser mode using the native Windows method.

[0494] At this time, the cloud streaming server can obtain alpha data through the renderer process and use the obtained alpha data to display the current frame in a form corresponding to RGBA on the GPU memory.

[0495] For example, the alpha data of the current frame obtained through the renderer process in the browser of the video cloud streaming server can be shared with the GPU process.

[0496] At this time, the renderer process may correspond to the rendering process for composing the screen to be displayed on the browser. For example, when an application execution screen is received from a web application server running a web application, the renderer processor can compute the received application execution screen and compose the screen so that it can be displayed on the browser of the video cloud streaming server.

[0497] At this time, the GPU process corresponds to a processing process for processing a screen shown to a user using a graphics processing unit (GPU), and can generate a video element stream such as H.264 / 265.

[0498] At this time, the RGBA values ​​of the current frame can be written to the frame buffer in the GPU memory.

[0499] At this time, the current frame can be divided into multiple tiles, and the area corresponding to each of the multiple tiles can be compared with the previous frame to detect whether the screen has changed.

[0500] For example, the GPU memory area that stores the current frame can be divided into NxN sized tiles, and then the screen can be detected by comparing the previous frame and the current frame for each divided tile area.

[0501] At this time, image cloud streaming requires rapid detection of areas where pixel differences occur between the previous and current frames. Conventionally, this process was handled via the CPU. This required retrieving the frame buffer from GPU memory to CPU memory. However, because transferring data from GPU memory to CPU memory is expensive, transferring minimal data is necessary.

[0502] Additionally, even if the data corresponding to the current frame is transferred to the CPU memory only when a screen change occurs, if the resolution of the frame is large, the pixel value transferred from the GPU memory to the CPU memory may become large, which may cause a decrease in performance.

[0503] Therefore, in the present invention, whether or not the screen changes on the GPU memory can be determined, but whether or not the change is determined on a tile basis rather than on the entire frame in order to minimize the amount of data transferred between memories.

[0504] At this time, the size of the multiple tiles can be determined by considering the service resolution and the data transfer speed from CPU memory to CPU memory.

[0505] For example, if the service resolution is large, the tile size can be set small so that the data size of the area corresponding to one tile is small.

[0506] For another example, when data transfer rates are smooth, the tile size can be set to be larger than when data transfer rates are slow, reducing the number of tiles to compare in a GPU memory control framework such as Compute Unified Device Architecture (CUDA).

[0507] At this time, the frame buffer in the GPU memory can be copied to the GPU memory control framework, and the kernel of the GPU memory control framework can determine whether the screen has changed on a tile-by-tile basis.

[0508] For example, in the present invention, a GPU memory control framework such as CUDA can be used to determine screen changes on a tile-by-tile basis, thereby selecting the minimum amount of data to be transferred to CPU memory.

[0509] At this time, the GPU memory control framework can compare the previous frame stored in GPU memory with the current frame copied from the frame buffer.

[0510] That is, while processing the previous frame, a comparison between the previous frame stored in GPU memory and the current frame copied from the frame buffer can be performed on the GPU memory control framework.

[0511] Additionally, the processor (2020) transfers the tiles in which the screen changes to the CPU memory.

[0512] At this time, the pixel values ​​corresponding to the tiles where the screen change occurred can be transferred to the CPU memory.

[0513] That is, instead of transferring the entire area of ​​the current frame to the CPU memory, pixel values ​​can be transferred from GPU memory to CPU memory only for the tiles where the pixel difference from the previous frame occurs.

[0514] Additionally, the processor (2020) calculates a change area for tiles where a screen change has occurred on the CPU memory.

[0515] At this time, the CPU memory can encode the changed area by examining the pixel values ​​of the area corresponding to the changed tile.

[0516] That is, even in CPU memory, the changed area among the entire frame area can be calculated for image encoding, but according to the present invention, the changed area can be quickly obtained by performing calculations only on the area corresponding to the tile transferred from GPU memory, not the entire frame area.

[0517] At this time, the change area can be calculated by performing a loop for the number of tiles where the screen change occurred on the CPU memory.

[0518] That is, instead of performing CPU loops for the number of pixels contained in the frame, we can calculate the change area by performing CPU loops only for the number of tiles where screen changes have occurred.

[0519] Additionally, the processor (2020) encodes the change area on the CPU memory and transmits it to the user terminal.

[0520] Memory (2030) stores the current frame, previous frame, and change area.

[0521] In addition, the memory (2030) stores various information generated in the process of providing an image cloud streaming service based on efficient split screen processing according to an embodiment of the present invention as described above.

[0522] In some embodiments, the memory (2030) may be configured independently from the cloud streaming server to support functions for providing an image cloud streaming service based on efficient split-screen processing. In this case, the memory (2030) may function as a separate large-capacity storage device and may include control functions for performing operations.

[0523] Additionally, the cloud streaming server configured as described above may be implemented with one or more servers.

[0524] Meanwhile, the cloud streaming server is equipped with memory and can store information within the device. In one embodiment, the memory is a computer-readable medium. In one embodiment, the memory may be a volatile memory unit, and in another embodiment, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various other embodiments, the storage device may include, for example, a hard disk drive, an optical disk drive, or any other mass storage device.

[0525] In this way, through a cloud streaming server, screen change areas can be detected quickly and efficiently in image cloud streaming services based on Native Browser Mode.

[0526] Additionally, by minimizing the amount of data transferred between GPU memory and GPU memory for screen change detection in the image cloud, system performance degradation can be minimized.

[0527] Additionally, latency can be minimized when streaming images to the cloud by detecting pixel change areas on a tile-by-tile basis in GPU memory.

[0528] The functional operations and implementations of the subject matter described herein may be implemented as digital electronic circuits, or may be implemented as computer software, firmware, or hardware that includes the structures disclosed herein and their structural equivalents, or a combination of one or more of these. The implementations of the subject matter described herein may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program storage medium for controlling the operation of a processing system or for execution by the same.

[0529] The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter for affecting a machine-readable radio signal, or a combination of one or more of these.

[0530] As used herein, the term "system" or "device" encompasses any mechanism, device, or machine for processing data, including, for example, a programmable processor, a computer, or a multiprocessor or computer. In addition to hardware, a processing system may include code that forms an execution environment for a computer program upon request, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0531] A computer program (also known as a program, software, software application, script, or code) may be written in any programming language, including compiled or interpreted languages, a priori or procedural languages, and may be deployed in any form, including as a standalone program, module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a single file provided to a requested program, in multiple interacting files (e.g., a file storing one or more modules, subprograms, or portions of code), or as part of a file containing other programs or data (e.g., one or more scripts stored within a markup language document). A computer program may be deployed to be executed on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0532] Meanwhile, computer-readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, including semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks such as internal hard disks or external disks, magneto-optical disks and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[0533] Implementations of the subject matter described herein may be implemented in a computing system that includes a back-end component, such as a data server, a middleware component, such as an application server, a front-end component, such as a user computer having a web browser or graphical user interface through which a user can interact with implementations of the subject matter described herein, or any combination of one or more of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network.

[0534] While this specification contains details of a number of specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Likewise, certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features may operate in a particular combination and may initially be described as being claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be modified into a subcombination or variation of a subcombination.

[0535] Furthermore, while this specification depicts operations in the drawings in a particular order, this should not be construed as requiring that such operations be performed in the particular or sequential order depicted to achieve desired results, or that all depicted operations be performed. In certain instances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components of the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products.

[0536] As such, this specification is not intended to limit the invention to the specific terms presented. Accordingly, while the invention has been described in detail with reference to the examples described above, those skilled in the art will appreciate that modifications, variations, and variations may be made to these examples without departing from the scope of the invention. The scope of the invention is indicated by the claims below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the invention.

[0537] As described above, the method for providing a video cloud streaming service based on efficient transparency processing according to the present invention and the device therefor are not limited to the configurations and methods of the embodiments described above, but the embodiments may be configured by selectively combining all or part of the embodiments so that various modifications can be made.

[0538] According to the present invention, a cloud streaming server can display a current frame on GPU memory using alpha data acquired from a renderer process, determine whether there is a change in alpha data between frames on the GPU memory, encode the alpha data by transferring it to CPU memory when the alpha data has changed, and stream the encoded alpha data in the CPU memory and the encoded RGB data in the GPU memory to a user terminal. Accordingly, the amount of data transferred between memories can be minimized, thereby preventing degradation of service performance due to data transfer.

[0539] In addition, according to the present invention, the cloud streaming server can display the current frame on the GPU memory using the ALPHA data obtained from the renderer process, determine whether the alpha data on the GPU memory changes between frames on a tile basis, and if the alpha data has changed, transmit the alpha data corresponding to the number of changed tiles to the CPU memory and encode it, and stream the alpha data encoded in the CPU memory and the RGB data encoded in the GPU memory to the user terminal. Accordingly, the amount of data transferred between memories can be minimized, thereby preventing degradation of service performance due to data transfer.

[0540] Furthermore, according to the present invention, a cloud streaming server can determine whether there is a screen change between the current frame and the previous frame on a tile-by-tile basis in GPU memory, transmit the tile where the screen change occurred to CPU memory, calculate the change area for the tile where the screen change occurred in CPU memory, encode the change area in CPU memory, and transmit it to the user terminal. Accordingly, the amount of data transferred between memories can be minimized, thereby preventing service performance degradation due to data transfer.

Claims

1. Cloud streaming server, A step of displaying the current frame on the GPU memory using alpha data obtained through the renderer process; A step of determining whether there is a change between frames in the alpha data on the GPU memory; When the above alpha data has changed, a step of transmitting the alpha data to the CPU memory and encoding it; and A step of streaming alpha data encoded in the CPU memory and RGB data encoded in the GPU memory to a user terminal. A method for providing a video cloud streaming service, characterized in that it includes:

2. In claim 1, The above judging steps are Further comprising a step of copying the frame buffer in the GPU memory to the GPU memory control framework, A method for providing a video cloud streaming service, characterized in that the kernel of the GPU memory control framework determines whether there is a change between frames in the alpha data.

3. In claim 2, The above GPU memory control framework A method for providing a video cloud streaming service, characterized in that the alpha data of the previous frame stored in GPU memory is compared with the alpha data of the current frame copied from the frame buffer.

4. In claim 3, The above CPU memory is A method for providing a video cloud streaming service, characterized in that the method comprises examining the above alpha data and encoding an alpha data change area.

5. In claim 1, The above encoded RGB data is A method for providing a video cloud streaming service, characterized by corresponding encoded video element stream data.

6. In claim 1, A method for providing a video cloud streaming service, characterized in that the browser of the above cloud streaming server is operated in a native Windows manner.

7. In claim 1, The above user terminal A method for providing a video cloud streaming service, characterized in that the encoded alpha data and the encoded RGB data are each decoded, and the decoded alpha data is applied to the decoded RGB data to display a screen with transparency applied to the user.

8. A processor that displays the current frame on the GPU memory using the alpha data obtained from the renderer process, determines whether the alpha data on the GPU memory changes between frames, and if the alpha data has changed, transmits the alpha data to the CPU memory to encode it, and streams the alpha data encoded in the CPU memory and the RGB data encoded in the GPU memory to the user terminal; and Memory for storing the above alpha data A cloud streaming server comprising:

9. Cloud streaming server, A step of displaying the current frame on the GPU memory using the alpha data obtained from the renderer process; A step of determining whether there is a change between frames of the alpha data on a tile basis on the GPU memory; When the above alpha data has changed, a step of encoding by transferring the alpha data corresponding to the number of changed tiles to the CPU memory; and A step of streaming alpha data encoded in the CPU memory and RGB data encoded in the GPU memory to a user terminal. A method for providing a video cloud streaming service, characterized in that it includes:

10. Cloud streaming server, A step for determining whether the screen has changed between the current frame and the previous frame on a tile-by-tile basis in GPU memory; A step of transferring tiles where screen changes have occurred to CPU memory; A step of calculating a change area for a tile in which a screen change has occurred on the CPU memory; and A step of encoding the change area on the CPU memory and transmitting it to the user terminal. A method for providing an image cloud streaming service, characterized by including:

Images