Methods, apparatus, devices, media, and products for processing video streams

By merging cloud gaming and live streaming video frames on the server side and using target identifiers for association and alignment, the problems of screen asynchrony and image quality loss in cloud gaming live streaming are solved, thus improving the viewing experience for viewers.

CN123123594APending Publication Date: 2026-11-17BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510622063.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

During cloud gaming live streaming, traditional video stream merging solutions result in asynchronous video feeds and image quality loss, impacting the viewer's experience.

Method used

The server directly merges cloud gaming video frames and live streaming video frames, associates and aligns them through target identifiers, and generates a combined video stream to avoid image quality loss caused by encoding and decoding and to force image alignment.

Benefits of technology

It improves the user's live streaming experience, avoids the problem of out-of-sync images, and enhances picture quality and smoothness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123594A_ABST
    Figure CN123123594A_ABST
Patent Text Reader

Abstract

Embodiments of this disclosure relate to methods, apparatus, devices, media, and products for processing video streams. The method includes, in response to a user initiating a cloud game for live streaming on a live streaming client, sending a cloud game video stream for the cloud game to the live streaming client via a first transmission channel, the cloud game video stream including cloud game video frames and target identifiers of the cloud game video frames. The method also includes receiving a live video stream for the live streaming client via a second transmission channel, the live video frames in the live video stream being associated with the target identifiers. The method further includes combining the cloud game video frames and the live video frames based on the target identifiers to generate a combined video stream. The method also includes sending the combined video stream for live streaming the cloud game. This method directly merges the cloud game screen and the live stream screen on the server, avoiding image quality loss due to encoding / decoding, and by associating the cloud game video frames and the live video frames, forcibly aligning them during merging, thus improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments disclosed herein generally relate to the field of live streaming, and specifically to methods, apparatuses, devices, media, and program products for processing video streams. Background Technology

[0002] Currently, the content of live streaming is becoming increasingly rich and diverse. With this diversification, cloud gaming live streaming has gradually emerged as a popular choice among viewers, and its streaming quality and audience interaction are also improving.

[0003] During cloud gaming live streaming, it's typically necessary to merge the live video stream from the streaming client with the cloud gaming video stream from the server before transmitting the merged stream to the audience. Viewers can then access and watch the merged stream via a content delivery network (CDN). However, there are still issues requiring improvement and further research regarding the merging of video streams during cloud gaming live streaming. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and program product for processing video streams.

[0005] According to a first aspect of this disclosure, a method for processing a video stream is provided. The method includes, in response to a user initiating a cloud game for live streaming at a live streaming terminal, sending a cloud game video stream for the cloud game to the live streaming terminal via a first transmission channel, the cloud game video stream including cloud game video frames and target identifiers of the cloud game video frames. The method also includes, via a second transmission channel, receiving a live video stream for the live streaming terminal, the live video frames in the live video stream being associated with target identifiers. The method further includes, based on the target identifiers, combining the cloud game video frames and the live video frames to generate a combined video stream. The method also includes, sending the combined video stream for live streaming the cloud game.

[0006] In a second aspect of this disclosure, a method for processing a video stream is provided. The method includes receiving a cloud gaming video stream for live streaming from a server via a first transmission channel in response to a user initiating live streaming of a cloud game on a live streaming end. The cloud gaming video stream includes cloud gaming video frames and target identifiers for the cloud gaming video frames. The method further includes associating a live video frame corresponding to a cloud gaming video frame in the live streaming video stream for the live streaming end with the target identifier. The method also includes sending the live video stream to the server via a second transmission channel for live streaming a combined video stream generated from the cloud gaming video frames and the live video frames, the live video stream including the live video frames and target identifiers associated with the live video frames.

[0007] In a third aspect of this disclosure, an apparatus for processing video streams is provided. The apparatus includes a video stream transmitting module configured to, in response to a user initiating a live-streamed cloud game on a live-streaming terminal, transmit a cloud game video stream for the cloud game to the live-streaming terminal via a first transmission channel, the cloud game video stream including cloud game video frames and target identifiers of the cloud game video frames; a video stream receiving module configured to receive a live-stream video stream for the live-streaming terminal via a second transmission channel, the live-stream video frames in the live-stream being associated with target identifiers; a video frame combining module configured to combine the cloud game video frames and the live-stream video frames based on the target identifiers to generate a combined video stream; and a video stream transmitting module configured to transmit the combined video stream for live-streaming the cloud game.

[0008] In a fourth aspect of this disclosure, an apparatus for processing video streams is provided. The apparatus includes a video stream receiving module configured to receive a cloud game video stream for the cloud game from a server via a first transmission channel in response to a user initiating a live-streamed cloud game on a live-streaming terminal. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. An association module is configured to associate a live-stream video frame corresponding to a cloud game video frame in the live-stream video stream for the live-streaming terminal with the target identifier. A video stream sending module is configured to send the live-stream video stream to a server via a second transmission channel for live-streaming a combined video stream generated from the cloud game video frames and the live-stream video frames. The live-stream video stream includes live-stream video frames and target identifiers associated with the live-stream video frames.

[0009] In a fifth aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the methods according to the first and second aspects of this disclosure.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0011] In a seventh aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0012] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0014] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0015] Figure 2 The illustration shows a schematic diagram of an example method for processing a video stream according to some embodiments of the present disclosure;

[0016] Figure 3 The illustration shows a schematic diagram of another example method for processing a video stream according to some embodiments of the present disclosure;

[0017] Figure 4 The illustration shows a schematic diagram of an example flow for processing a video stream according to some embodiments of the present disclosure;

[0018] Figure 5 The illustration shows a schematic diagram of another example architecture for processing video streams according to some embodiments of the present disclosure;

[0019] Figure 6 The illustration shows a schematic block diagram of an apparatus for processing video streams according to some embodiments of the present disclosure;

[0020] Figure 7 The illustration shows a schematic block diagram of another apparatus for processing video streams according to some embodiments of the present disclosure;

[0021] Figure 8 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0022] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0030] In today's live streaming industry, enhancing the viewing experience is becoming increasingly important. With the diversification of live streaming content, cloud gaming live streaming has gradually emerged as a popular choice among viewers. In this context, the merging of the live stream and the cloud gaming stream has become a crucial process, significantly impacting the viewer's experience. Traditional merging solutions rely heavily on the network and encoding performance of the live streaming client, requiring certain equipment specifications.

[0031] There are two approaches to merging traditional live streaming and cloud gaming video streams. The first approach involves locally combining the cloud gaming audio and video with the live streaming audio and video before pushing it to the viewer. This places high demands on the uplink network bandwidth and encoding performance of the live streaming client; if these requirements are not met, the viewer will experience blurry and choppy live streams. The second approach involves pushing the cloud gaming audio and video and the live streaming audio and video separately to a cloud-based merging service. The cloud gaming audio and video are directly captured in the cloud and pushed to the merging service, where they are then merged before being pushed to the viewer. While this reduces the uplink bandwidth and encoding burden on the live streaming client compared to the first approach, the merging node cannot align the two streams because they are pushed from different endpoints. This leads to desynchronization between the live stream and the cloud gaming streams. Furthermore, cloud-based merging involves an additional encoding and decoding step compared to local merging, resulting in image quality loss and additional costs. This leads to increased resource consumption and a decreased viewing experience for the viewer.

[0032] To address the aforementioned and other potential problems, embodiments of this disclosure propose a method for processing video streams. In this method, after a user initiates a cloud game for live streaming on a live streaming client, a cloud game video stream is first sent to the live streaming client via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers for those frames. After sending the cloud game video stream, a live video stream is received from the live streaming client via a second transmission channel. The live video frames in the live video stream are associated with the target identifiers. Next, upon receiving the live video stream, the server combines the cloud game video frames and the live video frames according to the target identifiers to generate a combined video stream. Finally, the combined video stream is sent for live streaming the cloud game. This method directly merges the cloud game footage and the live stream footage on the server generating the cloud game video stream, avoiding image quality loss due to cloud game image encoding / decoding. Furthermore, by associating the cloud game video frames and the live video frames, the cloud game video stream and the live video stream are forcibly aligned during merging, preventing desynchronization and improving the user's live streaming experience.

[0033] It is understood that this disclosure is only described in conjunction with relevant embodiments and is not intended to limit the scope of protection of this disclosure. Any technical solutions in other disclosures that fall within the scope of protection of this disclosure should be protected.

[0034] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1The illustration shows an example environment in which the devices and / or methods of embodiments of this disclosure can be implemented. In environment 100, a live streaming client 102 and a server 112 are used to generate a combined video stream 122. Any suitable computing device can be used as the live streaming client 102 and the server 112 in the example environment. For example, the live streaming client 102 can be a smart device, such as a smartphone, and the server 112 can be a cloud server, which can be used to run cloud applications, such as cloud gaming. Data transmission between the live streaming client 102 and the server 112 is performed through transmission channels. The first transmission channel 108 and the second transmission channel 110 can use the same transmission protocol for data transmission, or they can use different data transmission protocols, which is not limited herein. Cloud gaming 114 runs on the server 112, and users can use a cloud gaming-related software development kit (SDK) on the live streaming client 102 to start and play cloud gaming 114.

[0035] Examples of live streaming device 102 include, but are not limited to, mobile phones (e.g., smartphones), laptop computers, portable digital assistants (PDAs), e-book readers, portable game consoles, portable media players, game consoles, set-top boxes (STBs), smart TVs (TVs), personal computers, laptop computers, in-vehicle computers (e.g., navigation units), etc. Examples of server device 112 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices, multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0036] like Figure 1 As shown, when a user wants to stream a cloud game, they can start the cloud game 114 via the streaming client 102. The request to start the cloud game is sent to the server 112 via the streaming client 102. After receiving the user's request, the server 112 can start the cloud game 114. Then, the server 112 begins generating a cloud game video stream 116 for the cloud game 114. When generating cloud game video frames 118 in the cloud game video stream 116, the server 112 also determines a target identifier 120 corresponding to the cloud game video frame 118. In one example, the target identifier 120 can be used to indicate the generation time of the cloud game video frame 118; for example, the target identifier 120 can be used to indicate the specific time when the cloud game video frame 118 was generated, and the time information in the target identifier 120 can be presented in the form of a timestamp. In another example, the target identifier 120 can be any suitable identifier that can identify a cloud game video frame.

[0037] Next, server 112 can use the first transmission channel 108 to send a cloud game video stream 116 associated with cloud game 114 to live streaming client 102. The cloud game video stream 116 includes cloud game video frames 118 and a target identifier 120 for the cloud game video frames 118. Additionally, server 112 will also obtain the cloud game audio stream corresponding to the cloud game video stream and send the cloud game audio stream simultaneously when sending the cloud game video stream.

[0038] After receiving the cloud gaming video stream 116, the live streaming client 102 simultaneously acquires the user's live stream of the cloud game, for example, acquiring the live video stream 104 for the user. Then, the client 102 associates the corresponding live video frame in the live video stream 104 with the target identifier 120 included in the cloud gaming video stream 116 that corresponds to the cloud gaming video frame 118. For example, the target identifier 120 corresponding to the cloud gaming video frame 118 is stored in association with the live video frame 106, such as storing the target identifier in the metadata of the live video frame 106, thereby associating the live video frame 106 with the target identifier 120, so as to facilitate the subsequent association of the live video frame 106 with the cloud gaming video frame 118. Then, the live streaming client 102 sends the live video stream 104 to the server 112 via the second transmission channel 110.

[0039] After receiving the live video stream 104, the server 112 further uses the target identifier 120 associated with the live video frame 106 to combine the live video frame 106 and the cloud gaming video frame 116, thereby obtaining the combined video stream 122. It should be noted that the acquisition and generation of the cloud gaming video stream 116 and the generation of the combined video stream 122 are both performed on the server 112.

[0040] Finally, after generating the combined video stream 122, the server 112 sends the combined video stream 122 to broadcast the cloud game 114 live. For example, the server 112 sends the combined video stream 122 to a Content Delivery Network (CDN), which then sends the combined video stream 122 to viewers to broadcast the cloud game 114 live. Additionally, cloud game audio frames and cloud game video frames are transmitted to the live streaming client 102 via a first transmission channel 108 using a first transmission protocol, and the live audio stream and live video stream for the live streaming client are transmitted to the server 112 via a second transmission channel 110 using a second transmission protocol.

[0041] This method allows for the direct merging of cloud gaming and live streaming video streams on the server side, avoiding image quality loss due to cloud gaming video encoding and decoding. Furthermore, by associating cloud gaming and live streaming video frames, the merging process forces alignment between the cloud gaming and live streaming video streams, preventing desynchronization and improving the user experience when watching live streams.

[0042] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 The illustration depicts an example method 200 for processing a video stream according to some embodiments of the present disclosure. This example method 200 can be derived from... Figure 1 It can be executed on server 102 or any other suitable computing device.

[0043] like Figure 2 As shown, in example method 200, at block 202, in response to a user starting a cloud game for live streaming on the live streaming client, a cloud game video stream for the cloud game is sent to the live streaming client via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames.

[0044] In some embodiments, after cloud gaming 114 is started, server 112 first collects cloud gaming video stream 116 for cloud gaming 114, and during the collection of cloud gaming video stream 116, determines the target identifier 120 corresponding to cloud gaming video frame 118 in cloud gaming video stream 116. Then, it uses the first transmission channel 108 to send cloud gaming video stream 116, including cloud gaming video frame 118 and target identifier 120, to live streaming client 102. The process of collecting cloud gaming video frame 118 is performed on server 112.

[0045] Additionally, when server 112 collects the cloud game video stream 116 of cloud game 114, it also collects the cloud game audio stream of cloud game 114. The collected cloud game audio stream is associated with the cloud game video stream 116. After collecting the cloud game audio stream, it further transmits the cloud game audio stream to live streaming client 102 using the first transmission channel 108. It should be noted that during the process of transmitting the cloud game video stream 116 and cloud game audio stream to live streaming client 102 using the first transmission channel 108, a first transmission protocol for the first transmission channel 108 can also be used to align the cloud game video stream 116 and cloud game audio stream. The first transmission protocol can be a Real-Time Communication (RTC) protocol, or any other transmission protocol capable of aligning the cloud game video stream 116 and cloud game audio stream; this application does not impose any restrictions on this.

[0046] Then, at box 204, a live video stream for the live streaming client is received via the second transmission channel, and the live video frames in the live video stream are associated with the target identifier.

[0047] In some embodiments, during the process of the server 112 receiving the live video stream 104 from the live streaming terminal 102 using the second transmission channel 110, metadata of the live video frames for the live video stream 104 is acquired. In one example, the metadata of the live video frame 106 includes a target identifier 120 corresponding to the cloud gaming video frame 118. In another example, the metadata of the live video frame 106 also includes other information about the live video frame 106, such as pixel information, color information, etc. After acquiring the metadata of the live video frame 106, the server 112 also determines the target identifier 120 corresponding to the live video frame 106 in the metadata.

[0048] In some embodiments, after the server 112 receives the live video stream 104 from the live streaming terminal 102, it will also decode the received live video stream. By decoding the live video stream 104, the compressed live video frame 106 can be decompressed to obtain a live video with a higher bitrate.

[0049] Additionally, during the process of receiving the live video stream 104, the server 112 also utilizes the second transmission channel 110 to receive the live audio stream from the live streaming client 102. The received live audio stream is associated with the live video stream 104. It should be noted that during the process of receiving the live video stream 104 and the live audio stream using the second transmission channel 110, a second transmission protocol for the second transmission channel 110 can also be used to align the live video stream 104 and the live audio stream. In one example, the second transmission protocol can be the same as the first transmission protocol. In another example, the second transmission protocol can be a different transmission protocol than the first transmission protocol. This application does not impose any limitations here.

[0050] Next, at box 206, based on the target identifier, the cloud gaming video frame and the live streaming video frame are combined to generate a combined video stream. It should be noted that the process of combining the cloud gaming video frame 118 and the live streaming video frame 106 is performed on the server side 112.

[0051] In some embodiments, after obtaining the live video frame 106 at the server 112, the obtained live video frame 106 and the acquired cloud gaming video frame 118 can be further aligned according to the target identifier 120. Aligning the live video frame 106 and the cloud gaming video frame 118 can synchronize the live broadcast and the cloud gaming broadcast, reducing the problem of screen asynchrony when broadcasting cloud gaming 114. Subsequently, the aligned cloud gaming video frame 118 and the live video frame 106 are combined to obtain the combined video stream 122.

[0052] In some embodiments, for ease of description, the cloud gaming video frame 118 in the cloud gaming video stream 116 sent by server 112 to live streaming client 102 is referred to as the first cloud gaming video frame. The target identifier can be a timestamp, which represents the specific time when the cloud gaming video frame was generated. During the alignment of the cloud gaming video frame and the live streaming video frame, it is also necessary to obtain the second cloud gaming video frame generated when the live streaming video frame 106 is received, and to obtain a reference timestamp for the second cloud gaming video frame. It is understood that the specific time corresponding to the reference timestamp should be later than the specific time corresponding to the timestamp; that is, the second cloud gaming video frame is a newer cloud gaming video frame in time compared to the first cloud gaming video frame.

[0053] After obtaining the reference timestamp, the delay between the reference timestamp and the current timestamp can be further determined. In one example, the specific time corresponding to the reference timestamp is 0.5 seconds later than the specific time corresponding to the current timestamp. In this case, the delay between the reference timestamp and the current timestamp is 0.5 seconds. Therefore, the cloud gaming video frame can be delayed by this delay, for example, 0.5 seconds, to align with the received live video frame. For example, the cloud gaming video frames and their corresponding identifiers in the cloud gaming video stream can be cached on the server side. Then, the cloud gaming video frame corresponding to the received target identifier can be found from the cached cloud gaming video stream using the received target identifier. In this way, the first cloud gaming video frame acquired on the server 112 can be actively delayed, and the first cloud gaming video frame can be aligned with the live video frame and combined into a single stream, so that the live stream and the cloud gaming screen are synchronized.

[0054] Finally, at box 208, the combined video stream is sent for live streaming of the cloud game. After obtaining the combined video stream, server 112 can use it for live streaming.

[0055] In some embodiments, after determining the combined video stream 122, the server 112 sends the determined combined video stream 122 to the CDN, and then the CDN can send the combined video stream 122 to the viewer to live stream cloud gaming 114.

[0056] Additionally, server 112 will also mix the live audio stream received from live streaming client 102 with the cloud gaming audio stream corresponding to cloud gaming video stream 116 to obtain a mixed audio stream. Finally, the mixed audio stream will be sent to CDN and then sent to the viewer's client via CDN to live stream cloud gaming 114.

[0057] This method merges cloud gaming and live streaming feeds directly on the server side, avoiding image quality loss due to encoding and decoding. Furthermore, by associating cloud gaming and live streaming video frames, the merged streams are forcibly aligned to prevent desynchronization and improve the user experience when watching live streams.

[0058] The above combination Figure 2 A schematic diagram of an example method 200 for processing a video stream, according to some disclosed embodiments, is described below. Figure 3 The illustration depicts another example method 300 for processing a video stream according to some disclosed embodiments. This example method 300 can be derived from... Figure 1 The live streaming client 102 or any other suitable computing device can be used to execute the program.

[0059] like Figure 3 As shown, in example method 300, at block 302, in response to a user starting a cloud game for live streaming on the live streaming client, a cloud game video stream for the cloud game is received from the server via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames.

[0060] In some embodiments, after a user starts cloud gaming 114 and receives cloud gaming video stream 116 from server 112 on live streaming terminal 102, the cloud gaming video stream 116 will be further identified as including cloud gaming video frame 118 and target identifier 120 corresponding to cloud gaming video frame 118.

[0061] Then, at box 304, the live video frame corresponding to the cloud gaming video frame in the live video stream for the live streaming client is associated with the target identifier.

[0062] In some embodiments, after a user starts cloud gaming 114 and receives cloud gaming video stream 116 from server 112 at live streaming terminal 102, the target identifier 120 included in the cloud gaming video stream 116 can be written into the metadata of live video frame 106 of live video stream 104. In this way, live video frame 106 can be associated with target identifier 120.

[0063] Finally, at box 306, a live video stream is sent to the server via the second transmission channel for live streaming of a combined video stream generated from cloud gaming video frames and live video frames, the live video stream including the live video frames and target identifiers associated with the live video frames.

[0064] In some embodiments, before sending the live video stream 104 from the live streaming terminal 102 to the server 112, the live video stream 104 is encoded and then sent to the server 112 via the second transmission channel 110. Compressing the live video stream 104 can improve the data transmission efficiency of the live video stream 104 in the second transmission channel 110.

[0065] Additionally, when the live streaming terminal 102 sends the live video stream 104 to the server 112, the live streaming terminal 102 will also obtain the live audio stream corresponding to the live video stream 104 and send the live audio stream corresponding to the live video stream 104 to the server 112 using the second transmission channel 110.

[0066] This method merges cloud gaming and live streaming feeds directly on the server side, avoiding image quality loss due to encoding and decoding. Furthermore, by associating cloud gaming and live streaming video frames, the merged streams are forcibly aligned to prevent desynchronization and improve the user experience when watching live streams.

[0067] The above combination Figure 3 A schematic diagram of an example method 300 for processing a video stream, according to some disclosed embodiments, is described below. (The following is in conjunction with...) Figure 4 A schematic diagram illustrating an example flow for processing a video stream according to some disclosed embodiments.

[0068] like Figure 4 As shown in Example 400, the following explanation uses a live stream of cloud gaming by broadcaster 412 as an example. First, when broadcaster 412 starts the cloud game for live streaming, then at 414, the cloud game video stream is captured at the cloud game server. The cloud game server can be a set of containers used to run cloud game instances on the server side. Next, after capturing the cloud game video stream, at 406, the captured cloud game video stream is transmitted via the server-side RTC to the cloud game RTC server located at 404. The cloud game can be represented by `rtc_app_id`. Additionally, during the capture of the cloud game video stream, the cloud game audio stream is also captured, and a target identifier for the cloud game video stream is determined. When transmitting the cloud game video stream to the cloud game RTC server, the cloud game audio stream and the target identifier are also transmitted together. In one example, the target identifier is a timestamp.

[0069] Then, at position 402, the cloud game video frames are transmitted to the live streaming client via the client's RTC. This live streaming client can be used to open a live streaming room and broadcast the corresponding cloud game. The live streaming client can also be used to provide audio and video streaming services for the live streaming client. After receiving the cloud game audio and video, the live streaming client transmits the cloud game audio and video to the broadcaster at position 408 (position 412).

[0070] During the process of broadcaster 412 receiving cloud gaming audio and video for cloud gaming, at point 418, the broadcaster 412's camera is also captured to obtain live video frames for broadcaster 412 and acquire the live video stream for broadcaster 412. Next, at point 410, broadcaster 412 transmits the captured broadcaster camera information, i.e., the live video stream, to the live streaming client. The live streaming client uses the real-time communication protocol with the cloud gaming server to transmit the live video stream to the cloud gaming RTC server located at point 404. Additionally, the live streaming client also associates the timestamp in the received cloud gaming video stream with the live video frames in the live video stream to align the live video frames and cloud gaming video frames. Then, it further transmits the live audio, video, and timestamp to the cloud gaming RTC server located at point 404.

[0071] Then, after the cloud gaming RTC server receives the live video stream and timestamp from the live streaming client, it transmits the live video stream and timestamp to the cloud gaming server located at 414 via the server RTC at 406. Additionally, at 414, the cloud gaming server also captures the latest cloud gaming video frame and actively delays the cloud gaming video frame based on the delay between the timestamp and the reference timestamp of the latest cloud gaming video frame to align the cloud gaming video frame with the received live video frame. Subsequently, at 416, the aligned live video frame and the latest cloud gaming video frame are merged and sent to the content delivery network 420. Finally, the viewer's client 424 pulls the stream at 422 to obtain the merged live and cloud gaming images to watch the cloud gaming live stream. Additionally, the live audio stream and cloud gaming audio stream are also aligned and mixed; viewers at client 424 can also pull the mixed audio stream at 422 to watch the cloud gaming live stream.

[0072] The above combination Figure 4 A schematic diagram of an example flow for processing a video stream, according to some disclosed embodiments, is described below. (The following is in conjunction with...) Figure 5 A schematic diagram illustrating another example architecture for processing video streams according to some embodiments of the present disclosure.

[0073] like Figure 5As shown, in Example 500, combined with the previous Example 400, the example architecture includes a live streaming client 508 and a cloud gaming server 528. The live streaming client 508 can serve as... Figure 1 In the live streaming client 102, the cloud gaming server can be used as... Figure 1 The server-side component 112 is used for streaming. The live streaming client 508 provides the ability to open a live streaming room. The streamer 502 uses the live streaming client 508 to open a live streaming room, select the cloud game to stream, and viewers can watch the live content on their viewing devices. The live streaming client 508 integrates the cloud gaming SDK 510 and the live streaming SDK 512. The cloud gaming SDK 510 provides the ability to play cloud games; the streamer 502 can open cloud games within the live streaming client 508 using the cloud gaming SDK 510. The live streaming SDK 512 provides the ability to open a live streaming room and can also collect data from the streamer 502's camera and microphone. The live streaming SDK 512 can also push the collected audio and video data to the live streaming RTC 524 for subsequent merging. The cloud gaming server 528 provides the runtime environment for running the game 530 and also provides game streaming and merging capabilities. The cloud gaming server 528 integrates a cloud gaming streaming module 534 and a merging module 538. The cloud gaming streaming module 534 can capture and push the audio and video of the cloud game to the live streaming client 508, allowing the streamer to play the cloud game. Simultaneously, the cloud gaming streaming module 534 also transmits the captured cloud game audio and video to the merging module 538. The merging module 538 receives the cloud game audio and video from the cloud gaming streaming module 534 and the live audio and video from the live streaming client 508, calculates the latency between the live audio and video and the cloud game audio and video to align them, and then merges the aligned audio and video.

[0074] First, streamer 502 starts game 530 (a cloud game) running on cloud gaming server 528 at live streaming client 508 and at 504. Then, cloud gaming streaming module 534 captures the cloud game audio and video for game 530 at 532 and determines the timestamp. Next, cloud gaming server 528 transmits the cloud game audio and video and timestamp to cloud gaming RTC 518 at 520, and then the cloud game RTC 518 transmits the cloud game audio and video and timestamp to live streaming client 508 at 522. While receiving the cloud game audio and video and timestamp, live streaming SDK 514 also captures the streamer 502's camera and microphone at 506, i.e., captures live streaming audio and video. Finally, live streaming SDK 514 obtains the timestamp at 512 and writes the timestamp into the metadata of the live video frame.

[0075] Next, at point 516, the live streaming SDK 514 transmits the streamer's camera, microphone, and timestamp to the live streaming RTC 524, which means transmitting the live audio and video and timestamp for the live streaming client 508 to the live streaming RTC 524. Then, at point 526, the live streaming RTC 524 transmits the streamer's camera, microphone, and timestamp to the merging module 538 in the cloud gaming server 528, which means transmitting the live audio and video and timestamp for the live streaming client 508 to the merging module 538. Upon receiving the live audio and video and timestamp, the merging module 538 also retrieves the latest cloud gaming video frame from the cloud gaming streaming module 534 at point 536 and determines the reference timestamp for the latest cloud gaming video frame. At the merging module 538, it determines the latency of the cloud gaming video frame based on the timestamp and the reference timestamp, and then actively delays the cloud gaming video frame to align it with the live streaming video frame. In one example, if the delay between the reference timestamp and the timestamp is 1 second, then at the merge module 538, the cloud gaming video frame is actively delayed by 1 second to align the live video frame and the cloud gaming video frame. After the live video frame and the cloud gaming video frame are aligned, the live video frame and the cloud gaming video frame are combined to obtain a combined video stream, which is then pushed to CDN 544 at 546.

[0076] Finally, viewers at 540 pull the combined video stream from CDN 544 at 542 to watch the live stream. Additionally, the merging module 538 aligns and mixes the live audio frames and cloud gaming audio frames to obtain a mixed audio frame, which is then sent to CDN 544 at 546. Viewers can pull the mixed audio stream at 542 simultaneously with the combined video stream for watching the cloud gaming live stream.

[0077] like Figure 6 As shown, the device 600 includes a video stream sending module 602, configured to send a cloud game video stream for the cloud game to the live streaming terminal via a first transmission channel in response to a user starting a cloud game for live streaming on the live streaming terminal. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. A video stream receiving module 604 is configured to receive a live video stream for the live streaming terminal via a second transmission channel. The live video frames in the live video stream are associated with the target identifiers. A video frame combining module 606 is configured to combine cloud game video frames and live video frames based on the target identifiers to generate a combined video stream. A video stream sending module 608 is configured to send the combined video stream for live streaming the cloud game.

[0078] In some embodiments, the video stream receiving module 604 includes: a metadata acquisition module configured to acquire metadata for live video frames in the live video stream; and an identifier determination module configured to determine a target identifier for a cloud gaming video frame corresponding to the live video frame from the metadata.

[0079] In some embodiments, the video stream receiving module 604 further includes a decoding module configured to generate a decoded live video stream by decoding the live video stream in response to receiving the live video stream.

[0080] In some embodiments, the video frame combining module 606 includes: an alignment module configured to align cloud gaming video frames and live streaming video frames based on a target identifier; and a combining module configured to combine cloud gaming video frames and live streaming video frames to generate a combined video stream.

[0081] In some embodiments, the cloud gaming video frame is a first cloud gaming video frame, the target identifier is a timestamp, and the alignment module includes: a reference timestamp acquisition module configured to acquire a reference timestamp of a second cloud gaming video frame generated when a live video frame is received; a latency determination module configured to determine the latency for the cloud gaming video frame based on the timestamp and the reference timestamp; and a delay module configured to align the first cloud gaming video frame with the live video frame by delaying the first cloud gaming video frame.

[0082] In some embodiments, the video stream sending module 608 includes: a capture module configured to capture cloud gaming video frames for cloud gaming; an identifier determination module configured to determine a target identifier for the cloud gaming video frames; and a sending module configured to send a cloud gaming video stream including the cloud gaming video frames and the target identifier to a live streaming terminal via a first transmission channel.

[0083] In some embodiments, the apparatus 600 further includes: an audio stream acquisition module configured to acquire a cloud gaming audio stream for cloud gaming; and an audio stream transmission module configured to transmit the cloud gaming audio stream to a live streaming client via a first transmission channel in association with the cloud gaming video stream.

[0084] In some embodiments, the apparatus 600 further includes: an audio stream receiving module configured to receive a live audio stream for a live streaming client via a second transmission channel, the live audio stream being associated with a live video stream; a mixing module configured to generate a mixed audio stream by mixing the cloud gaming audio stream and the live audio stream; and an audio stream sending setting configured to send the mixed audio stream in association with the combined video stream.

[0085] In some embodiments, the cloud gaming video stream and cloud gaming audio stream are aligned using a first transport protocol for a first transport channel, and the live video stream and live audio stream are aligned using a second transport protocol for a second transport channel.

[0086] In some embodiments, the generation of the cloud gaming video stream and the generation of the combined video stream are both performed at the server running the cloud game.

[0087] like Figure 7 As shown, the device 700 includes a video stream receiving module 702, configured to receive a cloud game video stream for the cloud game from a server via a first transmission channel in response to a user starting a cloud game for live streaming on a live streaming terminal. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. An association module 704 is configured to associate the live video frames corresponding to the cloud game video frames in the live video stream for the live streaming terminal with the target identifiers. A video stream sending module 706 is configured to send a live video stream to a server via a second transmission channel for live streaming a combined video stream generated from the cloud game video frames and the live video frames. The live video stream includes live video frames and target identifiers associated with the live video frames.

[0088] In some embodiments, the association module 704 includes a writing module configured to write the target identifier into the metadata of the live video frame in the live video stream.

[0089] In some embodiments, the video stream sending module 706 further includes: an encoding module configured to generate an encoded live video stream by encoding the live video stream; and a video stream sending module configured to send the encoded live video stream to a server via a second transmission channel.

[0090] In some embodiments, the apparatus 700 further includes: an audio stream acquisition module configured to acquire a live audio stream associated with a live video stream; and an audio stream transmission module configured to transmit a live audio stream to a live streaming client via a second transmission channel.

[0091] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Figure 1The computing device corresponding to the live streaming terminal 102 or server 102 can be implemented using device 800. As shown in the figure, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of device 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0092] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] The various processes and handling described above, such as methods 200, 300, and examples 400 to 500, can be executed by processing unit 801. For example, in some embodiments, methods 200, 300, and examples 400 to 500 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of example methods 200, 300, and examples 400 to 500 described above can be performed.

[0094] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0095] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0096] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0097] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0098] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing a video stream, comprising: In response to a user starting a cloud game for live streaming on a live streaming client, a cloud game video stream for the cloud game is sent to the live streaming client via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. The live video stream for the live streaming client is received via a second transmission channel, and the live video frames in the live video stream are associated with the target identifier. Based on the target identifier, the cloud gaming video frames and the live streaming video frames are combined to generate a combined video stream; as well as The combined video stream is sent for live streaming of the cloud game.

2. The method according to claim 1, wherein receiving the live video stream for the live streaming terminal via the second transmission channel comprises: Obtain metadata for each live video frame from the live video stream; as well as The target identifier corresponding to the cloud gaming video frame of the live video frame is determined from the metadata.

3. The method according to claim 2, wherein receiving the live video stream for the live streaming terminal via the second transmission channel further comprises: In response to receiving the live video stream, a decoded live video stream is generated by decoding the live video stream.

4. The method of claim 1, wherein combining the cloud gaming video frames and the live streaming video frames to generate a combined video stream based on the target identifier comprises: Based on the target identifier, align the cloud gaming video frame and the live streaming video frame; as well as The cloud gaming video frames and the live streaming video frames are combined to generate the combined video stream.

5. The method according to claim 4, wherein the cloud gaming video frame is a first cloud gaming video frame, the target identifier is a timestamp, and aligning the cloud gaming video frame and the live streaming video frame includes: Obtain the reference timestamp of the second cloud gaming video frame generated when the live video frame is received; Based on the timestamp and the reference timestamp, the latency for the cloud gaming video frame is determined; as well as The first cloud gaming video frame is aligned with the live video frame by delaying the first cloud gaming video frame by the time delay.

6. The method according to claim 1, wherein sending a cloud gaming video stream for the cloud game to the live streaming terminal via a first transmission channel comprises: Collect cloud game video frames for the cloud game; Determine the target identifier of the cloud gaming video frame; as well as The cloud game video stream, including the cloud game video frame and the target identifier, is sent to the live streaming terminal via the first transmission channel.

7. The method according to claim 6, further comprising: Collect cloud game audio streams for the cloud game; as well as The cloud game audio stream is sent to the live streaming terminal via the first transmission channel in association with the cloud game video stream.

8. The method according to claim 7, further comprising: The live audio stream for the live streaming client is received via the second transmission channel, and the live audio stream is associated with the live video stream; A mixed audio stream is generated by mixing the cloud gaming audio stream and the live streaming audio stream; as well as The mixed audio stream is sent in association with the combined video stream.

9. The method of claim 8, wherein the cloud gaming video stream and the cloud gaming audio stream are aligned using a first transmission protocol for the first transmission channel, and the live video stream and the live audio stream are aligned using a second transmission protocol for the second transmission channel.

10. The method according to claim 1, wherein the generation of the cloud gaming video stream and the generation of the combined video stream are both performed at the server running the cloud game.

11. A method for processing a video stream, comprising: In response to a user starting a cloud game for live streaming on the live streaming client, a cloud game video stream for the cloud game is received from the server via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. Associate the live video frame corresponding to the cloud game video frame in the live video stream of the live streaming terminal with the target identifier; as well as The live video stream is sent to the server via a second transmission channel for live streaming of a combined video stream generated from the cloud gaming video frames and the live video frames, the live video stream including the live video frames and the target identifier associated with the live video frames.

12. The method of claim 11, wherein associating the live video frame corresponding to the cloud gaming video frame in the live video stream for the live streaming terminal with the target identifier includes: Write the target identifier into the metadata of the live video frame in the live video stream.

13. The method of claim 11, wherein sending the live video stream to the server via a second transmission channel for live streaming a combined video stream generated from the cloud gaming video frames and the live video frames further comprises: The encoded live video stream is generated by encoding the live video stream. as well as The encoded live video stream is sent to the server via a second transmission channel.

14. The method of claim 11, further comprising: Obtain the live audio stream associated with the live video stream; as well as The live audio stream is sent to the live streaming client via the second transmission channel.

15. An apparatus for processing a video stream, comprising: The video stream sending module is configured to, in response to a user starting a cloud game for live streaming on the live streaming terminal, send a cloud game video stream for the cloud game to the live streaming terminal via a first transmission channel. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. The video stream receiving module is configured to receive a live video stream for the live streaming client via a second transmission channel, wherein the live video frames in the live video stream are associated with the target identifier; The video frame combining module is configured to combine the cloud gaming video frames and the live streaming video frames based on the target identifier to generate a combined video stream; as well as The video stream sending module is configured to send the combined video stream for live streaming of the cloud game.

16. An apparatus for processing a video stream, comprising: The video stream receiving module is configured to receive a cloud game video stream for the cloud game from the server via a first transmission channel in response to a user starting a cloud game for live streaming on the live streaming terminal. The cloud game video stream includes cloud game video frames and target identifiers of the cloud game video frames. The association module is configured to associate the live video frame corresponding to the cloud game video frame in the live video stream of the live streaming terminal with the target identifier; as well as The video stream sending module is configured to send the live video stream to the server via a second transmission channel for live streaming a combined video stream generated from the cloud gaming video frames and the live video frames, wherein the live video stream includes the live video frames and the target identifier associated with the live video frames.

17. An electronic device comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-14.

18. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-14 when executed by a processor.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.