Method, device, equipment, medium and product for processing video stream

Abstract

The embodiment of the invention relates to a method and device for processing a video stream, equipment, a medium and a product. The method includes determining, at a client, a video stream for live broadcast. The method further comprises determining whether to adjust the attribute of the target video frame in the video stream based on the performance of the client. And in response to determining to adjust the attribute of the target video frame, adjusting the color value of the target pixel in the target video frame. The method also includes displaying a target video frame in the video stream having the adjusted target pixel. According to the method, the adjustment of the attribute of the target video frame of the video stream is sunk from the stream pushing end to the stream pulling client, so that the problem that the image quality is reduced in the transmission process of the video stream is reduced to a great extent, and the attribute of the target video frame in the video stream can be adjusted according to the performance conditions of different devices; the display difference of pictures between different devices is reduced as much as possible, and the experience of users for watching live broadcast is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure generally relate to the field of live broadcasting, and more particularly to methods, devices, equipment, media, and program products for processing video streams. Background Art

[0002] At present, in the field of live broadcasting, in addition to the interactive gameplay between anchors and guests, anchors and audiences, and guests and audiences becoming more and more popular, it is also becoming more and more popular among different groups of people. Common interactive gameplay can bring a lively atmosphere to anchors, guests and audiences, and can even be used for socializing and become a bridge for communication between people. Among other more flexible interactive gameplays, some gameplays that allow audiences, guests, etc. to join the live broadcast room anytime and anywhere for fun interactions are more popular. In addition to the fun brought by the interaction itself, it will also increase the communication between anchors, guests and audiences, liven up the atmosphere of the entire live broadcast room, and improve the enthusiasm and live broadcast experience of anchors, guests and audiences.

[0003] Nowadays, with the gradual upgrading of interactive gameplay, people have higher and higher requirements for the quality of video streams transmitted during live broadcasts. High-quality video streams can make anchors, guests and viewers more willing to stay in the live broadcast room for interaction, and can also improve the visual experience of all parties when watching the live broadcast, which plays an important role in maintaining the popularity of the live broadcast room and the live broadcast effect. Therefore, improving the quality of video streams during live broadcasts has become a research topic that has received more and more attention. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, medium, and program product for processing a video stream.

[0005] According to a first aspect of the present disclosure, a method for processing a video stream is provided. The method includes determining, at a client, a video stream for live broadcast. The method also includes determining, based on the performance of the client, whether to adjust the properties of a target video frame in the video stream. The method also includes adjusting the color value of a target pixel in the target video frame in response to determining that the properties of the target video frame are to be adjusted. The method also includes displaying the target video frame in the video stream with the adjusted target pixel.

[0006] In a second aspect of the present disclosure, there is provided an apparatus for processing a video stream. The apparatus includes a video stream determination module configured to determine a video stream for live broadcast at a client; an attribute adjustment determination module configured to determine whether to adjust the attributes of a target video frame in the video stream based on the performance of the client; a color value adjustment module configured to, in response to determining to adjust the attributes of the target video frame, adjust the color value of a target pixel in the target video frame; and a target video frame display module configured to display the target video frame in the video stream having the adjusted target pixel.

[0007] In a third aspect of the present disclosure, there is provided an electronic device including at least one processor; and a storage device for storing at least one program, which when executed by the at least one processor causes the at least one processor to implement the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, which when executed by a processor implements the method according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, there is provided a computer program product. The computer program product includes a computer program, which when executed by a processor implements the method according to the first aspect of the present disclosure.

[0010] It should be understood that the content described in this part is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By describing the exemplary embodiments of the present disclosure in more detail in conjunction with the drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent, wherein in the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.

[0012] Figure 1 A schematic diagram of an exemplary environment in which the devices and / or methods of some embodiments of the present disclosure may be implemented;

[0013] Figure 2 A schematic diagram of an exemplary method for processing a video stream according to some embodiments of the present disclosure;

[0014] Figure 3 A schematic diagram of an example of a flowchart for processing a video stream according to some embodiments of the present disclosure;

[0015] Figure 4Schematic diagram showing an example of another flowchart for processing a video stream according to some embodiments of the present disclosure;

[0016] Figure 5 Schematic diagram showing an example of yet another flowchart for processing a video stream according to some embodiments of the present disclosure;

[0017] Figure 6 Schematic diagram showing an example of still another flowchart for processing a video stream according to some embodiments of the present disclosure;

[0018] Figure 7 Schematic diagram showing an example of a video stream processing frame link according to some embodiments of the present disclosure;

[0019] Figure 8 Schematic block diagram showing an apparatus for processing a video stream according to some embodiments of the present disclosure;

[0020] Figure 9 Schematic block diagram showing an example device suitable for implementing multiple embodiments of the present disclosure.

[0021] In the respective drawings, the same or corresponding reference numerals denote the same or corresponding parts. Detailed Description

[0022] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0023] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, when receiving the user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested to be executed will require the acquisition and use of the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operations of the technical solutions of the present disclosure according to the prompt message.

[0025] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving the user's active request may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It should be understood that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0028] In the description of the embodiments of the present disclosure, the term "including" and its similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions below.

[0029] In today's live broadcast scenario, the picture quality of the video stream transmitted during the live broadcast is becoming increasingly important and has become an important reference index for increasing the popularity of the live broadcast room and the dwell time in the live broadcast room. During the multi-person live broadcast, the pictures shown by the anchor or the guest side are decoded from the real-time communication stream published by the subscribed cloud game. There is a process from encoding to transmission and then to decoding on this link, and this process is usually accompanied by picture quality loss. Therefore, the pictures shown by the anchor or the guest are of certain picture quality degradation compared with the original images during game rendering, which leads to a decrease in the quality of the video stream received by the anchor or the guest and even the audience, seriously affecting the viewing experience of the live broadcast room and possibly leading to a decline in the popularity of the live broadcast room and even a reduction in the number of people in the live broadcast room. Under the traditional solution, in the solution for multi-person live broadcast (such as multi-person live broadcast cloud game), a single room, multi-user, one-way stream solution is used. Therefore, for the users of cloud games such as the anchor and the guest who join the room, the pictures previewed locally all come from the same real-time communication stream published by the cloud game server, so as to minimize the quality gap of the pictures seen by different users and improve the live broadcast viewing experience of users using different devices.

[0030] However, in practical applications, since the types and performances of the devices used by different users are inconsistent, and since there are problems of picture quality degradation in the transmission link of the video stream, when the client receives the transmitted video stream, the picture already has problems of picture quality degradation. At the same time, due to problems such as the performance or type of the device, it may not be able to present the video stream to the user in the best quality completely correctly. This results in the inability to obtain the optimal display effect among different device manufacturers under a single transmission link.

[0031] To at least solve the above and other potential problems, embodiments of the present disclosure propose a method for processing a video stream. In this method, the computing device can first determine the video stream for live broadcast at the client. Among them, the video stream is obtained by decoding the original video stream. Then, after the client obtains the video stream, it will determine whether to adjust the attributes of the target video frame in the video stream according to the performance of the client. Then, after determining that the attributes of the target video frame need to be adjusted, the color values of the target pixels in the target video frame are adjusted. Finally, the computing device displays the target video frame in the video stream with the adjusted target pixels at the client. Through this method, by sinking the adjustment of the attributes of the target video frame of the video stream from the push stream end to the pull stream client, the problem of picture quality degradation caused by the video stream during transmission is greatly reduced, and the attributes of the target video frame in the video stream can be adjusted according to the performance of different devices, minimizing the display differences between different devices and improving the user experience of watching the live broadcast.

[0032] It can be understood that the present disclosure is only described in combination with related embodiments, and does not limit the protection scope of the present disclosure with related embodiments. Any technical solutions in other disclosures that fall within the protection scope of the present disclosure should be protected.

[0033] The embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings. Among them Figure 1 shows an example environment in which the devices and / or methods of the embodiments of the present disclosure can be implemented. In environment 100, the computing device 102 can be a client, and the processing of the video stream is completed at the client. For example, in a direct broadcast scenario, the computing device 102 is a client for the host or a client for the guest. In addition, the computing device 102 can also obtain the performance of its own device and determine whether to perform the adjustment of the target video frame in the video stream according to the level of the performance.

[0034] Examples of the computing device 102 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, distributed computing environments including any one of the above systems or devices, and the like.

[0035] As Figure 1 shown, the computing device 102 may determine a video stream 106 for live streaming. Among them, the video stream 106 is obtained by the computing device 102 processing the original video stream received via a Real Time Communication (RTC) transmission channel. For example, the video stream 106 is obtained by decoding the original video stream. In one example, the video stream is a video stream of cloud games from a cloud game server.

[0036] Next, after obtaining the video stream 106 for live streaming, the computing device 102 further determines the performance 112 of the computing device 102, and determines whether to adjust the attributes 114 of the target video frame 108 in the video stream 106 according to the obtained performance 112.

[0037] When the computing device 102 determines that its own performance 112 meets the condition that the video frames in the video stream can be adjusted, it can further determine to perform corresponding adjustments on the attributes 114 of the target video frame 108 in the video stream 106. Additionally, if the performance does not meet the condition that the video frames in the video stream can be adjusted, the attributes 114 of the target video frame 108 are not adjusted.

[0038] Then, after the computing device 102 determines that the attributes 114 of the target video frame 108 need to be adjusted, it adjusts the color value 104 of the target pixel 110 in the target video frame 108. Thus, the adjustment of the attributes 114 of the target video frame 108 is achieved by adjusting the color value 104 of the target pixel 110 in the target video frame 108. Among them, the adjustment of the color value 104 can be performed in the first color space or in the second color space, and the first color space is different from the second color space.

[0039] The adjustment of the color value 104 affects the change of the attributes 114 of the target video frame 108. For example, after adjusting the color value 104 of the target pixel 110, the sharpness, color, etc. included in the attributes 114 that can be adjusted thereby can be adjusted.

[0040] Finally, after the adjustment of the color value 104 of the target pixel 110 in the target video frame 108 is completed, the video stream 106 including the adjusted target video frame 108 is displayed on the display screen of the computing device 102 acting as a client.

[0041] With this method, by sinking the adjustment of the attributes of the target video frame of the video stream from the streaming end to the pulling client, the problem of image quality degradation caused during the transmission of the video stream is largely reduced, and the attributes of the target video frame in the video stream can be adjusted according to the performance of different devices, minimizing the display differences of the images between different devices and improving the user experience of watching the live broadcast.

[0042] The above combines Figure 1 with a schematic diagram of an example environment in which the devices and / or methods of some embodiments of the present disclosure may be implemented. Below, in combination with Figure 2 a schematic diagram of an example method 200 for processing a video stream according to some embodiments of the present disclosure is described. This example method may be executed by Figure 1 the computing device 102 in

[0043] As Figure 2 shown, in the example method 200, at block 202, a video stream for live broadcast is determined at the client. The client may be Figure 1 the computing device 102 in

[0044]

[0045] Next, at block 204, based on the performance of the client, it is determined whether to adjust the attributes of the target video frame in the video stream. For example, the performance of the computing device 102 may be represented by a performance score.

[0046] In some embodiments, a scoring model for the performance score of computing device 102 can be utilized to obtain the performance score of computing device 102. For example, a machine learning model can be used to obtain the parameters of computing device 102, and based on the obtained parameters, the performance score of computing device 102 can be determined. The parameters of computing device 102 can include information such as the processor model and the memory size. Subsequently, when it is determined that the performance score is greater than the threshold score, it can be determined that the performance 112 of computing device 102 meets the requirements, and it is determined that the attributes 114 of the target video frame 108 in video stream 106 can be adjusted accordingly. Additionally, the training of the machine learning model used for calculating the performance score can be adjusted by the user according to the actual situation, and this application does not limit it here.

[0047] In another example, the performance score of computing device 102 can also be determined according to a pre-set method for calculating the performance score. For example, the performance score of computing device 102 can be determined according to a pre-set calculation function for the performance score. The calculation function can include various parameters for calculation, and the parameters can be obtained using computing device 102. Additionally, the calculation function can be set by the user according to the actual situation, and this application does not limit it here.

[0048] In some embodiments, in addition to determining whether to adjust the attributes 114 of the target video frame 108 in video stream 106 according to the performance 112 of computing device 102, it can also be determined whether to adjust the attributes 114 of the target video frame 108 in video stream 106 according to the device type of computing device 102.

[0049] In one example, computing device 102 first obtains a set of device types of a group of devices having the target performance available for adjusting attributes 114, and then determines whether the device type of computing device 102 belongs to one of the device types in the set of device types. If the device type of computing device 102 belongs to one of the device types in the set of device types, it indicates that computing device 102 has the target performance.

[0050] For example, when it is determined that the device type of computing device 102 belongs to one of the device types in the set of device types, it can be determined that the attributes 114 of the target video frame 108 in video stream 106 can be adjusted on computing device 102. Additionally, when the device type of computing device 102 does not belong to one of the device types in the set of device types, it is determined not to adjust the attributes 114 of the target video frame 108 in video stream 106.

[0051] Additionally, it is only possible to determine an adjustment to the attribute 114 of the target video frame 108 in the video stream 106 when the performance score of the performance 112 of the computing device 102 reaches a threshold score and the device type of the computing device 102 belongs to one of a group of device types.

[0052] After the computing device 102 determines that the performance 112 meets the standard, it can further determine to make corresponding adjustments to the attribute 114 of the target video frame 108 in the video stream 106. Additionally, if the performance does not meet the standard, no adjustment is made to the attribute 114.

[0053] Then, at block 206, in response to determining an adjustment to the attribute of the target video frame, the color value of the target pixel in the target video frame is adjusted. For example, the attribute 114 can include multiple ones, such as brightness, sharpness, color, and contrast, etc.

[0054] In some embodiments, the adjustment for the attribute 114 needs to be performed in different color spaces. For example, the adjustment for the gray value in the attribute 114 can be performed in the second color space, and the adjustment for the contrast, color, etc. in the attribute 114 can be performed in the first color space. Among them, the first color space is the Red Green Blue (RGB) color space, and the second color space is the Luminance - Blue Chrominance - Red Chrominance (YCbCr) color space. Additionally, the computing device 102 can switch the color space between the first color space and the second color space. Additionally, the first color space and the second color space can be any color spaces that can implement this solution, and this application does not limit them here.

[0055] Subsequently, after determining that the attribute 114 of the target video frame 108 in the video stream 106 can be adjusted accordingly, the color value of the target pixel 110 in the target video frame 108 is further adjusted.

[0056] In some embodiments, the computing device 102 also obtains the texture length and texture width of the texture of the target video frame 108, and uses the obtained texture length and texture width to determine the pixel step for the target pixel 110. Subsequently, according to the determined pixel step, combined with the target pixel 110, a group of pixels for the target pixel 110 can be determined. In one example, the obtained texture length and texture width are used to determine the pixel step for the target pixel 110. For example, 8 pixel steps can be obtained, and 8 pixels located around the target pixel 110 are determined according to the obtained 8 pixel steps. Additionally, the calculation of the pixel step can be defined by the user himself / herself, and this application does not limit it here.

[0057] Subsequently, the computing device 102 may obtain the color value of the target pixel 110 and a set of color values of a set of pixels located around the target pixel 110. Additionally, the obtaining of the color value of the target pixel 110 and the set of color values of the set of pixels located around the target pixel 110 is implemented in the first color space. It should be noted that the number of the set of pixels located around the target pixel 110 can be defined by the user himself / herself, and this application does not limit it here.

[0058] Subsequently, the computing device 102 converts the color value of the target pixel 110 and the set of color values of the set of pixels located around the target pixel 110 from the first color space to the second color space, and then determines the gray value of the luminance component for the target pixel 110 and a set of gray values of the luminance components for the set of pixels in the second color space. After obtaining the gray value of the luminance component for the target pixel and the set of gray values of the luminance components for the set of pixels, the gray value of the target pixel can also be adjusted.

[0059] In some embodiments, the computing device 102 determines a set of gradients corresponding to the target pixel 110 according to the gray value and the set of gray values. For example, a gray gradient is calculated by combining the difference between the gray value of the target pixel and the gray value of a pixel in the set of pixels with the distance between the two pixels. Then, the maximum gradient in the set of gradients is further determined. Subsequently, a sharpness parameter for adjusting the target video frame is also determined, and the obtained sharpness parameter and the maximum gradient are used to adjust the gray value of the luminance component.

[0060] Subsequently, after the gray value of the luminance component is adjusted, the adjusted luminance component can also be used to update the color value of the target pixel 110 so as to adjust the sharpness in the attribute 114 of the target video frame 108. For example, the target pixel with the adjusted luminance component is converted from the second color space to the first color space for rendering and displaying the adjusted video stream 106 on the screen at the computing device 102.

[0061] In some embodiments, in addition to adjusting the sharpness in the attribute 114, the color in the attribute 114 can also be adjusted, and then the target video frame 108 with the adjusted attribute is rendered to display the adjusted video stream 106 in the computing device 102.

[0062] First, it is determined that the target video frame 108 is within the first color space, and then a denormalization operation is performed on the color value 104 of the target pixel 110 in the target video frame 108. For example, the first color space is mapped from the first range to the second range. In one example, the first color space is mapped from the first range [0, 1.0] to the second range [-0.5, 0.5].

[0063] Subsequently, the contrast parameter for the target video frame 108 is also obtained, and the obtained contrast parameter is used to adjust the denormalized color value 104. In one example, when the first color space is the RGB color space, the three color value components of R, G, and B corresponding to the color value 104 are respectively calculated with the contrast parameter, for example, multiplied, so as to obtain the adjusted color value, and then the adjusted color value is renormalized, that is, the first color space is remapped from the second range [-0.5, 0.5] to the first range [0, 1.0].

[0064] The brightness parameter for the target video frame 108 can also be obtained, and the renormalized color value is adjusted according to the obtained brightness parameter. For example, the brightness parameter is added to the renormalized color value. Then, the target video frame 108 with adjusted attributes can be rendered to display the adjusted video stream 106 in the computing device 102.

[0065] It should be noted that the adjustment of the sharpness in the attribute 114 of the target video frame 108 or the adjustment of the color in the attribute 114 of the target video frame 108 can be executed separately. Additionally, the adjustment of the sharpness in the attribute 114 of the target video frame 108 or the adjustment of the color in the attribute 114 of the target video frame 108 can be executed serially. In one example, the computing device can first determine whether the performance score of the computing device 102 is greater than the first threshold score and / or whether the type of the computing device 102 belongs to the first group of device types. If it is determined that the performance score of the computing device 102 is greater than the first threshold score and / or the type of the computing device 102 belongs to the first group of device types, the sharpness in the attribute 114 can be adjusted first. The computing device can then determine whether the performance score of the computing device 102 is greater than the second threshold score and / or whether the type of the computing device 102 belongs to the second group of device types. If it is determined that the performance score of the computing device 102 is greater than the second threshold score and / or the type of the computing device 102 belongs to the second group of device types, the color in the attribute 114 is adjusted. Additionally, the first threshold score and the second threshold score can be the same or different; the first group of device types and the second group of device types can be the same, different, or partially the same. In another example, the color in the attribute 114 can be adjusted first, and then the sharpness in the attribute 114 can be adjusted. In yet another example, the adjustments of the sharpness and color for the attribute 114 can be executed in parallel, and the adjustments for the sharpness and color are respectively executed in different color spaces. The conditions for determining whether to perform the attribute adjustment in the above examples can be the same or different.

[0066] The execution order of the adjustments for sharpness and color in Attribute 114 can be defined by the user themselves, and this application places no restrictions here.

[0067] Finally, at block 208, the target video frame in the video stream with the adjusted target pixel is displayed.

[0068] It should be noted that after the computing device 102 receives the video stream 106, the adjustment of Attribute 114 of the target pixel 110 of the target video frame 108 in the video stream 106 is performed on the computing device 102, and the adjusted target video frame 108 can be re-rendered into the video stream 106 and then displayed on the display screen of the computing device 102.

[0069] With this method, by sinking the adjustment of the attributes of the target video frame of the video stream from the pushing end to the pulling client, the problem of image quality degradation caused during the transmission of the video stream is largely reduced, and the attributes of the target video frame in the video stream can be adjusted according to the performance of different devices, minimizing the display differences between different devices as much as possible and improving the user experience of watching live broadcasts.

[0070] As described above in conjunction with Figure 2 A schematic diagram of an example method for processing a video stream according to some embodiments of the disclosure is described. The following is a schematic diagram in conjunction with Figure 3 A schematic diagram of an example of a flowchart for processing a video stream according to some embodiments of the present disclosure is described.

[0071] As Figure 3 shown in Example 300, taking the example of a multi-person live cloud game in a live broadcast scenario, after the live broadcast starts at 302, the client needs to subscribe to the cloud game video stream at 304. After the client receives the subscribed cloud game video stream (hereinafter referred to as the video stream), it decodes the received video stream at 306, and then the client takes over the rendering of the decoded video stream at 308.

[0072] Among them, after the takeover rendering in the client, since there will be a problem of image quality degradation in the decoded video stream, it is necessary to enhance the image quality of the decoded video stream in the client.

[0073] Subsequently, at 310, it is determined whether to perform sharpness adjustment. After it is determined to adjust the sharpness, at 312, the color values of the target pixel and the surrounding pixels are sampled and converted into grayscale values. Among them, the sampling of the color values of the target pixel and the surrounding pixels is performed in the first color space.

[0074] Subsequently, at 314, the gradient is calculated to detect the edge intensity and direction of the target pixel. Based on the calculated edge intensity and direction, the maximum gradient direction can be further selected at 316, and the image sharpness can be enhanced according to the sharpness parameter. Among them, the gradient magnitudes of the target pixel and a group of pixels around the target pixel are different. Additionally, the process of adjusting the sharpness is implemented in the two-color space.

[0075] Next, at 318, the luminance component is adjusted in the second color space and then converted back to the first color space. After the color space is converted back to the first color space, at 320, the video stream where the target pixel with adjusted sharpness is located is rendered onto the screen.

[0076] Finally, at the client, the video stream where the video frame with adjusted sharpness is located is displayed, and the process ends at 322.

[0077] If it is determined not to adjust the sharpness, the video stream is directly rendered onto the screen at 320, and the process ends at 322.

[0078] Through this method, by sinking the adjustment of the sharpness of the target video frame for the video stream to the pull-stream client from the push-stream end, the problem of image quality degradation caused during the transmission of the video stream is largely reduced, improving the user experience of watching the live broadcast.

[0079] The above combines Figure 3 to describe a schematic diagram of an example of a flowchart for processing a video stream according to some disclosed embodiments. The following combines Figure 4 to describe a schematic diagram of an example of another flowchart for processing a video stream according to some embodiments of the present disclosure.

[0080] As Figure 4 shown, in Example 400, in combination with the previous Example 300, after the business in the client takes over the rendering, at 402, it is determined whether to perform color adjustment. After it is determined to perform color adjustment, at 404, the first color space is removed from normalization, that is, the first color space is de-normalized, and the range of the first color space is mapped from the first range [0, 1.0] to the second range [-0.5, 0.5].

[0081] Next, at 406, contrast adjustment is performed. The r, g, and b components of the de-normalized color are multiplied by the contrast parameter. Subsequently, at 408, the color is re-normalized, and the first color space is re-normalized to [0, 1.0], that is, the first color space is remapped from the second range [-0.5, 0.5] to the first range [0, 1.0].

[0082] Then, at 410, brightness adjustment is performed by adding the brightness parameter to the re-normalized color values, and at 412, the video stream with the adjusted color of the target pixel is rendered onto the screen. Subsequently, the process ends at 414.

[0083] If it is determined not to adjust the color, the video stream is directly rendered onto the screen at 412, and the process ends at 414.

[0084] With this method, by sinking the adjustment of the color of the target video frame for the video stream from the pushing end to the pulling client, the problem of image quality degradation caused during the transmission of the video stream is largely reduced, improving the user experience of watching the live broadcast.

[0085] The above combines Figure 4 with the schematic diagram of an example of another flowchart for processing a video stream according to some disclosed embodiments. The following combines Figure 5 with the schematic diagram of an example of yet another flowchart for processing a video stream according to some embodiments of the present disclosure. This example can be executed by the computing device 102 in Figure 1 or any suitable device as a client.

[0086] As Figure 5 shown, in Example 500, combining the previous Example 300 and Example 400, after the business in the client takes over the rendering, first at 502, it is determined whether to perform sharpness adjustment. After it is determined to adjust the sharpness, at 504, the color values of the target pixel and the surrounding pixels are sampled and converted into grayscale values. Among them, the sampling of the color values of the target pixel and the surrounding pixels is performed in the first color space.

[0087] Subsequently, at 506, the gradient is calculated to detect the edge strength and direction of the target pixel. According to the calculated edge strength and direction, the maximum gradient direction can be further selected at 508, and the image sharpness is enhanced according to the sharpness parameter. Among them, the gradient magnitudes of the target pixel and a group of pixels around the target pixel are different. Additionally, the process of adjusting the sharpness is implemented in the second color space.

[0088] Next, at 510, the brightness component is adjusted in the second color space and then converted back to the first color space. After the color space is converted back to the first color space, further at 512, it is determined whether to perform color adjustment. After it is determined to adjust the color, at 514, the first color space is removed from normalization, that is, the first color space is de-normalized, and the range of the first color space is mapped from the first range [0, 1.0] to the second range [-0.5, 0.5].

[0089] Next, at 516, contrast adjustment is performed. The r, g, and b components of the denormalized color are multiplied by the contrast parameter. Subsequently, at 518, the color is renormalized, and the first color space is renormalized to [0, 1.0], that is, the first color space is remapped from the second range [-0.5, 0.5] to the first range [0, 1.0].

[0090] Then, at 520, brightness adjustment is performed. The brightness parameter is added to the renormalized color value, and at 522, the video stream where the target pixel with adjusted color is located is rendered onto the screen. Subsequently, this process ends at 524.

[0091] If it is determined that neither sharpness nor color is adjusted, the video stream is directly rendered onto the screen at 522, and this process ends at 524.

[0092] Through this method, by sinking the adjustment of the sharpness and color of the target video frame of the video stream from the push end to the pull client, the problem of image quality degradation caused during the transmission of the video stream is greatly reduced, and the display difference between the images of different devices is minimized as much as possible, improving the user experience of watching the live broadcast.

[0093] The above combines Figure 5 to describe a schematic diagram of an example of another flowchart for processing a video stream according to some disclosed embodiments. The following combines Figure 6 to describe a schematic diagram of an example of another flowchart for processing a video stream according to some embodiments of the present disclosure.

[0094] As Figure 6 shown, in Example 600, in combination with the previous Example 500, first at 602, it is determined whether color adjustment is to be performed. After it is determined that color adjustment is to be performed, at 604, the first color space is removed from normalization, that is, the first color space is denormalized, and the range of the first color space is mapped from the first range [0, 1.0] to the second range [-0.5, 0.5].

[0095] Next, at 606, contrast adjustment is performed. The r, g, and b components of the denormalized color are multiplied by the contrast parameter. Subsequently, at 608, the color is renormalized, and the first color space is renormalized to [0, 1.0], that is, the first color space is remapped from the second range [-0.5, 0.5] to the first range [0, 1.0].

[0096] Then, at 610, brightness adjustment is performed by adding the brightness parameter to the re-normalized color values. Subsequently, at 612, it is determined whether to perform sharpness adjustment. After determining that sharpness adjustment is to be performed, at 614, the color values of the target pixel and its surrounding pixels are sampled and converted into grayscale values. Among them, the sampling of the color values of the target pixel and its surrounding pixels is performed in the first color space.

[0097] Subsequently, at 616, the gradient is calculated to detect the edge strength and direction of the target pixel. Based on the calculated edge strength and direction, the maximum gradient direction can be further selected at 618, and the image sharpness can be enhanced according to the sharpness parameter. Among them, the gradient magnitudes of the target pixel and a group of pixels around the target pixel are different. Additionally, the process of adjusting sharpness is implemented in the second color space.

[0098] Next, at 620, the brightness component is adjusted in the second color space and then converted back to the first color space. After the color space is converted back to the first color space, the video stream with adjusted sharpness is rendered onto the screen at 622, and the process ends at 624.

[0099] If it is determined that neither sharpness nor color is to be adjusted, the video stream is directly rendered onto the screen at 622, and the process ends at 624.

[0100] Through this method, by sinking the adjustment of the sharpness and color of the target video frame of the video stream from the push end to the pull client, the problem of image quality degradation caused during the transmission of the video stream is greatly reduced, the display differences between different devices are minimized as much as possible, and the user experience of watching the live broadcast is improved.

[0101] The above is combined with Figure 6 to describe a schematic diagram of another flowchart example for processing a video stream according to some disclosed embodiments. The following is combined with Figure 7 to describe a schematic diagram of an example of a picture link for processing a video stream according to some embodiments of the present disclosure.

[0102] As Figure 7 shown, in Example 700, taking the live broadcast of a multiplayer cloud game in a live broadcast scenario as an example, the cloud game 702 includes a virtual avatar application 712, and virtual avatars can be generated within the virtual avatar application 712. For example, at 714, virtual avatar picture frame generation can be performed, where the resolution of the generated picture frame is 720p. Additionally, the resolution of the generated picture frame can be determined according to the performance of the device running the cloud game or can be a pre-set resolution.

[0103] Next, at 710, through inter-process communication, at 704, screen capture is performed on the frame of the generated virtual image, and then the captured frame is encoded at 706, and a video stream is generated.

[0104] Subsequently, at 708, the video stream frame is pushed to the server, and the video stream is pushed to the multi-person live broadcast - host 718 end via the real-time communication server 716, that is, pushed to the client side. The host side pulls the video stream at 720 via the real-time communication server 716, and decodes the pulled video stream at 722.

[0105] Subsequently, at 724, quality gain is performed on the decoded video stream, and the video stream with deteriorated image quality due to the transmission process is adjusted to reduce the impact caused by the deteriorated image quality.

[0106] Finally, the video stream with quality gain is rendered at 726 to display the adjusted video stream at the client side.

[0107] At the multi-person live broadcast - guest 728, the video stream is also pulled via the real-time communication server 716, the pulled video stream is decoded, and quality gain is also performed on the decoded video stream at 730, and then the adjusted video stream is rendered at the client side to be displayed on the display screen of the client.

[0108] Through this method, by sinking the adjustment of the attributes of the target video frame of the video stream from the push end to the pull client side, the problem of deteriorated image quality caused by the video stream during transmission is greatly reduced, the display difference of the images between different devices is minimized as much as possible, and the user experience of watching the live broadcast is improved.

[0109] As Figure 8 shown, the apparatus 800 includes a video stream determination module 802 configured to determine a video stream for live broadcast at the client; an attribute adjustment determination module 804 configured to determine whether to adjust the attributes of the target video frame in the video stream based on the performance of the client; a color value adjustment module 806 configured to, in response to determining to adjust the attributes of the target video frame, adjust the color value of the target pixel in the target video frame; and a target video frame display module 808 configured to display the target video frame in the video stream with the adjusted target pixel.

[0110] In some embodiments, the video stream determination module 802 includes: an original video stream receiving module configured to receive an original video stream for live broadcast at the client; and an original video stream decoding module configured to determine the video stream for live broadcast by decoding the original video stream.

[0111] In some embodiments, the attribute adjustment determination module 804 includes: a performance score determination module configured to determine a performance score for the performance of the client; and an attribute adjustment determination module configured to determine to adjust the attributes of the target video frame in the video stream in response to the performance score being greater than a threshold score.

[0112] In some embodiments, the attribute adjustment determination module 804 includes: a set of device type determination modules configured to determine a set of device types of a set of devices having target performance available for adjusting the attributes of video frames in the video stream; and an attribute adjustment determination module configured to determine that the client has the target performance for adjusting the attributes of the target video frame in the video stream in response to the type of the client belonging to the set of device types.

[0113] In some embodiments, where the attribute includes sharpness, the color value is the color value of the target pixel in the first color space, and the color value adjustment module 806 includes: a color value and a set of color value acquisition module configured to acquire the color value of the target pixel and a set of color values of a set of pixels located around the target pixel in the first color space; a grayscale value and a set of grayscale value determination module configured to determine, based on the color value and the set of color values, the grayscale value of the luminance component for the target pixel and a set of grayscale values of the luminance components for the set of pixels in the second color space; a grayscale value adjustment module configured to adjust the grayscale value of the luminance component based on the grayscale value and the set of grayscale values; and a color value update module configured to update the color value of the target pixel in the target video frame based on the adjusted luminance component to adjust the sharpness of the target video frame.

[0114] In some embodiments, the grayscale value adjustment module includes: a set of gradient determination modules configured to determine, based on the grayscale value and the set of grayscale values, a set of gradients corresponding to the target pixel; a sharpness parameter determination module configured to determine a sharpness parameter for adjusting the target video frame; and a grayscale value adjustment module configured to adjust the grayscale value of the luminance component based on the maximum gradient in the set of gradients and the sharpness parameter.

[0115] In some embodiments, the color value update module includes: a color space conversion module configured to update the color value of the target pixel by performing a color space conversion on the adjusted luminance component.

[0116] In some embodiments, the color value adjustment module 806 further includes: a texture length and texture width acquisition module configured to acquire the texture length and texture width of the texture for the target video frame; a pixel step determination module configured to determine a pixel step for the target pixel based on the texture width and the texture length; and a set of pixel determination modules configured to determine a set of pixels based on the pixel step and the target pixel.

[0117] In some embodiments, where the attribute includes color, and the color value adjustment module 806 further includes: a denormalization module configured to denormalize the color value of a target pixel in a target video frame; a contrast parameter acquisition module configured to acquire a contrast parameter for the target video frame; a denormalized color value adjustment module configured to adjust the denormalized color value based on the contrast parameter; a renormalization module configured to renormalize the adjusted color value; a brightness parameter acquisition module configured to acquire a brightness parameter for the target video frame; and a normalized color value adjustment module configured to adjust the normalized color value based on the brightness parameter.

[0118] In some embodiments, the denormalization module includes: a color value mapping module configured to map the color value of the target pixel from a first range to a second range, the second range being different from the first range.

[0119] In some embodiments, the target attribute includes at least one of the following: displacement, transparency, and scaling.

[0120] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 102 in can be implemented using the device 900. As shown, the device 900 includes a central processing unit (CPU) 901, which can execute various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The CPU 901, ROM 902, and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0121] Multiple components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disc, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0122] The various processes and treatments described above, such as method 200 and examples 300 to 700, may be executed by processing unit 901. For example, in some embodiments, method 200 and examples 300 to 700 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of the example methods 200 and examples 300 to 700 described above may be performed.

[0123] The present disclosure may be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for performing various aspects of the present disclosure.

[0124] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of a computer-readable storage medium (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not construed to be an instantaneous signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0125] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing devices, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0126] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0127] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of a method, apparatus (system), and computer program product according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer - readable program instructions.

[0128] These computer - readable program instructions can be provided to a processing unit of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that, when the instructions are executed by the processing unit of the computer or other programmable data - processing apparatus, a device is created that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner. Thus, the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0129] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to generate a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0130] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0131] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing a video stream, comprising: Determine the video stream for live broadcast on the client side; Determining whether to adjust a property of a target video frame in the video stream based on the performance of the client; In response to determining that the attribute of the target video frame is to be adjusted, adjusting a color value of a target pixel in the target video frame; and The target video frame in the video stream with the adjusted target pixel is displayed.

2. The method according to claim 1, wherein determining the video stream for live broadcast at the client comprises: Receiving, at the client, an original video stream for live broadcast; as well as The video stream for live broadcast is determined by decoding the original video stream.

3. The method of claim 1, wherein determining whether to adjust the attributes of the target video frame in the video stream comprises: determining a performance score for the performance of the client; and In response to the performance score being greater than a threshold score, determining to adjust a property of a target video frame in the video stream.

4. The method of claim 1, wherein determining whether to adjust a property of a target video frame in the video stream comprises: determining a set of device types for a set of devices having target capabilities that can be used to adjust properties of video frames in the video stream; In response to the type of the client belonging to the group of device types, it is determined that the client has the target capability for adjusting the attribute of the target video frame in the video stream.

5. The method of claim 1, wherein the attribute comprises sharpness, the color value is a color value of the target pixel in a first color space, and adjusting the color value of the target pixel in the target video frame comprises: Acquire the color value of the target pixel and a group of color values ​​of a group of pixels located around the target pixel in the first color space; Determining, based on the color value and the set of color values, a grayscale value for a luminance component of the target pixel and a set of grayscale values ​​for the luminance component of the set of pixels in a second color space; Based on the grayscale value and the set of grayscale values, adjusting the grayscale value of the brightness component; as well as Based on the adjusted luma component, a color value of the target pixel in the target video frame is updated to adjust the sharpness of the target video frame.

6. The method according to claim 5, wherein adjusting the grayscale value of the brightness component comprises: determining a set of gradients corresponding to the target pixel based on the grayscale value and the set of grayscale values; Determining a sharpness parameter for adjusting the target video frame; as well as The grayscale value of the luminance component is adjusted based on a maximum gradient in the set of gradients and the sharpness parameter.

7. The method of claim 5, wherein updating the color value of the target pixel in the target video frame to adjust the sharpness of the target video frame comprises: The color value of the target pixel is updated by performing a color space conversion on the adjusted luminance component.

8. The method according to claim 5, wherein adjusting the color value of the target pixel in the target video frame further comprises: Obtaining a texture length and a texture width of a texture for the target video frame; determining a pixel step size for the target pixel based on the texture width and the texture length; as well as Based on the pixel step size and the target pixel, the group of pixels is determined.

9. The method of claim 1 , wherein the attribute comprises color, and adjusting the color value of a target pixel in the target video frame comprises: Denormalizing the color value of the target pixel in the target video frame; Obtaining a contrast parameter for the target video frame; adjusting the denormalized color values ​​based on the contrast parameter; Renormalizing the adjusted color value; Obtaining brightness parameters for the target video frame; as well as Based on the brightness parameter, the normalized color value is adjusted.

10. The method of claim 9, wherein denormalizing the color value of the target pixel in the target video frame comprises: A color value of the target pixel is mapped from a first range to a second range, the second range being different from the first range.

11. The method according to claim 1, wherein the video stream is a video stream for cloud gaming.

12. An apparatus for processing a video stream, comprising: A video stream determination module, configured to determine a video stream for live broadcasting on a client; an attribute adjustment determination module, configured to determine whether to adjust an attribute of a target video frame in the video stream based on the performance of the client; A color value adjustment module, configured to adjust a color value of a target pixel in the target video frame in response to determining to adjust the attribute of the target video frame; as well as The target video frame display module is configured to display the target video frame in the video stream with the adjusted target pixels.

13. An electronic device comprising: at least one processor; as well as A storage device, used to store at least one program, when the at least one program is executed by the at least one processor, so that the at least one processor implements the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images