Live image processing method, device and equipment, live system and storage medium

Abstract

Embodiments of the present disclosure provide a live image processing method and device, equipment, a live system and a storage medium. A high-quality sample image and a low-quality sample image including a face and a target scene can be obtained, and then a generator to be trained is used to generate a target image based on the low-quality sample image; a target loss is determined based on a discrimination result of a respective face region in the target image and a high-quality sample image and a discrimination result of a respective target scene region, and the generator and the discriminator are trained based on the target loss, and the trained generator is used as a quality enhancement network. After the quality enhancement network is trained, the live image can be enhanced by using the quality enhancement network. In this way, the face and the target scene in the high-quality image after quality enhancement generated by the finally trained generator are more natural, a better quality enhancement effect is obtained, and the consumption of processing resources is not increased.

Description

Technical Field

[0001] This disclosure relates to the field of live streaming technology, and in particular to a live streaming image processing method, apparatus, device, live streaming system, and storage medium. Background Technology

[0002] In live streaming, to provide users with better image quality and an enhanced visual experience, the live stream is typically enhanced before being shown to them. For scenarios like game streaming and esports streaming, the live stream includes large areas of the game or esports scene, as well as smaller areas of faces. Therefore, when enhancing the live stream, the focus is usually on the larger scene areas, using enhancement methods suitable for those scenes. However, because faces and scene scenes differ significantly, enhancement methods suitable for the scene may not be suitable for faces, resulting in unnatural-looking enhanced faces and affecting the overall visual experience. Summary of the Invention

[0003] This disclosure provides a live streaming image processing method, apparatus, device, live streaming system, and storage medium.

[0004] According to a first aspect of the present disclosure, a live streaming image processing method is provided, the method comprising:

[0005] Obtain low-quality live images from the live video stream, wherein the low-quality live images include face regions and target scene regions;

[0006] The low-quality live stream image is input into a pre-trained image quality enhancement network to obtain a high-quality live stream image after image quality enhancement. The image quality enhancement network is obtained in the following way: acquiring a first sample image pair, which includes a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; generating a target image based on the low-quality sample image using a generator to be trained; performing pixel-by-pixel judgment on the target image and the high-quality sample image using a discriminator to be trained; determining a target loss based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image respectively by the discriminator to be trained; training the generator and the discriminator to be trained based on the target loss; and using the trained generator as the image quality enhancement network.

[0007] The high-quality live stream image is pushed to the viewer's client or displayed.

[0008] According to a second aspect of the present disclosure, a live streaming image processing apparatus is provided, the apparatus comprising:

[0009] The acquisition module is used to acquire low-quality live images from the live video stream, wherein the low-quality live images include face regions and target scene regions;

[0010] An image quality enhancement module is used to input the low-quality live image into a pre-trained image quality enhancement network to obtain a high-quality live image after image quality enhancement. The image quality enhancement network is obtained as follows: First sample image pairs are acquired, including high-quality and low-quality sample images of the same scene content, which includes faces and target scenes; a generator to be trained generates a target image based on the low-quality sample images; a discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample images; a target loss is determined based on the discrimination results of the face regions and target scene regions in the target image and the high-quality sample images, respectively; the generator and the discriminator to be trained are trained based on the target loss; and the trained generator is used as the image quality enhancement network.

[0011] The processing module is used to push the high-quality live stream image to the viewer's client or display it.

[0012] According to a third aspect of the present disclosure, a live streaming system is provided, including a broadcaster client, a live streaming server, and a viewer client.

[0013] The broadcaster client is used to capture live video streams and send them to the live streaming server;

[0014] The live streaming server is used to acquire low-quality live images from the live video stream, the low-quality live images including face regions and target scene regions; input the low-quality live images into a pre-trained image enhancement network to obtain high-quality live images after image enhancement; and push the high-quality live images after image enhancement to the viewer's client.

[0015] The image quality enhancement network is obtained as follows: A first sample image pair is acquired, comprising a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; a target image is generated based on the low-quality sample image using a generator to be trained; a discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample image; a target loss is determined based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image, respectively, by the discriminator to be trained; the generator and the discriminator to be trained are trained based on the target loss; and the trained generator is used as the image quality enhancement network.

[0016] The viewer client is used to display the received live image to the user after receiving the enhanced live image.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device including a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein when the processor executes the computer instructions, it can implement the method mentioned in the first aspect above.

[0018] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed, implement the method mentioned in the first aspect above.

[0019] In this embodiment, high-quality sample images and low-quality sample images, including faces and target scenes, can be acquired. Then, a generator to be trained is used to generate a target image based on the low-quality sample images. A discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample images. Based on the discriminator's judgment results of the face regions and target scene regions in each of the two frames, a target loss is determined. The generator and discriminator are trained based on the target loss, and the trained generator is used as an image quality enhancement network. After training the image quality enhancement network, it can be used to enhance live stream images.

[0020] This embodiment employs a discriminator with pixel-by-pixel judgment capability. The discriminator can separately judge the face region and target scene region in the target image generated by the generator, determining whether they approximate a high-quality image. This continuously encourages the generator to generate face regions and target scene regions that more closely resemble realistic high-quality images. In this way, the faces and target scenes in the final trained, enhanced high-quality image are more natural, resulting in better image enhancement without increasing processing resource consumption.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0023] Figure 1 This is a schematic diagram of a live broadcast image according to an embodiment of this disclosure.

[0024] Figure 2This is a schematic diagram of a generative adversarial network according to an embodiment of the present disclosure.

[0025] Figure 3 This is a schematic diagram of a live image processing method according to an embodiment of the present disclosure.

[0026] Figure 4 This is a schematic diagram of a training method for an image enhancement network according to an embodiment of the present disclosure.

[0027] Figure 5 This is a schematic diagram of a training method for an image enhancement network according to an embodiment of the present disclosure.

[0028] Figure 6 This is a schematic diagram of a method for generating a first sample image pair according to an embodiment of this disclosure.

[0029] Figure 7 This is a schematic diagram of the logical structure of a live image processing device according to an embodiment of the present disclosure.

[0030] Figure 8 This is a schematic diagram of the structure of a live streaming system according to an embodiment of the present disclosure.

[0031] Figure 9 This is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0036] In live streaming scenarios, to enhance the user's viewing experience and provide a higher-definition viewing experience, the live stream can be enhanced using image processing or deep learning techniques before being displayed. This enhances the details in the live stream, making them clearer and richer. Enhancing the live stream image improves the overall visual effect, thereby improving the user's viewing experience.

[0037] For live streaming scenarios such as game streaming and esports streaming, Figure 1 As shown, live streams include not only game scenes and esports events, but also facial images of gamers and commentators. Given that game and esports scenes occupy a large portion of the frame, while faces only make up a small part, current image enhancement primarily focuses on game and esports scenes. Enhancement methods suitable for these scenes are applied to the live stream, resulting in good enhancement effects. However, because game and esports scenes differ significantly from facial images in content and style, enhancement methods suitable for these scenes are not suitable for facial images. Applying scene-appropriate enhancement methods to faces results in unnatural-looking faces in the enhanced live stream, negatively impacting the overall visual quality of the live stream.

[0038] For example, current methods for enhancing live stream footage typically involve pre-training a neural network to perform the enhancement. However, current technology often only considers game scenes and their corresponding enhancement methods during training. This can lead to the neural network overfitting to a single game scene enhancement method, resulting in poor generalization ability. Applying the same enhancement method to different content can cause abnormal facial enhancement effects, negatively impacting the overall live stream quality and the user's viewing experience.

[0039] To overcome the above problems, this disclosure provides a live image processing method.

[0040] To better understand the solutions provided in the embodiments of this disclosure, a brief introduction to Generative Adversarial Networks (GANs) is given below. A GAN is a type of neural network commonly used to generate images and text. Figure 2 As shown, this neural network typically includes a generator (G) and a discriminator (D). The main idea behind this type of neural network is to train the network by having the generator and discriminator continuously compete against each other during training. Taking the generation of high-quality enhanced images using a generative adversarial network as an example, the generator can generate corresponding high-quality images based on some low-quality images. Then, the generated high-quality images and real high-quality images are input into the discriminator, which determines the authenticity of the images—that is, which is the generated high-quality image and which is the real high-quality image. The network parameters of the generator and discriminator can then be adjusted based on the determination results to train both. The goal of the training process is to make the high-quality images generated by the generator as close as possible to the real high-quality images, so that the discriminator cannot distinguish between them. At the same time, it is also to make the discriminator as accurate as possible in determining which is the generated image and which is the real image. The entire training process is a continuous competition between the generator and the discriminator, so that the generator can ultimately generate realistic, near-real high-quality images.

[0041] The live image processing method provided in this disclosure can pre-train an image quality enhancement network for enhancing the image quality of live images. Specifically, to obtain the image quality enhancement network, a generator and a discriminator to be trained can be pre-selected to construct a generative adversarial network (GAN), which is then trained. The generator in the trained GAN is then used as the image quality enhancement network.

[0042] During training, to ensure the image enhancement network achieves good and natural enhancement effects on facial regions and target scene regions such as game and sports scenes in live images, sample image pairs that simultaneously include facial regions and target scene regions can be used to train the generative adversarial network. These sample image pairs include both high-quality and low-quality sample images corresponding to the same scene content.

[0043] The low-quality sample image can then be input into the generator of the generative adversarial network (GAN) to generate an enhanced target image. Both the target image and the high-quality sample image can then be input into the discriminator. To enable the discriminator to simultaneously determine the authenticity of face regions and target scene regions in the image, the discriminator in this embodiment can be a network capable of pixel-by-pixel image evaluation; that is, the discriminator can determine whether each pixel in the image is close to the high-quality sample image. To accurately determine different types of image content, the discriminator can learn semantic features of the image, thereby determining whether face regions and target scene regions are close to the high-quality sample image. Then, based on the discriminator's determination results for face regions in the generated target image and high-quality sample image, and the discriminator's determination results for target scene regions in the generated target image and high-quality sample image, a target loss can be determined. The network parameters of the generator and discriminator are continuously adjusted based on the target loss to train both, and the trained generator is used as the image enhancement network.

[0044] After training the image enhancement network, low-quality live images can be obtained from the live video stream. Then, the image enhancement network can be used to enhance the image quality of the low-quality live images to obtain high-quality live images.

[0045] In previous technologies, discriminators could only determine whether an entire image resembled a high-quality image. Therefore, the trained generator could only effectively enhance scenes such as games and events that constituted a large portion of the image. This embodiment employs a discriminator with pixel-by-pixel judgment capabilities. This discriminator can separately judge the face region and the target scene region in the target image generated by the generator, determining whether they resemble a high-quality image. This continuously encourages the generator to generate face regions and target scene regions that more closely resemble realistic high-quality images. In this way, the faces and target scenes in the final enhanced high-quality image generated by the trained generator appear more natural, resulting in better image enhancement.

[0046] In some scenarios, the live image processing method provided in this disclosure can be executed by the live streaming server. That is, after the broadcaster's client acquires the live video stream, it sends it to the live streaming server, which then enhances the image quality before pushing it to the viewer's client. In some scenarios, if the device on which the broadcaster's client resides has strong computing power, the method can also be executed by the broadcaster's client. That is, the broadcaster's client enhances the live image before pushing it to the server. Alternatively, in some scenarios, the method can also be executed by the viewer's client. After acquiring the live video stream, the viewer's client enhances it before displaying it. For example, the viewer's client can set up an image quality enhancement function; after enabling this function, the aforementioned image quality enhancement operation is performed before displaying the live image.

[0047] The image enhancement processing in this embodiment refers to enhancing a low-quality image to obtain a high-quality image with the same or even higher resolution. For example, the edges in the image can be sharpened or enhanced to make the details in the image clearer and richer.

[0048] It should be noted that the high-quality images and low-quality images in the embodiments of this disclosure are relative. High-quality images have better image quality than low-quality images, that is, higher resolution, higher clarity, or richer details, etc.

[0049] The target scenario in this disclosure can be a game scenario, a competition scenario, or other scenarios set based on live streaming requirements. These target scenarios and face scenarios have certain differences, and the image enhancement methods applicable to them are also different.

[0050] For example, in some embodiments, the live stream image may be a game live stream image or a sports event live stream image, and the target scene may be a game scene or a sports event scene.

[0051] The following combination Figure 3 The live image processing method provided in this disclosure includes the following steps:

[0052] S302. Obtain a low-quality live video image from the live video stream, wherein the low-quality live video image includes a face region and a target scene region;

[0053] In step S302, a low-quality live stream image can be obtained from the live video stream. This low-quality live stream image is relative to the high-quality live stream image and refers to an image that requires image quality enhancement processing. The live stream image can be a game live stream image, an event live stream image, etc., and typically includes a target scene area and a face area. For example, the target scene could be a game scene, and the face could be the face of a game player.

[0054] S304. Input the low-quality live image into a pre-trained image quality enhancement network to obtain an enhanced live image. The image quality enhancement network is obtained in the following way: acquiring a first sample image pair, which includes a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; generating a target image based on the low-quality sample image using a generator to be trained; performing pixel-by-pixel judgment on the target image and the high-quality sample image using a discriminator to be trained; determining a target loss based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image respectively, and training the generator and the discriminator to be trained based on the target loss; and using the trained generator as the image quality enhancement network.

[0055] In step S304, considering that some broadcasters' shooting equipment has poor performance and the captured live images have poor image quality, the low-quality live images can be enhanced. For example, the resolution of the live images can be increased, the clarity of details in the live images can be improved, and so on, to obtain high-quality live images with better image quality.

[0056] To enhance the image quality of low-quality live stream images, an image enhancement network can be pre-trained. The low-quality live stream image can then be input into the image enhancement network, which outputs a high-quality enhanced live stream image.

[0057] In order for the image enhancement network to achieve good enhancement effects on both facial and target scene regions in live images, the image enhancement network can be designed according to... Figure 4 It was trained in the manner shown.

[0058] First, a large number of sample image pairs (hereinafter referred to as the first sample image pairs) can be obtained. Each first sample image pair includes high-quality sample images and low-quality sample images for the same scene content. The images in the first sample image pairs all include faces and target scenes.

[0059] To obtain an image enhancement network, a generative adversarial network can be trained using the first sample image, and then the generator of the trained generative adversarial network can be used as the image enhancement network.

[0060] A generative adversarial network (GAN) to be trained can be constructed. The generator of the GAN can be a network with image enhancement capabilities, such as an SPSR network generator. To ensure the trained generator achieves good image enhancement for both faces and target scenes, a discriminator with pixel-wise judgment capabilities can be selected to adversarially challenge the generator. This discriminator can learn semantic features of faces or target scenes and judge the realism of faces or target scenes based on these learned features. This ensures that the faces and target scenes in the enhanced images generated by the generator are close to high-quality images.

[0061] After constructing the generative adversarial network (GAN), a low-quality sample image from the first sample image pair can be input into the generator. The generator can then generate an enhanced, high-quality target image based on this low-quality image. The target image and the high-quality sample image can then be input into the discriminator. The discriminator performs pixel-by-pixel judgment on the target image and the high-quality sample image, determining whether each pixel represents the real high-quality sample image or the generated target image. Since the location of the face in the target image and the high-quality sample image is known, the training objective is to enable the generator to generate as realistic high-quality face regions and target scene regions as possible, so that the discriminator cannot distinguish between the high-quality sample image and the generated target image, while simultaneously enabling the discriminator to accurately identify the high-quality sample image and the generated target image as much as possible. Therefore, based on the discriminator's discrimination results for face regions in the target image and high-quality sample images (i.e., which is the high-quality sample image and which is the generated target image), and the discriminator's discrimination results for target scene regions in the target image and high-quality sample images (i.e., which is the high-quality sample image and which is the generated target image), the target loss can be determined. Then, the network parameters of the generator and discriminator can be continuously adjusted using the target loss until the network parameters converge, resulting in a trained generator and discriminator. The trained generator can then be used as the image enhancement network.

[0062] S306. Send the high-quality live stream image to the viewer's client or display it.

[0063] In step S306, after using the image enhancement network to enhance the image quality of the low-quality live broadcast image to obtain a high-quality live broadcast image, the high-quality live broadcast image can be sent to the viewer's client or displayed directly to the user.

[0064] Generally, to reduce data transmission volume, live video streams typically use YUV format images. In some embodiments, if the low-quality live images in the live video stream are in YUV format, after acquiring the live video stream, it can first be decoded to obtain low-quality YUV format live images. Then, the low-quality YUV format live images can be converted to RGB format low-quality live images, and finally, the converted RGB format live images can be input into the image enhancement network.

[0065] In some embodiments, considering that image quality enhancement typically requires converting low-resolution images into high-resolution images to achieve better enhancement results, a network with image super-resolution reconstruction capabilities can be selected as the generator to be trained. This network can reconstruct low-resolution images into high-resolution images. For example, a generator from an SPSR network can be used as the generator to be trained; of course, other networks with similar capabilities can also be selected.

[0066] In some embodiments, considering that the discriminator needs to have pixel-by-pixel judgment capabilities, that is, the discriminator needs to determine whether each pixel in the image is close to a high-quality image. Meanwhile, since the content of the face region and the target scene region differs significantly, in order to accurately distinguish between these two types of images, the discriminator can learn the semantic features of the game scene and the semantic features of the face, and make judgments based on these semantic features. Taking these factors into account, the U-Net network can be selected as the discriminator.

[0067] In some embodiments, during the training of a generative adversarial network (GAN), to obtain a generator and discriminator with better performance and higher accuracy, in constructing the target loss, in addition to using the common loss from the GAN training process (e.g., the loss determined based on the discriminator's discrimination results between generated and real images), a consistency constraint can be further introduced as a loss. For example, such as... Figure 5 As shown, a large number of second sample image pairs can be obtained. Each pair includes a high-quality face image and a low-quality face image corresponding to the same face. The generative adversarial network (GAN) is then trained using these second sample image pairs to obtain a discriminator with good face region discrimination ability, hereinafter referred to as the first discriminator. Similarly, a large number of third sample image pairs can be obtained. Each pair includes a high-quality target scene image and a low-quality target scene image corresponding to the same target scene. The GAN is then trained using these third sample image pairs to obtain a discriminator with good target scene discrimination ability, hereinafter referred to as the second discriminator.

[0068] When determining the target loss, a first loss can be determined based on the discrimination results of the discriminator to be trained on the face regions in the target image and the high-quality sample image, and the discrimination results of the discriminator on the target scene regions in the target image and the high-quality sample image, respectively. Simultaneously, a second loss can be determined based on the similarity between the discriminator's discrimination results on the face regions in the target image and the first discriminator's discrimination results, and the similarity between the discriminator's discrimination results on the target scene regions in the target image and the second discriminator's discrimination results, respectively. Then, the target loss is determined based on the first and second losses. For example, the sum of the two losses, or their weighted average, can be used as the target loss. The network parameters of the generator and discriminator to be trained can then be continuously adjusted based on the target loss.

[0069] Since the first discriminator is a pre-trained network with strong facial recognition capabilities, the discriminator to be trained should identify facial regions in the target image as closely as possible to the first discriminator's results. Similarly, the discriminator to be trained should identify target scene regions in the target image as closely as possible to the second discriminator's results. In other words, the discriminator's results should be as consistent as possible with the first and second discriminators. This can then be used as a constraint to train the generative adversarial network, resulting in better performance for both the generator and discriminator after training.

[0070] The first and second discriminators can be pre-trained. Of course, they can also be trained simultaneously with the discriminator to be trained; this disclosure does not impose any restrictions.

[0071] In some embodiments, to accelerate the convergence speed during the training of the generative adversarial network (GAN), the generator to be trained can also be a pre-trained network. For example, considering that the target scene usually occupies a large area in a live image, while the face only occupies a small part, meaning the generator mainly enhances the target scene, a large number of third-sample image pairs can be obtained first. Each third-sample image pair includes a high-quality target scene image and a low-quality target scene image corresponding to the same target scene. Then, the GAN can be trained using the third-sample image pairs to obtain a generator with good enhancement effect on the target scene, and this generator can then be used as the generator to be trained.

[0072] In some embodiments, a low-quality face image can be obtained by performing a degradation operation on a high-quality face image. Similarly, a low-quality target scene image can also be obtained by performing a degradation operation on a high-quality target scene image. The degradation operation includes one or more operations such as adding random noise, blurring, and compression.

[0073] In some embodiments, such as Figure 6 As shown, to obtain a large number of first sample image pairs, high-quality and low-quality face images corresponding to the same face can be acquired, as well as high-quality and low-quality target scene images corresponding to the same target scene. Then, a face region can be cropped from the high-quality face image, and this cropped face region can replace any image region in the high-quality target scene image to obtain a high-quality sample image. Similarly, a face region can be cropped from the low-quality face image, and this cropped face region can replace the region in the low-quality target scene image that is located at the same pixel position as any of the aforementioned image regions to obtain a low-quality sample image corresponding to the high-quality sample image.

[0074] In some embodiments, to diversify the first sample image pairs, multiple sets of first sample image pairs can be obtained by randomly adding cropped face regions to different regions of a high-quality target scene image (or a low-quality target scene). By placing face regions in different positions within the image, the diversity of the sample images can be increased, thereby improving the performance of the trained image enhancement network.

[0075] To further illustrate the live image processing method provided in this disclosure, a specific embodiment will be used as an example for explanation below.

[0076] In game live streaming scenarios, the live image includes both the game scene and the faces of the players. When enhancing the image quality, considering the large area occupied by the game scene, an enhancement method suitable for the game scene is usually chosen, resulting in unnatural effects after face enhancement. To overcome this problem, this embodiment provides a solution for enhancing the image quality of game live streaming images, specifically including the following steps:

[0077] (1) Constructing a generative adversarial network

[0078] Generative adversarial networks (GANs) can be constructed. The generator in the GAN can be a generator from an SPSR network, and the discriminator in the GAN can be a U-Net network, in order to achieve pixel-level discrimination.

[0079] (2) Construction of face image set and game scene image set

[0080] Paired sample image sets can be constructed, creating a high-quality image set and a low-quality image set for game scenes and face scenes respectively, with a one-to-one correspondence between images in the high-quality and low-quality image sets. High-quality images are high-resolution images obtained through capturing, collecting, cleaning, and filtering, while low-quality images are low-resolution images obtained after various data degradation operations. Degradation operations include, but are not limited to, random noise, blurring, compression, etc.

[0081] (3) Pre-training of generator and discriminator

[0082] By pre-training a generative adversarial network (GAN) using pairs of high-quality and low-quality game scene images, a superior game image enhancement generator, Ggame, and a superior game image enhancement discriminator, Dgame, were obtained. The generator Ggame is responsible for enhancing low-quality, low-resolution images into high-quality, high-resolution images. The discriminator network Dgame is responsible for judging the output images of the generator, determining whether they approximate real high-quality images. The game image enhancement discriminator Dgame primarily learns the semantic features of game scenes to determine whether the input image approximates a real high-quality game scene image.

[0083] Similarly, the constructed generative adversarial network can be pre-trained using pairs of high-quality and low-quality face images, resulting in a superior face enhancement generator Gface and a superior game image enhancement discriminator Dface. The game image enhancement discriminator Dface primarily learns the semantic features of the game scene to determine whether the input image closely resembles a real, high-quality face image.

[0084] (4) Construction of a sample image set containing faces and game scenes

[0085] Get paired game scene images {h g ,l g}, where h g Indicates high-quality game graphics, l g This indicates low-quality game images, while simultaneously acquiring paired face images {h f ,l f}, where h f Indicates a high-quality face image, l f This represents a low-quality face image. Let R represent a randomly selected contiguous region on the game image. Replace this region in a high-quality game image with a high-quality face image, and replace this region in a low-quality game image with a low-quality face image, resulting in sample image pairs {h}. n ,l n}, where hn Indicates a high-quality sample image, l n This represents a low-quality sample image. To obtain more diverse sample image pairs, an R region can be randomly selected from the image and replaced with a human face.

[0086] (5) Training of the image enhancement network

[0087] Considering that a large area of ​​the game live stream images is the game scene, the generator mainly enhances the game scene. In order to speed up the convergence speed during the network training process, the pre-trained Ggame in step (3) can be used as the generator to be trained. Then, the l in the sample image pair can be used. n The input is fed into the generator to be trained, which generates an enhanced target image. Then, the target image and h can be compared... n The input is fed into the discriminator (U-Net network) to be trained. Simultaneously, the target image is fed into the pre-trained game image enhancement discriminator Dgame and face image enhancement discriminator Dface, respectively. Based on the discriminator to be trained, the target image and h are analyzed. n The first loss is determined based on the results of identifying the face region and the game scene region in the target image. Simultaneously, a second loss is determined based on the similarity between the discriminator's (before training) results for identifying the face region and Dface's results, and the similarity between the discriminator's (before training) results for identifying the game scene region and Dgame's results. A target loss is obtained based on the first and second losses. This target loss is then used to train both the generator and the discriminator, and the trained generator is used as the image enhancement network.

[0088] (6) Use the trained image enhancement network to enhance the image quality of game live stream images.

[0089] The system acquires the game live stream video stream, decodes it to obtain a low-quality game live stream image in YUV format, converts the low-quality YUV game live stream image to a low-quality RGB format game live stream image, and inputs it into a trained image enhancement network to obtain a high-quality RGB format game live stream image after image enhancement. After being converted back to YUV format game live stream image, it is pushed to the viewer's client for display of the image enhancement game live stream image.

[0090] This embodiment trains an image enhancement network that can simultaneously improve the image quality of game scenes and facial images, thereby enhancing the overall image quality of game live streaming scenes and improving the user's viewing experience. Furthermore, it does not increase computing resources or processing time during deployment.

[0091] It is easy to understand that the solutions described in the above embodiments can be combined when there is no conflict, and not all of them will be listed in this disclosure.

[0092] Accordingly, this disclosure also provides a live streaming image processing apparatus, such as... Figure 7 As shown, the device 70 includes:

[0093] The acquisition module 71 is used to acquire low-quality live images from the live video stream, wherein the low-quality live images include a face region and a target scene region;

[0094] Image quality enhancement module 72 is used to input the low-quality live image into a pre-trained image quality enhancement network to obtain a high-quality live image after image quality enhancement. The image quality enhancement network is obtained based on the following method: acquiring a first sample image pair, the first sample image pair including a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; generating a target image based on the low-quality sample image using a generator to be trained; performing pixel-by-pixel judgment on the target image and the high-quality sample image using a discriminator to be trained; determining a target loss based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image respectively by the discriminator to be trained; training the generator and the discriminator to be trained based on the target loss; and using the trained generator as the image quality enhancement network.

[0095] The processing module 73 is used to push the high-quality live image to the viewer's client or display it.

[0096] The specific steps of the above-mentioned device in performing the live image processing method can be referred to the description in the above method embodiments, and will not be repeated here.

[0097] Furthermore, this disclosure also provides a live streaming system, such as... Figure 8 As shown, the live streaming system includes a broadcaster client, a live streaming server, and a viewer client.

[0098] The broadcaster client is used to capture live video streams and send them to the live streaming server;

[0099] The live streaming server is used to acquire low-quality live images from the live video stream, the low-quality live images including face regions and target scene regions; input the low-quality live images into a pre-trained image enhancement network to obtain high-quality live images after image enhancement; and push the high-quality live images after image enhancement to the viewer's client.

[0100] The image quality enhancement network is obtained as follows: A first sample image pair is acquired, comprising a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; a target image is generated based on the low-quality sample image using a generator to be trained; a discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample image; a target loss is determined based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image, respectively, by the discriminator to be trained; the generator and the discriminator to be trained are trained based on the target loss; and the trained generator is used as the image quality enhancement network.

[0101] The viewer client is used to display the received live image to the user after receiving the enhanced live image.

[0102] The specific steps of the live streaming system in performing the live streaming image processing method can be found in the description of the above method embodiments, and will not be repeated here.

[0103] Furthermore, embodiments of this disclosure also provide a device, such as... Figure 9 As shown, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 that can be executed by the processor 91. When the processor 91 executes the computer instructions, it implements the method described in any of the above embodiments.

[0104] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0105] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0106] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0107] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0109] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A live streaming image processing method, characterized in that, The method includes: Obtain low-quality live images from the live video stream, wherein the low-quality live images include face regions and target scene regions; The low-quality live stream image is input into a pre-trained image quality enhancement network to obtain a high-quality live stream image after image quality enhancement. The image quality enhancement network is obtained as follows: A first sample image pair is acquired, comprising a high-quality sample image and a low-quality sample image for the same scene content, including a face and a target scene; a generator to be trained generates a target image based on the low-quality sample image; a discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample image; based on the discrimination results of the discriminator on the face region and the target scene region in the target image and the high-quality sample image, a target loss is determined; the generator and the discriminator to be trained are trained based on the target loss; and the trained generator is used as the image quality enhancement network. The discriminator is a discriminator capable of pixel-by-pixel image judgment and can learn semantic features of faces or target scenes, determining the authenticity of faces or target scenes based on the learned semantic features. The high-quality live stream image is pushed to the viewer's client or displayed.

2. The method according to claim 1, characterized in that, The low-quality live stream image is a game live stream image or an event live stream image; the target scene is a game scene or an event scene; and / or The low-quality live images in the live video stream are in YUV format. Inputting the low-quality live images into a pre-trained image quality enhancement network includes: converting the low-quality live images in YUV format into RGB format images; and inputting the converted RGB format images into the image quality enhancement network.

3. The method according to claim 1, characterized in that, The generator to be trained is a network capable of image super-resolution reconstruction, and / or The discriminator to be trained is a U-Net network.

4. The method according to any one of claims 1-3, characterized in that, The target loss is determined based on the discrimination results of the face regions in the target image and the high-quality sample images, and the discrimination results of the target scene regions in the target image and the high-quality sample images, respectively, by the discriminator to be trained. The first loss is determined based on the discrimination results of the face regions in the target image and the high-quality sample image, as well as the discrimination results of the target scene regions in the target image and the high-quality sample image, respectively, by the discriminator to be trained. The second loss is determined based on the similarity between the discrimination results of the discriminator to be trained and the first discriminator on the face region in the target image, and the similarity between the discrimination results of the discriminator to be trained and the second discriminator on the target scene region in the target image. Wherein, the first discriminator is a discriminator in a generative adversarial network trained using the second sample image pair; the second discriminator is a discriminator in a generative adversarial network trained using the third sample image pair; the second sample image pair includes a high-quality face image and a low-quality face image corresponding to the same face, and the third sample image pair includes a high-quality target scene image and a low-quality target scene image corresponding to the same target scene; The target loss is determined based on the first loss and the second loss.

5. The method according to claim 4, characterized in that, The low-quality face image is obtained by performing a degradation operation on the high-quality face image, and the low-quality target scene image is obtained by performing a degradation operation on the high-quality target scene image; the degradation operation includes one or more of the following: adding random noise, blurring, and compression.

6. The method according to claim 1, characterized in that, The generator to be trained is a generator in a generative adversarial network trained using third sample images. The third sample image pairs include high-quality and low-quality target scene images corresponding to the same target scene; and / or The first sample image pair is obtained based on the following method: acquiring a high-quality face image and a low-quality face image corresponding to the same face, and acquiring a high-quality target scene image and a low-quality target scene image corresponding to the same target scene; cropping a face region from the high-quality face image, and replacing any region in the high-quality target scene image with the cropped face region to obtain the high-quality sample image; cropping a face region from the low-quality face image, and replacing a region in the low-quality target scene image that is located at the same pixel position as the any region with the cropped face region to obtain the low-quality sample image.

7. The method according to claim 6, characterized in that, The first sample image pair includes multiple groups, and the face regions in the high-quality sample images and the low-quality sample images in different groups of the first sample image pair are located at different positions in the image.

8. A live streaming image processing device, characterized in that, The device includes: The acquisition module is used to acquire low-quality live images from the live video stream, wherein the low-quality live images include face regions and target scene regions; The image quality enhancement module is used to input the low-quality live image into a pre-trained image quality enhancement network to obtain a high-quality live image after image quality enhancement. The image quality enhancement network is obtained as follows: First sample image pairs are acquired, including high-quality and low-quality sample images of the same scene content, which includes faces and target scenes; a generator to be trained generates a target image based on the low-quality sample images; a discriminator to be trained performs pixel-by-pixel judgment on the target image and the high-quality sample images, and the discriminator further refines the image quality based on the low-quality sample images. The target loss is determined by the discrimination results of the face regions in the target image and the high-quality sample images, as well as the discrimination results of the target scene regions. The generator and the discriminator to be trained are trained based on the target loss, and the trained generator is used as the image quality enhancement network. The discriminator is a discriminator with the function of performing pixel-by-pixel discrimination of the image, and the discriminator can learn the semantic features of the face or the target scene, and judge the authenticity of the face or the target scene based on the learned semantic features. The processing module is used to push the high-quality live image to the viewer's client or display it.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein the processor executes the computer instructions to implement the method as described in any one of claims 1-7.

10. A live streaming system, characterized in that, This includes the broadcaster's client, the live streaming server, and the viewer's client. The broadcaster client is used to capture live video streams and send them to the live streaming server; The live streaming server is used to acquire low-quality live streaming images from the live video stream, the low-quality live streaming images including face regions and target scene regions; The low-quality live image is input into a pre-trained image enhancement network to obtain a high-quality live image after image enhancement, and the image-enhanced live image is pushed to the viewer's client. The image enhancement network is obtained as follows: A first sample image pair is acquired, comprising a high-quality sample image and a low-quality sample image for the same scene content, the scene content including a face and a target scene; a generator to be trained is used to generate a target image based on the low-quality sample image; a discriminator to be trained is used to perform pixel-by-pixel judgment on the target image and the high-quality sample image; a target loss is determined based on the discrimination results of the face region and the target scene region in the target image and the high-quality sample image, respectively, by the discriminator to be trained; the generator to be trained and the discriminator to be trained are trained based on the target loss; and the trained generator is used as the image enhancement network. The discriminator is a discriminator capable of pixel-by-pixel image judgment, and can learn semantic features of the face or target scene, determining the authenticity of the face or target scene based on the learned semantic features. The viewer client is used to display the received live image to the user after receiving the enhanced image.

Images