Method for improving quality of image media

Abstract

The invention relates to a method for improving the quality of an image media. The quality of an image stream is improved by using an artificial intelligence enhancement module which is built in a client device and is trained in advance. The AI enhancement module is pre-trained in the server with a neural network to analyze differences between the original image generated by the server and the compressed image. Wherein the AI enhancement module improves the quality of the compressed image by using a mathematical equation obtained by analyzing the difference between the original image generated by the server and the compressed image. Therefore, the improved image is visually closer to the original image than the compressed image.

Description

[0001] This application is a divisional application of an invention patent application with an application date of June 19, 2019, application number 201910532952.0, and invention name “Method for Improving the Quality of Image Media”. Technical Field

[0002] The present invention relates to a method for improving the quality of image media, and more particularly to a method for improving the quality of image media by using a pre-trained artificial intelligence (AI) enhancement module built into a client device. Background Art

[0003] In recent years, online games have become increasingly popular around the world. With the development of cloud-based computing systems and technologies, a technology has been developed that allows servers to stream game content to provide services.

[0004] Traditionally, such cloud online game services are provided by having the server perform almost all the calculations. In other words, when providing online game services, a specific application is executed in the server to generate a virtual 3D environment containing many 3D (Three Dimensional) objects, including 3D objects that can be controlled or moved by players. Then, based on the player's control results, the server renders these 3D objects and the virtual 3D environment into a 2D (Two Dimensional) game screen for display on the player's device. Next, the server encodes and compresses the rendered image into a 2D image stream and transmits it to the player's device. The player's device only needs to decode the received 2D image stream and "play" it without performing 3D rendering calculations. However, there are still several issues that need to be noted in such cloud online game services, such as the high load of the server when providing 3D rendering programs for a large number of players at the same time, the reduction in image quality caused by encoding compression and streaming programs, and the large amount of communication bandwidth consumed by transmitting 2D image streams over the network.

[0005] One existing method to solve the problem of image quality degradation is to increase the resolution of the original image generated by the game application on the server side and increase the bitrate when transmitting the image, that is, to reduce the compression ratio when the server encodes the original image into a 2D image stream. However, it is obvious that doing so will cause the server load and bandwidth consumption to increase significantly due to the high resolution and high transmission bitrate of the image. For example, assuming that the frame rate and the encoding compression ratio are both constant, when the resolution of the original image generated by the game application on the server side is increased from 720p to 1080p, the server's computing load and the required network transmission bitrate will increase by 2.25 times. In contrast, if an attempt is made to reduce the server load or network bandwidth consumption, the image quality of the game image will be sacrificed. Therefore, it becomes a dilemma to simultaneously obtain perfect image quality and economical bandwidth consumption.

[0006] Another way to solve this problem is to reduce the resolution of the original image generated by the game application on the server side, or to encode the original image into a 2D image stream at a higher compression rate, or both. In this way, the bandwidth consumption of transmitting the 2D image stream through the network is reduced, although the picture quality of the game image will also be sacrificed. At the same time, an image enhancement technology is used on the client device. Once the 2D image stream is received, the client device decodes the image stream and uses the image enhancement technology to improve the visual effect of the image. Histogram equalization (HE) is one of the most commonly used methods for improving image contrast because of its simplicity and efficiency. However, HE may cause excessive contrast enhancement and feature loss problems, resulting in an unnatural appearance and loss of details in the processed image. In addition, not only HE but all other image enhancement technologies known in the art encounter the same dilemma, that is, they all try to use the same set of algorithms to process various images with completely different picture contents, which is not feasible. Take the cloud online game service as an example. The picture content of the original image generated by the server will change significantly due to changes in the game scene. For example, a city game scene may make the original image of the game contain many simple and clear appearance outlines and different but roughly the same color. A dark cave game scene will make the original image of the game full of monotonous and low-tone and low-chroma colors, but with irregular but inconspicuous landscape outlines. A lush garden scene will make the original image of the game contain many lively and colorful objects with detailed and complex outlines. Undoubtedly, no existing enhancement technology can provide good image enhancement effects for various different scenes with completely different picture contents.

[0007] In addition, another shortcoming of these existing image enhancement technologies is that although the mathematical expressions of these existing image enhancement technologies can improve the image effects such as contrast, sharpness, saturation, etc., these expressions and their parameters are completely unrelated to the original image generated by the server. Therefore, the enhancement process of these existing image enhancement technologies will never make the enhanced image visually closer to its corresponding original image, and therefore the game players on the client side cannot fully enjoy the image effects of the original image generated by the game application on the server side. Summary of the invention

[0008] Therefore, the main purpose of the present invention is to provide a method for improving the quality of image media. This method uses a pre-trained artificial intelligence (AI) enhancement module built into a client device to improve the quality of the image stream. The AI ​​enhancement module is pre-trained in a server with a neural network to analyze the difference between the original image generated by the server and the compressed image. Among them, the AI ​​enhancement module uses a mathematical formula obtained by analyzing the difference between the original image generated by the server and the compressed image to improve the quality of the compressed image. Thereby, the enhanced image will be visually closer to the original image than the compressed image.

[0009] Another object of the present invention is to provide a method for improving the quality of image media, which can reduce the computing load of the server, improve the quality of the image played on the client device, and save the consumption of the transmission bandwidth between the server and the client device.

[0010] Another object of the present invention is to provide a method for improving the quality of image media, wherein the original image generated by the server belongs to several different scenes (scene-mode), and the original image images of different scenes have different picture contents. The decompressed images corresponding to the original images of different scenes are image enhanced by the same AI enhancement module using different sets of weighted parameters corresponding to the scene. Therefore, the method of the present invention can provide good and stable image enhancement effects for various original images with completely different picture contents.

[0011] To achieve the above-mentioned purpose, an embodiment of the method for improving the quality of image media of the present invention includes:

[0012] Step (A): executing a first application in a server; the first application generates a plurality of original images according to at least one instruction; the plurality of original images are encoded and compressed by an encoder in the server to generate a plurality of encoded images;

[0013] Step (B): executing a second application in a client device remote from the server; the second application is related to and cooperates with the first application; thereby, the client device can be operated by a user and generate and send the command to the server;

[0014] Step (C): the client device transmits the command to the server via a network, and then receives the plurality of encoded images generated by the server and corresponding to the command via the network;

[0015] Step (D): The client device decodes the plurality of encoded images into a plurality of decoded images, and uses an AI enhancement module to enhance the quality of the plurality of decoded images to generate a plurality of enhanced images; wherein the AI ​​enhancement module processes the plurality of decoded images by analyzing at least one mathematical expression obtained in advance by differences between the plurality of decoded images and the corresponding plurality of original images; thereby, the plurality of enhanced images obtained will be visually closer to the plurality of original images than the plurality of decoded images; and

[0016] Step (E): The client device outputs the enhanced image to a screen as an output image to be played.

[0017] In one embodiment, the at least one mathematical expression of the AI ​​enhancement module is defined by a training procedure executed by an artificial neural network module in the server; the training procedure includes:

[0018] Step (a): executing the first application in a training mode to generate a plurality of training original images;

[0019] Step (b): encoding the plurality of original training images into a plurality of coded training images by the encoder;

[0020] Step (c): decoding the plurality of training coded images into a plurality of training decoded images by a training decoder in the server;

[0021] Step (d): the artificial neural network module receives the plurality of training decoded images and uses at least one training mathematical expression to process the training decoded images one by one to generate a plurality of training output images; the at least one training mathematical expression comprises a plurality of training weighting parameters;

[0022] Step (e): using a comparison and training module to compare the differences between the training output image and the corresponding training original image one by one, and adjusting the plurality of training weighting parameters of the at least one training mathematical expression accordingly; the plurality of training weighting parameters are adjusted to minimize the differences between the training output image and the corresponding training original image; each time the plurality of training weighting parameters are adjusted, the plurality of adjusted training weighting parameters are fed back to the at least one training mathematical expression for processing the next training decoded image in step (d);

[0023] Among them, after performing a predetermined number of comparisons of the training output images with the corresponding training original images, and a predetermined number of adjustments to the training weighting parameters, the multiple training weighting parameters finally obtained will be applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0024] In one embodiment, when the training decoded image and the training output image have the same color format, the artificial neural network module is a residual network module; and in step (d), each of the training output images is the sum of the corresponding training decoded image and the output of the residual network module processing the training decoded image.

[0025] In one embodiment, in step (e), the comparison and training module uses a discriminator to compare the difference between the training output image and the corresponding training original image, and converges the generative adversarial networks loss (GAN loss) and adjusts the training weighting parameters accordingly.

[0026] In one embodiment, the discriminator of the training and comparison module is trained in the following manner:

[0027] The training original image includes n channels, wherein n is a positive integer greater than 2; the training decoded image includes m channels, wherein m is a positive integer greater than 2;

[0028] In step (d), the artificial neural network module processes the m-channel training decoded image and generates an n-channel training output image; the n-channel training output image and the corresponding m-channel training decoded image are combined to generate a plurality of simulated false samples with m+n channels; and the n-channel training original image and the corresponding m-channel training decoded image are combined to generate a plurality of simulated true samples with m+n channels;

[0029] In step (e), the simulated false samples of the m+n channel and the simulated true samples of the m+n channel are fed back to the discriminator of the comparison and training module for training the discriminator to detect and distinguish the simulated false samples from the simulated true samples.

[0030] In one embodiment, the color format of the training original image is YUV420, and the color format of the training decoded image is RGB or YUV420;

[0031] In step (d), the artificial neural network module includes a first neural network and a second neural network; the second neural network is a convolutional neural network (CNN); the first neural network receives and processes the plurality of training decoded images to generate a plurality of first output images X2 having the same encoding format as the training original image; the second neural network receives and processes the plurality of first output images X2 to generate a plurality of second output images; the first output images X2 and the second output images are added to generate the training output images;

[0032] In step (e), the comparison and training module includes a first comparator and a second comparator; the first comparator compares the difference between the first output image X2 and the corresponding training original image for training the first neural network; the second comparator compares the difference between the training output image and the corresponding training original image for training the second neural network.

[0033] In one embodiment, the step of the first neural network receiving and processing the training decoded image in YUV420 color format comprises:

[0034] Extracting Y-part data from the training decoded image, and processing the Y-part data of the training decoded image by the first neural network with a standard size to generate Y-part output data;

[0035] Extracting UV-part data from the training decoded image, and processing the UV-part data of the training decoded image by the first neural network with two times magnification to generate UV-part output data with N channels;

[0036] The Y-part output data and the UV-part output data are added to generate the training output image.

[0037] In one embodiment, the step of the first neural network receiving and processing the training decoded image in YUV420 color format comprises:

[0038] The training decoded image includes N channels, and N is a positive integer greater than 2;

[0039] Extracting Y-part data from the training decoded image;

[0040] Extracting UV-part data from the training decoded image, and processing the UV-part data of the training decoded image using the first neural network with two-fold magnification to generate UV-part output data with N-1 channels;

[0041] The Y-part data and the UV-part data are processed by a concatenation function Concat (concatenates) to generate the training output image.

[0042] In one embodiment, the at least one mathematical expression of the AI ​​enhancement module of the client device includes a plurality of weighting parameters; the weighting parameters are associated with the difference between the decoded image and the corresponding original image, and are defined by a training program executed by an artificial neural network module in the server; wherein the weighting parameters are one of the following: pre-stored in the client device and downloaded from the server to the client device when the client device executes the second application.

[0043] In one embodiment, the plurality of original images generated by the first application may be divided into a plurality of groups of scenes, each of which includes a plurality of the original images;

[0044] The plurality of weighting parameters used by the AI ​​enhancement module in the client device are also divided into a plurality of groups, each group comprising a plurality of the weighting parameters and corresponding to one of the scenarios;

[0045] The multiple decoded images corresponding to the original images of different scenes will be subjected to image enhancement processing by the same AI enhancement module using the group of weighted parameters corresponding to the scene among the multiple different groups of weighted parameters;

[0046] When all of the plurality of different sets of weighting parameters are pre-stored in the client device, each time the scene changes, the set of weighting parameters corresponding to the changed new scene will be applied to the AI ​​enhancement module to generate the enhanced image;

[0047] Among them, when the multiple different groups of weighted parameters are all stored on the server side, whenever the scene changes, the group of weighted parameters corresponding to the changed new scene will be downloaded from the server to the client device, and then applied to the AI ​​enhancement module to generate the enhanced image.

[0048] In one embodiment, in the training mode, the plurality of original training images are high-quality images; the plurality of original training images of high quality are images that meet one or more of the following properties: having a higher resolution (Resolution) such as 4K resolution or higher, having a larger field of view (FOV) such as 150 degrees or higher, having a higher frame rate (FPS) such as 60FPS or higher, having binocular stereo information (stereo vision), having visual depth information, having additional color information such as infrared or ultraviolet, having better audio such as multi-channel, high sampling frequency (44khz or higher) or high resolution (audio bit depth 24bit or higher);

[0049] The following step is further added between step (a) and step (b): image degradation processing; using a degradation module to degrade the plurality of training original image images to reduce their image quality, so as to obtain degraded training original image images that meet one or more of the following properties: reducing the resolution by sampling (reducing to 720p, 1080i or lower), reducing the field of view by cropping (reducing to 120 degrees or lower, or cropping the original image with an aspect ratio of 16:9 to 4:3), reducing the frame rate by discarding frames (reducing to 30FPS or lower), extracting binocular stereo information into a single frame by capturing, discarding visual depth information, discarding visual depth information, converting audio into stereo dual channels, reducing the sampling frequency (22.1khz or lower), reducing the resolution (16bit or lower), adding noise (such as Gaussian or Laplacian noise), blurring (such as Gaussian blurring);

[0050] In step (b), the plurality of degraded original training images are encoded and compressed by the encoder to form a plurality of the encoded training images.

[0051] In one embodiment, in step (d), in the training mode, the artificial neural network module receives several frames of the training decoded image and generates and outputs at least one frame of the training output image accordingly; and, in step (D), in the client device, the AI ​​enhancement module receives several frames of the decoded image and generates and outputs at least one frame of the enhanced image accordingly.

[0052] In one embodiment, the artificial neural network module receives and processes the training decoded image and an audio; and the AI ​​enhancement module receives and processes the decoded image and another audio.

[0053] To achieve the above object, the present invention provides another embodiment of a method for improving the quality of image media, comprising:

[0054] A client device is provided that includes an AI enhancement module, wherein the AI ​​enhancement module includes at least one predefined mathematical expression, and the at least one mathematical expression includes at least one weighting parameter; the at least one weighting parameter of the at least one mathematical expression is predefined by a training program executed by an artificial neural network module in a server;

[0055] Receiving a plurality of encoded images by the client device;

[0056] The client device decodes the plurality of encoded images into a plurality of decoded images, and processes the plurality of decoded images using the at least one mathematical expression and the at least one weighting parameter of the AI ​​enhancement module to generate a plurality of enhanced images; and

[0057] The client device outputs the enhanced image to a screen as an output image to be played.

[0058] To achieve the above object, the present invention provides another embodiment of a method for improving the quality of image media, comprising:

[0059] Decoding a plurality of encoded images using a client device, wherein the plurality of encoded images are generated by encoding a plurality of original images;

[0060] The client device decodes the plurality of encoded images into a plurality of decoded images, and uses an AI enhancement module to enhance the quality of the plurality of decoded images to generate a plurality of enhanced images; wherein the AI ​​enhancement module processes the plurality of decoded images by analyzing at least one mathematical expression obtained in advance by differences between the plurality of decoded images and the corresponding plurality of original images; thereby, the plurality of enhanced images obtained will be visually closer to the plurality of original images than the plurality of decoded images; and

[0061] The client device outputs the enhanced image to a screen as an output image to be played;

[0062] The at least one mathematical expression of the AI ​​enhancement module of the client device includes a plurality of weighting parameters; the weighting parameters are associated with the difference between the decoded image and the corresponding original image;

[0063] The plurality of original images may be divided into a plurality of groups of scenes, each of which includes a plurality of the original images;

[0064] The plurality of weighting parameters used by the AI ​​enhancement module in the client device are also divided into a plurality of groups, each group comprising a plurality of the weighting parameters and corresponding to one of the scenarios;

[0065] In the step of using the AI ​​enhancement module on the client device to enhance the quality of the multiple decoded images to generate multiple enhanced images, the multiple decoded images corresponding to the original images of different scenes will be subjected to image enhancement processing by the same AI enhancement module using the group of weighted parameters corresponding to the scene among the multiple different groups of weighted parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 The system for improving the quality of video streams transmitted via a network of the present invention is schematically introduced;

[0067] Figure 2 is a schematic diagram of an embodiment of a system architecture for improving the quality of an image stream transmitted via a network according to the present invention;

[0068] Figure 3 is a schematic diagram of an embodiment of a method for improving the quality of an image stream transmitted via a network according to the present invention;

[0069] Figure 4 is a schematic diagram of a first embodiment of a training procedure of the artificial neural network module 105 of the present invention;

[0070] Figure 5 is a schematic diagram of a second embodiment of the training procedure of the artificial neural network module 105 of the present invention;

[0071] Figure 6 is a schematic diagram of a third embodiment of the training procedure of the artificial neural network module 105 of the present invention;

[0072] Figure 7 Yes Figure 6 A schematic diagram of an embodiment of a training procedure for a discriminator is shown;

[0073] Figure 8 , which discloses an embodiment of the training process of the neural network of the present invention, wherein the original image is YUV420, and the output image is RGB or YUV420;

[0074] Fig. 9 is a schematic diagram of an embodiment of a procedure for processing a decoded image in YUV420 format according to the present invention;

[0075] Fig.10 is a schematic diagram of another embodiment of a procedure for processing a decoded image in YUV420 format according to the present invention;

[0076] Fig.11 A schematic diagram of another embodiment (fourth embodiment) of the training procedure of the artificial neural network module of the present invention;

[0077] Fig.12 A schematic diagram of another embodiment (fifth embodiment) of the training procedure of the artificial neural network module according to the present invention;

[0078] Fig.13 is a schematic diagram of another embodiment of a method for improving the quality of an image stream according to the present invention;

[0079] Fig.14 Schematic diagram of yet another embodiment (sixth embodiment) of the training procedure of the artificial neural network module of the present invention;

[0080] Fig.15 is a schematic diagram of another embodiment of a method for improving the quality of an image stream according to the present invention;

[0081] Fig.16 is a schematic diagram of another embodiment (seventh embodiment) of the training procedure of the artificial neural network module of the present invention;

[0082] Fig.17 is a schematic diagram of another embodiment of a method for improving the quality of an image stream according to the present invention;

[0083] Fig.18 is a schematic diagram of yet another embodiment (eighth embodiment) of the training procedure of the artificial neural network module of the present invention;

[0084] Fig.19 is a schematic diagram of yet another embodiment of a method for improving the quality of an image stream according to the present invention;

[0085] Fig. 20 It is a schematic diagram of yet another embodiment of the procedure of the present invention for processing a decoded image in YUV420 format.

[0086] Explanation of the accompanying drawings: 1, 501 ~ server; 2, 21, 22, 23, 502 ~ client device; 3 ~ base station; 30 ~ router; 4 ~ network; 100, 200 ~ application (App); 101, 201 ~ memory; 102 ~ encoding; 103 ~ streaming; 104 ~ network equipment; 105 ~ artificial neural network module; 106 ~ neural network; 107 ~ decoding module; 108 ~ comparison and training module; 202 ~ network module; 203 ~ decoding module; 204 ~ AI enhancement module; 205 ~ output module; 301-308, 400-466, 511-596, 601-607 ~ steps. DETAILED DESCRIPTION

[0087] The present invention relates to a method for improving the quality of an image stream using a pre-trained artificial intelligence (AI) enhancement module built into a client device. The AI ​​enhancement module is pre-trained in a server with a neural network to analyze the difference between an original image generated by the server and a compressed image. The AI ​​enhancement module uses a mathematical formula obtained by analyzing the difference between the original image generated by the server and the compressed image to improve the quality of the compressed image. As a result, the enhanced image is visually closer to the original image than the compressed image.

[0088] One application of the present invention is cloud-based online games, in which a player uses a client device to connect to a server via a network to play a game provided by the server. The server can respond to commands input by the player and generate corresponding video images. Thus, for example, a player can execute a move command on a client device. This move command is sent to the server via the network, and then the server calculates an image based on the move command, and sends the image back to the client device for playback. In many games, the server generates a 2D image containing several 3D rendered objects in the visible range.

[0089] See also Figure 1, which schematically introduces the system of improving the quality of image streaming transmitted via a network of the present invention. A server 1 is used to provide a service provided by an application executed on the server 1; the service may be, but is not limited to, a cloud online game service. A plurality of client devices 21, 22, 23 may connect (login) to the server 1 via a network 4 to use the service provided by the application executed on the server 1. In this embodiment, the network 4 is the Internet, and the client devices 21, 22, 23 may be any type of electronic device that can be connected to the Internet, such as (but not limited to) a smartphone 21, a digital drawing tablet, a laptop computer 22, a desktop computer 23, an electronic game console, or even a smart TV. Some of the client devices 21, 22 are wirelessly connected to the network 4 via a wireless communication base station 3 or a wireless router 30, while others are wired to the network 4 via a network router or a network share. The application executed on the server 1 generates a virtual 3D environment which includes a plurality of 3D objects; some of the 3D objects can be moved or destroyed according to the user's operation, while others cannot. In a preferred embodiment, for each client device, the application will have an independent running instance; that is, each application only provides the service to one client device, but multiple applications can be executed simultaneously in the server 1 to provide services to multiple client devices. The client devices 21, 22, 23 are connected to the server 1 via the network 4 to receive the images generated by the application and including at least a portion of the 3D objects. The architecture and functions of the system of the present invention will be described through Figure 2 And its related instructions are described in detail.

[0090] Figure 21 is a schematic diagram of an embodiment of the system architecture of the present invention. The application (App) 100 is an application (usually a 3D game program) stored in the memory 101 and executed on the server 1, which can generate a 3D screen rendering result composed of a series of original image images. The encoding 102 and the streaming 103 are respectively an encoding module and a streaming module, which can receive the multiple original image images generated by the application 100, and encode and string them into a 2D image stream. The 2D image stream is then transmitted to the remote client device 2 through the network 4 via the network device 104 of the server. Each client device 2 is pre-installed with an application 200, which is stored in the memory 201 of the client device 2 and can be associated with and cooperate with the application 100 on the server 1. The application 200 of the client device 2 can establish a link with the application 100 of the server 1, and receive the encoded 2D image stream from the server 1 through the network module 202. The encoded 2D image stream is then decoded by the decoding module 203 to generate a decoded image. Due to these encoding, streaming and decoding procedures, the quality of the decoded image will obviously be much worse than the original image. The AI ​​enhancement module 204 built into the client device 2 can enhance the quality of those decoded images to generate corresponding enhanced images. Among them, the AI ​​enhancement module 204 processes the multiple decoded images by analyzing at least one mathematical expression obtained by the difference between the multiple decoded images and the corresponding multiple original images; thereby, the obtained enhanced image will be visually closer to the original image than the decoded image. Afterwards, the enhanced image is output (played) on the screen (display panel) of the client device 2 via the output module 205. In the present invention, the mathematical expression used by the AI ​​enhancement module 204 of the client device 2 is defined by a training program executed by the artificial neural network module 105 located on the server 1. The artificial neural network module 105 is arranged in the server 1 and includes: an artificial neural network 106, a decoding module 107 and a comparison and training module 108. The embodiment of the training procedure of the artificial neural network module 105 of the present invention will be described in detail later.

[0091] Figure 3 FIG. 1 is a schematic diagram of an embodiment of a method of improving the quality of an image stream transmitted via a network according to the present invention. Figure 2 and Figure 3 The system and architecture of the present invention shown in the figure generally include the following steps:

[0092] Step 301: A first application is executed in a server. The first application generates a plurality of original images according to at least one instruction (step 302). Afterwards, the original images are encoded and compressed by an encoder in the server (step 303) to generate a plurality of encoded images. The plurality of encoded images are then transmitted to a client device via a network in the form of a 2D image stream (step 304).

[0093] A second application is executed in a client device away from the server (step 305). The second application is related to and cooperates with the first application, whereby the client device can be operated by a user and generate and send instructions to the server to enjoy the service provided by the first application of the server. The client device transmits the instruction to the server via the network, and then receives the encoded image generated by the server and corresponding to the instruction via the network. Then, the client device decodes (step 306) the multiple encoded images into multiple decoded images, and uses an AI enhancement module (step 307) to enhance the quality of the multiple decoded images to generate multiple enhanced images. The AI ​​enhancement module processes the multiple decoded images by analyzing at least one mathematical expression obtained in advance by the difference between the multiple decoded images and the corresponding multiple original images; thereby, the obtained enhanced image will be visually closer to the original image than the decoded image. Afterwards, the client device outputs (step 308) the enhanced image to the screen (display panel) as the output image to be played.

[0094] In the present invention, the at least one mathematical expression used by the AI ​​enhancement module in the client device includes a plurality of weighted parameters. The plurality of weighted parameters are associated with the difference between the decoded image and the corresponding original image, and are defined by a training program executed by an artificial neural network module in the server. In one embodiment of the present invention, the weighted parameters are pre-stored in the client device. In another embodiment, the weighted parameters are downloaded from the server to the client when the client device executes the second application.

[0095] In one embodiment of the present invention, the picture content contained in the original image generated by the server will change dramatically due to different game scenes. For example, a city game scene may make the original image of the game contain many simple and clear appearance contours and different but roughly the same color system. Another dark cave game scene will make the original image of the game full of monotonous and low-tone and low-chroma colors, but with irregular but inconspicuous landscape contours. And another lush garden scene will make the original image of the game contain many lively and colorful objects with detailed and complex contours. The method of the present invention uses an array of different weighting parameters to adapt to these different game scenes respectively, whereby the quality of the output image enhanced by the same AU enhancement module can be maintained at a high-quality and stable level, even if the picture content of the original image changes dramatically.

[0096] Preferably, the multiple original images generated by the first application can be divided into multiple scenes (scene-modes), each scene includes multiple original images. The multiple weighting parameters are also divided into multiple groups, each group includes multiple weighting parameters and corresponds to one of the scenes. The multiple decoded images corresponding to the original images of different scenes will be processed by the same AI enhancement module using the group of weighting parameters corresponding to the scene in the multiple different groups of weighting parameters. In one embodiment of the present invention, the multiple different groups of weighting parameters are all pre-stored in the client device, and whenever the scene changes, the group of weighting parameters corresponding to the changed new scene will be used in the AI ​​enhancement module to generate the enhanced image. In another embodiment, the multiple different groups of weighting parameters are all stored in the server side, and whenever the scene changes, the group of weighting parameters corresponding to the changed new scene will be transmitted from the server to the client device, and then used in the AI ​​enhancement module to generate the enhanced image.

[0097] Figure 4 1 is a schematic diagram of a first embodiment of the training procedure of the artificial neural network module 105 of the present invention. In the present invention, the mathematical expression used by the AI ​​enhancement module 204 of the client device 2 is trained and defined by a training procedure executed by the artificial neural network module 105 in the server 1. The training procedure includes the following steps:

[0098] Step 400: Execute the first application in a training mode to generate a plurality of training original images (step 401);

[0099] Step 402: Encode the plurality of original training images into a plurality of encoded training images by the encoder;

[0100] Step 403: Decoding the plurality of training coded images into a plurality of training decoded images by a training decoder in the server;

[0101] Step 404: The artificial neural network module receives the plurality of training decoded images and processes the training decoded images one by one using at least one training mathematical expression to generate a plurality of training output images (Step 405); the at least one training mathematical expression comprises a plurality of training weighting parameters; and

[0102] Step 406: Use the comparison and training module to compare the differences between the training output image and the corresponding training original image one by one, and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly; the multiple training weighting parameters will be adjusted to minimize the differences between the training output image and the corresponding training original image; each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the at least one training mathematical expression for processing the next training decoded image in step 404. After a predetermined number of comparisons between the training output images and the corresponding training original images and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters obtained after the training is finally completed (step 407) will be retrieved and applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0103] In the first embodiment of the present invention, the training decoded image is input into the artificial neural network module to generate the corresponding training output image. Then, the training output image and the corresponding training original image are compared to calculate the difference value. Then, a mathematical optimization algorithm such as the Adam algorithm, stochastic gradient descent (SGD), or forward root mean square gradient descent (RMSProp) is used to learn the weighting parameters (usually referred to as weight weightw, bias b) of the artificial neural network so that the difference value is as small as possible, so that the training output image can be closer to its corresponding training original image. Different methods can be used to calculate the difference value (or approximation) to suit different needs; for example: mean square error (MSE), L1 regularization (using absolute value error), peak signal-to-noise ratio (PSNR), structure similarity (SSIM), generative adversarial networks loss (GAN loss) and / or other methods. In the first embodiment, the following methods are used to calculate the difference value: (1) weighted average of MSE, L1, and GAN loss; (2) MSE; (3) GAN loss and simultaneous training of the discriminator; (4) weighted average of MSE and edge of MSE. More details of the training procedure will be described later.

[0104] Figure 5 1 is a schematic diagram of a second embodiment of the training procedure of the artificial neural network module 105 of the present invention. In the present invention, the training procedure of the second embodiment includes the following steps:

[0105] Step 410: Execute the first application in a training mode to generate a plurality of training original images (Step 411), wherein the color format of the plurality of training original images is RGB;

[0106] Step 412: Encode the plurality of original training images into a plurality of coded training images by the encoder;

[0107] Step 413: Decoding the plurality of training coded images into a plurality of training decoded images by a training decoder in the server;

[0108] Step 414: In the second embodiment, when the color format of the training decoded image and the training output image is the same (both are RGB in the second embodiment), a residual network module (also known as a convolutional neural network (CNN)) may be used in the artificial neural network module; the output of the residual network module for processing the corresponding training decoded image is summed up with the corresponding training decoded image (step 415); then, the sum of the output of the residual network module and the corresponding training decoded image is output as the training output image (step 416); and

[0109] Step 417: Use the comparison and training module to compare the differences between the training output image and the corresponding training original image one by one (calculate the difference value), and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly; the multiple training weighting parameters will be adjusted to minimize the difference between the training output image and the corresponding training original image; each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the artificial neural network for processing the next training decoded image in step 414. After a predetermined number of comparisons of the training output images with the corresponding training original images and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters obtained after the training is finally completed (step 418) will be retrieved and applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0110] Figure 6 1 is a schematic diagram of a third embodiment of the training procedure of the artificial neural network module 105 of the present invention. In the third embodiment, the comparison and training module uses a discriminator to compare the difference between the training output image and the corresponding training original image and adjust the training weighting parameters accordingly. The training procedure of the third embodiment includes the following steps:

[0111] Step 420: Execute the first application in a training mode to generate a plurality of original training images (Step 421), wherein the plurality of original training images include n channels, wherein n is a positive integer greater than 2;

[0112] Step 422: Encode the plurality of original training images into a plurality of coded training images by the encoder;

[0113] Step 423: Decode the plurality of training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images include m channels, wherein m is a positive integer greater than 2; and

[0114] Step 424: The artificial neural network module receives the multiple training decoded images (m channels) and uses at least one training mathematical expression to process the training decoded images one by one to generate multiple training output images (n channels) (step 425); the at least one training mathematical expression includes multiple training weighting parameters; the training output images (n channels) and the corresponding training decoded images (m channels) are combined (step 426) to generate multiple training combined images (having m+n channels); then, these training combined images are fed back to a discriminator (step 427) for analyzing the quality of the training output images, thereby training the artificial neural network.

[0115] Figure 7 Yes Figure 6 The training procedure of the discriminator shown in FIG. 1 includes the following steps:

[0116] Step 430: Execute the first application in a training mode to generate a plurality of original training images (Step 431), wherein the plurality of original training images include n channels, wherein n is a positive integer greater than 2;

[0117] Step 432: Encode the plurality of original training images into a plurality of coded training images by the encoder;

[0118] Step 433: Decode the plurality of training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images include m channels, wherein m is a positive integer greater than 2; and

[0119] Step 434: The artificial neural network module receives the plurality of training decoded images and uses at least one training mathematical expression to process the training decoded images (m channels) one by one to generate a plurality of training output images (step 435); the at least one training mathematical expression comprises a plurality of training weighting parameters; the training output images comprise n channels;

[0120] Step 436: The n-channel training output image and its corresponding m-channel training decoded image are combined to generate a plurality of false samples having m+n channels; and the n-channel training original image and its corresponding m-channel training decoded image are combined to generate a plurality of true samples having m+n channels (Step 437); and

[0121] Step 438: The simulated false samples of the m+n channel and the simulated true samples of the m+n channel are fed back to the discriminator of the comparison and training module for training the discriminator to detect and distinguish the simulated false samples from the simulated true samples.

[0122] In the artificial neural network 105 (such as Figure 2 After the AI ​​enhancement module 204 (as shown) is properly trained on the server 1 side, the obtained weighting parameters (weight w, bias b) will be applied to the AI ​​enhancement module 204 in the client device. The AI ​​enhancement module 204 and its associated weighting parameters (weight w, bias b) are stored in the client device 2. Thereafter, each time the client device receives and decodes the encoded image contained in the 2D image stream from the server, each of the encoded images will be processed by the AI ​​enhancement module to generate an enhanced image. The client device then plays the multiple enhanced images as output images on its screen. The neural network can learn and enhance the color, brightness and details of the image. Since some details of the original image may be damaged or lost during the encoding and streaming process, a properly trained neural network can repair these damaged or lost details. In an embodiment of the present invention, the neural network of the AI ​​enhancement module requires the following information to operate:

[0123] Related functions and parameters:

[0124] X: input image;

[0125] Conv2d(X,a,b,c,d,w,b): Executed on X; amount of output channels is a (amount of outputchannel=a); kernel size is b (kernel_size=b); stride is c (stride=c); padding size=2d convolution with bias of d (padding size=2d convolution with bias of d); the weighting parameters for this training are kernel w (kernel w) and bias b (bias b);

[0126] Conv2dTranspose(X,a,b,c,w,b)): performed on X; amount of output channel = a; kernel size = b; stride = c; cropping size = 2d transpose convolution with bias of d; weighting parameters for this training are kernel w and bias b;

[0127] σ(X): nonlinear activation function working on X;

[0128] uint8(x): used to control and limit the floating point value x between 0 and 255 (including 255). u uses the unconditional rounding method and is converted to unsigned int8.

[0129] R(X,w): residual blocks working on X, including many conv2d and batchnorm, each of which contains its own weighted parameters for training (for more information, please refer to the following website: https: / / stats.stackexchange.com / questions / 246928 / what-exactly-is-a-residual-learning-block-in-the-context-of-deep-residual-networ).

[0130] Since the input and output images may have different color formats, such as RGB, YUV420, YUV444, etc., the following will discuss more about the input and output images with different color formats.

[0131] The first case: the original image is RGB and the output image is also RGB.

[0132] This case is the simplest, because both the input and output images are RGB images. In order to increase the processing speed, a relatively large kernel size (e.g. 8x8, stride=4 in convolution and transposed convolution structures) is used to speed up the calculation as quickly as possible to cope with the high resolution of Full HD images. In this case, a residual network is used to make convergence easier and more stable.

[0133] Related functions and parameters:

[0134] X: input image, which is in RGB format, and each color is in unsigned int8 format;

[0135]

[0136] Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d,w_1,b_1)),w_2,b_2)+X2)*128+128);

[0137] w_1 is a matrix whose size is b*b*3*a; b_1 is a vector whose size is a;

[0138] w_2 is a matrix whose size is b*b*3*a; b_2 is a vector whose size is 3;

[0139] The parameters used are:

[0140] The resolution of X is 1280x720;

[0141] a=128, b=10, c=5, d=0, σ=leaky relu with alpha=0.2;

[0142] a=128, b=9, c=5, d=4, σ=leaky relu with alpha=0.2;

[0143] a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2;

[0144] If the client device has a faster processing speed, the following mathematical formula can be used:

[0145] Y=uint8((Conv2dTranspose(R(σ(Conv2d(X2,a,b,c,d,w_1,b_1)),w_R),w_2,b_2)+X2)*128+128);

[0146] w_1 is a matrix whose size is b*b*3*a; b_1 is a vector whose size is a;

[0147] w_2 is a matrix whose size is b*b*3*a; b_2 is a vector whose size is 3;

[0148] Where R is a residual block with n layers;

[0149] It contains many neural network layers, each of which has its own trained weighted parameters collectively referred to as w_R;

[0150] The parameters used are:

[0151] a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2; n=2;

[0152] a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2; n=6.

[0153] The second case: the original image is YUV420 and the output image is RGB or YUV444;

[0154] If the input original image is YUV420 and the output image is RGB or YUV444, the residual network cannot be directly applied to this situation due to the different resolutions and formats of the input and output images. The method of the present invention first decodes the YUV420 input image, and then uses another neural network (called A network, where N=3) to process the decoded image and obtain an image in RGB or YUV444 format (called X2). Then, this X2 image is sent to the neural network (residual network) described in the first case for training. In addition, the same training method is also applied to the A network to compare the difference between X2 and the original image, thereby training the A network.

[0155] X_y is the Y value of the input image in YUV420 format, in unsigned int8 format;

[0156] X_uv is the uv of the input image in YUV420 format, and its format is unsigned int8;

[0157]

[0158] X2=Conv2d(X2_y,3,e,1,w_y,b_y)+Conv2dTranspose(X2_uv,3,f,2,w_uv,b_uv);

[0159] w_y is a matrix whose size is e*e*1*3; b_y is a vector whose size is 3;

[0160] w_uv is a matrix whose size is f*f*3*2; b_uv is a vector whose size is 3;

[0161] The above is the first embodiment of the A network (neural network number A);

[0162] Finally, the mathematical formula used to generate the output image is the same as the mathematical formula used in the first case above when both the input and output images are in RGB format:

[0163] Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d,w_1,b_1)),w_2,b_2)+X2)*128+128);

[0164] w_1 is a matrix whose size is b*b*3*a; b_1 is a vector whose size is a;

[0165] w_2 is a matrix whose size is b*b*3*a; b_2 is a vector whose size is 3;

[0166] The parameters used are the same as those used above when both the input and output images are in RGB format:

[0167] The resolution of X is 1280x720;

[0168] a=128, b=8, c=4, d=0, e=1, f=2, σ=leaky relu with alpha=0.2;

[0169] a=128, b=8, c=4, d=0, e=1, f=2, σ=leaky relu with alpha=0.2.

[0170] See also Figure 8 , which discloses an embodiment of the training process of the neural network of the present invention, wherein the original image is YUV420 and the output image is RGB or YUV420. The training process of the neural network includes the following steps:

[0171] Step 440: Execute the first application in a training mode to generate a plurality of original training images, wherein the plurality of original training images are in RGB or YUV444 format;

[0172] Step 441: Encode the plurality of original training images into a plurality of coded training images by the encoder;

[0173] Step 442: Decode the plurality of training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images are in YUV420 format;

[0174] Step 443: The artificial neural network module includes a first neural network and a second neural network; the first neural network (also referred to as A network) receives the plurality of training decoded images and uses at least one training mathematical expression to process the training decoded images (YUV420) one by one to generate a plurality of first output images X2 (also referred to as X2; as shown in step 444) having the same encoding format as the training original image; the at least one training mathematical expression includes a plurality of training weighting parameters;

[0175] Step 445: The second neural network is a convolutional neural network (CNN). The second neural network (also referred to as a CNN network) receives the first output image X2 and processes the first output image X2 one by one using at least one training mathematical expression to generate a plurality of second output images. The at least one training mathematical expression includes a plurality of training weighting parameters. Then, the first output image X2 and the second output image are added (step 446) to generate the training output image (step 447).

[0176] The comparison and training module includes a first comparator and a second comparator; in step 448, the first comparator compares the difference between the first output image X2 and its corresponding training original image for training the first neural network; in step 449, the second comparator compares the difference between the training output image and its corresponding training original image for training the second neural network.

[0177] Figure 9 1 is a schematic diagram of an embodiment of a procedure for processing a decoded image in YUV420 format according to the present invention. The procedure for processing a decoded image in YUV420 format according to the present invention comprises:

[0178] Step 451: The first neural network receives and processes the training decoded image in YUV420 color format, including:

[0179] Step 452: extracting Y-part data from the training decoded image, and processing the Y-part data of the training decoded image by the neural network with a standard size (original size) to generate Y-part output data with N channels (e.g., stride value Stride=1 in convolution; as shown in step 454);

[0180] Step 453: extract UV-part data from the training decoded image, and process the UV-part data of the training decoded image by a neural network with two times magnification to generate UV-part output data with N channels (e.g., stride value Stride=2 in transposed convolution; as in step 455);

[0181] Step 456: Add the Y-part output data and the UV-part output data to generate the training output image (step 457).

[0182] The third case: the original image is YUV420 and the output image is YUV444, which is processed in another faster way.

[0183] If the input image is YUV420 and the output image is YUV444, in addition to the aforementioned method, there is another way to implement the first neural network (A network), which is a special case with faster speed. The decoded image in YUV420 format is first converted into an image in YUV444 format (also called X2) by the first neural network (A network); then, X2 is sent to the aforementioned neural network (residual network) for training. In addition, the same training method is also implemented in the A network to compare the difference between X2 and the original image, so as to train the A network.

[0184] X_y is the Y value of the input image in YUV420 format, in unsigned int8 format;

[0185] X_uv is the uv of the input image in YUV420 format, and its format is unsigned int8;

[0186]

[0187]

[0188] X3_uv=Conv2dTranspose(X2_uv,2,2,2,w_uv,b_uv);

[0189] w_uv is a matrix whose size is 2*2*2*2; b_uv is a vector whose size is 2;

[0190] X2 = concat(X2_y, X3_uv);

[0191] The above is another embodiment of the A network (neural network number A), wherein the "concat" function connects the input in the direction of the channel;

[0192] Finally, the mathematical formula used to generate the output image is the same as the mathematical formula used in the first case above when both the input and output images are in RGB format:

[0193] Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d,w_1,b_1)),w_2,b_2)+X2)*128+128);

[0194] w_1 is a matrix whose size is b*b*3*a; b_1 is a vector whose size is a;

[0195] w_2 is a matrix whose size is b*b*3*a; b_2 is a vector whose size is 3;

[0196] The parameters used are the same as those used above when both the input and output images are in RGB format:

[0197] The resolution of X is 1280x720;

[0198] a=128, b=10, c=5, d=0, σ=leaky relu with alpha=0.2;

[0199] a=128, b=9, c=5, d=4, σ=leaky relu with alpha=0.2;

[0200] a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2.

[0201] Fig.10 FIG. 2 is a schematic diagram of another embodiment of the process of processing a decoded image in YUV420 format according to the present invention. Fig.10 The process of processing the decoded image in YUV420 format according to the present invention includes:

[0202] Step 461: The first neural network receives and processes a training decoded image in a YUV420 color format by the following steps, wherein the training decoded image includes N channels, and N is a positive integer greater than 2;

[0203] Step 462: extract the Y-part data in the training decoded image to generate Y-part output data;

[0204] Step 463: extract UV-part data from the training decoded image, and use the first neural network with two times magnification to process the UV-part data of the training decoded image to generate UV-part output data with N-1 channels (e.g., stride value Stride=2 in transposed convolution; as shown in step 464);

[0205] Step 465: Process the Y-part data and the UV-part data with a concatenation function Concat (concatenates) to generate the training output image (step 466).

[0206] The fourth case: the original image is YUV420 and the output image is also YUV420.

[0207] If the input image is YUV420 and the output image is also YUV420, the processing will be similar to the RGB-to-RGB method mentioned above. However, due to the different input and output formats, different convolution methods will be applied to different channels. For example, when the core size of the neural network is 8x8 and the stride value is 4 to process the Y-part of the image, the neural network can be changed to a core size of 4x4 and a stride value of 2 to process the UV-part of the image.

[0208] X_y is the Y value of the input image in YUV420 format, in unsigned int8 format;

[0209] X_uv is the uv of the input image in YUV420 format, and its format is unsigned int8;

[0210]

[0211] X3=σ(Conv2d(X2_y,a,b,c,w_y,b_y)+Conv2d(X2_uv,a,b / 2,c / 2,w_uv,b_uv));

[0212] w_y is a matrix whose size is b*b*1*a; b_y is a vector whose size is a;

[0213] w_uv is a matrix whose size is (b / 2)*(b / 2)*2*a; b_uv is a vector whose size is a;

[0214] X4_y=Conv2dTranspose(X3,1,b,c,w_1,b_1)+X2_y;

[0215] X4_uv=Conv2dTranspose(X3,2,b / 2,c / 2,w_2,b_2)+X2_uv;

[0216] w_1 is a matrix whose size is b*b*1*a; b_1 is a vector whose size is 1;

[0217] w_2 is a matrix whose size is (b / 2)*(b / 2)*2*a; b_2 is a vector whose size is 2;

[0218] The above is another embodiment of the A network (neural network number A), wherein the "concat" function connects the input in the direction of the channel;

[0219] Final output:

[0220] Y_y = uint8(X4_y*128+128);

[0221] Y_uv=uint8(X4_uv*128+128);

[0222] Parameters used:

[0223] a=128, b=8, c=4, d=0, e=2, f=2, σ=leaky relu with alpha=0.2.

[0224] The detailed description of the parameters used in the present invention is as follows:

[0225] Training parameters:

[0226] The initial value of the weighting parameter is based on Gaussian distribution, with mean=0 and stddev=0.02;

[0227] The Adam algorithm is used during training, with learning rate = 1e-4, beta1 = 0.9;

[0228] Mini batch size = 1;

[0229] The primary error function is:

[0230] 100*(L2+L2e)+λ*L1+γ*D+α*Lg;

[0231] The standard values ​​of the parameters used are:

[0232] λ=0,γ=0,α=0;

[0233] λ=0,γ=0,α=100;

[0234] λ=0,γ=1,α=0;

[0235] λ=10,γ=0,α=0;

[0236] λ=10,γ=0,α=100;

[0237] λ=10,γ=1,α=0;

[0238] in:

[0239] L2=mena((TY) 2 ), where mean refers to the average value and T is the training target;

[0240] L1 = mena(|TY|); where mean refers to the average value and T is the training target;

[0241] D is the Generative Adversarial Network loss (GAN loss), which uses the general GAN ​​training method to train the Discriminator to distinguish (X, Y) from (X, T);

[0242] The mathematical formula of Lg is:

[0243] For WxH images,

[0244] Y_dx(i,j)=Y(i+1,j)-Y(i,j)0<=i <W-1,0<=j<H

[0245] T_dx(i,j)=T(i+1,j)-T(i,j)0<=i <W-1,0<=j<H

[0246] Y_dy(i,j)=Y(i,j+1)-Y(i,j)0<=i <W,0<=j<H-1

[0247] T_dy(i,j)=T(i,j+1)-T(i,j)0<=i <W,0<=j<H-1

[0248] L g =meam((T dx -Y dx ) 2 )+mean((T dy -Y dy ) 2 )

[0249] In RGB mode, the training target T is the original image of the RGB game image;

[0250] In YUV444 mode, the training target T is the original original image of the RGB game image;

[0251] In RGB->RGB and YUV420->YUV420 modes, L2e=0;

[0252] In YUV420->RGB and YUV420->YUV444 modes,

[0253] L 2e =mean((T-X2) 2 ).

[0254] From the above description, it can be seen that the method of improving the quality of image media transmitted via a network of the present invention has the following advantages:

[0255] The neural network can be trained at any time according to various images with different contents, so as to perform different enhancement effects on different image contents; for example, for images with cartoon style, realistic style or different scenes, different weighting parameters w, b can be pre-stored in the client device or dynamically downloaded to the client device;

[0256] Regarding the method of determining which mode the original image should belong to, the neural network on the server side can automatically determine the mode of the original image and transmit such information to the client device; because the content of the original image is consistent, this determination process can be performed periodically by the server, for example, once per second; however, in another embodiment, the process of determining the image mode can also be performed periodically by the client device, for example, once every few seconds, depending on the computing power of the client device;

[0257] The training is conducted based on real video images, and the degree of enhancement can be actually measured; for example, when the method of the present invention is used to enhance a video image with a resolution of 1280x720 and a bit rate of 3000, the PSNR value of a similar scene can be increased by about 1.5 to 2.2 db, which can prove that the method of the present invention can indeed truly improve the quality of the output image and make the output image visually closer to the quality of the original image; and the present invention is different from those well-known image enhancement technologies, which can only increase the contrast, smoothness and color filtering of the output image, but cannot make the output image visually closer to the original image like the present invention;

[0258] By simplifying the model of the neural network algorithm and using large cores and large stride values, the resolution of the neural network can be quickly reduced and the processing speed of the model can be greatly improved; even client devices with limited computing power can achieve the goal of 60fps and HD resolution output images; and

[0259] By bringing the color format (YUV420 and RGB) conversion work into the neural network and taking advantage of the lower resolution of the UV channel than the Y channel, the step value of the UV channel is set to half of that of the Y channel, which can increase the calculation speed of the neural network.

[0260] In addition to the above-mentioned embodiments, the method of improving the quality of image media transmitted via the network of the present invention has many different extended applications; for example: increasing display resolution (Resolution), expanding the picture size, increasing time resolution (i.e., increasing the frame rate Frames Per Second, abbreviated as FPS), multi-frame input, predicted pictures, offline applications, etc., as described below.

[0261] See also Fig.11, is a schematic diagram of another embodiment (fourth embodiment) of the training procedure of the artificial neural network module of the present invention. In the present invention, the mathematical expression used by the AI ​​enhancement module 204 of the client device 2 is trained and defined by a training procedure executed by the artificial neural network module 105 in the server 1. Fig.11 Another embodiment of the training program shown includes the following steps:

[0262] Step 511: In a training mode, a training video is provided which is composed of a plurality of high-quality original training images; the plurality of high-quality original training images may be real-time videos generated by, for example, game software or other applications, or may be pre-recorded videos or pre-stored videos in memory (for example, but not limited to: MP4, 3GP, AVI, MKV, WMV, MPG, VOB, FLV, SWF, MOV, RMVB and other video formats). In this embodiment, the plurality of high-quality training original images have relatively good or excellent image quality, such as images meeting one or more of the following properties: (a) having a higher resolution, such as 4K or 8K resolution, (b) having a larger field of view (FOV), such as 150 degrees, 180 degrees or even 360 degrees, (c) having a higher frame rate (Frame per Second, FPS), such as 60 FPS or 120 FPS, (d) having binocular stereo information, (e) having visual depth information, (f) having additional color information such as infrared, ultraviolet or others (reference URL: https: / / photo.stackexchange.com / questions / 83923 / why-dont-cameras-offer-more-than-3-colour-channels-or-do-they), or / and (g) having better audio such as multi-channel, high sampling frequency (44 khz or higher) or high resolution (audio bit depth 24 bit or above), etc.

[0263] Step 512: Image degradation processing: Degrade the plurality of original training images by a degradation module to reduce the image quality, so as to obtain degraded original training images that meet one or more of the following properties, such as but not limited to: (a) reducing the resolution by sampling (reducing to 720p, 1080i or lower), (b) reducing the field of view by cropping (reducing to 120 degrees or lower, or cropping the original image with an aspect ratio of 16:9 to 4:3), (c) discarding frames The above methods include (i) reducing the frame rate (to 30 FPS or lower), (ii) extracting the stereovision into a single frame by means of capture, (iii) discarding the visual depth information, (iv) discarding the visual depth information, (v) converting the audio into stereo, reducing the sampling frequency (22.1 kHz or lower), reducing the resolution (16 bit or lower), (v) adding noise (such as Gaussian or Laplacian noise), or / and (i) blurring (such as Gaussian blurring), etc.

[0264] Step 513: Encode and compress the plurality of degraded original training images into a plurality of coded training images by an encoder.

[0265] Step 514: Decode and decompress the plurality of training encoded images into a plurality of training decoded images by a training decoder in the server.

[0266] Step 515: The artificial neural network module in the server receives the plurality of training decoded images and processes the training decoded images one by one using at least one training mathematical expression to generate a plurality of training output images (Step 516). The at least one training mathematical expression includes a plurality of training weighting parameters.

[0267] Step 517: Use the comparison and training module to compare the differences between the training output image and the corresponding training original image provided in step 511 one by one, and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly. The multiple training weighting parameters will be adjusted to minimize the difference between the training output image and the corresponding training original image. Each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the at least one training mathematical expression for processing the next training decoded image in step 515. After a predetermined number of comparisons of the training output images with the corresponding training original images and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters obtained after the training is finally completed will be retrieved and applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0268] At Fig.11 In another embodiment of the present invention, the training decoded image is input into the artificial neural network module to generate the corresponding training output image. Then, the training output image and the corresponding training original image are compared to calculate the difference value. Then, a mathematical optimization algorithm such as the Adam algorithm, stochastic gradient descent (SGD), or forward root mean square gradient descent (RMSProp) is used to learn the weighting parameters (usually referred to as weight w, bias b) of the artificial neural network, so that the difference value is as small as possible, so that the training output image can be closer to its corresponding high-quality training original image. Different methods can be used to calculate the difference value (or approximation) to meet different requirements; for example: mean square error (MSE), L1 regularization (using absolute value error), peak signal-to-noise ratio (PSNR), structure similarity (SSIM), generative adversarial network loss (GAN loss) and / or other methods. In this embodiment, the following methods are used to calculate the difference value: (1) weighted average of MSE, L1, and GAN loss; (2) MSE; (3) GAN loss and simultaneous training of the discriminator; (4) weighted average of MSE and edge of MSE.

[0269] See also Fig.12 , is a schematic diagram of another embodiment (fifth embodiment) of the training procedure of the artificial neural network module of the present invention. In the training procedure of this embodiment, the input of the neural network may not be just one picture, but may be several pictures. By using the information of the previous pictures to help enhance the current image, a more optimized enhancement effect that is closer to the original image can be obtained. Fig.12 The training procedure of the illustrated embodiment includes the following steps:

[0270] Step 521: Provide a training video consisting of a plurality of training original images in a training mode; the plurality of training original images may be real-time videos generated by, for example, game software or other applications, or may be pre-recorded or pre-stored in memory. In this embodiment, the frames of the plurality of training original images may be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer.

[0271] Step 522: Encode and compress the plurality of original training images by an encoder into an image stream consisting of a plurality of training encoded images, wherein the frames are represented by Encoded_Frame(1), Encoded_Frame(2), ..., Encoded_Frame(N-1), Encoded_Frame(N), respectively, where N is a positive integer.

[0272] Step 523: A training decoder in the server sequentially decodes and decompresses the plurality of training coded images one by one into a plurality of training decoded images, whose frames are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(N-1), and Decoded_Frame(N), respectively, where N is a positive integer; and extracts the frame of the Nth training decoded image corresponding to the Nth training original image Frame(N) and one or more frames before it, such as Decoded_Frame(N-2), Decoded_Frame(N-1), and Decoded_Frame(N), to proceed to the next step 524. When the value of N-2 or N-1 is less than 1, the frame of Decoded_Frame(1) replaces Decoded_Frame(N-2) and Decoded_Frame(N-1).

[0273] Step 524: The artificial neural network module in the server receives the multiple training decoded images and sequentially processes the corresponding Nth training decoded image and the previous one or more training decoded images, i.e., Decoded_Frame(N-2), Decoded_Frame(N-1), Decoded_Frame(N), for each training original image (Frame(N), where N=3, 4, 5, ...), using at least one training mathematical expression to generate a corresponding training output image frame (also called Predict_Frame(N), as shown in step 525). The at least one training mathematical expression includes multiple training weighting parameters. In other words, the artificial neural network module receives input of several (e.g., three) training decoded image frames Decoded_Frame(N-2), Decoded_Frame(N-1), Decoded_Frame(N) each time and outputs a training output image frame Predict_Frame(N), as shown in step 525.

[0274] Step 526: Use the comparison and training module to compare the difference between the frame Predict_Frame(N) of the training output image and the frame Frame(N) of the corresponding training original image provided in step 521 one by one, and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly. The multiple training weighting parameters will be adjusted to minimize the difference between the frame Predict_Frame(N) of the training output image and the frame Frame(N) of the corresponding training original image. Then, the weights of the neural network are learned by a mathematical optimization algorithm (such as ADAM / SGD / RMSProp) to minimize the error. Each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the at least one training mathematical expression for processing the next training decoded image in step 524. After a predetermined number of comparisons of the training output images with the corresponding training original images and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters finally obtained will be applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression. Due to different requirements, different methods can be used to calculate the error (or similarity), for example, MSE (minimum square error), L1 (absolute value error), PSNR (peak signal-to-noise ratio), SSIM (Structure similarity), GAN loss (generative adversarial networks loss), etc. The present invention actually uses the following methods: (i) weighted average of MSE, L1 and GAN loss; (ii) MSE; (iii) GAN loss (simultaneous training of Discriminator); (iv) weighted average of MSE and Edge's MSE.

[0275] in accordance with Fig.12 The neural network training procedure shown in FIG. 1 generates an output frame by inputting a plurality of frames, and the corresponding embodiment of the method for improving the quality of the image media of the present invention can be as follows: Fig.13 shown. Fig.13 FIG. 1 is a schematic diagram of another embodiment of a method for improving the quality of an image stream according to the present invention, which comprises the following steps:

[0276] Step 531: A server 501 provides a plurality of original images (the frames of the plurality of original images can be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer, as shown in step 531). The plurality of original images can be real-time videos generated by, for example, game software or other applications, or can be videos pre-recorded or pre-stored in memory. Afterwards, the original images are encoded and compressed by an encoder (step 532) in the server 501 to generate a plurality of encoded images, whose frames are represented by Encoded_Frame(1), Encoded_Frame(2), ..., Encoded_Frame(N-1), Encoded_Frame(N), where N is a positive integer. The plurality of encoded images are then transmitted to the client device via the network in the form of a 2D image stream (step 533).

[0277] It is worth mentioning that the embodiment described in step 531 to step 533 is an embodiment when the encoded image is transmitted through a network. However, in another embodiment of the present invention, the encoded image and its 2D image stream can also be a video file pre-recorded or pre-stored in a portable memory medium (such as a flash drive, an optical disk, a hard disk, etc.), and the client device directly reads the video file to obtain the multiple encoded images or its 2D image stream, without receiving it through the network. In other words, the present invention can also use the AI ​​enhancement module to enhance the image quality in the client device in an offline state; only after reading the video file from the portable memory medium, the image quality can be improved by the AI ​​enhancement module before outputting it for playback. Therefore, the same neural network training method of the present invention can be used for videos from other sources; such as images recorded by a camera, videos recorded as files, etc. Using the same neural network and machine learning method, the video quality of the aforementioned images or videos during playback can also be improved. For example, playing DVD resolution images on a 4K TV, playing videos recorded by a mobile phone on a wide-screen TV, or increasing frames for slow motion playback.

[0278] Step 534: The client device 502 receives and decodes the plurality of encoded images (step 534) into a plurality of decoded images, whose frames are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(N-1), Decoded(N), respectively, where N is a positive integer; and extracts the Nth decoded image corresponding to the Nth original image Frame(N) and the frames of one or more previous training decoded images, such as Decoded_Frame(N-2), Decoded_Frame(N-1), Decoded_Frame(N), to proceed to the next step 524. When the value of N-2 or N-1 is less than 1, the frame of Decoded_Frame(1) replaces Decoded_Frame(N-2) and Decoded_Frame(N-1). Furthermore, an AI enhancement module (also called an enhanced neural network model, as shown in step 535) is used to input the Nth decoded image and one or more previous frames, that is, input Decoded_Frame(N-2), Decoded_Frame(N-1), Decoded_Frame(N), to generate a corresponding enhanced image frame as the output image (also called Enhanced_Frame(N), as shown in step 536). In other words, the AI ​​enhancement module will accept the input of several (for example, three) decoded image frames Decoded_Frame(N-2), Decoded_Frame(N-1), Decoded_Frame(N) each time and output an enhanced image frame Enhanced_Frame(N), as shown in step 536. Among them, at least one mathematical expression and its weighted parameters used by the enhanced neural network model in the AI ​​enhancement module are obtained by Fig.12 The same mathematical expressions and their weighting parameters described for the training procedure of the illustrated embodiment.

[0279] In this embodiment, if the neural network is written as a mathematical expression, it will be in the following form:

[0280] X = Decode (Video stream);

[0281] Y(N)=Network(X(N-2),X(N-1),X(n),W);

[0282] Where X = (X(1), X(2), X(3), ...); where X(N) is the Nth frame in the decompressed video, i.e. Decoded_Frame(N);

[0283] The Network is a neural network or a machine learning algorithm;

[0284] W is the trained weight parameter;

[0285] Y(N) is the Nth frame after enhancement, that is, Enhanced_Frame(N).

[0286] At Fig.12 and Fig.13 In the illustrated embodiment, the first few frames including the Nth decoded frame are used as input values ​​of the neural network, and the Nth frame after enhancement (or training) is generated accordingly. However, in another embodiment of the present invention, the frame of the Nth decoded image itself and its previous and next frames (i.e., Decoded_Frame(N-1), Decoded_Frame(N), Decoded_Frame(N+1)) may be used as input values ​​of the neural network, and the Nth frame after enhancement (or training) is generated accordingly. Alternatively, the number of input frames is not necessarily 3, but may be a different number. Since the training procedure of the neural network used in this embodiment and the implementation method of its use in improving the quality of the image stream are generally similar to the aforementioned Fig.12 and Fig.13 The embodiments shown are not described in detail.

[0287] See also Fig.14 , is a schematic diagram of another embodiment (sixth embodiment) of the training procedure of the artificial neural network module of the present invention. In the training procedure of this embodiment, the input of the neural network may not only be a simple picture, but also include audio data. Fig.14 The training procedure of the illustrated embodiment includes the following steps:

[0288] Step 541: In a training mode, a training video is provided which includes a plurality of training original images and audio. The plurality of training original images and audio are represented by Frame (1), Frame (2), ..., Frame (N-1), Frame (N), where N is a positive integer. The plurality of training original images and audio can be real-time video and audio generated by, for example, game software or other application programs, or can be pre-recorded or pre-stored in memory.

[0289] Step 542: Encode and compress the multiple training original images and audios by an encoder into an image stream consisting of multiple training encoded images and audios, represented by Encoded_Frame(1), Encoded_Frame(2), ..., Encoded_Frame(N-1), Encoded_Frame(N), respectively, where N is a positive integer.

[0290] Step 543: A training decoder is used to sequentially decode and decompress the plurality of training coded images one by one into a plurality of training decoded images, which are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(N-1), and Decoded_Frame(N), respectively, where N is a positive integer. Furthermore, according to the frame rate (FPS) of the training coded image stream, the decoded audio is divided into a plurality of audio segments of predetermined time length, which are represented by Audio(1), Audio(2), ..., Audio(N-1), and Audio(N), respectively, and each audio segment Audio(N) corresponds to a training coded image Decoded_Frame(N). For example, if the frame rate of the training encoded image stream is 25FPS, it means that each frame needs to be matched with an audio segment of 1 / 25 second, or 0.04 second. Therefore, in step 543, the received, decoded and decompressed audio will be divided into multiple audio segments in units of 0.04 seconds.

[0291] Step 544: The artificial neural network module receives the plurality of training decoded image and audio segments and processes each of them using at least one training mathematical expression to generate a corresponding training output image and audio segment (as in step 545). The at least one training mathematical expression includes a plurality of training weighting parameters.

[0292] Step 546: Use the comparison and training module to compare the differences between the training output image and audio segment and the corresponding training original image Frame (N) and original audio segment provided in step 541 one by one, and adjust the plurality of training weighting parameters of the at least one training mathematical expression accordingly. The plurality of training weighting parameters will be adjusted to minimize the differences between the training output image and audio segment and the corresponding training original image and audio segment. Since the adjustment method of the weighting parameters described in this embodiment is the same as that described above, it will not be repeated here.

[0293] in accordance with Fig.14 The neural network training procedure shown in FIG. 1 is a method for improving the quality of image media of the present invention, and the method includes a decoded image of an audio segment as an input to generate an output image and an audio segment. Fig.15 shown. Fig.152 is a schematic diagram of another embodiment of a method for improving the quality of an image stream containing audio according to the present invention, which comprises the following steps:

[0294] Step 551: A server 501 provides a plurality of original images and audios (the plurality of original images can be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer). Afterwards, the original images and audios are encoded and compressed by an encoder in the server (step 552) to generate a plurality of encoded images and audios. The plurality of encoded images and audios are then transmitted to the client device via the network in the form of a 2D image stream containing audio (step 553). As described above, in another embodiment, the encoded 2D image stream containing audio can also be a video file pre-recorded or pre-stored in a portable memory medium (such as a flash drive, a CD, a hard disk, etc.), and the client device directly reads the video file to obtain the 2D image stream containing audio.

[0295] Step 554: The client device 502 receives and decodes the plurality of encoded images and audios into a plurality of decoded image and audio segments. Furthermore, an AI enhancement module (also referred to as an enhanced neural network model, such as step 555) is used to input the Nth training decoded image and the corresponding audio segment to generate a corresponding enhanced image and audio segment (also referred to as Enhanced_Frame(N) and Enhanced_Audio(N), where N is a positive integer, such as step 556) as the output image and audio. Among them, at least one mathematical expression and its weighted parameters used by the enhanced neural network model in the AI ​​enhancement module are obtained by Fig.14 The same mathematical expressions and their weighting parameters described for the training procedure of the illustrated embodiment.

[0296] In this embodiment, if the neural network is written as a mathematical expression, it will be in the following form:

[0297] X=Decode(Video / Audio stream);

[0298] Y(N)=Network(X(N-2),X(N-1),X(n),W);

[0299] Where X = (X(1), X(2), X(3), ...); where X(N) is the Nth frame in the decompressed video and the corresponding audio segment;

[0300] The Network is a neural network or a machine learning algorithm;

[0301] W is the trained weight parameter;

[0302] Y(N) is the enhanced Nth frame and the corresponding audio segment, namely, Enhanced_Frame(N) and Enhanced_Audio(N).

[0303] See also Fig.16 , is a schematic diagram of another embodiment of the training procedure of the artificial neural network module of the present invention. In the training procedure of this embodiment, the input of the neural network can be a picture with a lower frame rate, and the neural network generates a picture with a higher frame rate, which can improve the frame rate of the video. Fig.16 The training procedure of the illustrated embodiment includes the following steps:

[0304] Step 561: Provide a training video consisting of a plurality of training original images in a training mode; the plurality of training original images may be real-time videos generated by, for example, game software or other applications, or may be pre-recorded or pre-stored in memory. In this embodiment, the plurality of training original images (training video) have a relatively high frame rate (e.g., 60 FPS or higher) and the frames contained therein per second may be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer.

[0305] Step 562: The frame rate of the plurality of original training images (training videos) is reduced by periodically discarding frames. For example, odd or even frames are discarded so that the number of frames per second of the decelerated images is halved (that is, the frame rate is halved, for example, reduced to 30 FPS). The decelerated images can be represented by L_Frame(1), L_Frame(2), ..., L_Frame(M-1), L_Frame(M), where N=2M and M is a positive integer. In other words, the content of L_Frame(M) is the same as Frame(2M), and the 2M-1th frame Frame(2M-1) in the original image will be discarded. Next, the decelerated images are encoded and compressed and then decoded and decompressed to obtain the decoded training decoded images, whose frames per second can be represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(M-1), Decoded_Frame(M), as shown in step 563. In this embodiment, since half of the frames of the original image are discarded, the frame rate is reduced from 60FPS to 30FPS. Therefore, the Mth frame Decoded_Frame(M) of the training decoded image is the result of encoding and decoding the 2Mth frame Frame(2M) of the original image.

[0306] Step 564: The artificial neural network module in the server receives the multiple training decoded images and uses at least one training mathematical expression to process the corresponding Mth training decoded image and one or more previous or subsequent training decoded images one by one, i.e., Decoded_Frame(Mn), ..., Decoded_Frame(M), ..., Decoded_Frame(M+m), to generate two corresponding training output images (also called Predict_Frame(N-1), Predict_Frame(N), where N=2M, as shown in step 565). Where n and m are integers greater than or equal to 0, and when Mn is less than 1, Decoded_Frame(1) is used to replace Decoded_Frame(Mn), and if M+m is greater than 2M, Decoded_Frame(2M) is used to replace Decoded_Frame(M+m) to input into the artificial neural network module for processing. The at least one training mathematical expression includes multiple training weighting parameters. In other words, the artificial neural network module accepts input of several (e.g., three, four or more) training decoded images Decoded_Frame(Mn),…,Decoded_Frame(M),…,Decoded_Frame(M+m) each time and outputs two training output images Predict_Frame(N-1) and Predict_Frame(N), as shown in step 565.

[0307] Step 566: Use the comparison and training module to compare the differences between the training output images Predict_Frame(N-1), Predict_Frame(N) and the corresponding training original images Frame(N-1), Frame(N) provided in step 561, and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly. The multiple training weighting parameters will be adjusted to minimize the differences between the training output images Predict_Frame(N-1), Predict_Frame(N) and the corresponding training original images Frame(N-1), Frame(N). Then, the weights of the neural network are learned by a mathematical optimization algorithm (such as ADAM / SGD / RMSProp) to minimize the error. Each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the at least one training mathematical expression for processing the next training decoded image in step 564. After a predetermined number of training output images are compared with the corresponding training original images, and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters finally obtained will be applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0308] in accordance with Fig.16 The neural network training procedure shown in FIG. 1 generates an output frame by inputting a plurality of frames, and the corresponding embodiment of the method for improving the quality of the image media of the present invention can be as follows: Fig.17 shown. Fig.17 is a schematic diagram of another embodiment of a method for improving the quality of an image stream according to the present invention, which comprises the following steps:

[0309] Step 571: A server 501 provides a plurality of original images with a relatively low frame rate (e.g., 30 FPS) (the plurality of original images can be represented by Frame(1), Frame(2), ..., Frame(M-1), Frame(M), where M is a positive integer, as shown in step 571). The plurality of original images can be real-time videos generated by, for example, game software or other applications, or can be pre-recorded or pre-stored videos in memory. Afterwards, the original images are encoded and compressed by an encoder (step 532) in the server 501 to generate a plurality of encoded images, which are represented by Encoded_Frame(1), Encoded_Frame(2), ..., Encoded_Frame(M-1), Encoded_Frame(M). The plurality of encoded images are then transmitted to the client device via the network in the form of a 2D image stream (step 573).

[0310] Step 574: The client device 502 receives and decodes the plurality of encoded images into a plurality of decoded images, whose frames are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(M-1), and Decoded(M); and extracts one or more training decoded images such as Decoded_Frame(Mn), ..., Decoded_Frame(M), ..., and Decoded_Frame(M+m) from the Mth decoded image and its preceding or succeeding image to proceed to the next step 575. Where n and m are integers greater than or equal to 0, and when Mn is less than 1, Decoded_Frame(1) is used to replace Decoded_Frame(Mn); if M+m is greater than 2M, Decoded_Frame(2M) is used to replace Decoded_Frame(M+m) to input into the artificial neural network module for processing. Afterwards, an AI enhancement module (also called an enhanced neural network model, as shown in step 575) is used to input the Mth training decoded image and one or more previous or subsequent training decoded images, that is, input Decoded_Frame(Mn), ..., Decoded_Frame(M), ..., Decoded_Frame(M+m) to generate two corresponding enhanced images as output images (also called Enhanced_Frame(N-1) and Enhanced_Frame(N), as shown in step 576). In other words, the AI ​​enhancement module will accept the input of several (for example, three, four or more) decoded image frames Decoded_Frame(Mn), ..., Decoded_Frame(M), ..., Decoded_Frame(M+m) each time and output two enhanced image frames Enhanced_Frame(N-1) and Enhanced_Frame(N), and play these enhanced images at a speed of 60FPS, as shown in step 576. Among them, at least one mathematical expression and its weighted parameters used by the enhanced neural network model in the AI ​​enhancement module are obtained by Fig.16 The same mathematical expressions and their weighting parameters described for the training procedure of the illustrated embodiment.

[0311] In this embodiment, if the neural network is written as a mathematical expression, it will be in the following form:

[0312] X=Decode(30FPS Video stream);

[0313] Y(2M-1),Y(2M)=Network(X(Mn),…,X(M),…,X(M+n),W);

[0314] Where X = (X(1), X(2), X(3), ...); where X(M) is the Mth frame in the decompressed video;

[0315] The Network is a neural network or a machine learning algorithm;

[0316] W is the trained weight parameter;

[0317] Y(N) is the Nth frame after enhancement, ie, Enhanced_Frame(N), where N=2M.

[0318] See also Fig.18 , is a schematic diagram of another embodiment of the training procedure of the artificial neural network module described in the present invention. Sometimes the network will drop frames. Through this training procedure, the artificial neural network module can have the function of automatically filling in the predicted frames in advance or filling in the lost frames. In the training procedure of this embodiment, the input of the neural network may not only be one frame, but may be several frames. The information of the previous frames is used to help fill in or predict the lost frames. Fig.18 The training procedure of the illustrated embodiment includes the following steps:

[0319] Step 581: Provide a training video consisting of a plurality of training original images in a training mode; the plurality of training original images may be real-time videos generated by, for example, game software or other applications, or may be pre-recorded or pre-stored in memory. In this embodiment, the frames of the plurality of training original images may be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer.

[0320] Step 582: The plurality of original training images are encoded and compressed by an encoder to form an image stream consisting of a plurality of training encoded images, and then the plurality of training encoded images are decoded and decompressed one by one in sequence by a training decoder to form a plurality of training decoded images, whose frames are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(N-1), and Decoded_Frame(N), respectively, where N is a positive integer.

[0321] Step 583: Assume that the Nth frame Decoded_Frame(N) of the training decoded image is lost or needs to be predicted. At this time, extract one or more frames Decoded_Frame(Nn), ..., Decoded_Frame(N-1) before the Nth training decoded image frame to proceed to the next step 524. Where n is a positive integer and n is less than N.

[0322] Step 584: The artificial neural network module in the server receives the frames of the plurality of training decoded images and uses at least one training mathematical expression to process one or more frames of the corresponding training decoded images before the Nth frame, i.e., Decoded_Frame(Nn), ..., Decoded_Frame(N-1), one by one, to generate a corresponding training output image (also called Predict_Frame(N), as shown in step 585). If a number of consecutive frames are lost or need to be predicted, the frame Predict_Frame(N) of the training output image generated in this step 584 can be used as the Nth frame Decoded_Frame(N) of the training decoded image and sent back to the artificial neural network module for calculating the next frame Predict_Frame(N+1) of the training output image. By analogy, the artificial neural network module can continuously calculate (predict) the subsequent frames of the training output image by continuing to send the frames Predict_Frame(N) and Predict_Frame(N+1) of the training output image it generates back to the artificial neural network module; in other words, in this embodiment, the frames of the several training decoded images before the Nth frame (Decoded_Frame(Nn),…,Decoded_Frame(N-1)) can be input into the artificial neural network module to generate the subsequent frames after the Nth frame (Predict_Frame(N), Predict_Frame(N+1),…,Predict_Frame(N+m)), where m is a positive integer, as shown in step 585.

[0323] Step 586: Use the comparison and training module to compare the differences between the frames Predict_Frame(N), Predict_Frame(N+1),…, Predict_Frame(N+m) of the multiple training output images and the frames Frame(N), Frame(N+1),…, Frame(N+m) of the corresponding training original images provided in step 581, and adjust the multiple training weighting parameters of the at least one training mathematical expression accordingly. The multiple training weighting parameters will be adjusted to minimize the difference between the training output image Predict_Frame(N) and the corresponding training original image Frame(N). Then, the weights of the neural network are learned by a mathematical optimization algorithm (such as ADAM / SGD / RMSProp) to minimize the error. Each time the multiple training weighting parameters are adjusted, the multiple adjusted training weighting parameters will be fed back to the at least one training mathematical expression for processing the next training decoded image in step 584. After a predetermined number of training output images are compared with the corresponding training original images, and a predetermined number of training weighting parameter adjustment procedures are performed, the multiple training weighting parameters finally obtained will be applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

[0324] in accordance with Fig.18 The neural network training program shown in FIG. 1 generates output images of the Nth frame and the subsequent frames by inputting several frames before the Nth frame. The corresponding embodiment of the method for improving the quality of image media of the present invention can be as follows: Fig.19 shown. Fig.19 is a schematic diagram of yet another embodiment of a method for improving the quality of an image stream according to the present invention, which comprises the following steps:

[0325] Step 591: A server 501 provides a plurality of original images (the frames of the plurality of original images can be represented by Frame(1), Frame(2), ..., Frame(N-1), Frame(N), where N is a positive integer, as shown in step 591). The plurality of original images can be real-time videos generated by, for example, game software or other applications, or can be videos pre-recorded or pre-stored in memory. Afterwards, the original images are encoded and compressed by an encoder (step 592) in the server 501 to generate a plurality of encoded images, whose frames are represented by Encoded_Frame(1), Encoded_Frame(2), ..., Encoded_Frame(N-1), Encoded_Frame(N). The plurality of encoded images are then transmitted to the client device via the network in the form of a 2D image stream (step 593).

[0326] Step 594: The client device 502 receives and decodes the plurality of encoded images (step 594) into a plurality of decoded images, whose frames are represented by Decoded_Frame(1), Decoded_Frame(2), ..., Decoded_Frame(N-1), and Decoded(N). Next, assuming that the Nth frame or a number (m) of frames starting from the Nth frame are lost due to network instability during the network transmission process, as shown in step 594, the present invention will extract one or more frames of the training decoded images before the Nth frame, that is, Decoded_Frame(Nn), ..., Decoded_Frame(N-1) to proceed to the next step 595.

[0327] Step 595: Use the AI ​​enhancement module (also known as the enhanced neural network model) to input one or more frames of the training decoded image before the Nth frame, that is, Decoded_Frame(Nn),…,Decoded_Frame(N-1), to generate one or more (m+1) corresponding enhanced images as output images, whose frames are Enhanced_Frame(N),…,Enhanced_Frame(N+m), as shown in step 596). In other words, the AI ​​enhancement module will accept the input of several decoded image frames Decoded_Frame(Nn),…,Decoded_Frame(N-1) each time and output one or more enhanced images Enhanced_Frame(N),…,Enhanced_Frame(N+m) accordingly, as shown in step 596. Among them, at least one mathematical expression and its weighted parameters used by the enhanced neural network model in the AI ​​enhancement module are obtained by Fig.18 The same mathematical expressions and their weighting parameters described for the training procedure of the illustrated embodiment.

[0328] In this embodiment, if the neural network is written as a mathematical expression, it will be in the following form:

[0329] X = Decode (Video stream);

[0330] Y(N),…,Y(N+m)=Network(X(Nn),…,X(n),W);

[0331] Where X = (X(1), X(2), X(3), ...); where X(N) is the Nth frame in the decompressed video, i.e. Decoded_Frame(N);

[0332] The Network is a neural network or a machine learning algorithm;

[0333] W is the trained weight parameter;

[0334] Y(N) is the Nth frame after enhancement, that is, Enhanced_Frame(N).

[0335] Fig. 20 This is another embodiment of the present invention's procedure for processing decoded images in YUV420 format. In the implementation, assuming that each frame has m channels (such as RGB video, m=3), X(N-k+1), ..., X(N-1), X(N) will be merged and overlapped (Concat) according to the channels and regarded as an input of mk channels. If each frame of the original video has m channels and k frames are input, the input can be regarded as mk channels. Based on this principle, Fig. 20 The process of processing the decoded image in YUV420 format according to the present invention includes:

[0336] Step 601: Receive and process a training decoded image in YUV420 color format, wherein the training decoded image includes 3 channels, and n frames are concat-merged (stacked) into 3n channels, where n is a positive integer.

[0337] Step 602: extracting Y-part data from the training decoded image to generate Y-part output data having 1n channels;

[0338] Step 603: extracting UV-part data from the training decoded image, which has 2n channels;

[0339] Step 604: Process the UV-part data of the training decoded image using a neural network with two times magnification to generate UV-part output data with 2n channels (e.g., stride value Stride=2 in transposed convolution; as in step 604);

[0340] Step 605: Use the concatenation function Concat (concatenates) to overlap (merge) the Y-part output data and the UV-part output data;

[0341] Step 606: Receive the superimposed data in step 605 with a multi-layer neural network and generate 3k channels (where k is a constant greater than 1 and a positive integer) accordingly;

[0342] Step 607: Output the data generated in step 606 as the training output image with k frames and 3-channel RGB.

[0343] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. The scope of protection of the present invention shall be subject to the content of the patent application. Any modification and change of the present invention by anyone familiar with the technology may not deviate from the inventive spirit and the scope of the present invention, and is still covered by the content of the patent application of the present invention.

Claims

1. A method for improving the quality of image media, characterized in that: The following steps are involved: A first application is executed in a server; the first application generates a plurality of original images according to at least one instruction; the plurality of original images are encoded and compressed by an encoder in the server to generate a plurality of encoded images; A second application is executed in a client device far from the server; the second application is associated with and cooperates with the first application; thereby, the client device is operated by a user and generates and sends the command to the server; The client device transmits the command to the server via a network, and then receives the plurality of encoded images generated by the server and corresponding to the command via the network; The client device decodes the plurality of encoded images into a plurality of decoded images, and uses an AI enhancement module to enhance the quality of the plurality of decoded images to generate a plurality of enhanced images; wherein the AI ​​enhancement module processes the plurality of decoded images by analyzing at least one mathematical expression obtained in advance by differences between the plurality of decoded images and the corresponding plurality of original images; thereby, the plurality of enhanced images obtained will be visually closer to the plurality of original images than the plurality of decoded images; and The client device outputs the enhanced image to a screen as an output image to be played; The at least one mathematical expression of the AI ​​enhancement module of the client device includes a plurality of weighting parameters; the weighting parameters of the client device are associated with the difference between the decoded image and the corresponding original image; The multiple original images generated by the first application are divided into multiple groups of scenes, each scene includes multiple original images; the multiple original images generated by the server may have drastic changes because they belong to different scenes; The plurality of weighted parameters used by the AI ​​enhancement module in the client device are also divided into a plurality of groups, each group comprising a plurality of the weighted parameters and corresponding to one of the scenarios; The multiple decoded images corresponding to the original images of different scenes will be subjected to image enhancement processing by the same AI enhancement module using the group of weighted parameters corresponding to the scene among the multiple different groups of weighted parameters.

2. The method according to claim 1, characterized in that The multiple different groups of weighted parameters of the client device are pre-stored in the client device; wherein, whenever the scene changes, the group of weighted parameters corresponding to the changed new scene will be applied to the AI ​​enhancement module to generate the enhanced image.

3. The method according to claim 1, characterized in that The weighted parameters of the client device are downloaded from the server to the client device only when the client device executes the second application; wherein, whenever the scene changes, the set of weighted parameters corresponding to the changed new scene will be downloaded from the server to the client device and then applied to the AI ​​enhancement module to generate the enhanced image.

4. The method according to claim 1, characterized in that The at least one mathematical expression of the AI ​​enhancement module is defined by a training procedure executed by an artificial neural network module in the server; the training procedure includes: Step (a): executing the first application in a training mode to generate a plurality of training original images; Step (b): encoding the plurality of original training images into a plurality of coded training images by the encoder; Step (c): decoding the plurality of training coded images into a plurality of training decoded images by a training decoder in the server; Step (d): the artificial neural network module receives the plurality of training decoded images and uses at least one training mathematical expression to process the training decoded images one by one to generate a plurality of training output images; the at least one training mathematical expression comprises a plurality of training weighting parameters; Step (e): using a comparison and training module to compare the differences between the training output image and the corresponding training original image one by one, and adjusting the plurality of training weighting parameters of the at least one training mathematical expression accordingly; the plurality of training weighting parameters are adjusted to minimize the differences between the training output image and the corresponding training original image; each time the plurality of training weighting parameters are adjusted, the plurality of adjusted training weighting parameters are fed back to the at least one training mathematical expression for processing the next training decoded image in step (d); After a predetermined number of comparisons of the training output images with the corresponding training original images and a predetermined number of adjustments of the training weighting parameters, the multiple training weighting parameters finally obtained will be applied in the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

5. The method according to claim 4, characterized in that In step (e), the comparison and training module uses a discriminator to compare the difference between the training output image and the corresponding training original image, and converges the generative adversarial network loss and adjusts the training weighting parameters accordingly; The discriminator of the training and comparison module is trained in the following manner: The training original image includes n channels, wherein n is a positive integer greater than 2; the training decoded image includes m channels, wherein m is a positive integer greater than 2; In step (d), the artificial neural network module processes the m-channel training decoded image and generates an n-channel training output image; the n-channel training output image and the corresponding m-channel training decoded image are combined to generate a plurality of simulated false samples with m+n channels; and the n-channel training original image and the corresponding m-channel training decoded image are combined to generate a plurality of simulated true samples with m+n channels; In step (e), the simulated false samples of the m+n channel and the simulated true samples of the m+n channel are fed back to the discriminator of the comparison and training module for training the discriminator to detect and distinguish the simulated false samples from the simulated true samples.

6. The method according to claim 4, characterized in that In the training mode, the plurality of original training images are high-quality images; the plurality of original high-quality training images are images that meet one or more of the following properties: images with 4K or 8K resolution, 150-degree, 180-degree or 360-degree field of view, 60FPS or 120FPS frame rate, binocular stereo information, visual depth information, infrared or ultraviolet color information, multi-channel audio, 44khz or higher sampling frequency or 24bit or higher resolution; The following steps are further added between step (a) and step (b): image degradation processing; using a degradation module to degrade the plurality of training original image images to reduce their image quality, so as to obtain degraded training original image images that meet one or more of the following properties: reducing resolution by sampling, reducing field of view by cropping, reducing frame rate by discarding frames, extracting binocular stereo information from a single frame by capturing, discarding visual depth information, discarding visual depth information, converting audio into stereo dual channels, reducing sampling frequency, reducing resolution, adding noise, and blurring; In step (b), the plurality of degraded original training images are encoded and compressed by the encoder to form a plurality of the encoded training images.

7. A method for improving the quality of image media, characterized in that: include: Using a client device to receive a plurality of encoded images, the plurality of encoded images are generated by encoding a plurality of original images; The client device decodes the plurality of encoded images into a plurality of decoded images, and the client device uses an AI enhancement module to enhance the quality of the plurality of decoded images to generate a plurality of enhanced images; wherein the AI ​​enhancement module processes the plurality of decoded images by analyzing at least one mathematical expression obtained in advance by differences between the plurality of decoded images and the corresponding plurality of original images; thereby, the plurality of enhanced images obtained will be visually closer to the plurality of original images than the plurality of decoded images; and The client device outputs the enhanced image to a screen as an output image to be played; The at least one mathematical expression of the AI ​​enhancement module of the client device includes a plurality of weighting parameters; the weighting parameters of the client device are associated with the difference between the decoded image and the corresponding original image; The plurality of original images are divided into a plurality of scenes, each scene includes a plurality of the original images; the plurality of original images may have drastic changes because they belong to different scenes; The plurality of weighted parameters used by the AI ​​enhancement module in the client device are also divided into a plurality of groups, each group comprising a plurality of the weighted parameters and corresponding to one of the scenarios; The multiple decoded images corresponding to the original images of different scenes will be subjected to image enhancement processing by the same AI enhancement module using the group of weighted parameters corresponding to the scene among the multiple different groups of weighted parameters.

8. The method according to claim 7, characterized in that The plurality of weighting parameters of the at least one mathematical expression are predefined by a training process executed by an artificial neural network module in a server; The training process is performed in the server and includes: Step (a): providing a plurality of original training images in a training mode; Step (b): encoding the plurality of original training images into a plurality of coded training images by an encoder; Step (c): decoding the plurality of training coded images into a plurality of training decoded images by a training decoder; Step (d): the artificial neural network module receives the plurality of training decoded images and uses at least one training mathematical expression to process the training decoded images one by one to generate a plurality of training output images; the at least one training mathematical expression comprises a plurality of training weighting parameters; Step (e): using a comparison and training module to compare the differences between the training output image and the corresponding training original image one by one, and adjusting the plurality of training weighting parameters of the at least one training mathematical expression accordingly; the plurality of training weighting parameters are adjusted to minimize the differences between the training output image and the corresponding training original image; each time the plurality of training weighting parameters are adjusted, the plurality of adjusted training weighting parameters are fed back to the at least one training mathematical expression for processing the next training decoded image in step (d); Among them, after performing a predetermined number of comparisons of the training output images with the corresponding training original images, and a predetermined number of adjustments to the training weighting parameters, the multiple training weighting parameters finally obtained will be applied to the AI ​​enhancement module of the client device as weighting parameters of its mathematical expression.

9. The method according to claim 8, characterized in that The plurality of different sets of weighted parameters are pre-stored in the client device or in the server; When the plurality of different sets of weighting parameters are pre-stored in the client device, each time the scene changes, the set of weighting parameters corresponding to the changed new scene will be applied to the AI ​​enhancement module to generate the enhanced image; When the multiple different groups of weighted parameters are stored on the server side, whenever the scene changes, the group of weighted parameters corresponding to the changed new scene will be downloaded from the server to the client device and then applied to the AI ​​enhancement module to generate the enhanced image.

10. The method according to claim 8, characterized in that In the training mode, the plurality of original training images are high-quality images; the plurality of original high-quality training images are images that meet one or more of the following properties: images with 4K or 8K resolution, 150-degree, 180-degree or 360-degree field of view, 60FPS or 120FPS frame rate, binocular stereo information, visual depth information, infrared or ultraviolet color information, multi-channel audio, 44khz or higher sampling frequency or 24bit or higher resolution; The following steps are further added between step (a) and step (b): image degradation processing; using a degradation module to degrade the plurality of training original image images to reduce their image quality, so as to obtain degraded training original image images that meet one or more of the following properties: reducing resolution by sampling, reducing field of view by cropping, reducing frame rate by discarding frames, extracting binocular stereo information from a single frame by capturing, discarding visual depth information, discarding visual depth information, converting audio into stereo dual channels, reducing sampling frequency, reducing resolution, adding noise, and blurring; In step (b), the plurality of degraded original training images are encoded and compressed by the encoder to form a plurality of the encoded training images.

Images