Color enhancement model training method and device, image enhancement method and device and electronic equipment
By enhancing the color of the game live broadcast screen and converting the SDR format to HDR format using the color enhancement model, the problem of dark tones in the game live broadcast is solved, and the color expression and user experience of the picture are improved.
Patent Information
- Application Number
- CN202411138750.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
AI Technical Summary
In game live broadcast, many classic games only support standard dynamic range (SDR) format, resulting in the tone of the live broadcast screen being gray, unable to make full use of the color expressiveness of modern display devices, affecting the screen display effect.
By determining the color enhancement parameters based on the SDR and HDR images of the reference game, performing color enhancement, and training the game color enhancement model, it is used to adjust and enhance the target game screen to achieve the conversion from SDR to HDR and improve the color richness of the picture.
It improves the colorful richness and visual effects of the game live broadcast screen, optimizes the user's viewing experience, and enhances the detailed display and overall beauty of the screen.
Smart Images

Figure CN119090789A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a color enhancement model training method, an image enhancement method, a device and an electronic device. Background Art
[0002] In the field of game live broadcast technology, ultra-high-definition live broadcast has become the norm. Modern live broadcast equipment and platforms usually support 1080P, 2K, and even 4K resolutions to provide viewers with extremely clear live videos. However, despite the significant improvement in resolution, the color problem in live broadcasts is often overlooked. In particular, many classic and popular games were produced several years ago and often only support the standard dynamic range (SDR) format. This will cause the overall color tone of the game screen in the live broadcast to be slightly gray, unable to fully display the color expression that modern display devices can provide, affecting the display effect of the game live broadcast screen. Summary of the invention
[0003] In view of this, an object of the present invention is to provide a color enhancement model training method, an image enhancement method, a device and an electronic device.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0005] In a first aspect, the present invention provides a color enhancement model training method, comprising:
[0006] Determining color enhancement parameters of each of a plurality of second standard dynamic range images of a target game according to a plurality of first standard dynamic range images of a reference game and the corresponding first high dynamic range images thereof;
[0007] Performing color enhancement on each second standard dynamic range image according to the color enhancement parameter of each second standard dynamic range image to obtain an initial high dynamic range image corresponding to each second standard dynamic range image;
[0008] Performing color adjustment on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game;
[0009] Based on each second standard dynamic range image and its corresponding second high dynamic range image, a basic color enhancement model is trained to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing the target game screen.
[0010] In an optional embodiment, the step of determining the color enhancement parameters of each of the plurality of second standard dynamic range images of the target game based on the plurality of first standard dynamic range images of the reference game and the corresponding first high dynamic range images thereof comprises:
[0011] According to each first standard dynamic range image and its corresponding first high dynamic range image, a basic color parameter estimation model is trained to obtain a game color parameter estimation model;
[0012] Using the game color parameter estimation model, generating initial color enhancement parameters for each second standard dynamic range image;
[0013] The initial color enhancement parameters of each second standard dynamic range image are used as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
[0014] In an optional embodiment, all second standard dynamic range images are arranged in a set order;
[0015] After the step of generating initial color enhancement parameters of each second standard dynamic range image using the game color parameter estimation model, the color enhancement model training method further includes:
[0016] Using the initial color enhancement parameters of the first second standard dynamic range image as its smoothed color enhancement parameters;
[0017] Performing smoothing processing according to the smoothed color enhancement parameters of the nth second standard dynamic range image and the initial color enhancement parameters of the n+1th second standard dynamic range image, obtaining the smoothed color enhancement parameters of the n+1th second standard dynamic range image, and obtaining the smoothed color enhancement parameters of each second standard dynamic range image; n is a positive integer;
[0018] The smoothed color enhancement parameters of each second standard dynamic range image are used as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
[0019] In an optional implementation, the step of performing color adjustment on each initial high dynamic range image to obtain each second high dynamic range image includes:
[0020] Obtaining a preset brightness parameter range, contrast parameter range, and saturation parameter range that match the screen color requirements of the target game;
[0021] For each initial high dynamic range image, color adjustment is performed on the initial high dynamic range image according to the brightness parameter range, the contrast parameter range, and the saturation parameter range to obtain each second high dynamic range image.
[0022] In a second aspect, the present invention provides an image enhancement method, comprising:
[0023] Use a preset recognition model to perform scene recognition on video images in live video;
[0024] When the scene of the video image is a target game scene, the color of the video image is enhanced by using a preset game color enhancement model to obtain an enhanced video image;
[0025] Among them, the game color enhancement model is obtained according to the color enhancement model training method described in any one of the aforementioned embodiments.
[0026] In an optional implementation, after the step of performing scene recognition on the video image in the live video using a preset recognition model, the method further includes:
[0027] When the scene of the video image is a target game scene, a game screen area and a non-game screen area of the video image are determined using a preset segmentation model;
[0028] The game screen area of the video image is color enhanced using a preset game color enhancement model, and the non-game screen area of the video image is color enhanced using a preset color enhancement algorithm to obtain an enhanced video image.
[0029] In a third aspect, the present invention provides a color enhancement model training device, comprising:
[0030] A parameter determination module, for determining color enhancement parameters of each of a plurality of second standard dynamic range images of a target game according to a plurality of first standard dynamic range images of a reference game and their corresponding first high dynamic range images;
[0031] A color enhancement module, configured to perform color enhancement on each second standard dynamic range image according to the color enhancement parameter of the second standard dynamic range image, so as to obtain an initial high dynamic range image corresponding to each second standard dynamic range image;
[0032] A color adjustment module, configured to perform color adjustment on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game;
[0033] A model training module is used to train a basic color enhancement model based on each second standard dynamic range image and its corresponding second high dynamic range image to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing the target game screen.
[0034] In a fourth aspect, the present invention provides an image enhancement device, comprising:
[0035] A scene recognition module, used to perform scene recognition on video images in live video using a preset recognition model;
[0036] An image enhancement module, used for, when the scene of the video image is a target game scene, using a preset game color enhancement model to perform color enhancement on the video image to obtain an enhanced video image;
[0037] Among them, the game color enhancement model is obtained according to the color enhancement model training method described in any one of the aforementioned embodiments.
[0038] In a fifth aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the color enhancement model training method described in any one of the aforementioned embodiments and / or the image enhancement method described in any one of the aforementioned embodiments are implemented.
[0039] In a sixth aspect, the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the color enhancement model training method described in any one of the aforementioned embodiments and / or the image enhancement method described in any one of the aforementioned embodiments are implemented.
[0040] The color enhancement model training method, image enhancement method, device and electronic device provided by the present invention include: determining the color enhancement parameters of each of the multiple second standard dynamic range images of the target game according to the multiple first standard dynamic range images and the corresponding first high dynamic range images of the reference game; and color enhancing each second standard dynamic range image based on each color enhancement parameter to obtain each initial high dynamic range image; then color adjusting each initial high dynamic range image to obtain each second high dynamic range image that matches the color requirements of the target game screen; training the basic color enhancement model based on each second standard dynamic range image and the corresponding second high dynamic range image to obtain the game color enhancement model, which is used to color enhance the image containing the target game screen. By using the high dynamic range image and the standard dynamic range image of the reference game to obtain the color enhancement parameters of the game, and training to obtain the game color enhancement model for color enhancement of the game live broadcast screen, the game color enhancement model can be used to improve the color richness of the game live broadcast screen, and enhance the visual effect of the game live broadcast screen, thereby optimizing the user's viewing experience.
[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0043] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown;
[0044] Figure 2 A schematic diagram of a process flow of a color enhancement model training method provided by an embodiment of the present invention is shown;
[0045] Figure 3 A schematic diagram of a process of an image enhancement method provided by an embodiment of the present invention is shown;
[0046] Figure 4 An example diagram of an image enhancement method provided by an embodiment of the present invention is shown;
[0047] Figure 5 Another example diagram of the image enhancement method provided by the embodiment of the present invention is shown;
[0048] Figure 6 A functional module diagram of a color enhancement model training device provided by an embodiment of the present invention is shown;
[0049] Figure 7 A functional module diagram of an image enhancement device provided by an embodiment of the present invention is shown.
[0050] Icon: 100 - electronic device; 110 - processor; 120 - memory; 130 - communication module; 300 - color enhancement model training device; 310 - parameter determination module; 330 - color enhancement module; 350 - color adjustment module; 370 - model training module; 400 - image enhancement device; 410 - scene recognition module; 430 - image enhancement module. DETAILED DESCRIPTION
[0051] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0053] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0054] Please refer to Figure 1 , is a block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a processor 110, a memory 120, and a communication module 130, and each component is directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.
[0055] The processor 110 is used to read / write data or programs stored in the memory 120 and execute corresponding functions. It can be a general-purpose processor, including a CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be a DSP digital signal processor, an ASIC application-specific integrated circuit, an FPGA off-the-shelf programmable gate array or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.
[0056] The memory 120 is used to store programs or data. The memory 120 can be a RAM (Random Access Memory), a ROM (Read Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electric Erasable Programmable Read-Only Memory), etc.
[0057] The communication module 130 is used to perform signaling or data communications with other devices.
[0058] Understandably, Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0059] The above-mentioned electronic device 100 will be used as an execution subject to execute each step in each method provided in the embodiment of the present invention and achieve corresponding technical effects.
[0060] See also Figure 2 , is a flow chart of a color enhancement model training method provided in an embodiment of the present invention.
[0061] Step S202, determining color enhancement parameters of each of a plurality of second standard dynamic range images of a target game according to a plurality of first standard dynamic range images of a reference game and their corresponding first high dynamic range images.
[0062] It is understandable that compared with the SDR format, HDR (High Dynamic Range) format images can represent more colors, which can make the overall color of the image richer. For example, HDR10 is an image format that uses 10-bit data to represent color. So when live streaming a game that only supports the SDR format, you can upgrade the live image to an HDR10 format image by color-enhancing the game screen in the live image to improve the richness of the game screen color in the live image.
[0063] For games that only support the SDR format (this type of game is referred to as a target game in the embodiment of the present invention), if the images of these games are to be color enhanced, it is necessary to obtain color enhancement parameters for generating HDR10 images of these games. Then, some SDR images and their corresponding HDR10 images can be collected to obtain the color enhancement rules for converting SDR images into HDR10 images, and the color enhancement rules can be used to obtain the color enhancement parameters of the SDR images of these games.
[0064] Although there are some open source image datasets in HDR format, such as MIT-Adobe FiveK, HDR+BurstPhotography, HDR-SYNTH, HDR-REAL, RAISE, etc., these image datasets usually use 16-bit or 32-bit data to represent colors, and the contents of these image datasets are often natural scenery and portraits, and the colors of natural scenery and portraits are different from those in game scenes, that is, these image datasets are not suitable for game scenes.
[0065] Therefore, the embodiment of the present invention obtains multiple SDR images and their corresponding HDR10 images by recording game videos. Some existing stand-alone games support both SDR format and HDR10 format (the embodiment of the present invention refers to such games as reference games). You can obtain multiple first standard dynamic range images, i.e., first SDR images, by recording the game video when the game is running in SDR format, and obtain multiple corresponding first high dynamic range images, i.e., first HDR10 images by recording the game video when the game is running in HDR10 format. It can be understood that the first standard dynamic range image refers to the SDR image of the reference game, and the first high dynamic range image refers to the HDR10 image of the reference game.
[0066] Then, based on the multiple first standard dynamic range images and their corresponding first high dynamic range images, the color enhancement parameters of the multiple second standard dynamic range images of the target game are determined. It should be noted that in order to ensure accurate color enhancement parameters, the picture content of the first standard dynamic range image and its corresponding first high dynamic range image should be consistent.
[0067] It can be understood that the embodiment of the present invention obtains the first standard dynamic range image and the first high dynamic range image by recording the game video, which can not only reduce the cost of obtaining image data and improve the efficiency of image data acquisition, but also ensure that the color of the obtained image data matches the game scene, thereby providing important data support for improving the color of the live game screen.
[0068] Step S204: performing color enhancement on each second standard dynamic range image according to its color enhancement parameter to obtain an initial high dynamic range image corresponding to each second standard dynamic range image.
[0069] Step S206, color adjustment is performed on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game.
[0070] In this embodiment, for each second standard dynamic range image, the second standard dynamic range image can be color enhanced according to the color enhancement parameters of the second standard dynamic range image to obtain a corresponding initial high dynamic range image, and then an initial high dynamic range image corresponding to each second standard dynamic range image is obtained. It can be understood that the SDR image of the target game is converted into an HDR10 image using the color enhancement parameters.
[0071] It should be understood that each game has different requirements for screen color. For example, for cute-style games, the screen color requires bright colors; for horror-style games, the screen color requires dim colors. Since the color enhancement parameters of the second standard dynamic range image are obtained based on the standard dynamic range image and the high dynamic range image of the reference game, the initial high dynamic range image obtained based on the color enhancement parameter processing matches the color requirements of the reference game. Then, after obtaining the initial high dynamic range image, it is necessary to perform color adjustment on it to obtain a second high dynamic range image that matches the screen color requirements of the target game. It can be understood that the second standard dynamic range image refers to the SDR image of the target game, and the second high dynamic range image refers to the color-adjusted HDR10 image of the target game.
[0072] Step S208, based on each second standard dynamic range image and its corresponding second high dynamic range image, the basic color enhancement model is trained to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing a target game screen.
[0073] In this embodiment, based on each second standard dynamic range image and its corresponding second high dynamic range image, a basic color enhancement model can be trained to obtain a game color enhancement model for color enhancement of an image containing a target game screen.
[0074] It is understandable that the basic color enhancement model is an Image-to-Image black box model. For example, the basic color enhancement model can adopt EDRN (Enhanced Deep Residual Networks for Single Image Super-Resolution), which has efficient image enhancement capability and good generalization performance, and can maintain image details and quality.
[0075] Optionally, in order to meet the high real-time requirements of live video broadcasting and to increase the speed at which the game color enhancement model performs color enhancement on the target game screen in the live broadcast image, the cropped EDRN can be used as the basic color enhancement model for training. For example, the number of residual blocks of the cropped EDRN is reduced to 8, and the number of channels is reduced to 1 / 4 of the original, i.e., 64. In this way, using the cropped EDRN to train the game color enhancement model can reduce the complexity of the model calculation, and can maintain a good image enhancement effect while improving processing efficiency.
[0076] It can be seen that based on the above steps, according to the multiple first standard dynamic range images of the reference game and their corresponding first high dynamic range images, the color enhancement parameters of the multiple second standard dynamic range images of the target game are determined; and each second standard dynamic range image is color enhanced based on each color enhancement parameter to obtain each initial high dynamic range image; each initial high dynamic range image is color adjusted to obtain each second high dynamic range image that matches the color requirements of the target game screen; based on each second standard dynamic range image and its corresponding second high dynamic range image, the basic color enhancement model is trained to obtain a game color enhancement model, which is used to color enhance the image containing the target game screen. By using the high dynamic range image and the standard dynamic range image of the reference game to obtain the color enhancement parameters of the game, and training the game color enhancement model for color enhancement of the game live broadcast screen, the game color enhancement model can be used to improve the color richness of the game live broadcast screen, and enhance the visual effect of the game live broadcast screen, thereby optimizing the user's viewing experience.
[0077] Optionally, for step S202, an embodiment of the present invention provides a possible implementation method.
[0078] Step S202-1: training a basic color parameter estimation model according to each first standard dynamic range image and its corresponding first high dynamic range image to obtain a game color parameter estimation model.
[0079] Step S202-3, using the game color parameter estimation model to generate initial color enhancement parameters for each second standard dynamic range image.
[0080] Step S202-5: Using the initial color enhancement parameters of each second standard dynamic range image as its color enhancement parameters, to obtain the color enhancement parameters of each second standard dynamic range image.
[0081] It is understandable that the method for determining the color enhancement parameters of each second standard dynamic range image can adopt the white box mode of Image-to-Transformation. This mode includes a variety of methods, such as Exposure (exposure adjustment) algorithm, DeepUPE (Deep Universal Photo Enhancement, deep universal image enhancement) model, DeepLPF (Deep Low Pass Filter, deep low-pass filter) algorithm, Image-Adaptive-3D Look-UpTable, i.e. Image-Adaptive-3DLUT (Image Adaptive Three-Dimensional Lookup Table) model, etc. These methods can obtain color enhancement parameters that can be applied to images by learning the color enhancement rules.
[0082] Since the Image-Adaptive-3DLUT model can automatically adjust the color mapping according to the image content, it can make the color enhancement effect of the image more natural and consistent. Therefore, the embodiment of the present invention adopts the Image-Adaptive-3DLUT model to obtain the color enhancement parameters.
[0083] In this embodiment, each first standard dynamic range image and its corresponding first high dynamic range image are used to train the basic color parameter estimation model, i.e., the Image-Adaptive-3DLUT model, so that the Image-Adaptive-3DLUT model learns the color enhancement rules of the pictures in the game scene, and the trained Image-Adaptive-3DLUT model is the game color parameter estimation model.
[0084] Then, for each second standard dynamic range image, the second standard dynamic range image is input into the game color parameter estimation model, and the initial color enhancement parameters of the second standard dynamic range image are generated by using the game color parameter estimation model, and the initial color enhancement parameters are used as the color enhancement parameters, that is, the color enhancement parameters of each second standard dynamic range image are obtained. Then, the second standard dynamic range image is color enhanced according to the initial color enhancement parameters, and the enhanced image obtained is the initial high dynamic range image.
[0085] It is understandable that the game color parameter estimation model is used to estimate the color enhancement parameters of a single image. If the color enhancement parameters output by the game color parameter estimation model are directly used to enhance the color of consecutive video frames in the live scene, the brightness and color of the live video may fluctuate greatly due to the large difference in color enhancement parameters between consecutive video frames, resulting in flickering. This will reduce the stability of the overall video quality and affect the user's viewing experience. Therefore, after step S202-3, the embodiment of the present invention also provides a possible implementation method to solve the problem of flickering caused by large fluctuations in video brightness and color.
[0086] Step S202-7: using the initial color enhancement parameters of the first second standard dynamic range image as its smoothed color enhancement parameters.
[0087] Step S202-11, performing smoothing processing according to the smoothed color enhancement parameters of the nth second standard dynamic range image and the initial color enhancement parameters of the n+1th second standard dynamic range image, obtaining the smoothed color enhancement parameters of the n+1th second standard dynamic range image, and obtaining the smoothed color enhancement parameters of each second standard dynamic range image; n is a positive integer.
[0088] Step S202-13: Using the smoothed color enhancement parameters of each second standard dynamic range image as its color enhancement parameters, to obtain the color enhancement parameters of each second standard dynamic range image.
[0089] In this embodiment, the multiple second standard dynamic range images of the target game are video frames in a game video, that is, the second standard dynamic range images are temporally correlated, so their temporal order in the game video can be set as their arrangement order. In addition, each initial color enhancement parameter includes the same number of color transformation matrices and their corresponding initial weight values. Then, based on adjacent second standard dynamic range images, the initial weight values in the initial color enhancement parameters can be smoothed to obtain smoothed color enhancement parameters.
[0090] For ease of understanding, the following description is made by taking the total number of second standard dynamic range images as n and an initial color enhancement parameter having k initial weight values as an example.
[0091] The k initial weight values in the initial color enhancement parameters of the first second standard dynamic range image are all used as smoothing weight values, that is, the k smoothing weight values of the first second standard dynamic range image are obtained. Then, for the second second standard dynamic range image, the Moving Average method is used, and according to the preset formula, based on the k smoothing weight values of the first second standard dynamic range image and the k initial weight values in the initial color enhancement parameters of the second second standard dynamic range image, the k smoothing weight values of the second second standard dynamic range image are calculated, that is, the k color transformation matrices of the second second standard dynamic range image and their corresponding smoothing weight values are obtained, and the smoothing color enhancement parameters of the second second standard dynamic range image are obtained. And the preset formula is:
[0092]
[0093] in, represents the i-th smoothing weight value of the n+1-th second standard dynamic range image; ω(n+1) i represents the i-th initial weight value of the n+1-th second standard dynamic range image; represents the i-th smoothing weight value of the n-th second standard dynamic range image; α represents the smoothing coefficient and α∈[0,1]. Optionally, the smoothing coefficient α can be set to 0.9.
[0094] It can be understood that, according to the preset formula, based on the k smoothing weight values of the nth second standard dynamic range image and the k initial weight values of the n+1th second standard dynamic range image, the k smoothing weight values of the n+1th second standard dynamic range image are calculated, that is, the k color transformation matrices of the n+1th second standard dynamic range image and their corresponding smoothing weight values are obtained, and then the smoothing color enhancement parameters of the n+1th second standard dynamic range image are obtained.
[0095] In a similar manner as above, each second standard dynamic range image is traversed to obtain the smooth color enhancement parameters of each second standard dynamic range image. The smooth color enhancement parameters of each second standard dynamic range image are used as its color enhancement parameters, thereby obtaining the color enhancement parameters of each second standard dynamic range image.
[0096] It can be understood that the embodiment of the present invention smoothes the initial color enhancement parameters based on two adjacent second standard dynamic range images to obtain smooth color enhancement parameters. The brightness and color of the initial high dynamic range image obtained by processing with the smooth color enhancement parameters are smoother. Therefore, by training the color enhancement model based on the second high dynamic range image obtained from the initial high dynamic range image, the game color enhancement model can ensure a smooth transition of brightness and color between consecutive video frames when performing color enhancement on the live game screen, thereby ensuring the stability and consistency of the live video screen.
[0097] Optionally, for step S206, an embodiment of the present invention provides a possible implementation method.
[0098] Step S206-1, obtaining a preset brightness parameter range, contrast parameter range, and saturation parameter range that match the screen color requirements of the target game.
[0099] Step S206 - 3 , for each initial high dynamic range image, color adjustment is performed on the initial high dynamic range image according to the brightness parameter range, the contrast parameter range, and the saturation parameter range to obtain each second high dynamic range image.
[0100] As mentioned above, after obtaining the initial high dynamic range image, it is necessary to perform color adjustment on it to obtain a second high dynamic range image that matches the screen color requirements of the target game. Then, color adjustment parameters that match the screen color requirements of the target game can be pre-set, and these color adjustment parameters can be used to perform color adjustment on the initial high dynamic range image to obtain the corresponding second high dynamic range image.
[0101] It is understandable that brightness is an important part of picture color adjustment. Many games that only support SDR format contain a lot of low-light environments or night scenes, and the pictures in these scenes are usually dim and difficult to show details. Then the brightness of the initial high dynamic range image can be adjusted according to the preset brightness parameter range that matches the picture color requirements of the target game to increase the brightness of the game picture and make the details in dark scenes clearer.
[0102] Contrast is used to reflect the layering of the picture. You can also adjust the contrast of the initial high dynamic range image according to the preset contrast parameter range that matches the color requirements of the target game to highlight the key elements in the game screen and make the contrast between the characters, objects and scenes in the game screen more obvious. This will make it easier for users to focus on important details in the game and enhance the overall beauty of the picture.
[0103] Saturation is used to reflect the vividness of the color of the picture. Many classic games may not have as rich colors as recent games due to their age. Then you can also adjust the saturation of the initial high dynamic range image according to the preset saturation parameter range that matches the color requirements of the target game to make the game picture more vivid and lively.
[0104] It can be understood that, by adjusting the brightness, contrast and saturation of each initial high dynamic range image respectively according to the brightness parameter range, the contrast parameter range and the saturation parameter range, each second high dynamic range image can be obtained.
[0105] Based on the above obtained game color enhancement model, the embodiment of the present invention also provides an image enhancement method, see Figure 3 , is a flow chart of an image enhancement method provided by an embodiment of the present invention.
[0106] Step S210, using a preset recognition model to perform scene recognition on the video image in the live video.
[0107] Step S212, when the scene of the video image is a target game scene, the color of the video image is enhanced using a preset game color enhancement model to obtain an enhanced video image.
[0108] In this embodiment, a multi-classification model can be pre-trained as a recognition model to perform multi-classification using the recognition model, that is, to identify whether the live video image is a non-game scene or a game scene and which game scene it is. Then, during the live video broadcast, the recognition model can be used to perform scene recognition on the video image in the live video. If the scene of the video image is the target game scene, the game color enhancement model obtained by the above training is used to perform color enhancement on the video image to obtain an enhanced video image.
[0109] It is understandable that the recognition model may also adopt a binary classification model, that is, to identify whether the live video image is a non-game scene or a game scene. If the video image is a game scene, the game corresponding to the game scene is determined, and the game color enhancement model corresponding to the game is used to enhance the color of the video image to obtain an enhanced video image. If the video image is a non-game scene, the color of the video image is enhanced using a preset color enhancement algorithm to obtain an enhanced video image.
[0110] Optionally, when the recognition model is a binary classification model, the recognition model can adopt a structure including seven combination layers, a global average pooling layer (Global Average Pool), a fully connected layer and a normalization layer (Softmax). And each combination layer includes a convolution layer (Convolution) with a convolution kernel size of 3 and a sliding step size of 2 and an activation layer (LeakyRelu). Through the stacking of convolutional neural networks (Convolutional Neural Network) and global average pooling, effective features are extracted to improve the accuracy of recognition.
[0111] For ease of understanding, the present invention provides two example diagrams using the target game being the League of Legends game as an example. Figure 4 , which shows a picture of the original graphics of the League of Legends game; see Figure 5 , which shows the enhanced image obtained by color enhancement of the original image using the game color enhancement model of the embodiment of the present invention. It can be seen that compared with the original image, the enhanced image has richer colors, clearer details, and a more vivid and layered overall image, which greatly enhances the user's visual experience. In addition, since the color-enhanced game live broadcast screen has significant advantages, user acceptance and usage rate have also increased, by about 15%, which fully demonstrates the user's demand and preference for high-quality image quality, and also shows that the present invention has great potential in improving live broadcast quality and attracting users.
[0112] In addition, the present invention can also be combined with high-resolution live broadcast technology to achieve ultra-high-definition live broadcast quality such as 2K HDR and 4K HDR, further improving the overall quality of live broadcast, and users can enjoy an unprecedented visual feast in live broadcast. High resolution plus color enhancement makes the game screen not only clearer, but also more vivid and realistic, so that users feel as if they are in the game world when watching live broadcasts, and every detail is lifelike, greatly enhancing the immersiveness and pleasure of watching.
[0113] It is understandable that in the process of live video broadcasting, most of the live video image may be the game screen, and a small part of the live video image may be the portrait screen of the anchor, game commentator or game contestant. However, the color enhancement requirements of game scenes and portraits are different. Game scenes require high contrast and bright colors to highlight details, while characters require soft colors to maintain natural skin color and realism. In order to perform appropriate color enhancement processing for different types of pictures to ensure the overall display effect of the live picture. Therefore, after step S210, an embodiment of the present invention also provides a possible implementation method.
[0114] Step S214, when the scene of the video image is a target game scene, a preset segmentation model is used to determine a game screen area and a non-game screen area of the video image.
[0115] Step S216, using a preset game color enhancement model to perform color enhancement on the game screen area of the video image, and using a preset color enhancement algorithm to perform color enhancement on the non-game screen area of the video image to obtain an enhanced video image.
[0116] In this embodiment, when the scene of the video image is a target game scene, a preset segmentation model can be used to determine the game screen area and the non-game screen area of the video image. For example, the segmentation model segments the video image to obtain a mask image for distinguishing the game screen area and the non-game screen area.
[0117] Then, the game color enhancement model obtained through the training is used to enhance the color of the game screen area of the video image based on the mask image to obtain a first enhanced image. And a preset color enhancement algorithm, such as a color enhancement algorithm that matches the color requirements of the portrait screen, is used to enhance the color of the non-game screen area of the video image based on the mask image to obtain a second enhanced image. Then, the first enhanced image and the second enhanced image are fused to obtain an enhanced video image.
[0118] Optionally, the segmentation model can adopt a deep learning model that combines the DeepLabV3 segmentation model and ResNet18 as the backbone network. DeepLabV3 is an advanced semantic segmentation model with strong segmentation performance and accuracy. By using ResNet18 as the backbone network, the feature extraction capability of ResNet (residual network) is applied to the DeepLabV3 segmentation model, which can effectively extract image features and achieve high-quality segmentation, thereby improving the efficiency and accuracy of segmentation.
[0119] It can be understood that the embodiment of the present invention divides the game screen area and the non-game screen area in the video image through the segmentation model, so as to adopt different color enhancement strategies for different types of screens. This can not only highlight the details in the game screen, but also ensure the naturalness and realism of the non-game screen such as the portrait screen, and improve the overall display effect of the live screen.
[0120] In order to perform the corresponding steps in the above color enhancement model training method, a method for implementing a color enhancement model training device is given below. Figure 6, is a functional module diagram of the color enhancement model training device 300 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the color enhancement model training device 300 provided in this embodiment are the same as those of the above embodiment. For the sake of brief description, the corresponding contents in the above embodiment may be referred to for the parts not mentioned in this embodiment. The color enhancement model training device 300 includes:
[0121] The parameter determination module 310 is used to determine the color enhancement parameters of each of the plurality of second standard dynamic range images of the target game according to the plurality of first standard dynamic range images of the reference game and the corresponding first high dynamic range images.
[0122] The color enhancement module 330 is used to perform color enhancement on each second standard dynamic range image according to the color enhancement parameter of the second standard dynamic range image to obtain an initial high dynamic range image corresponding to each second standard dynamic range image.
[0123] The color adjustment module 350 is used to perform color adjustment on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game.
[0124] The model training module 370 is used to train the basic color enhancement model based on each second standard dynamic range image and its corresponding second high dynamic range image to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing a target game screen.
[0125] Optionally, the parameter determination module 310 is also used to: train a basic color parameter estimation model according to each first standard dynamic range image and its corresponding first high dynamic range image to obtain a game color parameter estimation model; use the game color parameter estimation model to generate initial color enhancement parameters for each second standard dynamic range image; use the initial color enhancement parameters of each second standard dynamic range image as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
[0126] Optionally, the parameter determination module 310 is further used to: use the initial color enhancement parameters of the first second standard dynamic range image as its smoothed color enhancement parameters; perform smoothing according to the smoothed color enhancement parameters of the nth second standard dynamic range image and the initial color enhancement parameters of the n+1th second standard dynamic range image to obtain the smoothed color enhancement parameters of the n+1th second standard dynamic range image, and obtain the smoothed color enhancement parameters of each second standard dynamic range image; n is a positive integer; use the smoothed color enhancement parameters of each second standard dynamic range image as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
[0127] Optionally, the color adjustment module 350 is also used to: obtain a preset brightness parameter range, contrast parameter range, and saturation parameter range that match the screen color requirements of the target game; for each initial high dynamic range image, perform color adjustment on the initial high dynamic range image according to the brightness parameter range, contrast parameter range, and saturation parameter range to obtain each second high dynamic range image.
[0128] In order to perform the corresponding steps in the above-mentioned image enhancement method, an implementation method of an image enhancement device is given below. Figure 7 , is a functional module diagram of an image enhancement device 400 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the image enhancement device 400 provided in this embodiment are the same as those of the above embodiment. For the sake of brief description, for matters not mentioned in this embodiment, reference may be made to the corresponding contents in the above embodiment. The image enhancement device 400 includes:
[0129] The scene recognition module 410 is used to perform scene recognition on the video image in the live video using a preset recognition model;
[0130] The image enhancement module 430 is used to perform color enhancement on the video image using a preset game color enhancement model to obtain an enhanced video image when the scene of the video image is a target game scene.
[0131] Optionally, the image enhancement module 430 is also used to: when the scene of the video image is a target game scene, determine the game screen area and non-game screen area of the video image using a preset segmentation model; perform color enhancement on the game screen area of the video image using a preset game color enhancement model, and perform color enhancement on the non-game screen area of the video image using a preset color enhancement algorithm to obtain an enhanced video image.
[0132] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the color enhancement model training method and / or the image enhancement method disclosed in the embodiment of the present invention are implemented.
[0133] An embodiment of the present invention further provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the color enhancement model training method and / or the image enhancement method disclosed in the embodiment of the present invention are implemented.
[0134] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0135] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0136] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A color enhancement model training method, characterized in that: include: Determining color enhancement parameters of each of a plurality of second standard dynamic range images of a target game according to a plurality of first standard dynamic range images of a reference game and their corresponding first high dynamic range images; Color-enhance each second standard dynamic range image according to its color enhancement parameter to obtain an initial high dynamic range image corresponding to each second standard dynamic range image; Performing color adjustment on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game; Based on each second standard dynamic range image and its corresponding second high dynamic range image, a basic color enhancement model is trained to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing the target game screen.
2. The color enhancement model training method according to claim 1, characterized in that: The step of determining the color enhancement parameters of each of the plurality of second standard dynamic range images of the target game according to the plurality of first standard dynamic range images of the reference game and the corresponding first high dynamic range images thereof comprises: According to each first standard dynamic range image and its corresponding first high dynamic range image, a basic color parameter estimation model is trained to obtain a game color parameter estimation model; Using the game color parameter estimation model, generating initial color enhancement parameters for each second standard dynamic range image; The initial color enhancement parameters of each second standard dynamic range image are used as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
3. The color enhancement model training method according to claim 2, characterized in that: All second standard dynamic range images are arranged in a set order; After the step of generating initial color enhancement parameters of each second standard dynamic range image using the game color parameter estimation model, the color enhancement model training method further includes: Using the initial color enhancement parameters of the first second standard dynamic range image as its smoothed color enhancement parameters; Performing smoothing processing according to the smoothed color enhancement parameters of the nth second standard dynamic range image and the initial color enhancement parameters of the n+1th second standard dynamic range image, obtaining the smoothed color enhancement parameters of the n+1th second standard dynamic range image, and obtaining the smoothed color enhancement parameters of each second standard dynamic range image; n is a positive integer; The smoothed color enhancement parameters of each second standard dynamic range image are used as its color enhancement parameters to obtain the color enhancement parameters of each second standard dynamic range image.
4. The color enhancement model training method according to claim 1, characterized in that: The step of performing color adjustment on each initial high dynamic range image to obtain each second high dynamic range image comprises: Obtaining a preset brightness parameter range, contrast parameter range, and saturation parameter range that match the screen color requirements of the target game; For each initial high dynamic range image, color adjustment is performed on the initial high dynamic range image according to the brightness parameter range, the contrast parameter range, and the saturation parameter range to obtain each second high dynamic range image.
5. An image enhancement method, characterized in that: include: Use a preset recognition model to perform scene recognition on video images in live video; When the scene of the video image is a target game scene, the color of the video image is enhanced by using a preset game color enhancement model to obtain an enhanced video image; Wherein, the game color enhancement model is obtained according to the color enhancement model training method described in any one of claims 1 to 4.
6. The image enhancement method according to claim 5, characterized in that: After the step of performing scene recognition on the video image in the live video using the preset recognition model, the method further includes: When the scene of the video image is a target game scene, a game screen area and a non-game screen area of the video image are determined using a preset segmentation model; The game screen area of the video image is color enhanced using a preset game color enhancement model, and the non-game screen area of the video image is color enhanced using a preset color enhancement algorithm to obtain an enhanced video image.
7. A color enhancement model training device, characterized in that: include: A parameter determination module, for determining color enhancement parameters of each of a plurality of second standard dynamic range images of a target game according to a plurality of first standard dynamic range images of a reference game and their corresponding first high dynamic range images; A color enhancement module, configured to perform color enhancement on each second standard dynamic range image according to the color enhancement parameter of the second standard dynamic range image, so as to obtain an initial high dynamic range image corresponding to each second standard dynamic range image; A color adjustment module, configured to perform color adjustment on each initial high dynamic range image to obtain each second high dynamic range image; the second high dynamic range image matches the screen color requirement of the target game; A model training module is used to train a basic color enhancement model based on each second standard dynamic range image and its corresponding second high dynamic range image to obtain a game color enhancement model; the game color enhancement model is used to perform color enhancement on an image containing the target game screen.
8. An image enhancement device, characterized in that: include: A scene recognition module, used to perform scene recognition on video images in live video using a preset recognition model; An image enhancement module, used for, when the scene of the video image is a target game scene, using a preset game color enhancement model to perform color enhancement on the video image to obtain an enhanced video image; Wherein, the game color enhancement model is obtained according to the color enhancement model training method described in any one of claims 1 to 4.
9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the color enhancement model training method described in any one of claims 1 to 4 and / or the image enhancement method described in any one of claims 5 to 6 are implemented.
10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the color enhancement model training method described in any one of claims 1 to 4, and / or the image enhancement method described in any one of claims 5 to 6.
