Image processing method and system, target model acquisition method, and device and medium

Abstract

An image processing method and system, a target model acquisition method and apparatus, and a device and a medium, which belong to the technical field of image processing. The image processing method comprises: acquiring a plurality of first images to be processed, wherein the first images use a first color coding format; calling a target model that matches the first color coding format, and processing the plurality of first images, so as to obtain a plurality of images to be transcoded that conform to the first color coding format, wherein a dynamic range of the images to be transcoded is higher than a dynamic range of the first images; on the basis of a second color coding format and the first color coding format, processing the images to be transcoded into second images, wherein the second images conform to the second color coding format; and inputting a plurality of second images into a codec, such that the codec codes the plurality of second images into a high-dynamic-range video.

Description

Image processing method, system, target model acquisition method, device and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 20, 2023, with application number 202310736133.4 and invention name “Image processing method, system, target model acquisition method, device and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, system, target model acquisition method, device and medium. Background Art

[0003] As high dynamic range (HDR) shooting and video display technologies mature, HDR video has gradually developed. Compared to traditional standard dynamic range (SDR) video, HDR video has a greater dynamic range and a wider color gamut, capable of displaying higher contrast and richer colors. Therefore, many consumer-grade ultra-high definition (UHD) TVs have HDR video playback capabilities.

[0004] With the development of HDR video, it is necessary to convert images of different formats into HDR video. In related technologies, images of different formats are generally sent to the codec for conversion and then encoded into HDR video. Under this method, the conversion quality of images of different formats is uneven and the conversion quality is low.

[0005] Overview

[0006] The present disclosure provides an image processing method, the method comprising:

[0007] Acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0008] Invoking a target model that matches the first color coding format to process the plurality of first images to obtain a plurality of images to be transcoded that conform to the first color coding format; wherein a dynamic range of the images to be transcoded is higher than a dynamic range of the first images;

[0009] Based on a second color coding format and the first color coding format, processing the image to be transcoded into a second image, where the second image complies with the second color coding format;

[0010] The plurality of second images are input into a codec so that the codec encodes the plurality of second images into a high dynamic range video.

[0011] In a second aspect of the present disclosure, a method for acquiring a target model is provided, the method comprising:

[0012] Obtaining a video sample and a plurality of first image samples in a first color encoding format;

[0013] Inputting a plurality of the first image samples into a to-be-trained model to obtain image samples to be transcoded output by the to-be-trained model; wherein the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples;

[0014] Based on the first color coding format and the second color coding format, processing the image samples to be transcoded into converted image samples that comply with the second color coding format; wherein the converted image samples are input into a codec to obtain a high dynamic range video;

[0015] Decoding a plurality of second image samples from the video sample based on a target image sample, and updating the parameters of the to-be-trained model based on the target image sample and the second image sample; wherein the target image sample is any one of the to-be-transcoded image sample, the intermediate image sample, and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the to-be-transcoded image into the converted image sample;

[0016] The model to be trained after multiple updates is determined as the target model corresponding to the first color coding format.

[0017] According to a third aspect of the present disclosure, an image processing system is provided, the system comprising: a plurality of target models, different target models corresponding to different first color coding formats;

[0018] an acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0019] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of first images, and obtain a plurality of to-be-transcoded images conforming to the first color coding format; wherein the dynamic range of the to-be-transcoded images is higher than the dynamic range of the first images;

[0020] a post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0021] The codec module is configured to input the second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0022] In an embodiment of the first aspect of the present disclosure, multiple first images to be processed can be obtained. A target model corresponding to a first color coding format can then be called to process the multiple first images, resulting in multiple images to be transcoded in the first color coding format. Subsequently, based on a second color coding format and the first color coding format, the images to be transcoded are processed into second images, and the multiple second images are input into a codec, causing the codec to encode the multiple second images into high dynamic range videos. Because the dynamic range of the images to be transcoded is higher than the dynamic range of the first images, the second images obtained based on the images to be transcoded are also high dynamic range images, thereby enabling the editing of high dynamic range videos.

[0023] In the process of converting images in different first color coding formats into HDR videos, the present disclosure undergoes two processing stages. In the pre-processing stage, the dynamic range of the first image is increased by using a target model that matches the first color coding format. In the post-processing stage, the image to be transcoded is processed into an image that conforms to the second color coding format. The image is then fed into the codec and encoded into HDR video. This can achieve at least the following effects:

[0024] In the pre-processing stage, the processing of the target model is matched with the color encoding format of the first image. Due to the matching between the two, the conversion quality of the dynamic range of the first image by the target model is improved, thereby first images of different color encoding formats can be converted into high-quality images to be transcoded;

[0025] In the post-processing stage, the image to be transcoded output by the target model is processed into a second image that conforms to the second color coding format, and then input into the codec to be encoded as an HDR video. As a result, the codec directly encodes the second image into an HDR video without having to perform internal conversion on images that do not conform to the second color coding format. On the one hand, this reduces the workload of the codec and improves the efficiency of image conversion. On the other hand, since the codec does not need to convert the color coding format, it can avoid the problem of low quality of the encoded HDR video caused by the codec's lack of the ability to convert some of the first color coding formats. This can avoid the problem of uneven conversion quality for images of different formats, so that first images of different color coding formats can all be converted into high-quality HDR videos.

[0026] An embodiment of the second aspect of the present disclosure can obtain video samples and first image sets corresponding to different first color coding formats, and decode the video samples based on a third color coding format to obtain multiple second image samples. Then, the model to be trained is trained with the multiple first image samples as input and the multiple second image samples as supervision to obtain a target model that matches the different first color coding formats; wherein the target image samples are any one of the image samples to be transcoded, the intermediate image samples, and the transcoded image samples belonging to the second color coding format output by the model to be trained during the training process, the intermediate image samples are images obtained in the process of converting the image to be transcoded into the transcoded image samples, the transcoded image samples are used to input the codec to obtain a high dynamic range video, the image samples to be transcoded belong to the first color coding format, and the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples.

[0027] In the embodiment of the second aspect, since a target model matching the first color coding format is trained using first image samples of different first color coding formats, the target model matches the first image samples it processes, so that images of different color coding formats can be converted into high-quality transcoded images by the target model. Furthermore, since the second image samples used for target model supervision are based on any one of the transcoded image samples, intermediate image samples, and transcoded image samples output by the trained model, during the training of the target model, loss calculation can be performed based on the image output by the target model (in the first color coding format), or based on any image of the color coding format obtained during the conversion of the image output by the target model into an image of the second color coding format, or based on the image input to the codec (transcoded image sample). In this way, the processing capability of the target model for the input image can be adapted to the image input to the codec (in the second color coding format), thereby improving the quality of the image input to the codec. Thus, after the quality of the image input to the codec is improved, the quality of the HDR video converted from the images of different first color coding formats is also improved, solving the problem of uneven conversion quality of images of different formats.

[0028] An embodiment of the present disclosure also discloses an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when executed by the processor, the image processing method as described in the first aspect or the target model acquisition method as described in the second aspect is implemented.

[0029] The present disclosure also discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the image processing method described in the first aspect of the present disclosure or the target model acquisition method described in the second aspect of the present disclosure.

[0030] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below.

[0031] BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following is a brief introduction to the drawings required for the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. It should be noted that the scales in the drawings are for illustration only and do not represent the actual scale.

[0033] FIG1 shows a schematic diagram of YUV422 sampling in an embodiment of the present disclosure;

[0034] FIG2 shows a color gamut distribution diagram;

[0035] FIG3 shows a schematic flow chart of steps of an image processing method in one embodiment of the present disclosure;

[0036] FIG4 a shows a schematic diagram of the processing flow of method A in an embodiment of the present disclosure;

[0037] FIG4 b shows a schematic flow chart of steps for processing an input image using a target model in an embodiment of the present disclosure;

[0038] FIG5 a shows a schematic diagram of the processing flow of method B in an embodiment of the present disclosure;

[0039] FIG5 b shows a schematic flow chart of steps for processing an input image using a target model in an embodiment of the present disclosure;

[0040] FIG6 is a schematic diagram showing a flow chart of steps for processing a first image by a target model according to an embodiment of the present disclosure;

[0041] FIG7 exemplarily shows a schematic diagram of the overall flow of an image processing method in a specific example;

[0042] FIG8 is a schematic diagram showing a step flow of a method for acquiring a target model in an embodiment of the present disclosure;

[0043] FIG9 shows a schematic diagram of a strategy for updating parameters of a model to be trained according to an embodiment of the present disclosure;

[0044] FIG10 is a schematic diagram showing another strategy for updating parameters of a model to be trained according to an embodiment of the present disclosure;

[0045] FIG11 is a schematic diagram showing a process of decoding a second image sample in an embodiment of the present disclosure;

[0046] FIG12 shows a schematic diagram of the framework structure of the image processing system according to an embodiment of the present disclosure;

[0047] FIG13 shows a schematic structural diagram of a first target model according to an embodiment of the present disclosure;

[0048] FIG14 shows a schematic structural diagram of a second target model according to an embodiment of the present disclosure;

[0049] Figures 15a-15c are schematic diagrams showing the structures of the second feature extraction branches with different receptive fields in an embodiment of the present disclosure;

[0050] FIG16 shows a schematic structural diagram of an M1×1 layer, an M3×3 layer, and an M5×5 layer in the second feature extraction branch in an embodiment of the present disclosure;

[0051] FIG17 shows a schematic diagram of a framework of a device for acquiring a target model in an embodiment of the present disclosure.

[0052] Detailed description

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0054] In the related art, it may be necessary to convert SDR video into HDR video, or to convert low dynamic range video into high dynamic range HDR video, wherein the color encoding format of the video frame decoded from the video file used for conversion can be RGB format and YUV format. The conversion process is to extract video frames from the video, input the video frames into the model to improve the dynamic range, and then encode them into HDR video. Among them, for video frames in RGB format, the input and output of the model are both RGB format images. The RGB image output by the model will be directly input into the codec, and the codec will convert the RGB format image into a YUV format image, and then encode it into HDR video.

[0055] However, in the RGB to YUV conversion method of the codec tool, its color gamut does not meet the requirements of the BT.2020 color gamut under HDR. As a result, the quality of HDR adaptation converted from RGB format video frames is poor. Compared with the HDR video converted from YUV format video frames, there is a significant difference in display effect.

[0056] In view of this, the present disclosure proposes a solution that can reduce the difference between HDR videos converted from video frames of different color coding formats. The core idea of ​​the solution is: according to the first color coding format of the video frame to be converted, call the adapted target model for processing to improve its dynamic range, then process the image output by the target model according to the second color coding format, obtain the image adapted to the second color coding format, and then send it to the codec to be encoded as an HDR video. In this way, for video frames of different first color coding formats, after the dynamic range is improved, the color coding format when input to the codec is the second color coding format, thus, there is no need for the codec to convert the color coding format of the input image, thereby avoiding the problem of poor HDR video quality caused by the color gamut of the codec not meeting the requirements of the BT.2020 color gamut under HDR when converting images in RGB format. In addition, images of different coding formats can be processed using a matching target model so that the processing power of the model is adapted to the color coding format of the input image, thereby improving the conversion quality of images of different color coding formats, and helping to improve the quality of HDR videos to which images of different color coding formats are converted.

[0057] Based on the above core concept, the present disclosure also provides a solution for reducing the difference between HDR videos converted from video frames in different color coding formats. The solution uses a first image set in different first color coding formats as training samples and a second image set in a third color coding format as supervision to train a target model adapted to different first color coding formats; wherein the third color coding format can be a second color coding format and an intermediate color coding format. In this way, during the model training process, loss calculation can be performed based on the image input to the codec, so that the target model's processing capability for the input image can be adapted to the image input to the codec (in the second color coding format). As a result, the quality of the images that need to be input to the codec after the images in different color coding formats are converted is improved. After the quality of the images input to the codec is improved, the quality of the HDR video encoded thereby is also improved synchronously, thereby solving the problem of uneven conversion quality of images in different color coding formats in the related art from the perspective of model construction.

[0058] The above solution proposed in the present disclosure is described in detail below.

[0059] Before introducing the image processing method of the present disclosure, some technical terms used in the present disclosure are first explained:

[0060] 1. Description of the basic format of HDR (HLG) video, as shown in Table 1 below, shows a schematic table of basic parameters of HDR (HLG) video:

[0061] Among them, sampling - 4:2:2, indicating the YUV422 format; quantization - 10bit, representing the P10 part of YUV422P10; color gamut - ITU-R BT.2020, indicating the BT.2020 color gamut;

[0062] High Dynamic Range – HLG standard, indicating HDR conversion for the HLG standard (along with other HDR standards such as HDR10 and HDR10+);

[0063] 2. Sampling of YUV422

[0064] YUV 4:2:2 sampling means that the U (chrominance) and V (saturation) components are half the size of the Y (luminance) component, with the Y and UV components sampled in a 2:1 ratio. Figure 1 shows a schematic diagram of YUV422 sampling. In Figure 1, the Y component is represented by a cross, and the UV component is represented by a circle. In the example in Figure 1, the Y data size is W × H (W represents width and H represents height), while the U and V data are w / 2 × H.

[0065] 3. BT.709 and BT.2020 color gamut

[0066] Color gamut or color space has the same meaning, both indicating a collection of colors and describing the color coverage range. Referring to FIG2 , a color gamut distribution diagram is shown. As shown in FIG2 , the closer the value is to the edge of the horseshoe shape, the higher the color saturation is. The color gamut close to the edge of the horseshoe shape is the R.2020 color gamut.

[0067] Table 1 - Basic parameters of HDR (HLG) video

[0068] In conjunction with the above technical terms, in one embodiment, an image processing method of the present disclosure is introduced. As shown in FIG3 , a schematic flow chart of the steps of the image processing method in one embodiment is shown. As shown in FIG3 , the method may specifically include the following steps:

[0069] Step S301: Acquire a plurality of first images to be processed.

[0070] The first image adopts a first color coding format.

[0071] In this embodiment, multiple first images can be decoded from the video to be processed. The video to be processed can be an SDR (Standard Dynamic Range) video or a video in a non-standard HDR video format. The multiple first images can be video frames in the video to be processed. In practice, each video frame in the video to be processed can be decoded as a first image.

[0072] Among them, the color coding format can represent the color coding method of the image. Generally speaking, the color coding formats of videos encoded using different color coding methods are different. Among them, the color coding formats of the video include RGB format, YUV format, etc. Different color coding formats are encoded in different color spaces. For example, for the RGB format, its color space is RGB space, that is, the value of the pixel in the image includes the value of the red component, the value of the green component and the value of the blue component; for the YUV format, its color space is YUV space, that is, the value of the pixel in the image includes the value of the brightness component, the value of the chroma component and the value of the saturation component.

[0073] In this embodiment, the first color encoding format may include an RGB format and a YUV format, wherein the first color encoding format of the first image may be an RGB format or a YUV format.

[0074] Step S302: calling a target model that matches the first color coding format, processing the plurality of first images, and obtaining a plurality of images to be transcoded that conform to the first color coding format;

[0075] The dynamic range of the image to be transcoded is higher than the dynamic range of the first image.

[0076] Among them, different first color coding formats can match different target models, such as the first image in YUV format can match the target model for YUV image processing, and the first image in RGB format can match the target model for RGB image processing, so that the target model and the color space of the first image to be processed are adapted, that is, the target model performs targeted processing on the first image within its color space.

[0077] The target model is used to improve the dynamic range of the first image.

[0078] Multiple first images can be batch-inputted into the target model, thereby simultaneously obtaining transcoded images corresponding to each of the multiple first images. After processing by the target model, transcoded images with a higher dynamic range can be obtained. The transcoded images are also in the first color encoding format. In other words, processing by the target model does not change the color encoding format of the first images, but improves their dynamic range.

[0079] Furthermore, during the process of processing the first image, the target model may not change the resolution and data storage format of the first image.

[0080] Step S303: Based on the second color coding format and the first color coding format, the image to be transcoded is processed into a second image, where the second image complies with the second color coding format.

[0081] The second color coding format may be a color coding format that complies with an HDR video encoding method, such as a YUV format. Thus, the first color coding format and the second color coding format may be the same or different. In the case where they are the same, that is, the first image is in a YUV format and the second image is also in a YUV format, when the image to be transcoded is processed into the second image based on the second color coding format and the first color coding format, the image to be transcoded may be converted into a target data storage format under the second color format, such as a YUV422 format or a YUV444 format.

[0082] When the first color coding format and the second color coding format are different, such as when the first image is in RGB format and the second image is in YUV format, the image to be transcoded can be processed into a second image according to the second color coding format based on the second color coding format and the first color coding format. The second image can be in YUV422 format or YUV444 format.

[0083] Step S304: inputting the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0084] After the image to be transcoded is converted into a second image, the second image may be directly input into a codec, wherein the codec may be an ffmpeg encoding tool, or other types of codecs, which are not particularly limited herein.

[0085] For example, using the image processing method of this embodiment, when the first input image is a video frame extracted from an SDR video, the HDR video obtained by processing is a video with a high dynamic range, that is, the SDR video is converted into an HDR video; when the first input image is a video frame extracted from a non-standard HDR video, the HDR video obtained by processing is a video with a high dynamic range, that is, the non-standard video is converted into a standard HDR video.

[0086] According to the technical solution of the embodiment of the present disclosure, on the first aspect, when the target model is used to process the first image, since the color coding format of the target model matches that of the first image, the conversion quality of the dynamic range of the first image by the target model can be improved, thereby first images of different color coding formats can be converted into high-quality images to be transcoded.

[0087] Secondly, after obtaining the image to be transcoded output by the target model, the image to be transcoded is further processed into a second image that conforms to the second color coding format. Thereafter, the second image is input into the codec to be encoded into an HDR video, so that the codec no longer needs to perform color coding conversion on the image that does not conform to the second color coding format. Therefore, the problem of low quality of the encoded HDR video caused by the codec not having the ability to convert the first color coding format into the second color coding format can be avoided, thereby improving the conversion quality of images of different formats, so that first images of different color coding formats can all be converted into high-quality HDR videos.

[0088] In one embodiment, before a first image is input into a target model for processing, the first image may be preprocessed based on a first color coding format to reduce the amount of image data input into the target model, thereby reducing the computational complexity of the target model and improving image processing efficiency. Alternatively, to ensure the quality of the processed image to be transcoded, the resolution of the first image may be increased. Specifically, a preprocessing procedure corresponding to the first color coding format may be determined, and then the preprocessing may be performed on the first image to obtain the image to be input.

[0089] Different first color coding formats correspond to different preprocessing, the computational complexity corresponding to the image to be input is smaller than that corresponding to the first image, or the resolution corresponding to the image to be input is larger than that corresponding to the first image.

[0090] Accordingly, when the target model is used to process the first image to obtain the image to be transcoded, the target model can be used to process the input image to obtain the image to be transcoded.

[0091] The case where the computational complexity of the image to be input is smaller than the computational complexity of the first image includes the case where the resolution of the image to be input is smaller than the resolution of the first image, and the values ​​of each pixel in the image to be input are limited to a smaller range. If the computational complexity of the image to be input is smaller than the computational complexity of the first image, the data volume of the image to be input can be smaller. In practice, the data volume of the image to be input can be reduced by converting the data storage method of the first image.

[0092] When the first image is in YUV format, its data storage methods include YUV422, YUV444, and YUV420. Different data storage methods correspond to different data volumes. Among them, the data volume of YUV444 is larger than that of YUV422, and the data volume of YUV422 is larger than that of YUV420. For example, if you want to reduce the data volume of the first image in YUV444 format,

[0093] At least one of the Y component, the U component, and the V component may be downsampled, such as the U component and the V component.

[0094] When the first image is in RGB format, the amount of computation for the first image can be reduced by reducing the RGB values ​​of the pixels. For example, the RGB values ​​of each pixel in the first RGB format image can be reduced by narrowing the RGB values ​​to a certain range to reduce the amount of computation required for the calculation. Generally, the RGB value range is 0 to 255, so the RGB value can be limited to 0 to 10, thereby reducing the amount of computation.

[0095] When the resolution of the input image is greater than the resolution of the first image, the quality of the processed image to be transcoded can be guaranteed, and the problem of low image quality caused by missing image information can be avoided. Specifically, the resolution of the first image can be increased by using methods such as nearest interpolation, bilinear interpolation, and bicubic interpolation to achieve the resolution improvement.

[0096] For example, for the first image in YUV420 format, the data size of the luminance channel is W×H (W represents width and H represents height), and the data size of the chroma and saturation channels are both W / 2×H / 2. The data belonging to the chroma channel in the first image can be stretched 2 times in the dimension of height H, and the data belonging to the saturation channel in the first image can be stretched 2 times in the dimension of height H, so that the height H / 2 is stretched to H, and the input image to be obtained in YUV422 format is obtained, and the resolution of the input image to be input is higher than the resolution of the first image.

[0097] In particular, while the computational complexity of the input image to the target model is small, it is also necessary to ensure that the input image does not lose much original information, so as to ensure the processing quality of the first image by the target model, and thus ensure the quality of the converted HDR video. Therefore, in an optional example of this embodiment, when the first color encoding format is a YUV format, the preprocessing may include: sampling the data belonging to different color channels in the first image separately to reduce or increase the resolution of the first image;

[0098] When the first color encoding format is RGB format, the preprocessing may refer to normalizing the RGB value of each pixel in the first image so that the RGB value of each pixel in the input image ranges from 0 to 1.

[0099] Among them, the resolution of the image to be input is the target resolution. When the resolution of the first image is greater than the target resolution, the data belonging to the brightness channel in the first image can be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel can be down-sampled to obtain the image to be input; when the resolution of the first image is less than the target resolution, the data belonging to the brightness channel in the first image can be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel can be up-sampled to obtain the image to be input.

[0100] For example, taking the first color format as YUV format and the target resolution as the resolution in YUV422 format as an example, if the first image is in YUV420 format, it does not meet the requirements of YUV422 and needs to be processed into a YUV422 image. Then, the data belonging to the chroma channel in the first image can be stretched 2 times in the dimension of height H, and the data belonging to the saturation channel in the first image can be stretched 2 times in the dimension of height H, that is, the height H / 2 is stretched to H; wherein, a public upsampling algorithm can be used, such as nearest neighbor interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic), so that the format of the image to be input is finally made into YUV422 format.

[0101] As another example, taking the first color format as YUV format and the target resolution as the resolution in YUV422 format, the resolution of the first image is in YUV444 format, which does not meet the requirements of YUV422 and needs to be processed into a YUV422 image. Then, the data belonging to the chroma channel in the first image can be downsampled by 2 times in the dimension of width W, and the data belonging to the saturation channel in the first image can be downsampled by 2 times in the dimension of width W, that is, the width W is downsampled to W / 2; wherein, a public upsampling algorithm can be used, such as nearest neighbor interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic).

[0102] Wherein, when the first color encoding format is RGB format, the RGB value of each pixel in the first image can be normalized so that the RGB value of each pixel in the input image is in the range of 0 to 1. Since the first image in RGB format is derived from an SDR video, after decoding according to the RGB data format, if the quantization value of the SDR video is dep, its RGB value range is [0, 2dep-1 ] For example, in the BT709 color space, if the quantization is 8 bits, the value range is [0, 255]. Then it can be normalized according to the following formula (1):

[0103] In formula (1), RGB_norm is the normalized RGB value of the pixel in the input image, and RGB is the RGB value of the pixel in the first image. The value range of RGB_norm is [0, 1], and the value range of RGB is [0, 255].

[0104] It should be noted that the RGB value corresponding to each pixel in the first image and the image to be input includes: the pixel value in the R (red) channel, the pixel value in the G (green) channel, and the pixel value in the B (blue) channel.

[0105] By adopting the above technical solution, after the RGB values ​​of each pixel in the first image in RGB format are normalized, the amount of calculation of the first image in the target model can be reduced, thereby improving efficiency; after the resolution of the first image in YUV format is improved, the first image input to the target model can contain more image features, thereby improving the quality of the processed image to be transcoded; after the resolution of the first image in YUV format is reduced, the amount of calculation corresponding to the first image input to the target model can be reduced, thereby improving the processing efficiency of the target model.

[0106] Of course, in another example of this embodiment, the method of reducing the amount of calculation of the first image may also be: reducing the range of values ​​of each pixel in the first image, or reducing the size of a channel in the color space.

[0107] For example, regardless of whether the first image is in YUV format or RGB format, the value of each pixel in the preprocessed image to be input is in the range of 0 to 1, or, regardless of whether the first image is in YUV format or RGB format, the preprocessed image to be input is in the color space corresponding to the first color coding format, and the size of the first channel is twice the size of the second channel and the size of the third channel.

[0108] For example, for the first image sample in YUV format, the size of the brightness channel is twice the size of the color channel and the size of the saturation channel, that is, the YUV422 format; for the first image sample in RGB format, the size of the green channel can be twice the size of the blue channel and the size of the red channel. Of course, the first channel, second channel and third channel can be selected in RGB according to actual needs, which will not be repeated here.

[0109] Among them, when the first color encoding format is the YUV format, the range of the value of each pixel point can be narrowed, such as converted to the range of 0 to 1; when the first color encoding format is the RGB format, the range of the value of each pixel point can also be narrowed, such as converted to the range of 0 to 1.

[0110] For another example, when the first color encoding format is RGB format, the size of a certain color channel of the first image can be reduced while retaining the sizes of other color channels, such as reducing the size of the R channel while retaining the sizes of the G channel and the B channel. However, this may cause partial distortion of the image color.

[0111] When the first color encoding format is a YUV format, the size of the color channel and the size of the saturation channel can be reduced, while the size of the brightness channel can be retained. This reduces the computational complexity while preserving the original color information of the first image. Specifically, in one example, as described above, the data belonging to the color channel and the data belonging to the saturation channel can be downsampled to reduce the size of the color channel and the size of the saturation channel. After preprocessing, the size of the brightness channel is twice the size of the color channel and the size of the saturation channel.

[0112] As described above, depending on the format of the video file from which the first image originates, the first color encoding format of the first image may be YUV format or RGB format. Different target models are used to improve the dynamic range of the first image for different color encoding formats.

[0113] The following describes in detail the processing of the first image in two color coding formats.

[0114] #1, the first color encoding format is YUV format.

[0115] When the first image is in YUV format, the first image includes a Y component, a U component, and a V component, that is, data of a luminance channel, data of a color channel, and data of a saturation channel. Since the first image in YUV format is preprocessed before being input into the target model, in one example, the preprocessing may be used to increase the resolution of the first image or to reduce the resolution of the first image. When reducing the resolution, the data belonging to the luminance channel may be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel may be downsampled.

[0116] Accordingly, the image to be input may include a first sampling image belonging to a brightness channel, a second sampling image belonging to a color channel, and a third sampling image belonging to a saturation channel, and the size of the first sampling image is larger than the second sampling image and the third sampling image.

[0117] In the case where the input image includes a first sample image, a second sample image, and a third sample image, since the sizes of the first sample image, the second sample image, and the third sample image are different, there are two ways to process the first image in YUV format using the target model:

[0118] In one method A, when the target model (target model M1-2 described in subsequent embodiments) processes the first sampling image, the second sampling image, and the third sampling image, the three are treated as feature maps of different feature channels. This method B can reduce the complexity of the target model and is beneficial to the development of the target model.

[0119] In another method B, the target model (the target model M1-1 described in subsequent embodiments) can use the corresponding sub-model to process the first sampling image, the second sampling image and the third sampling image respectively, and then splice the images output after the respective processing to obtain the image to be transcoded. This method A can reduce the computational complexity of the target model for image processing and improve processing efficiency.

[0120] Specifically, referring to FIG. 4a and FIG. 4b , FIG. 4a shows a schematic diagram of a processing flow of mode A, and FIG. 4b shows a schematic diagram of a flow of steps for processing an input image using a target model. As shown in FIG. 4a and FIG. 4b , the following steps may be specifically included:

[0121] Step S401: stitching the second sampling image and the third sampling image to obtain a stitched image having the same size as the first sampling image;

[0122] Among them, the format of the image input to the target model can be YUV422 format. In this case, the size of the first sampling image of the luminance channel is W×H, the second sampling image belonging to the color channel, and the third sampling image belonging to the saturation channel are W / 2×H respectively; the second sampling image and the third sampling image can be spliced ​​in the width dimension, so that the size of the spliced ​​image is W×H, which is the same as the size of the first sampling image.

[0123] Step S402: inputting the first sampled image and the stitched image into a target model, where the target model is used to increase the dynamic ranges of the first sampled image and the stitched image respectively.

[0124] Since the first sampling image and the stitched image have the same size, the target model can regard the first sampling image and the stitched image as two images in different feature spaces for parallel processing, such as parallel convolution to extract features, and then reconstruct the extracted feature maps in parallel to obtain the predicted images corresponding to the first sampling image and the stitched image, as shown in Figure 4a, including predicted image 1, predicted image 2 and predicted image 3. In practice, the predicted image exists in the target model in the form of data, such as a data matrix.

[0125] Among them, the predicted image 1 can be data of the brightness channel, the predicted image 2 can be data of the chrominance channel, and the predicted image 3 can be data of the saturation channel.

[0126] Step S403: obtaining an image to be transcoded based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

[0127] Since the target model does not change the resolution of the input image when processing it, the size of the predicted image corresponding to the first sampling image is W×H, and the size of the predicted image corresponding to the spliced ​​image is also W×H. Since the spliced ​​image is obtained by splicing the second sampling image and the third sampling image in the width dimension, the predicted image 2 and the predicted image 3 corresponding to the spliced ​​image can be fused with the predicted image 1 corresponding to the first sampling image to form the image shown in Figure 1, thereby obtaining the image to be transcoded.

[0128] By adopting the processing method of this embodiment, since the target model only needs to process the first sampling image and the stitched image in parallel to obtain the image to be transcoded, its model structure can be designed to be relatively simple, thereby reducing the software and hardware resources occupied by the target model and improving operating efficiency.

[0129] Specifically, referring to Figures 5a and 5b, Figure 5a shows a schematic diagram of the processing flow of method B, and Figure 5b shows a schematic diagram of the step flow of processing the input image using the target model. As shown in Figure 5a, the target model includes a first sub-model and a second sub-model, wherein the first sub-model is used to process the first sampled image of the luminance channel, and the second sub-model is used to process the second sampled image of the color channel and the third sampled image belonging to the saturation channel.

[0130] As shown in FIG5a and FIG5b, the method may specifically include the following steps:

[0131] Step S501: using a first sub-model to increase the dynamic range of a first sampled image to obtain a first predicted image.

[0132] Step S502: using the second sub-model to increase the dynamic range of the second sampled image and the third sampled image, to obtain a second predicted image corresponding to the first predicted image and the second sampled image, and a third predicted image corresponding to the third sampled image.

[0133] Step S503: obtaining an image to be transcoded based on the first predicted image, the second predicted image, and the third predicted image.

[0134] In this embodiment, the image input to the target model can be in YUV422 format. In this case, the size of the first sampled image of the luminance channel is W×H, and the size of the second sampled image belonging to the color channel and the third sampled image belonging to the saturation channel are W×H / 2. Since the sizes of the first sampled image and the second sampled image are different, and also different from the size of the third sampled image, while the sizes of the third sampled image and the second sampled image are the same, the first sampled image can be input into the first sub-model, and the dynamic range of the first sampled image can be processed by the first sub-model. Furthermore, the second and third sampled images can be input into the second sub-model, and the dynamic range of the second sampled image can be processed by the second sub-model, thereby obtaining the first predicted image output by the first sub-model, and the second and third predicted images output by the second sub-model.

[0135] Among them, the sizes of the second predicted image and the third predicted image output by the second sub-model are W / 2×H respectively, and the size of the first predicted image is W×H. In practice, the second predicted image and the third predicted image can be spliced ​​in the width dimension and then fused with the first predicted image to obtain the image to be transcoded.

[0136] It should be noted that the formats of the to-be-transcoded images obtained by the above two processing methods can both be YUV422 format.

[0137] By adopting the processing method of this embodiment, since the target model can process the first sampling image, the second sampling image and the third sampling image of different sizes respectively through the first sub-model and the second sub-model, compared with the solution of method A, the amount of data processed by each sub-model when processing each sampling image in parallel is reduced, thereby improving the calculation speed and thus improving the processing efficiency of each sub-model.

[0138] #2, the first color encoding format is RGB format.

[0139] When the first color encoding format is RGB, the first image is an RGB image. The values ​​of each pixel in the preprocessed input image can be in the range of 0 to 1. In this case, the input image can be directly input into the target model. The target model can extract and process the data in the three RGB color channels of the input image and reconstruct it into a high dynamic range image to be transcoded. For details, please refer to the subsequent target model processing example.

[0140] The structure of the target model adapted to the RGB format may be the same as the structure of the target model adapted to the YUV format in method A.

[0141] It should be noted that when the input image to the target model is in RGB format and the RGB value of each pixel is normalized to a range between 0 and 1, the image to be transcoded output by the target model is also in RGB format, and the RGB value of each pixel is normalized to a range between 0 and 1.

[0142] In some embodiments, when the target model processes the first image, in order to improve the accuracy of the target model's processing of the first image, retain the feature information of the first image while increasing the dynamic range, the target model can extract global, large area local, small area local, low-frequency area and other features of the image feature domain under receptive fields of different sizes when extracting image features, thereby extracting features with a high degree of expression for use in reconstructing the dynamic range.

[0143] Specifically, referring to FIG6 , a schematic flow chart of steps for processing the first image by the target model is shown. As shown in FIG6 , the steps may include:

[0144] Step S601: extract features from the first image to obtain an initial feature map.

[0145] Step S602: performing feature extraction on the initial feature map under receptive fields of different sizes to obtain multiple feature maps;

[0146] Among them, different feature maps correspond to receptive fields of different sizes.

[0147] Step S603: performing feature fusion on multiple feature maps to obtain a first fused feature map.

[0148] Step S604: reconstructing features of the first fused feature map to obtain an image to be transcoded.

[0149] In this embodiment, convolution processing can be performed on the first image to extract image features from the first image to obtain an initial feature map. Specifically, a 3x3 convolution layer can be used for convolution processing, and the feature channels can be expanded from 3 channels to nf channels (nf can be 64, 48, 32 or other).

[0150] Next, feature extraction is performed on the initial feature map at different scales. Different scales can be understood as receptive fields of different sizes, where different receptive fields use different convolution kernel sizes. By performing feature extraction on the initial feature map at different scales, subtle features, local detail features, and global features in the first image can be fully extracted.

[0151] In one example, three or more receptive fields of different sizes can be used to extract features from the initial features, thereby gradually transitioning from subtle details to global features to extract features on the first image in each transition process, thereby fully extracting features from the first image.

[0152] Next, feature fusion can be performed on multiple feature maps. When performing feature fusion, multiple feature maps can be first spliced ​​in the channel dimension, and then the spliced ​​feature maps are convolved using a 1×1 convolution kernel to obtain a first fused feature map. In this way, the first fused feature map can fuse features extracted under different receptive fields, thereby enhancing the degree of expression of the first image by the first fused feature map.

[0153] Afterwards, feature reconstruction is performed on the first fused feature map, wherein reconstructing the first fused feature map may refer to reconstructing each pixel in the first fused feature map under a high dynamic range, thereby obtaining an image to be transcoded.

[0154] It should be noted that, whether it is a target model for YUV format or a target model for RGB format, the target model's processing process of the first image can include the above-mentioned steps S601 to S604. The difference is that the model parameters used for feature extraction of the first image in different first color coding formats are different, that is, the model parameters of the target models for different first color coding formats can be different.

[0155] By adopting the above processing method, by performing feature extraction at different scales on the first image, the image features extracted under different receptive fields can be obtained, and then the subtle features, local detail features and global features of the first image can be extracted, thereby enhancing the expression degree of the first image by the first fusion feature map. As a result, the reconstructed transcoded image has high quality, can fully reflect the image features of the first image, and can ensure a high dynamic range, thereby improving the quality of subsequent HDR videos.

[0156] In some examples, the target model may further extract low-frequency region features from the first image, that is, features of low-frequency information. Specifically, after performing feature extraction on the first image to obtain an initial feature map, features belonging to low-frequency information in the initial feature map may be extracted based on a preset feature map to obtain a low-frequency feature map; and, feature extraction may be performed on the low-frequency feature map under the target receptive field to obtain a guided filtering feature map.

[0157] Correspondingly, after obtaining the first fused feature map, the guided filter feature map and the first fused feature map can be fused to obtain a second fused feature map. When performing feature reconstruction, the second fused feature map can be reconstructed to obtain the image to be transcoded.

[0158] The target receptive field is the receptive field of the receptive fields of different sizes except the receptive field with the largest size. Specifically, it can be the receptive field with the smallest size.

[0159] The preset feature map may refer to a feature map of a guide image. Specifically, the guide image may be the first image itself, a different version of the first image, or an image completely different from the first image. Similar to filtering operations, when extracting features from the initial feature map, the preset feature map influences the statistics of a region in the corresponding spatial neighborhood of the initial feature map, thereby preserving the features of the region specified in the initial feature map. For example, edge features may be preserved.

[0160] The guided filter may be used to perform guided filtering on the initial feature map to obtain a guided filtered feature map.

[0161] For example, assume that the preset feature map is the feature map of the guide image I, the input is the first image p, and the input is the image q (the guide filter feature map). The guide image I is set in advance according to the specific application, and can also be directly taken as the input image p. For the i-th pixel in the output image q, its calculation method can be expressed as the following formula (2):

[0162] In formula (2), i and j are pixel labels, W ij is the filter kernel function, which is defined by the following formula (3):

[0163] In formula (3), w k is the kth kernel function window, w is the number of pixels in the window, μ k and are the mean and variance of the guided image I within the window, and ε is the smoothing factor.

[0164] By using guided filtering to extract features from the initial feature map, effects such as noise reduction, detail smoothing, HDR compression, cutout, defogging, and joint sampling can be achieved, thereby improving the quality of feature extraction of the first image and thus improving the quality of the image to be transcoded.

[0165] Among them, after obtaining the guided filtering feature map, the guided filtering feature map and the first fusion feature map can be fused, so that the features extracted after denoising, detail smoothing, HDR compression, cutout, defogging or joint sampling of the first image are fused into the first fusion feature map, thereby making the second fusion feature map obtained after fusion also have the effects of denoising, detail smoothing, HDR compression, cutout, defogging or joint sampling.

[0166] In an optional fusion, the values ​​of each pixel in the guided filtering feature map and the first fused feature map can be weighted and summed according to preset weights. For example, the guided filtering feature map can correspond to a weight, and the first fused feature map can also correspond to a weight. Then, the two are weighted and summed to achieve the fusion of the two feature maps.

[0167] This approach allows for targeted fusion of the features of the guided filter feature map and the first fused feature map based on their respective importance, thereby improving the pertinence of image processing. For example, if the weight of the first fused feature map is large, the focus of characterization should be on the degree of feature expression of the first image; if the weight of the guided filter feature map is large, the focus of characterization should be on the clarity of feature expression of the first image (smoothing and denoising are both for improving clarity).

[0168] In an optional example of this embodiment, when the guided filtering feature map and the first fused feature map are fused, the values ​​of each pixel point in the guided filtering feature map can be used as weights to update the values ​​of each pixel point in the first fused feature map. Specifically, the values ​​of each feature point in the guided filtering feature map can be mapped to a target interval to obtain a mapped guided filtering feature map, and based on the values ​​of each feature point in the mapped guided filtering feature map, the values ​​of each feature point in the first fused feature map can be adjusted to obtain a second fused feature map; wherein the target interval ranges from 0 to 1.

[0169] In this example, the values ​​of each feature point in the guided filtering feature map can be normalized so that the value range of its feature points is 0 to 1. In this way, the normalized value of each feature point in the guided filtering feature map can be multiplied by the value of each feature point in the first fused feature map to adjust the value of each feature point in the first fused feature map to obtain a second fused feature map.

[0170] Using this method, the feature points in the first fused feature map can be adjusted according to the guided filtering feature map, so that the first fused feature map can be repaired using the guided filtering feature map, and then the deep fusion between the guided filtering feature map and the first fused feature map can be achieved. The second fused feature map after fusion can be a feature map with better clarity and stronger feature expression.

[0171] After processing through the target model, a to-be-transcoded image with a higher dynamic range can be obtained. The to-be-transcoded image is also in the first color coding format. Since the second image input to the codec needs to conform to the second color coding format, the color coding format of the to-be-transcoded image needs to be reconverted so that the color coding format of the to-be-transcoded image is in the second color coding format, thereby preventing the codec from converting the color coding format of the to-be-transcoded image. As described above, the to-be-transcoded image can be processed into the second image based on the second color coding format and the first color coding format.

[0172] In an example of this embodiment, when the first color coding format and the second color coding format are the same, only the size of each color channel of the image to be transcoded needs to be adjusted; and when the first color coding format and the second color coding format are different, the color coding format of the image to be transcoded can be converted.

[0173] Specifically, when the first color coding format is the same as the second color coding format, in the color space of the second color coding format, the sizes of different channels of the image to be transcoded in the color space are adjusted to obtain a second image;

[0174] In a case where the first color coding format is different from the second color coding format, the color coding format of the image to be transcoded is converted into the second color coding format to obtain a second image.

[0175] The second color encoding format may be a YUV format. If the first color encoding format is a YUV format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be sampled to obtain the second image. For example, if the image to be transcoded is in a YUV444 format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be downsampled by a factor of two in the width dimension to obtain the second image in a YUV422 format. For another example, if the image to be transcoded is in a YUV420 format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be upsampled by a factor of two in the height dimension to obtain the second image in a YUV422 format.

[0176] Of course, if the image to be transcoded is an image in the YUV422 format, there is no need to process the image to be transcoded, and the image can be directly input into the codec.

[0177] When the first color encoding format is RGB, the color encoding format of the image to be transcoded can be converted into a second color encoding format to obtain a second image. The second image can be in YUV422 format, which means that the image to be transcoded must first be converted into an image in a color space corresponding to YUV.

[0178] In a further example, the value of each pixel in the input image will be limited to the range of 0 to 1. At this time, the value of each pixel in the image to be transcoded will also be limited to the range of 0 to 1. In the image, whether it is in YUV format or RGB format, the value of each component in the color space is in a larger range. Therefore, it is necessary to restore the value of each pixel in the image to be transcoded to the range of each component in the color space.

[0179] Accordingly, when the image to be transcoded is processed into a second image based on the second color coding format and the first color coding format, the value of each pixel in the image to be transcoded can be inversely normalized and then mapped to the color space corresponding to the second color coding format to obtain a mapped image in the color space, and the value of each pixel in the mapped image can be converted into a pixel value under the second color coding format to obtain a second image.

[0180] Specifically, when the first color coding format is the YUV format, that is, the same as the second color coding format, it can be converted into a second image in the second color coding format by simply performing inverse normalization on the values ​​of each pixel in the transcoded image.

[0181] When the first color coding format is RGB format, that is, different from the second color coding format, it is necessary to first inversely normalize the value of each pixel in the image to be transcoded, and then map it to the color space corresponding to the second color coding format to obtain a mapped image in the color space; then convert the value of each pixel in the mapped image into a pixel value under the second color coding format to obtain a second image.

[0182] Specifically, when the first color encoding format is RGB format, the image to be transcoded is an image in the BT2020 color gamut. The formula for normalizing the RGB values ​​in the BT2020 color gamut can be seen as the following formula (4):

[0183] In formula (4), R2, G2, and B2 represent the normalized pixel values ​​corresponding to different color channels of the pixel in the BT2020 color gamut; the value range of R, G, and B is [0, 255].

[0184] According to formula (4), the RGB values ​​are inversely normalized in the BT2020 color gamut. For details, please refer to the following formula (5):

[0185] In formula (5), R1, G1, and B1 represent the inverse normalized pixel values ​​corresponding to different color channels of the pixel point in the BT2020 color gamut; R2, G2, and B2 represent the normalized pixel values ​​corresponding to different color channels of the pixel point in the BT2020 color gamut.

[0186] Next, the image obtained after inverse normalization can be mapped to the color space corresponding to the second color encoding format. Specifically, the inverse normalized pixel value of each pixel point can be converted into a pixel value under the YUV color domain. Specifically, the conversion can be made by referring to the following formula (6):

[0187] Among them, for the data in YUV format, it is stored in a narrow range, and the value range is Y∈[16×2 dep-8 ,235×2 dep-8 ],U∈[16×2 dep-8 ,240×2 dep-8 ],V∈[16×2 dep-8 ,240×2 dep-8 ]. When HDR video is quantized to 10 bits, Y∈[64,940], U∈

[0188] [64,960], V∈[64,960]; when YUV444 is converted to YCbCr, it can be performed according to the following formula (7):

[0189] According to the value range of YUV, we can get Yc∈[0,1],Cb∈[-0.5,0.5],Cr∈[-0.5,0.5]; then according to formula (7), taking dep as 10bit as an example, 2 dep-8 =2 10-8 =2 2 =4, the pixel value of each pixel in the image to be transcoded in the YUV444 format can be obtained, which can be determined according to the following formula (8): Y = (219*Yc+16)×2 dep-8 U=(224*Cb+128)×2 dep-8 V=(224*Cr+128)×2 dep-8 Formula (8)

[0190] Next, the obtained YUV444 format image can be converted into an image in YUV422 format. Specifically, the data of the chroma channel and the data of the saturation channel in the image are downsampled by 2 times in the width dimension, that is, the width W is downsampled to W / 2. After the conversion is completed, the size of the chroma channel data and the saturation channel data are both W / 2×H.

[0191] The image processing method of this embodiment is described below with reference to a specific example. FIG. 7 shows a schematic diagram of a specific flow chart of this embodiment, which specifically includes:

[0192] S1: Obtain an input video, and decode a video frame in a first color coding format, that is, a first image, from the input video.

[0193] As shown in FIG7 , the format of the input video is first determined. If it is an SDR type video, the first image in the RGB format is decoded. If it is a non-standard HDR type video, the first image in the YUV format is decoded.

[0194] For the first image in YUV format, the conversion is performed according to the process of steps S2-S4, as shown in the process of the left branch in Figure 7. For the first image in RGB format, the conversion is performed according to the process of steps S5-S8, as shown in the process of the right branch in Figure 7.

[0195] S2: Determine whether the first image is in YUV422 format. If so, directly input the first image into the target model. If not, convert the first image into a YUV422 sampling format.

[0196] Among them, when the first image is in YUV420 format, the UV component of the first image is stretched (upsampled) by 2 times in the height dimension, that is, the height is stretched from H / 2 to H. A public upsampling algorithm can be used, such as nearest neighbor interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic). After the conversion is completed, the size of the Y data is W×H, and the size of the U and V data is W / 2×H.

[0197] When the first image is in YUV444 format, the width dimension is downsampled by a factor of 2, that is, the width W is downsampled to W / 2. Public downsampling algorithms such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation can be used. After the conversion, the size of the U and V data is W / 2×H.

[0198] After the above preprocessing, a first sampled image of the first image in the Y component, a second sampled image of the first image in the U component, and a third sampled image of the first image in the V component can be obtained.

[0199] S31: Processing the first sample image, the second sample image, and the third sample image using the target model M1-1, where the target model M1-1 includes a first sub-model and a second sub-model.

[0200] Because the size of Y is different from that of U and V, the first sub-model is used to process the first sampling image, and the second sub-model is used to process the second sampling image and the third sampling image to obtain the processed first predicted image, second predicted image and third predicted image. Because the target model M1-1 does not change the resolution of the input and output, the output is still the 4:2:2 sampling format under the YUV format.

[0201] S32: Process the first sampling image, the second sampling image, and the third sampling image using the target model M1-2.

[0202] Because the size of Y is different from that of U and V, the size of U and V data is w / 2×H. Therefore, the second sampling image and the third sampling image can be merged in the width dimension, and the two W / 2 can be merged into 1 W. At this time, the second sampling image and the third sampling image have the same size as the first sampling image and are directly input into the target model M1-2. The target model M1-2 processes the spliced ​​image and the first sampling image as feature maps of two channels. The output processed image to be transcoded includes a predicted image corresponding to the spliced ​​image and a predicted image corresponding to the first sampling image. After the two predicted images are spliced, they are the image to be transcoded, which is still the 4:2:2 sampling format under the YUV format.

[0203] S33: The target model M1-1 or the target model M1-2 processes the first sample image, the second sample image, and the third sample image as follows (the processing processes of the first sample image, the second sample image, and the third sample image are the same, so they are all replaced by the input image):

[0204] S331, extracting features from the input image to obtain an initial feature map;

[0205] S332, performing feature extraction on a small-area local feature, a large-area local feature, and a global feature, including: performing feature extraction on the initial feature map under receptive fields of different sizes to obtain multiple feature maps; wherein the convolution kernel sizes used for feature extraction under receptive fields of different sizes are different;

[0206] S333, extracting low-frequency information of the initial feature map using guided filtering, including:

[0207] Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; and feature extraction is performed on the low-frequency feature map under the target receptive field to obtain a guided filtering feature map; wherein the target receptive field is the receptive field of the receptive fields of different sizes except the receptive field of the smallest size;

[0208] S334, performing feature fusion on the multiple feature maps to obtain a first fused feature map, and adjusting the values ​​of each feature point in the first fused feature map based on the guided filtering feature map to obtain a second fused feature map;

[0209] S335 , reconstructing features of the second fused feature map to obtain an image to be transcoded.

[0210] S4: Obtain the image to be transcoded. The image to be transcoded may also be an image in YUV422 format. The second color coding format may be YUV422 format. In this case, the image to be transcoded does not need to be processed and can be directly input into the codec to be encoded into HDR video.

[0211] For the first image in RGB format, the following processing is specifically included:

[0212] S5: Normalize the RGB values ​​of each pixel in the first image. Specifically, the first image may be an image in the BT709 color gamut, and the normalization may be performed according to the following formula (1):

[0213] In formula (9), R_norm is the normalized value of the pixel in the red channel, G_norm is the normalized value of the pixel in the green channel, and B_norm is the normalized value of the pixel in the blue channel.

[0214] S6: Input the input image obtained in S5 into the target model M2, where the structure of the target model M2 is the same as that of the target model M1-1. The processing process is as follows:

[0215] S61, extracting features from the input image to obtain an initial feature map;

[0216] S62, performing feature extraction on a small area local feature, a large area local feature, and a global feature, including: performing feature extraction on the initial feature map under receptive fields of different sizes to obtain multiple feature maps; wherein the convolution kernel sizes used for feature extraction under receptive fields of different sizes are different;

[0217] S63, extracting low-frequency information of the initial feature map using guided filtering, including:

[0218] Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; and feature extraction is performed on the low-frequency feature map under the target receptive field to obtain a guided filtering feature map; wherein the target receptive field is the receptive field of the receptive fields of different sizes except the receptive field of the smallest size;

[0219] S64, performing feature fusion on the multiple feature maps to obtain a first fused feature map, and adjusting the values ​​of each feature point in the first fused feature map based on the guided filtering feature map to obtain a second fused feature map;

[0220] S65, reconstructing features of the second fused feature map to obtain an image to be transcoded, wherein the RGB value of each pixel in the image to be transcoded is still in the range of 0 to 1, but is already in the BT2020 color gamut.

[0221] S7, performing inverse normalization on the pixel values ​​of the image to be transcoded in the BT2020 color gamut, which can be determined by referring to the above formula (5).

[0222] S8: Map the inverse normalized image to a color space in a YUV format to obtain a mapped image in a color space in a YUV format, wherein the pixel values ​​are YCbCr values, which can be specifically determined according to formula (6).

[0223] S9: Convert the value of each pixel point in the mapped image into a pixel value in a second color coding format to obtain a second image. Specifically, the conversion is performed according to formula (8).

[0224] The obtained second image is an image in the YUV444 format. In practice, its resolution (also known as data storage format or size) can be changed to obtain a second image in the YUV422 format for input to a codec.

[0225] S10: Input the second image obtained in step S9 into the codec to obtain an HDR video. In this case, it can be seen that the codec does not need to convert the RGB format image into the YUV format image, which can avoid the problem of low quality of the encoded HDR video caused by the codec not being able to convert the RGB format image into the YUV format under BT2020.

[0226] Based on the same inventive concept, the embodiment of the present disclosure further provides a method for obtaining a target model. Referring to FIG8 , a schematic flow chart of the steps of the method for obtaining a target model is shown. As shown in FIG8 , the method may specifically include the following steps:

[0227] Step S801: Acquire a video sample and a plurality of first image samples in a first color coding format.

[0228] In this embodiment, the video sample may be a standard HDR video, and the multiple first image samples may be derived from an SDR video sample or a non-standard HDR video sample. The video sample from which the multiple first image samples are derived and the standard HDR video may be videos with the same picture content, for example, a high dynamic range video and a low dynamic range video shot for the same scene, wherein the multiple first image samples are decoded from the low dynamic range video, and the image samples (second image samples) used as tags may be decoded from the high dynamic range video.

[0229] Among them, there may be multiple first image samples corresponding to different first color coding formats. For example, for the RGB format, there may be 100 first image samples; for the YUV format, there may also be 100 first image samples. First image samples of different formats are used to train different target models.

[0230] Step S802: input a plurality of first image samples into the model to be trained to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples.

[0231] The model to be trained may be trained using multiple first image samples as training samples. During the training process, image samples to be transcoded output by the model to be trained may be obtained, and the color encoding format of the image samples to be transcoded is the first color encoding format. When the multiple first image samples are input into the model to be trained, they may be input in batches.

[0232] Step S803: Based on the first color coding format and the second color coding format, the image sample to be transcoded is processed into a converted image sample that complies with the second color coding format;

[0233] The converted image samples are used to input the codec to obtain high dynamic range video.

[0234] Since the image to be transcoded needs to be converted and then input into the codec to be encoded into HDR video, in this embodiment, the conversion of the image sample to be transcoded can refer to the above-mentioned image processing method embodiment, and the conversion processing of the image to be transcoded can be sufficient, which will not be repeated here.

[0235] Step S804: based on the target image sample, a plurality of second image samples are decoded from the video sample, and based on the target image sample and the second image samples, the parameters of the to-be-trained model are updated;

[0236] The target image sample is any one of the image sample to be transcoded, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the image to be transcoded into the converted image sample.

[0237] During the training process of the model to be trained, the parameters of the model to be trained need to be updated. In the present disclosure, since the video sample is an HDR video, it can be used as supervision of the model to be trained. Since the image to be transcoded output by the model to be trained needs to be converted into a converted image, multiple intermediate image samples may be obtained during the conversion process. As described in the above-mentioned image processing embodiment, when the image sample to be transcoded is in RGB format, the corresponding intermediate image samples will be obtained in sequence through formula (5), formula (6) and formula (8). Therefore, the loss can be calculated based on any intermediate image sample, or based on the image sample to be transcoded or the converted image sample.

[0238] Accordingly, when decoding multiple second image samples from the video sample, the decoding can be performed according to the format of the target image sample, and the decoded second image samples have the same format as the target image sample. The format here can include: consistency of color coding format and consistency of data storage format.

[0239] Step S805: Determine the model to be trained after multiple updates as the target model corresponding to the first color coding format.

[0240] The to-be-trained model may be iteratively trained multiple times according to the above-described process of S802-S804 until a training termination condition is satisfied. The to-be-trained model obtained under the training termination condition may then be determined as the target model. The training termination condition may include convergence of the to-be-trained model, a loss between the target image sample and the second image sample being less than a preset loss, or a predetermined number of training cycles being completed. In practice, any of the above conditions may be selected.

[0241] According to the technical solution of the embodiment of the present disclosure, a target model matching the first color coding format is obtained by training using first image samples of different first color coding formats. Thus, the target model matches the first image samples it processes, so that images of different color coding formats can be converted into high-quality transcoded images by the target model. Furthermore, according to any one of the transcoded image samples, intermediate image samples, and transcoded image samples output by the trained model, during the training of the target model, loss calculation can be performed based on the image output by the target model (in the first color coding format), or based on any image of the color coding format obtained during the conversion of the image output by the target model into an image of the second color coding format, or based on the image input to the codec (transcoded image sample). In this way, the processing capability of the target model for the input image can be adapted to the image input to the codec (in the second color coding format), thereby improving the quality of the image input to the codec. Thus, after the quality of the image input to the codec is improved, the quality of the HDR video converted from the images of different first color coding formats is also improved, solving the problem of uneven conversion quality of images of different formats.

[0242] As described in the above-mentioned embodiment of the image processing method, the image input to the target model can be an image obtained by preprocessing the first image according to the first color encoding format. In this embodiment, the first image sample used as the training sample can also be an image obtained by preprocessing the original image according to the first color encoding format. Therefore, when the first color encoding format is RGB format, the value of each pixel of the first image sample is 0 to 1; when the first color encoding format is YUV format, the data storage format of the first image sample is YUV422 format.

[0243] In this way, the computational complexity of the model to be trained can be reduced and the operating efficiency can be improved.

[0244] In some examples, the first color encoding format is RGB format, and the second color encoding format is YUV422 format. In this case, the color encoding format of the target image sample is RGB format, and the data storage format is that the RGB value in the target image sample ranges from 0 to 1;

[0245] Alternatively, the color encoding format of the target image sample is the YUV format, and the data storage format is the YUV444 format or the YUV422 format.

[0246] As described above, when decoding the second image sample, decoding can be performed according to the color encoding format and data storage format of the target image sample. Since the target image sample is the image sample to be transcoded, the intermediate image sample, or the converted image sample output by the model to be trained, when the first color encoding format is the YUV format, since the first color encoding format and the second color encoding format are the same, the color encoding format of the image sample to be transcoded is also the second color encoding format, and conversion is not required. Therefore, the second image sample can be directly decoded from the video sample according to the YUV format.

[0247] When the first color encoding format is RGB format, since the first color encoding format and the second color encoding format are different, the color encoding format of the transcoded image sample is also RGB format. As mentioned above, when the RGB of each pixel in the first image sample input to the model to be trained is a normalized value, the RGB of each pixel in the image sample to be transcoded is also a normalized value. In practice, it needs to be converted into YUV422 format, and multiple intermediate image samples will be obtained. The multiple intermediate image samples include: image samples with RGB format but the RGB value of each pixel is an inverse normalized value, YCbCr image samples, and image samples in YUV444 format.

[0248] 9 and 10, schematic diagrams of strategies for updating the parameters of the training model are shown respectively. As shown in FIG9, the video sample used as a label is a standard HDR video sample. In one example, the target image sample can be an image sample to be transcoded, that is, its color encoding format is RGB format, and the RGB value of each pixel is in the range of 0 to 1. In this case, a second image sample can be decoded from the video sample used as a label in RGB format. As shown in FIG11, a schematic diagram of the process of decoding the second image sample is shown, which includes the following process:

[0249] First, because the data sampling format of the labeled video sample (HDR video file) is YUV422, the first step is to convert the input video format to YUV444 sampling format. Specifically, the YUV422 format video frame can be decoded from the video sample. Then, the data of the chroma channel in the video frame is stretched (upsampled) by 2 times in the width dimension, and the data of the saturation channel is stretched (upsampled) by 2 times in the width dimension, that is, the width is stretched from W / 2 to W. After the conversion is completed, the size of the luminance channel data, chroma channel data, and saturation channel data of the video frame are all W×H.

[0250] Then, the converted YUV444 format video frames are stored in a narrow range, with the value range of Y∈[16×2 dep-8 ,235×2 dep-8],U∈[16×2 dep-8 ,240×2 dep-8 ],V∈[16×2 dep-8 ,240×2 dep-8 When HDR video is quantized to 10 bits, Y∈[64,940], U∈[64,960], V∈[64,960]. Specifically, convert YUV444 to YCbCr according to the following formula (9):

[0251] In formula (10), according to the numerical range of YUV, we can obtain Yc∈[0,1], Cb∈[-0.5,0.5], Cr∈[-0.5,0.5].

[0252] After that, YcbCr is converted to RGB format, where the relationship between YcbCr and RGB under the BT2020 color gamut satisfies the following equation (10): Yc=0.2627R+0.6780G+0.0593B

[0253] The reverse solution can be used to obtain the converted RGB value. Specifically, the relationship between RGB and YCbCr satisfies the following equation (11):

[0254] According to the value range of Yc∈[0,1],Cb∈[-0.5,0.5],Cr∈[-0.5,0.5], we can get R∈[-0.7373,1.7373],G∈[-0.36795,1.36795],B∈[-0.9407,1.9407].

[0255] Therefore, through the relationship (11), an intermediate image sample that conforms to the RGB format under the BT2020 color gamut can be obtained, and then the RGB values ​​of each pixel in the intermediate image sample are normalized. Specifically, it can be processed according to formula (4), thereby obtaining a second image sample that conforms to the RGB format and whose RGB value is a normalized value. It can then be based on the second image sample and the image sample to be transcoded to update the parameters of the training model.

[0256] As shown in FIG10 , the video sample used as a label is a standard HDR video sample. In some examples, since the image sample to be transcoded needs to be converted into a converted image sample and then input into the codec to be encoded as an HDR video sample, the color encoding format of the target image sample can also be in YUV format, and the data storage format is in YUV444 format or YUV422 format.

[0257] When the target image sample is in YUV444 format or YUV422 format, the decoding process of the video sample used as a label can be avoided. For example, when the target image sample is in YUV444 format, it is only necessary to convert the input video format into the YUV444 sampling format. On the one hand, the decoding process is short, which can reduce unnecessary calculations and improve the efficiency of decoding the second image sample; on the other hand, the original image features can be retained as much as possible, so that when calculating the loss, the loss calculation accuracy can be improved, thereby improving the image processing accuracy of the model to be trained.

[0258] For another example, when the target image sample is in the YUV422 format, it is only necessary to decode the second image sample from the video sample, which reduces the computation process and improves the efficiency of decoding the second image sample.

[0259] Preferably, an intermediate image sample in a YUV444 format may be used as a target image sample. In this case, a second image sample in a YUV444 format may be decoded from a video sample serving as a tag.

[0260] In some embodiments, as described in the above-mentioned embodiment of the image processing method, when the color encoding format of the input first image sample is in YUV format, there may be two target models with different structures for processing the input image in the first color encoding format. In the process of training the target model, there are also two corresponding training methods, specifically, training method C and training method D:

[0261] In training method C, the first image sample can be first decomposed into an image of a Y component, an image of a U component, and an image of a V component. Specifically, before inputting the plurality of first image samples into the model to be trained, a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel can be obtained from the first image sample; and the second sampled image sample and the third sampled image sample can be spliced ​​together to obtain a spliced ​​image of the same size as the first sampled image sample.

[0262] Accordingly, the first sampling image samples and the spliced ​​image samples can be input into the model to be trained, and the image samples to be transcoded can be obtained based on the predicted image samples output by the model to be trained and corresponding to the first sampling image and the spliced ​​image samples respectively.

[0263] As described in the above-mentioned image processing method example, the first image sample is an image sample in YUV422 format. The second sampled image sample and the third sampled image sample can be spliced ​​in the width dimension to obtain a size of W×H. The size of the spliced ​​image sample is the same as the size of the first sampled image sample. The model to be trained can process the first sampled image sample and the spliced ​​image sample as image samples of different channels.

[0264] During the training process, the model to be trained can output a predicted image sample corresponding to the first sampled image sample and a predicted image sample corresponding to the spliced ​​image sample. The two predicted image samples are fused to obtain the image sample to be transcoded. Then, according to the process shown in Figures 9 and 10 above, the loss is calculated based on the second image sample and the target image sample (any one of the image sample to be transcoded, the intermediate image sample, and the converted image sample). The parameters of the model to be trained are updated based on the loss value.

[0265] Adopting the technical solution C of this embodiment can reduce the complexity of the target model, be beneficial to the training and development of the target model, and reduce the model's occupation of software and hardware resources.

[0266] In training mode D, a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel may be obtained from the first image sample;

[0267] Accordingly, the first sampling image sample can be input into the first sub-model, and the second sampling image sample and the third sampling image sample can be input into the second sub-model; and when the training model is updated, the parameters of the first sub-model can be updated based on the predicted image sample and the second image sample corresponding to the first sampling image output by the first sub-model; and the parameters of the second sub-model can be updated based on the second image sample and the predicted image sample corresponding to the second sampling image sample and the third sampling image sample output by the second sub-model respectively.

[0268] As described in the above image processing method example, the first image sample is in YUV422 format, and the second and third sampled image samples are half the width of the first sampled image sample. The model to be trained can include a first sub-model and a second sub-model to separately process sampled image samples of different sizes. Specifically, the first sub-model is used to process the first sampled image sample, and the second sub-model is used to process the second and third sampled image samples of the same size.

[0269] During the training process, the first sub-model can output a first predicted image sample corresponding to the first sampled image sample, and the second sub-model can output a second predicted image sample corresponding to the second sampled image sample, and a third predicted image sample corresponding to the third sampled image sample; the first predicted image sample, the second predicted image sample, and the third predicted image sample are data from different channels, and the three are fused to obtain the image sample to be transcoded. Then, according to the process shown in Figures 9 and 10 above, the loss is calculated based on the second image sample and the target image sample (any one of the image sample to be transcoded, the intermediate image sample, and the converted image sample), and the parameters of the first sub-model and the parameters of the second sub-model are updated according to the loss value.

[0270] It should be noted that, since the first sub-model and the second sub-model are included, during the training process, the parameters of the first sub-model and the parameters of the second sub-model need to be updated. Since the two process image data of different channels, they are updated in one model;

[0271] Sample data belonging to the first sub-model can be extracted from the target image samples and the second image samples, such as sample data corresponding to the brightness channel. In this way, sample data corresponding to the brightness channel are respectively extracted from the target image samples and the second image samples. Based on the two sample data, the loss corresponding to the first sub-model is calculated, and then the parameters of the first sub-model are updated according to the loss.

[0272] Similarly, sample data belonging to the second sub-model can be extracted from the target image samples and the second image samples, such as sample data corresponding to the hue and saturation channels. In this way, sample data corresponding to the bright hue and saturation channels are respectively extracted from the target image samples and the second image samples. Based on the two sample data, the loss corresponding to the second sub-model is calculated, and then the parameters of the second sub-model are updated according to the loss.

[0273] For example, taking the target image sample as an image sample in YUV444 format, sample data Y1 belonging to the Y channel can be extracted from the target image sample, and sample data Y1' of the Y channel can be extracted from the second image sample. The parameters of the first sub-model are updated based on the sample data Y1 and the sample data Y1'. Furthermore, sample data U1 belonging to the U channel can be extracted from the target image sample, and sample data U1' of the U channel can be extracted from the second image sample. Similarly, sample data V1 belonging to the V channel can be extracted from the target image sample, and sample data V1' of the V channel can be extracted from the second image sample. The parameters of the second sub-model are updated based on the sample data U1 and sample data U1', and the sample data V1 and sample data V1'.

[0274] In this way, the update accuracy of the training model can be improved.

[0275] By adopting this method D, the first sampling image samples, the second sampling image samples and the third sampling image samples of different sizes can be processed respectively by the first sub-model and the second sub-model. In this way, compared with the method C, the amount of data processed by each sub-model when processing each sampling image sample in parallel is reduced, thereby improving the model's computing speed and thus improving the processing efficiency of each sub-model.

[0276] The following is a specific example to illustrate the method of obtaining the target model. As shown in FIG9 to FIG11 , the method may include the following steps:

[0277] S1′, obtaining a video sample used as a label and a video sample used as an input, wherein a second image sample can be decoded from the video sample used as the label, and a first image sample can be decoded from the video sample used as the input, wherein the first image sample is in a first color coding format;

[0278] Specifically, the process of decoding the first image sample from the video sample as input may be as follows:

[0279] A video frame is decoded from the input video sample, i.e., the standard HDR video sample. Thereafter, the video frame is preprocessed corresponding to the first color encoding format. Specifically, for a video frame in a YUV format, its data storage format can be converted into a YUV422 format; for a video frame in an RGB format, the value of each pixel can be normalized, thereby reducing the amount of data storage.

[0280] Specifically, as shown in FIG11 , the process of decoding the second image sample from the standard HDR video sample may include the following steps:

[0281] Process 1: A video frame in a second color coding format, i.e., a YUV format video frame, can be decoded from the standard HDR video sample. The data storage format of the YUV format video frame is converted into a YUV422 format to obtain a second image sample SA in the YUV422 format.

[0282] Process 2: A video frame in a second color coding format, i.e., a YUV format video frame, can be decoded from the labeled video sample. The data storage format of the YUV format video frame is converted to a YUV444 format to obtain a second image sample SB in the YUV444 format.

[0283] Process 3: A video frame in a second color coding format, i.e., a YUV format video frame, can be decoded from the labeled video sample. The data storage format of the YUV format video frame is converted to the YUV444 format. Then, the color coding format of the YUV444 format video frame is converted to the RGB format. The values ​​of each pixel in the obtained RGB format video frame are normalized to obtain a second image sample SC in the RGB format.

[0284] The above three second image samples may be used in updating the model to be trained, specifically, as described in subsequent steps.

[0285] S2', for a first image sample whose first color encoding format is YUV, has the following training process A and training process B; wherein, the training process A refers to steps S21'-S24', and the training process B refers to steps S25'-S29':

[0286] S21′: acquiring a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0287] S22′: splicing the second sampled image sample and the third sampled image sample to obtain a spliced ​​image with the same size as the first sampled image sample;

[0288] S23′: inputting the first sampled image sample and the spliced ​​image sample into the to-be-trained model, and obtaining the to-be-transcoded image sample based on the predicted image sample output by the to-be-trained model and corresponding to the first sampled image and the spliced ​​image sample, respectively; wherein the format of the to-be-transcoded image sample is YUV422 format;

[0289] S24': Calculate the loss using the image sample to be transcoded (and the target image sample is the image sample to be transcoded) and the second image sample SA (YUV422 format), update the parameters of the model to be trained according to the obtained loss value, and determine the model to be trained after multiple updates as the target model.

[0290] Training process B, in which the model to be trained includes a first sub-model and a second sub-model, wherein:

[0291] S25′: acquiring a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0292] S26′: inputting the first sampled image sample into the first sub-model, and inputting the second sampled image sample and the third sampled image sample into the second sub-model;

[0293] S27′: determining a first loss value based on the second image sample SA and the predicted image sample output by the first sub-model corresponding to the first sampled image sample, and updating the parameters of the first sub-model based on the first loss value;

[0294] S28′: determining a second loss value based on the second image sample SA and the predicted image sample output by the second sub-model corresponding to the second sampled image sample and the third sampled image sample, and updating the parameters of the second sub-model based on the second loss value;

[0295] S29': Calculate the overall loss value based on the first loss value and the second loss value. Specifically, the overall loss value can be calculated using a weighted summation method, and then based on the overall loss value, fine-tune the parameters of the first sub-model and the second sub-model, and determine the model to be trained after multiple updates as the target model.

[0296] This target model can be used to process images in YUV422 format to improve the dynamic range of images in YUV422 format.

[0297] S3′, for a first image sample whose first color encoding format is RGB, has the following training process:

[0298] S31′: inputting the first image sample into the model to be trained;

[0299] S32′: Obtain an image sample to be transcoded corresponding to the first image sample output by the model to be trained, wherein the image sample to be transcoded is in RGB format and is in the BT2020 color gamut, and the value of each pixel in the image sample to be transcoded is 0 to 1;

[0300] S33': Transcode the image sample to be transcoded according to the YUV format to obtain the following intermediate image samples in sequence:

[0301] Intermediate image sample 1: a sample for inverse normalizing the values ​​of each pixel in the image sample to be transcoded. The RGB value of each pixel in intermediate image sample 1 is within the range of the BT2020 color gamut.

[0302] Intermediate image sample 2: obtained by mapping the values ​​of each pixel in intermediate image sample 1 to the YUV color gamut. The values ​​of each pixel in intermediate image sample 2 are within the YCbCr range.

[0303] Intermediate image sample 3: Convert the resolution of intermediate image sample 2 to change its data storage format. The format of intermediate image sample 3 is YUV444.

[0304] Converted image sample: The resolution of the intermediate image sample 3 is converted again to change its data storage format. The format of the converted image sample is YUV442.

[0305] S34': Update the parameters of the model to be trained, specifically including the following update methods:

[0306] Method 1: As shown in FIG9 , based on the image sample to be transcoded and the second image sample SC, a third loss value of the model to be trained is calculated, and the parameters of the model to be trained are updated based on the third loss value;

[0307] Method 2: As shown in FIG10 , based on the intermediate image sample 3 and the second image sample SB, a fourth loss value of the model to be trained is calculated, and the parameters of the model to be trained are updated based on the fourth loss value;

[0308] Method 3: Based on the converted image sample and the second image sample SA, a fifth loss value of the to-be-trained model is calculated, and the parameters of the to-be-trained model are updated based on the fifth loss value;

[0309] S35′: The model to be trained obtained in step S34′ after multiple updates is used as the target model corresponding to the RGB format.

[0310] Among them, any of the above-mentioned parameter updating methods can obtain a model to be trained with higher precision, but when adopting the above-mentioned method 2, compared with method 1, since it uses the YUV format image obtained after conversion and the second image sample SB to calculate the loss, the capability of the target model can match the HDR video to be converted, and the quality of the image to be transcoded can be improved; and when adopting the above-mentioned method 2, compared with method 3, since the process of converting from the YUV format image to the YUV444 format image is relatively simple, the efficiency of obtaining the second image sample can be improved, and since the YUV444 format image contains more image features, the accuracy of the loss calculation can be improved, thereby improving the accuracy of the target model.

[0311] Based on the same inventive concept, the present disclosure further provides an image processing system, as shown in FIG12 , which shows a schematic diagram of the framework structure of the system. As shown in FIG12 , the system may specifically include: multiple target models, where different target models correspond to different first color coding formats;

[0312] an acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0313] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of first images, and obtain a plurality of to-be-transcoded images conforming to the first color coding format; wherein the dynamic range of the to-be-transcoded images is higher than the dynamic range of the first images;

[0314] a post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0315] The codec module is configured to input the second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0316] The system embodiment may refer to the description of the above-mentioned image processing method embodiment, and will not be repeated here.

[0317] This image processing system can call different target models for processing first images in different first color coding formats, and the transcoded images output by different target models can enter different post-processing processes, so that first images in different color coding formats can all be matched and processed by the target models, thereby improving the quality of the transcoded images. The codec module can directly input the converted second image that conforms to the second color coding format into the codec, so that the codec directly encodes the second image into HDR video, eliminating the need for the codec to perform internal conversion on images that do not conform to the second color coding format. On the one hand, this reduces the workload of the codec and improves image conversion efficiency. On the other hand, it avoids the problem of low quality of the encoded HDR video caused by the codec's lack of conversion capabilities for some of the first color coding formats, thereby allowing first images in different color coding formats to be converted into high-quality HDR videos.

[0318] The structure of the target model is described below. In an optional example, regardless of whether the first color encoding format is the YUV format or the RGB format, the target model can include the following structure. As shown in FIG13 , a schematic diagram of the structure of the target model is shown. As shown in FIG13 , the target model can specifically include:

[0319] a first feature extraction branch, a plurality of second feature extraction branches connected to an output end of the first feature extraction branch, a first fusion module connected to the output ends of the plurality of second feature extraction branches, and a reconstruction module connected to the output end of the first fusion module; wherein different second feature extraction branches correspond to receptive fields of different sizes;

[0320] The first feature extraction branch is configured to perform feature extraction on an input image to obtain an initial feature map. The input image is the image to be input after preprocessing the first image, and the values ​​of each pixel in the image to be input meet a first preset condition, or, in a color space corresponding to the first color encoding format, the sizes of each color channel of the image to be input in the color space meet a second preset condition.

[0321] The second feature extraction branch is used to extract features from the initial feature map under the corresponding receptive field to obtain a feature map;

[0322] A first fusion module is used to perform feature fusion on multiple feature maps to obtain a first fused feature map;

[0323] The reconstruction module is used to reconstruct the features of the first fused feature map to obtain the image to be transcoded.

[0324] In this embodiment, the image to be input (or the first image) can be input into the first feature extraction branch. The first feature extraction branch can be a 3x3 convolution layer for performing convolution processing on the input image.

[0325] Since different second feature extraction branches in the multiple second feature extraction branches correspond to receptive fields of different sizes, different second feature extraction branches can perform convolution processing of different scales. It can be understood that the second feature extraction branch with a larger receptive field can correspond to a larger convolution kernel, and the second feature extraction branch with a smaller receptive field can correspond to a smaller convolution kernel.

[0326] Among them, the first fusion module may include a splicing unit and a fusion unit, the splicing unit is used to splice the feature maps output by multiple second feature extraction branches, and the fusion unit can be used to perform 1×1 convolution on the spliced ​​feature maps.

[0327] Among them, the reconstruction module can be used to reconstruct the features of the obtained first fusion feature map, thereby outputting a predicted image.

[0328] In a specific implementation of this example, referring to FIG14 , a schematic diagram of the structure of the second target model is shown. As shown in FIG14 , the target model may further include:

[0329] A third feature extraction branch connected to the output end of the first feature extraction branch, and a second fusion module connected to the output end of the first fusion module and the output end of the fourth feature extraction branch, and an input end of the reconstruction module connected to the output end of the second fusion module; wherein:

[0330] The third feature extraction branch is used to extract the features belonging to low-frequency information in the initial feature map by presetting the feature map to obtain a low-frequency feature map;

[0331] The fourth feature extraction branch in the second feature extraction branch is further used to extract features from the low-frequency feature map to obtain a guided filtering feature map; the receptive field corresponding to the fourth feature extraction branch is not the largest among the multiple second feature extraction branches;

[0332] A second fusion module is used to fuse the guided filter feature map and the first fused feature map to obtain a second fused feature map;

[0333] The reconstruction module is specifically used to reconstruct the features of the second fused feature map to obtain the image to be transcoded.

[0334] In some examples, the second fusion module is specifically used to map the values ​​of each pixel point in the guided filter feature map to a target interval to obtain a mapped guided filter feature map; and based on the values ​​of each pixel point in the mapped guided filter feature map, adjust the values ​​of each pixel point in the first fused feature map to obtain a second fused feature map; wherein the target interval ranges from 0 to 1.

[0335] The processing process can be described with reference to the embodiment of the above-mentioned image processing method, and will not be repeated here.

[0336] Among them, the first color coding format includes YUV format and RGB format. When the first color coding format is YUV format, as described in the embodiments of the above-mentioned image processing method and target model acquisition method, there may be target models with different structures. In this embodiment, the target model corresponding to the YUV format may include a first sub-model and a second sub-model, wherein the structure of the first sub-model and the structure of the second sub-model are the same, and both include: a first feature extraction branch, multiple second feature extraction branches, the first fusion module, and a reconstruction module.

[0337] In this example, although the target model includes two branches (the first sub-model and the second sub-model), the structure of each branch can be the same, that is, they all have the model structure shown in reference Figure 14. Thus, the image belonging to the Y component, the image belonging to the U component, and the image belonging to the V component in the first image can be processed by independent sub-models, thereby improving the processing accuracy and quality.

[0338] Of course, in another example, the target model corresponding to the YUV format can be a model with one branch, that is, a comprehensive model as shown in Figure 14. In this case, it is necessary to splice the data belonging to the U component (chrominance channel) and the V component (saturation channel) in the first image to obtain data of the same size as the data of the Y component (luminance channel). In this way, the spliced ​​image obtained after splicing has the same image size as the luminance channel, so that the target model can process the spliced ​​image and the Y component image as images of different channels in parallel, thereby reducing the complexity of the target model and the operating resources occupied by the target model, thereby improving processing efficiency.

[0339] 15a-15c , schematic structural diagrams of second feature extraction branches with different receptive fields are shown. As shown in FIG15a-15c , the plurality of second feature extraction branches include at least one second feature extraction branch of the first type and a second feature extraction branch of the second type:

[0340] The first type of second feature extraction branch includes a plurality of first feature extraction units connected in series, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels; the smaller the receptive field size of the second feature extraction branch, the smaller the convolution kernel size of the first convolution unit and the second convolution unit;

[0341] Among them, the second feature extraction branch of the second type includes: multiple third feature extraction units connected in series, and the third feature extraction units are connected in series in series, and the fifth convolution unit, the activation unit and the pooling unit are connected in series.

[0342] As shown in Figure 15a, a structural schematic diagram of the second feature extraction branch corresponding to the first receptive field in the second feature extraction branch of the first type is shown. As shown in Figure 15a, the second feature extraction branch corresponding to the first receptive field includes: multiple first feature extraction units connected in series in sequence, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches in parallel, and different first convolution branches correspond to different convolution kernels.

[0343] Among them, the second feature extraction branch shown in Figure 15a can be called the second feature extraction branch L1, L1 includes first feature extraction units connected in series, each first feature extraction unit includes a first convolution unit and a second convolution unit M3, wherein the first convolution unit can be a 3×3 convolution layer, the second convolution unit M3 includes two first convolution branches in parallel, and the branch L1-1 in the two first convolution branches includes an M1×1 layer, an activation function ReLu layer, an M1×1 layer, an activation function ReLu layer and an M1×1 layer connected in series.

[0344] The branch L1-2 in the two first convolution branches includes an M3×3 layer, an activation function ReLu layer, an M3×3 layer, an activation function ReLu layer, and an M3×3 layer connected in series.

[0345] Among them, the second convolution unit also includes a splicing subunit connected to the output ends of the two first convolution branches respectively, a 3×3 convolution layer, a GMP (global maximum pooling) layer, and two fully connected layers connected to the GMP (global maximum pooling) layer respectively, as well as a fusion subunit connected to the output end of a fully connected layer and branch L1-1, a fusion subunit connected to the output end of another fully connected layer and branch L1-2, and a feature addition layer connected to the output ends of the two fusion subunits at the same time.

[0346] The structures of the M1×1, M3×3, and M5×5 layers can be shown in Figure 16. They are roughly the same, consisting of a concatenated convolutional layer, a ReLu activation function layer, a 1×1 convolutional layer, a Sigmoid layer, and a residual branch. The residual branch is used to fuse the features output by the ReLu activation function layer with the features output by the Sigmoid layer. The difference is that the convolution kernel sizes used in the first convolution layer of the M1×1, M3×3, and M5×5 layers are different. The convolution kernel size used in the first convolution layer of the M1×1 layer is 1×1, the convolution kernel size used in the first convolution layer of the M3×3 layer is 3×3, and the convolution kernel size used in the first convolution layer of the M3×3 layer is 3×3, with a stride of 2.

[0347] As shown in Figure 15b, a structural schematic diagram of the second feature extraction branch corresponding to the second receptive field in the second feature extraction branch of the first type is shown. As shown in Figure 15b, the second feature extraction branch corresponding to the second receptive field includes: multiple first feature extraction units connected in series in sequence, the two first feature extraction units include a first convolution unit and a second convolution unit M5, the second convolution unit M5 includes two first convolution branches in parallel, and different first convolution branches correspond to different convolution kernels.

[0348] Among them, the second feature extraction branch shown in Figure 15b can be called the second feature extraction branch L2, L2 includes first feature extraction units connected in series, each first feature extraction unit includes a first convolution unit and a second convolution unit M5, wherein the first convolution unit can be a 3×3 convolution layer with a step size of 2, the second convolution unit M5 includes two first convolution branches in parallel, and the branch L2-1 in the two first convolution branches includes an M3×3 layer, an activation function ReLu layer, an M3×3 layer, an activation function ReLu layer and an M3×3 layer connected in series.

[0349] The branch L2-2 in the two first convolution branches includes an M5×5 layer, an activation function ReLu layer, an M5×5 layer, an activation function ReLu layer, and an M5×5 layer connected in series.

[0350] Among them, the second convolution unit M5 also includes a splicing subunit connected to the output ends of the two first convolution branches, a 3×3 convolution layer, a GMP (global maximum pooling) layer, and two fully connected layers connected to the GMP (global maximum pooling) layer, as well as a fusion subunit connected to the output end of one fully connected layer and branch L2-1, a fusion subunit connected to the output end of another fully connected layer and branch L2-2, and a feature addition layer connected to the output ends of the two fusion subunits. The structure of the M5×5 layer can be seen in Figure 16 in detail and will not be repeated here.

[0351] Among them, the convolution scale of the second feature extraction branch L1 is smaller than the convolution scale corresponding to the second feature extraction branch L2. Therefore, the size of the first receptive field is smaller than the size of the second receptive field. The second feature extraction branch under the first receptive field can be used to extract subtle features in the initial feature map, and the second feature extraction branch under the second receptive field can be used to extract local detail features in the initial feature map.

[0352] As shown in Figure 15c, a structural schematic diagram of the second feature extraction branch of the second type is shown, including: multiple second feature extraction units connected in series, the second feature extraction unit includes a third convolution unit, an activation unit and a pooling unit connected in series; wherein, connected in series with the last second feature extraction unit are a fourth convolution unit, a fusion subunit connected to the fourth convolution unit, an activation function layer ReLu connected to the fusion subunit, and a convolution layer connected to the activation function layer.

[0353] The third feature extraction unit includes a fourth convolution unit, an activation unit ReLu, and a pooling unit connected in series. The structure may include a convolution layer, an activation function layer ReLu, a maximum pooling layer, and a normalization layer connected in series.

[0354] Among them, the second feature extraction branch of the second type corresponds to the third receptive field, the size of the third receptive field is larger than the size of the second receptive field, and the second feature extraction branch under the third receptive field can be used to extract global features in the initial feature map.

[0355] The following describes the entire process from acquiring a target model to processing the first image to be processed using the target model and converting it into an HDR video, using a specific example:

[0356] First, according to steps S21'-S29', the target model M1 corresponding to the YUV format is trained, wherein the target model M1 includes a target model M1-1 and a target model M1-2; the target model M1-1 includes a first sub-model and a second sub-model, which are trained through steps S25'-S29', and the target model M1-2 is a comprehensive model, which is trained through steps S21'-S24' and has a branch. The structure of the branch can refer to the structure shown in Figure 14. Similarly, the structure of the first sub-model and the second sub-model can refer to the structure shown in Figure 14.

[0357] And, the target model M2 corresponding to the RGB format is trained according to the process shown in Figures 9-11 above, wherein the process of training the target model M2 can refer to the above steps S31'-S35'. In this example, in order to ensure the accuracy of the target model M2, when updating the parameters of the model to be trained, the selected target image samples can be image samples in YUV444 format, and the second image samples need to be decoded from the video samples used as labels in YUV444 format to obtain multiple second image samples SB, and then the parameters of the model to be trained are updated based on the second image samples SB and the image samples in YUV444 format (intermediate image samples 3).

[0358] Taking the video to be processed as SDR video as an example, the following process is performed:

[0359] Decode a first image in RGB format from the SDR video, where parameters of the SDR video should at least meet the following parameters: sampling (4:2:0 or 4:2:2 or 4:4:4), quantization (8 bit or 10 bit or 12 bit), color gamut (BT.470 or BT.601 or BT.709), and no high dynamic range.

[0360] Afterwards, the SDR video needs to be converted into an HDR video, the RGB value of each pixel in the decoded multiple first images is normalized, and then the normalized first image is input into the target model M2 to obtain the image to be transcoded output by the target model M2, wherein the image to be transcoded is an image under the BT2020 color gamut, and the RGB value of each pixel included therein ranges from 0 to 1;

[0361] Next, the image to be transcoded is sequentially subjected to inverse normalization, YUV color coding format conversion, and data storage format conversion to obtain a second image in YUV422 format;

[0362] After that, the second image is sent to the codec to obtain HDR video, where the parameters of the HDR video should at least meet the following parameters: sampling (4:2:2), quantization (10bit), color gamut (BT.2020, high dynamic range, in line with the HLG standard.

[0363] For example, if the video to be processed is a non-standard HDR video format, the video may be a non-professional HDR video from self-media, the Internet, etc., but it conforms to the dynamic range of HLG. The main process is as follows:

[0364] Decoding a first image in a YUV format from a video in a non-standard HDR video format, and converting a data storage format of the first image into a first image in a YUV422 format;

[0365] Then, the second sampled image belonging to the U component and the third sampled image belonging to the V component of the first image in the YUV422 format are spliced ​​to obtain a spliced ​​image, and the spliced ​​image and the first sampled image belonging to the Y component are input into the target model M1-2 to obtain an image to be transcoded, which is in the YUV422 format;

[0366] Alternatively, the second sampled image belonging to the U component and the third sampled image belonging to the V component in the first image in YUV422 format are input into the second sub-model in the target model M1-1, and the first sampled image belonging to the Y component is input into the first sub-model in the target model M1-1. According to the outputs of the first sub-model and the second sub-model, the image to be transcoded is obtained, and the image to be transcoded is in YUV422 format.

[0367] Next, the image to be transcoded is directly input into the codec to obtain an HDR video with a dynamic range that conforms to HLG. The parameters of the HDR video should at least meet the following parameters: sampling (4:2:2), quantization (10bit or 12bit), color gamut (BT.2020, high dynamic range, in line with the HLG standard.

[0368] In this case, the target model can adjust the high dynamic range / color of the non-standard HDR video so that the HDR video that does not meet the requirements can be output as a HDR video that meets the requirements through the target model.

[0369] Based on the same inventive concept, the present disclosure further provides an image processing device, as shown in FIG12 , comprising:

[0370] an acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0371] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of first images, and obtain a plurality of to-be-transcoded images in the first color coding format; wherein the dynamic range of the to-be-transcoded images is higher than the dynamic range of the first images;

[0372] a post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0373] The codec module is configured to input the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0374] Optionally, the device further comprises:

[0375] a first determining module, configured to determine a preprocessing corresponding to the first color coding format, wherein different first color coding formats correspond to different preprocessing;

[0376] a preprocessing module, configured to perform the preprocessing on the first image to obtain an image to be input; wherein the computational complexity of the image to be input is smaller than that of the first image, or the resolution of the image to be input is larger than that of the first image;

[0377] The step of using the target model to process the first image to obtain the image to be transcoded includes: using the target model to process the image to be input to obtain the image to be transcoded.

[0378] Optionally, the preprocessing module includes:

[0379] a first processing unit, configured to sample data belonging to different color channels in the first image separately when the first color encoding format is a YUV format, so as to reduce or increase a resolution of the first image;

[0380] The second processing unit is configured to normalize the RGB value of each pixel in the first image when the first color encoding format is RGB format, so that the RGB value of each pixel in the image to be input is in the range of 0 to 1.

[0381] Optionally, the value range of each pixel in the image to be input is 0 to 1; or, in the color space corresponding to the first color coding format, the size of the first channel in the image to be input is twice the size of the second channel and the size of the third channel.

[0382] Optionally, the first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a luminance channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and the size of the first sampled image is larger than the second sampled image and the third sampled image; the step of processing the image to be input using the target model includes:

[0383] splicing the second sampling image and the third sampling image to obtain a spliced ​​image with the same size as the first sampling image;

[0384] Inputting the first sampled image and the stitched image into the target model, wherein the target model is used to increase the dynamic range of the first sampled image and the stitched image respectively;

[0385] The image to be transcoded is obtained based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

[0386] Optionally, the first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a luminance channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and the size of the first sampled image is larger than the second sampled image and the third sampled image; the target model includes a first sub-model and a second sub-model, and the step of processing the image to be input using the target model includes:

[0387] increasing the dynamic range of the first sampled image using the first sub-model to obtain a first predicted image;

[0388] Using the second sub-model to increase the dynamic range of the second sampled image and the third sampled image, to obtain a second predicted image corresponding to the first predicted image and the second sampled image, and a third predicted image corresponding to the third sampled image;

[0389] The image to be transcoded is obtained based on the first predicted image, the second predicted image, and the third predicted image.

[0390] Optionally, the post-processing module includes:

[0391] a third processing unit, configured to, when the first color encoding format is the same as the second color encoding format, adjust sizes of different channels of the image to be transcoded in the color space of the second color encoding format to obtain a second image;

[0392] The fourth processing unit is configured to, when the first color coding format is different from the second color coding format, convert the color coding format of the image to be transcoded into the second color coding format to obtain the second image.

[0393] Optionally, the value of each pixel in the input image ranges from 0 to 1; and the post-processing module includes:

[0394] an inverse normalization unit, configured to inverse normalize the value of each pixel in the image to be transcoded, and then map the value to a color space corresponding to the second color encoding format to obtain a mapped image in the color space;

[0395] The conversion unit is configured to convert the value of each pixel in the mapped image into a pixel value in the second color coding format to obtain the second image.

[0396] Optionally, the step of processing the first image by the target model includes:

[0397] Performing feature extraction on the first image to obtain an initial feature map;

[0398] Performing feature extraction on the initial feature map under receptive fields of different sizes to obtain multiple feature maps; wherein different feature maps correspond to receptive fields of different sizes;

[0399] Performing feature fusion on the plurality of feature maps to obtain a first fused feature map;

[0400] Perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

[0401] Optionally, the pair of devices further comprises:

[0402] A low-frequency feature extraction module is used to extract features belonging to low-frequency information in the initial feature map based on a preset feature map to obtain a low-frequency feature map;

[0403] A multi-scale feature extraction module is used to extract features from the low-frequency feature map under a target receptive field to obtain a guided filtering feature map; wherein the target receptive field is the receptive field of different sizes except the receptive field with the largest size;

[0404] a fusion processing module, configured to fuse the guided filtering feature map and the first fused feature map to obtain a second fused feature map;

[0405] The step of performing feature reconstruction on the first fused feature map to obtain the image to be transcoded includes: performing feature reconstruction on the second fused feature map to obtain the image to be transcoded.

[0406] Optionally, the fusion processing module includes:

[0407] A mapping unit, configured to map the value of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map, wherein the target interval ranges from 0 to 1;

[0408] An adjustment unit is configured to adjust the values ​​of each feature point in the first fused feature map based on the values ​​of each feature point in the mapped guided filtering feature map to obtain the second fused feature map.

[0409] Based on the same inventive concept, the present disclosure further provides a device for acquiring a target model. FIG17 shows a schematic diagram of a framework of the device for acquiring a target model. The device includes:

[0410] a sample acquisition module, configured to acquire a video sample and a plurality of first image samples in a first color coding format;

[0411] an input module, configured to input a plurality of the first image samples into a model to be trained, and obtain image samples to be transcoded output by the model to be trained; wherein the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples;

[0412] a sample processing module, configured to process the image samples to be transcoded into converted image samples conforming to the second color coding format based on the first color coding format and the second color coding format; wherein the converted image samples are input to a codec to obtain a high dynamic range video;

[0413] an updating module, configured to decode a plurality of second image samples from the video sample based on a target image sample, and update parameters of the to-be-trained model based on the target image sample and the second image samples; wherein the target image sample is any one of the to-be-transcoded image sample, the intermediate image sample, and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the to-be-transcoded image into the converted image sample;

[0414] The acquisition module is used to determine the to-be-trained model after multiple updates as the target model corresponding to the first color coding format.

[0415] Optionally, the sample processing module includes:

[0416] a decoding unit, configured to decode a plurality of video frames from the video sample;

[0417] The conversion unit is configured to perform conversion processing on the plurality of video frames based on the color coding format and the data storage format of the target image samples, so as to obtain a plurality of second image samples.

[0418] Optionally, the first color encoding format is an RGB format, and the second color encoding format is a YUV422 format; wherein:

[0419] The color encoding format of the target image sample is the RGB format, and the data storage format is that the RGB value in the target image sample ranges from 0 to 1;

[0420] Alternatively, the color encoding format of the target image sample is the YUV format, and the data storage format is the YUV444 format or the YUV422 format.

[0421] Optionally, when the first color encoding format is RGB format, the value of each pixel of the first image sample is 0 to 1; when the first color encoding format is YUV format, the data storage format of the first image sample is YUV422 format.

[0422] Optionally, the device further comprises:

[0423] an extraction module, configured to obtain, from the first image sample, a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel;

[0424] a stitching processing module, configured to stitch the second sampling image sample and the third sampling image sample to obtain a stitched image with the same size as the first sampling image sample;

[0425] The input module is specifically used to input the first sampling image sample and the spliced ​​image sample into the model to be trained, and obtain the image sample to be transcoded based on the predicted image samples output by the model to be trained and corresponding to the first sampling image and the spliced ​​image sample respectively.

[0426] Optionally, the data storage format of the first image samples is YUV422 format, the model to be trained includes a first sub-model and a second sub-model, and before inputting the plurality of first image samples into the model to be trained, the apparatus further includes:

[0427] an extraction module, configured to obtain, from the first image sample, a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel;

[0428] The input module is specifically configured to: input the first sampled image sample into the first sub-model, and input the second sampled image sample and the third sampled image sample into the second sub-model;

[0429] The update module is specifically used to: update the parameters of the first sub-model based on the predicted image samples and the second image samples output by the first sub-model corresponding to the first sampling image; and update the parameters of the second sub-model based on the second image samples and the predicted image samples corresponding to the second sampling image samples and the third sampling image samples output by the second sub-model respectively.

[0430] An embodiment of the present disclosure further provides an electronic device, wherein a computer program stored in the electronic device enables a processor to execute the image processing method or the target model acquisition method as described in the embodiment of the present disclosure.

[0431] The embodiments of the present disclosure further provide a computer-readable storage medium, which stores a computer program that enables a processor to execute the image processing method or the target model acquisition method as described in the embodiments of the present disclosure.

[0432] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0433] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, commodity, or device that includes the element.

[0434] The above is a detailed introduction to an image processing method, system, target model acquisition method, device, equipment and medium provided by the present disclosure. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core ideas. At the same time, for general technical personnel in this field, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

[0435] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0436] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0437] References herein to "one embodiment," "an embodiment," or "one or more embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Furthermore, please note that instances of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0438] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0439] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present disclosure may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0440] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a plurality of first images to be processed, wherein the first images are in a first color coding format; Invoking a target model that matches the first color coding format, processing the plurality of the first images, and obtaining a plurality of images to be transcoded that conform to the first color coding format; wherein a dynamic range of the images to be transcoded is higher than a dynamic range of the first images; Based on a second color coding format and the first color coding format, processing the image to be transcoded into a second image, the second image conforming to the second color coding format; The plurality of second images are input into a codec so that the codec encodes the plurality of second images into a high dynamic range video.

2. The method according to claim 1, characterized in that: Before calling the target model matching the first color coding format and processing the plurality of first images, the method further comprises: determining a preprocessing corresponding to the first color coding format, wherein different first color coding formats correspond to different preprocessing; Performing the preprocessing on the first image to obtain an image to be input; wherein the amount of calculation corresponding to the image to be input is smaller than the amount of calculation corresponding to the first image, or the resolution corresponding to the image to be input is larger than the resolution corresponding to the first image; The calling of the target model matching the first color coding format to process the plurality of the first images comprises: The target model is used to process the plurality of images to be input, so as to obtain the plurality of images to be transcoded.

3. The method according to claim 2, characterized in that The preprocessing of the first image includes: When the first color encoding format is a YUV format, sampling data belonging to different color channels in the first image respectively to reduce or increase the resolution of the first image; When the first color encoding format is RGB format, the RGB value of each pixel in the first image is normalized so that the RGB value of each pixel in the image to be input is The RGB value range is 0 to 1.

4. The method according to claim 2, characterized in that: The value range of each pixel in the image to be input is 0 to 1; or, in the color space corresponding to the first color encoding format, the size belonging to the first channel in the image to be input is different from the size belonging to the second channel and the size belonging to the third channel; wherein the size belonging to the second channel is the same as the size belonging to the third channel.

5. The method according to any one of claims 2 to 4, characterized in that: The first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and a size of the first sampled image is larger than sizes of the second sampled image and the third sampled image; The step of processing the plurality of images to be input by using the target model to obtain the plurality of images to be transcoded includes: Splicing the second sampling image and the third sampling image to obtain a spliced ​​image; wherein the size of the spliced ​​image is the same as the size of the first sampling image; Inputting the first sampled image and the stitched image into the target model, wherein the target model is used to respectively improve the dynamic range of the first sampled image and the stitched image; The image to be transcoded is obtained based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

6. The method according to claim 2, characterized in that The first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and a size of the first sampled image is larger than sizes of the second sampled image and the third sampled image; The target model includes a first sub-model and a second sub-model, and the using the target model to process the plurality of images to be input to obtain the plurality of images to be transcoded includes: Using the first sub-model to improve the dynamic range of the first sampled image, to obtain a first predicted image; Using the second sub-model to improve the dynamic range of the second sampled image and the third sampled image, to obtain a second predicted image corresponding to the second sampled image and a third predicted image corresponding to the third sampled image; The image to be transcoded is obtained based on the first predicted image, the second predicted image and the third predicted image.

7. The method according to any one of claims 1 to 6, characterized in that: The step of processing the to-be-transcoded image into a second image based on the second color coding format and the first color coding format includes: When the first color coding format is the same as the second color coding format, in a color space of the second color coding format, adjusting the sizes of different channels of the image to be transcoded in the color space to obtain a second image; In a case where the first color coding format is different from the second color coding format, the color coding format of the image to be transcoded is converted into the second color coding format to obtain the second image.

8. The method according to claim 2, characterized in that: The value of each pixel in the image to be input is in the range of 0 to 1; and processing the image to be transcoded into a second image based on the second color coding format and the first color coding format includes: After inverse normalization, the values ​​of each pixel in the image to be transcoded are mapped to a color space corresponding to the second color encoding format to obtain a mapped image in the color space; The value of each pixel in the mapping image is converted into a pixel value in the second color coding format to obtain the second image.

9. The method according to any one of claims 1 to 8, characterized in that: When the target model processes the first image, it includes: Performing feature extraction on the first image to obtain an initial feature map; Performing feature extraction on the initial feature map under receptive fields of different sizes respectively to obtain multiple feature maps; wherein different feature maps correspond to receptive fields of different sizes; Performing feature fusion on the plurality of feature maps to obtain a first fused feature map; Perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

10. The method according to claim 9, characterized in that: After extracting features from the first image to obtain an initial feature map, the method further includes: Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; Under the target receptive field, feature extraction is performed on the low-frequency feature map to obtain the guided filtering feature Figure; wherein the target receptive field is the receptive field of the receptive fields of different sizes except the receptive field with the largest size; After obtaining the first fusion feature map, the method further includes: Fusing the guided filtering feature map and the first fused feature map to obtain a second fused feature map; The step of reconstructing the first fused feature map to obtain the image to be transcoded includes: Perform feature reconstruction on the second fused feature map to obtain the image to be transcoded.

11. The method according to claim 9, characterized in that: The step of fusing the guided filtering feature map and the first fused feature map to obtain a second fused feature map includes: Mapping the values ​​of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map, wherein the target interval ranges from 0 to 1; Based on the values ​​of each feature point in the mapped guided filtering feature map, the values ​​of each feature point in the first fused feature map are adjusted to obtain the second fused feature map.

12. A method for acquiring a target model, characterized in that: The method comprises: Acquire a video sample and a plurality of first image samples in a first color encoding format; Inputting a plurality of the first image samples into a model to be trained to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples; Based on the first color coding format and the second color coding format, the image samples to be transcoded are processed into converted image samples conforming to the second color coding format; wherein the converted image samples are used to input into a codec to obtain a high dynamic range video; Based on the target image sample, a plurality of second image samples are decoded from the video sample, and based on the target image sample and the second image sample, the parameters of the to-be-trained model are updated; wherein the target image sample is any one of the to-be-transcoded image sample, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the to-be-transcoded image into the converted image sample; The model to be trained after multiple updates is determined as the target model corresponding to the first color coding format.

13. The acquisition method according to claim 12, characterized in that: The step of decoding a plurality of second image samples from the video sample based on the target image sample comprises: Decoding a plurality of video frames from the video samples; Based on the color coding format and data storage format of the target image samples, the plurality of video frames are converted and processed respectively to obtain a plurality of the second image samples.

14. The acquisition method according to claim 12, characterized in that: The first color encoding format is RGB format, and the second color encoding format is YUV422 format; wherein: The color encoding format of the target image sample is the RGB format, and the range of the RGB value in the target image sample is 0 to 1; Alternatively, the color encoding format of the target image sample is the YUV format, and the data storage format is the YUV444 format or the YUV422 format.

15. The acquisition method according to claim 12, characterized in that: When the first color encoding format is RGB format, the value of each pixel of the first image sample is 0-1; when the first color encoding format is YUV format, the data storage format of the first image sample is YUV422 format.

16. The acquisition method according to claim 12, characterized in that: The data storage format of the first image samples is a YUV422 format. Before inputting the plurality of first image samples into the model to be trained, the method further includes: Acquire a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample; Splicing the second sampled image sample and the third sampled image sample to obtain a spliced ​​image with the same size as the first sampled image sample; Inputting a plurality of the first image samples into a model to be trained to obtain image samples to be transcoded output by the model to be trained, comprising: The first sampling image samples and the spliced ​​image samples are input into the model to be trained, and the image samples to be transcoded are obtained based on the predicted image samples output by the model to be trained and corresponding to the first sampling image and the spliced ​​image samples respectively.

17. The acquisition method according to claim 12, characterized in that: The data storage format of the first image samples is a YUV422 format, the model to be trained includes a first sub-model and a second sub-model, and before inputting the plurality of first image samples into the model to be trained, the method further includes: Acquire a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample; The step of inputting the plurality of first image samples into the model to be trained, obtaining the image samples to be transcoded output by the model to be trained, and updating the parameters of the model to be trained based on the target image samples and the second image samples, comprises: Inputting the first sampled image sample into the first sub-model, and inputting the second sampled image sample and the third sampled image sample into the second sub-model; Based on the second image sample and the predicted image sample output by the first sub-model and corresponding to the first sampled image, updating the parameters of the first sub-model; Based on the second image samples and the predicted image samples output by the second sub-model and corresponding to the second sampled image samples and the third sampled image samples respectively, the parameters of the second sub-model are updated.

18. An image processing system, characterized in that: The system is configured with a plurality of target models, and different target models correspond to different first color coding formats; the system comprises: An acquisition module, used for acquiring a plurality of first images to be processed, wherein the first images are in a first color coding format; a scheduling module, configured to call a target model matching the first color coding format, process the plurality of the first images, and obtain a plurality of images to be transcoded that conform to the first color coding format; wherein the dynamic range of the images to be transcoded is higher than the dynamic range of the first images; A post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the second image complies with the second color coding format; The codec module is configured to input the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

19. The system according to claim 18, characterized in that The target model comprises: a first feature extraction branch, a plurality of second feature extraction branches connected to the output end of the first feature extraction branch, a first fusion module connected to the output ends of the plurality of second feature extraction branches at the same time, and a reconstruction module connected to the output end of the first fusion module; wherein different second feature extraction branches correspond to receptive fields of different sizes; The first feature extraction branch is used to extract features from the first image to obtain an initial feature map; The second feature extraction branch is used to extract features from the initial feature map under the corresponding receptive field to obtain a feature map; The first fusion module is used to perform feature fusion on the plurality of feature maps to obtain a first fused feature map; The reconstruction module is used to perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

20. The system according to claim 19, characterized in that The target model further includes: a third feature extraction branch connected to the output end of the first feature extraction branch, and a second fusion module connected to the output end of the output end of the first fusion module, and the input end of the reconstruction module is connected to the output end of the second fusion module; wherein: The third feature extraction branch is used to perform feature extraction on the initial feature map using a preset feature map to obtain a low-frequency feature map; The fourth feature extraction branch in the second feature extraction branch is further used to extract features from the low-frequency feature map to obtain a guided filtering feature map; the receptive field corresponding to the fourth feature extraction branch is the receptive field except for the receptive field with the largest size; The second fusion module is used to fuse the guided filtering feature map and the first fused feature map to obtain a second fused feature map; The reconstruction module is specifically used to perform feature reconstruction on the second fused feature map to obtain the image to be transcoded.

21. The system according to claim 20, characterized in that The second fusion module is specifically used to map the values ​​of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map; and based on the values ​​of each feature point in the mapped guided filtering feature map, adjust the values ​​of each feature point in the first fused feature map to obtain the second fused feature map; The target interval ranges from 0 to 1.

22. The system according to claim 21, characterized in that The first color encoding format includes a YUV format, and the target model corresponding to the YUV format includes a first sub-model and a second sub-model, wherein; The structure of the first sub-model is the same as that of the second sub-model, and both include: the first feature extraction branch, a plurality of the second feature extraction branches, the first fusion module and the reconstruction module.

23. The system according to claim 21, characterized in that The plurality of second feature extraction branches include at least one second feature extraction branch of the first type and a second feature extraction branch of the second type; the receptive field of the second feature extraction branch of the second type is larger than the receptive field of the at least one second feature extraction branch of the first type; The second feature extraction branch of the first type includes a plurality of first feature extraction units connected in series, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels; wherein, the smaller the receptive field size of the second feature extraction branch, the smaller the convolution kernel size of the first convolution unit and the second convolution unit; The second feature extraction branch of the second type includes: a plurality of second feature extraction units connected in series in sequence, wherein the second feature extraction unit includes a third convolution unit, an activation unit and a pooling unit connected in series in sequence.

24. An electronic device, characterized in that: include: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the image processing method as described in any one of claims 1 to 11 or the target model acquisition method as described in any one of claims 12 to 17 is implemented.

25. A computer-readable storage medium, characterized in that: The computer program stored therein enables a processor to implement the image processing method described in any one of claims 1 to 11, or the target model acquisition method described in any one of claims 12 to 17 when executed.

Images