Image processing method and system, target model acquisition method and device, and medium

Abstract

The invention provides an image processing method and system, a target model obtaining method and device, equipment and a medium, and belongs to the technical field of image processing.The image processing method comprises the steps that a plurality of first images to be processed are obtained, and the first images adopt a first color coding format; calling a target model matched with the first color coding format, and processing the plurality of first images to obtain a plurality of to-be-transcoded images conforming to the first color coding format; wherein the dynamic range of the image to be transcoded is higher than that of the first image; based on a second color coding format and the first color coding format, the image to be transcoded is processed into a second image, and the second image accords with the second color coding format; and inputting the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, system, target model acquisition method, device and medium. Background Art

[0002] With the maturity of High Dynamic Range (HDR) shooting and video display technology, HDR video has gradually developed. Compared with traditional Standard Dynamic Range (SDR) video, HDR video has a larger dynamic range, a wider color gamut, and can show higher contrast and richer colors. Therefore, many consumer-grade Ultra-high definition (UHD) TVs are equipped with the function of playing HDR videos for HDR video playback.

[0003] With the development of HDR video, it is necessary to convert images of different formats into HDR video. In the related technology, images of different formats are generally sent to the codec for conversion and then encoded into HDR video. In this way, the conversion quality of images of different formats is uneven and the conversion quality is low. Summary of the invention

[0004] Based on the content of the background technology, the present disclosure proposes an image processing method, system, target model acquisition method, device, equipment and medium to improve the quality of HDR videos converted from images of different formats.

[0005] In a first aspect, the present disclosure provides an image processing method, the method comprising:

[0006] Acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0007] Invoking a target model that matches the first color coding format, processing the plurality of the first images, and obtaining a plurality of images to be transcoded that conform to the first color coding format; wherein a dynamic range of the images to be transcoded is higher than a dynamic range of the first images;

[0008] Based on a second color coding format and the first color coding format, processing the image to be transcoded into a second image, the second image conforming to the second color coding format;

[0009] The plurality of second images are input into a codec so that the codec encodes the plurality of second images into a high dynamic range video.

[0010] In a second aspect of the present disclosure, a method for acquiring a target model is provided, the method comprising:

[0011] Acquire a video sample and a plurality of first image samples in a first color encoding format;

[0012] Inputting a plurality of the first image samples into a model to be trained to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples;

[0013] Based on the first color coding format and the second color coding format, the image samples to be transcoded are processed into converted image samples conforming to the second color coding format; wherein the converted image samples are used to input into a codec to obtain a high dynamic range video;

[0014] Based on the target image sample, a plurality of second image samples are decoded from the video sample, and based on the target image sample and the second image sample, the parameters of the to-be-trained model are updated; wherein the target image sample is any one of the to-be-transcoded image sample, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the to-be-transcoded image into the converted image sample;

[0015] The model to be trained after multiple updates is determined as the target model corresponding to the first color coding format.

[0016] In a third aspect of the present disclosure, there is provided an image processing system, the system comprising: a plurality of target models, different target models corresponding to different first color coding formats;

[0017] An acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0018] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of the first images, and obtain a plurality of images to be transcoded that conform to the first color coding format; wherein the dynamic range of the images to be transcoded is higher than the dynamic range of the first images;

[0019] A post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0020] The codec module is configured to input the second image into a codec, so that the codec encodes a plurality of the second images into a high dynamic range video.

[0021] In the embodiment of the first aspect of the present disclosure, a plurality of first images to be processed can be obtained, and then a target model corresponding to a first color coding format can be called to process the plurality of first images to obtain a plurality of images to be transcoded in the first color coding format; thereafter, based on a second color coding format and the first color coding format, the images to be transcoded are processed into second images, and the plurality of second images are input into a codec, so that the codec encodes the plurality of second images into high dynamic range videos. Since the dynamic range of the image to be transcoded is higher than the dynamic range of the first image, the second image obtained based on the image to be transcoded is also an image with a high dynamic range, so that a high dynamic range video can be edited.

[0022] In the process of converting images of different first color coding formats into HDR videos, the present disclosure undergoes two processing stages. In the pre-processing stage, the dynamic range of the first image is improved by using a target model matching the first color coding format. In the post-processing stage, the image to be transcoded is processed into an image that conforms to the second color coding format, and then sent to the codec to be encoded into an HDR video, which can at least achieve the following effects:

[0023] In the pre-processing stage, the processing of the target model is matched with the color encoding format of the first image. Since the two are matched, the conversion quality of the dynamic range of the first image by the target model is improved, so that the first images of different color encoding formats can be converted into high-quality images to be transcoded;

[0024] In the post-processing stage, the image to be transcoded output by the target model is processed into a second image that conforms to the second color coding format, and then input into the codec to be encoded as an HDR video, thereby allowing the codec to directly encode the second image into an HDR video without the need to perform internal conversion on the image that does not conform to the second color coding format. On the one hand, the workload of the codec is reduced and the image conversion efficiency is improved; on the other hand, since the codec does not need to convert the color coding format, the problem of low quality of the encoded HDR video caused by the codec's lack of the ability to convert some of the first color coding formats can be avoided, thereby avoiding the problem of uneven conversion quality of images of different formats, so that first images of different color coding formats can all be converted into high-quality HDR videos.

[0025] An embodiment of the second aspect of the present disclosure can obtain video samples and first image sets corresponding to different first color coding formats, and decode the video samples based on a third color coding format to obtain multiple second image samples, and then use the multiple first image samples as input and the multiple second image samples as supervision to train a model to be trained to obtain a target model matching different first color coding formats; wherein the target image samples are any one of the image samples to be transcoded, intermediate image samples, and transcoded image samples belonging to the second color coding format output by the model to be trained during the training process, the intermediate image samples are images obtained in the process of converting the images to be transcoded into the transcoded image samples, the transcoded image samples are used to input a codec to obtain a high dynamic range video, the image samples to be transcoded belong to the first color coding format, and the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples.

[0026] In the embodiment of the second aspect, since the first image samples of different first color coding formats are used to train the target model matching the first color coding format, the target model matches the first image samples it processes, so that images of different color coding formats can be converted into high-quality images to be transcoded by the target model; and since the second image samples used for target model supervision are any one of the image samples to be transcoded, the intermediate image samples and the transcoded image samples output by the model to be trained, in the process of training the target model, the loss calculation can be performed based on the image output by the target model (first color coding format), or based on the image of any color coding format obtained during the process of converting the image output by the target model into the image of the second color coding format, or based on the image input to the codec (transcoded image sample). In this way, the processing capability of the target model for the input image can be suitable for the image input to the codec (in the second color coding format), thereby improving the quality of the image input to the codec, so that after the quality of the image input to the codec is improved, the quality of the HDR video converted from the images of different first color coding formats is also improved, solving the problem of uneven conversion quality of images of different formats.

[0027] The present disclosure also discloses an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the image processing method described in the first aspect or the target model acquisition method described in the second aspect is implemented.

[0028] The embodiment of the present disclosure also discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the image processing method described in the first aspect of the present disclosure or the target model acquisition method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings described below are only some implementation methods of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0030] Figure 1 A schematic diagram of YUV422 sampling in an embodiment of the present disclosure is shown;

[0031] Figure 2 A color gamut distribution diagram is shown;

[0032] Figure 3 A schematic diagram of the steps of an image processing method in one embodiment of the present disclosure is shown;

[0033] Figure 4a A schematic diagram of the processing flow of method A in an embodiment of the present disclosure is shown;

[0034] Figure 4b A schematic diagram of the steps of processing an input image using a target model in an embodiment of the present disclosure is shown;

[0035] Figure 5a A schematic diagram of the processing flow of method B in an embodiment of the present disclosure is shown;

[0036] Figure 5b A schematic diagram of the steps of processing an input image using a target model in an embodiment of the present disclosure is shown;

[0037] Figure 6 A schematic diagram of the process flow of the target model processing the first image in an embodiment of the present disclosure is shown;

[0038] Figure 7 The overall flow chart of the image processing method in a specific example is exemplarily shown;

[0039] Figure 8 A schematic diagram showing the steps of a method for acquiring a target model in an embodiment of the present disclosure is shown;

[0040] Fig. 9 A schematic diagram of a strategy for updating the parameters of a model to be trained according to an embodiment of the present disclosure is shown;

[0041] Fig.10 Another schematic diagram of a strategy for updating the parameters of a model to be trained in an embodiment of the present disclosure is shown;

[0042] Fig.11A schematic diagram showing a process of decoding a second image sample in an embodiment of the present disclosure is shown;

[0043] Fig.12 A schematic diagram showing the framework structure of the image processing system according to an embodiment of the present disclosure is shown;

[0044] Fig.13 A schematic diagram of the structure of the first target model in the embodiment of the present disclosure is shown;

[0045] Fig.14 A schematic diagram showing the structure of a second target model in an embodiment of the present disclosure is shown;

[0046] Figures 15a-15c A schematic diagram showing the structure of the second feature extraction branch with different receptive fields of different sizes in the embodiment of the present disclosure is shown;

[0047] Fig.16 A schematic diagram of the structure of the M1×1 layer, the M3×3 layer and the M5×5 layer in the second feature extraction branch in an embodiment of the present disclosure is shown;

[0048] Fig.17 A schematic diagram of the framework of a device for acquiring a target model in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0050] In the related technology, it may be necessary to convert SDR video into HDR video, or to convert low dynamic range video into high dynamic range HDR video, wherein the color encoding format of the video frame decoded from the video file used for conversion can be RGB format and YUV format. The conversion process is to extract video frames from the video, input the video frames into the model to improve the dynamic range, and then encode them into HDR video. Among them, for video frames in RGB format, the input and output of the model are both RGB format images, and the RGB images output by the model will be directly input into the codec, and the codec will convert the RGB format images into YUV format images, and then encode them into HDR video.

[0051] However, in the RGB to YUV conversion method of the codec tool, its color gamut does not meet the requirements of the BT.2020 color gamut under HDR. As a result, the quality of HDR adaptation converted from RGB format video frames is poor. Compared with the HDR video converted from YUV format video frames, there is a significant difference in display effect.

[0052] In view of this, the present disclosure proposes a solution that can reduce the difference between HDR videos converted by video frames of different color coding formats. The core concept of the solution is: according to the first color coding format of the video frame to be converted, call the adapted target model for processing to improve its dynamic range, then process the image output by the target model according to the second color coding format, obtain the image adapted to the second color coding format, and then send it to the codec to encode it as an HDR video. In this way, for video frames of different first color coding formats, after the dynamic range is improved, the color coding format when input to the codec is the second color coding format, thus, there is no need for the codec to convert the color coding format of the input image, thereby avoiding the problem of poor HDR video quality caused by the codec converting the image in RGB format because its color gamut does not meet the requirements of the BT2020 gamut under HDR. In addition, images of different coding formats can be processed using a matching target model so that the processing power of the model is adapted to the color coding format of the input image, thereby improving the conversion quality of images of different color coding formats, and helping to improve the quality of HDR videos converted to images of different color coding formats.

[0053] Based on the above core concept, the present disclosure also provides a solution for reducing the difference between HDR videos converted from video frames in different color coding formats. The solution uses a first image set in different first color coding formats as training samples and a second image set in a third color coding format as supervision, and trains a target model adapted to different first color coding formats; wherein the third color coding format can be a second color coding format and an intermediate color coding format. In this way, during the model training process, loss calculation can be performed based on the image input to the codec, so that the target model's processing capability for the input image can be suitable for the image (in the second color coding format) input to the codec. As a result, the quality of images in different color coding formats that need to be input to the codec after being converted is improved. After the quality of the image input to the codec is improved, the quality of the HDR video encoded thereby is also improved synchronously, thereby solving the problem of uneven conversion quality of images in different color coding formats in the related art from the perspective of model construction.

[0054] The above scheme proposed in the present disclosure is described in detail below.

[0055] Before introducing the image processing method of the present invention, some technical terms used in the present invention are first explained:

[0056] 1. Description of the basic format of HDR (HLG) video, as shown in Table 1 below, shows a schematic table of basic parameters of HDR (HLG) video:

[0057] Table 1-Basic parameters of HDR (HLG) video

[0058]

[0059]

[0060] Among them, sampling - 4:2:2, indicating the YUV422 format; quantization - 10bit, representing the P10 part of YUV422P10; color gamut - ITU-R BT2020, indicating the BT2020 color gamut; high dynamic range - HLG standard, indicating HDR conversion applicable to the HLG standard (there are also other HDR standards such as HDR10, HDR10+, etc.);

[0061] 2. Sampling of YUV422

[0062] YUV 4:2:2 sampling means that the data size of U (chrominance) and V (saturation) components is half of the Y (brightness) component sampling, and the Y component and UV component are sampled in a 2:1 ratio. Figure 1 As shown, a schematic diagram of YUV422 sampling is shown. Figure 1 In the figure, the Y component is represented by a cross × and the UV component is represented by a circle ○. Figure 1 In the example, the data size of Y is W×H (W represents width and H represents height), and the data of U and V are w / 2×H.

[0063] 3. BT.709 and BT.2020 color gamut

[0064] Color gamut or color space is a meaning, both refer to a collection of colors, describing the color coverage range, see Figure 2 As shown, a color gamut distribution diagram is shown, such as Figure 2 As shown, the closer the value is to the edge of the horseshoe, the higher the color saturation is, and the color gamut close to the edge of the horseshoe is the R.2020 color gamut.

[0065] In combination with the above technical terms, in one embodiment, an image processing method of the present disclosure is introduced, referring to Figure 3 As shown, a schematic diagram of the steps of an image processing method in an embodiment is shown. Figure 3As shown, the following steps may be specifically included:

[0066] Step S301: Acquire a plurality of first images to be processed.

[0067] The first image adopts a first color coding format.

[0068] In this embodiment, multiple first images can be decoded from the video to be processed. The video to be processed may be an SDR (Standard Dynamic Range) video or a video in a non-standard HDR video format. The multiple first images may be video frames in the video to be processed. In practice, each video frame in the video to be processed may be decoded as a first image.

[0069] Among them, the color coding format can represent the color coding method of the image. Generally speaking, the color coding formats of videos encoded by different color coding methods are different. Among them, the color coding formats of the video include RGB format, YUV format, etc. Different color coding formats are encoded in different color spaces. For example, for the RGB format, its color space is RGB space, that is, the value of the pixel in the image includes the value of the red component, the value of the green component and the value of the blue component; for the YUV format, its color space is YUV space, that is, the value of the pixel in the image includes the value of the brightness component, the value of the chroma component and the value of the saturation component.

[0070] In this embodiment, the first color encoding format may include an RGB format and a YUV format, wherein the first color encoding format of the first image may be an RGB format or a YUV format.

[0071] Step S302: calling a target model that matches the first color coding format, processing the plurality of first images, and obtaining a plurality of images to be transcoded that conform to the first color coding format;

[0072] The dynamic range of the image to be transcoded is higher than the dynamic range of the first image.

[0073] Among them, different first color coding formats can match different target models, such as the first image in YUV format can match the target model for YUV image processing, and the first image in RGB format can match the target model for RGB image processing, so that the target model and the color space of the first image to be processed are compatible, that is, the target model performs targeted processing on the first image within its color space.

[0074] The target model is used to improve the dynamic range of the first image.

[0075] Among them, multiple first images can be batch-inputted into the target model, so that the images to be transcoded corresponding to the multiple first images can be obtained at the same time. After being processed by the target model, the images to be transcoded with a higher dynamic range can be obtained, wherein the images to be transcoded are also in the first color coding format, that is, after being processed by the target model, the color coding format of the first image is not changed, but the dynamic range is improved.

[0076] Furthermore, during the process of processing the first image, the target model may not change the resolution and data storage format of the first image.

[0077] Step S303: Based on the second color coding format and the first color coding format, the image to be transcoded is processed into a second image, and the second image complies with the second color coding format.

[0078] The second color coding format may be a color coding format that complies with the coding method of the HDR video, such as the YUV format. In this way, the first color coding format and the second color coding format may be the same or different. In the same case, that is, the first image is in the YUV format and the second image is also in the YUV format, when the image to be transcoded is processed into the second image based on the second color coding format and the first color coding format, the image to be transcoded may be converted into a target data storage format under the second color format, such as the YUV422 format or the YUV444 format.

[0079] When the first color coding format and the second color coding format are different, such as the first image is in RGB format and the second image is in YUV format, based on the second color coding format and the first color coding format, the image to be transcoded can be processed into a second image according to the second color coding format, and the second image can be in YUV422 format or YUV444 format.

[0080] Step S304: input the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0081] After the image to be transcoded is converted into a second image, the second image may be directly input into a codec, wherein the codec may be an ffmpeg encoding tool, or other types of codecs, which are not particularly limited herein.

[0082] For example, using the image processing method of this embodiment, when the first input image is a video frame extracted from an SDR video, the HDR video obtained by processing is a video with a high dynamic range, that is, the SDR video is converted into an HDR video; when the first input image is a video frame extracted from a non-standard HDR video, the HDR video obtained by processing is a video with a high dynamic range, that is, the non-standard video is converted into a standard HDR video.

[0083] By adopting the technical solution of the embodiment of the present disclosure, on the first aspect, when the first image is processed by using the target model, since the color coding format of the target model matches that of the first image, the conversion quality of the dynamic range of the first image by the target model can be improved, thereby first images of different color coding formats can be converted into high-quality images to be transcoded.

[0084] Secondly, after obtaining the image to be transcoded output by the target model, the image to be transcoded is further processed into a second image that conforms to a second color coding format, and then the second image is input into the codec to be encoded into an HDR video, so that the codec no longer needs to perform color coding conversion on the image that does not conform to the second color coding format. Therefore, the problem of low quality of the encoded HDR video caused by the codec not having the ability to convert the first color coding format into the second color coding format can be avoided, thereby improving the conversion quality of images of different formats, so that first images of different color coding formats can all be converted into high-quality HDR videos.

[0085] In one embodiment, before the first image is input into the target model for processing, the first image may be preprocessed based on the first color coding format to reduce the amount of data of the image input into the target model, thereby reducing the amount of calculation of the target model and improving the image processing efficiency. Alternatively, in order to ensure the quality of the processed image to be transcoded, the resolution of the first image may be increased; specifically, the preprocessing corresponding to the first color coding format may be determined, and then the first image may be preprocessed to obtain the image to be input.

[0086] Among them, different first color coding formats correspond to different preprocessing, the calculation amount corresponding to the image to be input is smaller than the calculation amount corresponding to the first image, or the resolution corresponding to the image to be input is larger than the resolution corresponding to the first image.

[0087] Accordingly, when the first image is processed by using the target model to obtain the image to be transcoded, the input image can be processed by using the target model to obtain the image to be transcoded.

[0088] The case where the amount of calculation corresponding to the image to be input is smaller than the amount of calculation corresponding to the first image includes that the resolution of the image to be input is smaller than the resolution corresponding to the first image, and the values ​​of each pixel in the image to be input are limited to a smaller range. In the case where the amount of calculation corresponding to the image to be input is smaller than the amount of calculation corresponding to the first image, the amount of data of the image to be input can be smaller. In practice, the amount of data of the image to be input can be reduced by converting the data storage method of the first image.

[0089] When the first image is in YUV format, its data storage modes include YUV422, YUV444 and YUV420, and different data storage modes correspond to different data volumes. The data volume of YUV444 is larger than that of YUV422, and the data volume of YUV422 is larger than that of YUV420. For example, if the data volume of the first image in YUV444 format is to be reduced, at least one of the Y component, the U component and the V component may be downsampled, such as downsampling the U component and the V component.

[0090] When the first image is in RGB format, the amount of calculation for the first image can be reduced by reducing the RGB value of the pixel. For example, the RGB value of each pixel in the first image in RGB format can be reduced by reducing the RGB value to a certain range to reduce the amount of calculation for the pixel. Generally speaking, the range of RGB value is 0 to 255, and it can be limited to 0 to 10, thereby reducing the amount of calculation.

[0091] When the resolution of the input image is greater than the resolution of the first image, the quality of the processed image to be transcoded can be guaranteed, and the problem of low image quality caused by missing image information can be avoided. Specifically, the resolution of the first image can be improved by using nearest interpolation, bilinear interpolation, bicubic interpolation, etc.

[0092] For example, for the first image in YUV420 format, the data size of the brightness channel is W×H (W represents width and H represents height), and the data sizes of the chroma and saturation channels are both W / 2×H / 2. The data belonging to the chroma channel in the first image can be stretched by 2 times in the dimension of height H, and the data belonging to the saturation channel in the first image can be stretched by 2 times in the dimension of height H, so that the height H / 2 is stretched to H, and the input image in YUV422 format is obtained, and the resolution of the input image is higher than the resolution of the first image.

[0093] Among them, while the amount of calculation of the input image to the target model is small, it is also necessary to ensure that the input image does not lose much original information, so as to ensure the processing quality of the first image by the target model, and then ensure the quality of the converted HDR video. Therefore, in an optional example of this embodiment, when the first color encoding format is a YUV format, the preprocessing may refer to: sampling the data belonging to different color channels in the first image respectively to reduce or increase the resolution of the first image;

[0094] When the first color encoding format is an RGB format, the preprocessing may refer to: normalizing the RGB value of each pixel in the first image so that the RGB value of each pixel in the image to be input is in the range of 0 to 1.

[0095] Among them, the resolution of the image to be input is the target resolution. When the resolution of the first image is greater than the target resolution, the data belonging to the brightness channel in the first image can be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel can be down-sampled to obtain the image to be input; when the resolution of the first image is less than the target resolution, the data belonging to the brightness channel in the first image can be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel can be up-sampled to obtain the image to be input.

[0096] For example, taking the first color format as YUV format and the target resolution as the resolution in YUV422 format, if the first image is in YUV420 format, it does not meet the requirements of YUV422 and needs to be processed as a YUV422 image. The data belonging to the chroma channel in the first image can be stretched by 2 times in the dimension of height H, and the data belonging to the saturation channel in the first image can be stretched by 2 times in the dimension of height H, that is, the height H / 2 is stretched to H; wherein, a public upsampling algorithm can be used, such as nearest neighbor interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic), so that the format of the image to be input is finally made into YUV422 format.

[0097] As another example, taking the first color format as YUV format and the target resolution as the resolution in YUV422 format, the resolution of the first image is in YUV444 format, which does not meet the requirements of YUV422 and needs to be processed as a YUV422 image. The data belonging to the chroma channel in the first image can be downsampled by 2 times in the dimension of width W, and the data belonging to the saturation channel in the first image can be downsampled by 2 times in the dimension of width W, that is, the width W is downsampled to W / 2; wherein, a public upsampling algorithm can be used, such as nearest neighbor interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic).

[0098] Wherein, when the first color encoding format is RGB format, the RGB value of each pixel in the first image can be normalized so that the RGB value of each pixel in the input image is in the range of 0 to 1. Since the first image in RGB format is derived from an SDR video, after being decoded according to the RGB data format, if the quantization value of the SDR video is dep, its RGB value range is [0, 2 dep -1], for example, in the BT709 color space, if it is 8-bit quantization, the value range is [0,255]. Then it can be normalized according to the following formula (1):

[0099]

[0100] In formula (1), RGB_norm is the normalized RGB value of the pixel in the input image, and RGB is the RGB value of the pixel in the first image. The value range of RGB_norm is [0, 1], and the value range of RGB is [0, 255].

[0101] It should be noted that the RGB value corresponding to each pixel in the first image and the image to be input includes: a pixel value in the R (red) channel, a pixel value in the G (green) channel, and a pixel value in the B (blue) channel.

[0102] By adopting the above technical scheme, after the RGB values ​​of each pixel in the first image in RGB format are normalized, the amount of calculation of the first image in the target model can be reduced, thereby improving efficiency; after the resolution of the first image in YUV format is improved, the first image input to the target model can contain more image features, thereby improving the quality of the processed image to be transcoded; after the resolution of the first image in YUV format is reduced, the amount of calculation corresponding to the first image input to the target model can be reduced, thereby improving the processing efficiency of the target model.

[0103] Of course, in another example of this embodiment, the method of reducing the amount of calculation of the first image may also be: reducing the range of values ​​of each pixel in the first image, or reducing the size of a channel in the color space.

[0104] For example, no matter whether the first image is in YUV format or RGB format, the value of each pixel in the preprocessed image to be input is in the range of 0 to 1, including 0 and 1. Alternatively, no matter whether the first image is in YUV format or RGB format, in the color space corresponding to the first color encoding format, the size of the first channel of the preprocessed image to be input is twice the size of the second channel and the size of the third channel.

[0105] For example, for the first image sample in YUV format, the size of the brightness channel is twice the size of the color channel and the size of the saturation channel, that is, the YUV422 format; for the first image sample in RGB format, the size of the green channel may be twice the size of the blue channel and the size of the red channel. Of course, the first channel, the second channel and the third channel can be selected in RGB according to actual needs, which will not be repeated here.

[0106] Among them, when the first color encoding format is the YUV format, the range of the value of each pixel point can be narrowed, such as converted to the range of 0 to 1; when the first color encoding format is the RGB format, the range of the value of each pixel point can also be narrowed, such as converted to the range of 0 to 1.

[0107] For another example, when the first color encoding format is RGB format, the size of a certain color channel of the first image can be reduced while retaining the sizes of other color channels, such as reducing the size of the R channel while retaining the sizes of the G channel and the B channel. However, this may cause partial distortion of the image color.

[0108] Wherein, when the first color encoding format is a YUV format, the size of the color channel and the size of the saturation channel can be reduced, while the size of the brightness channel is retained, thereby reducing the amount of calculation while retaining the original color information of the first image. Specifically, in one example, as described above, the data belonging to the color channel and the data belonging to the saturation channel can be downsampled to reduce the size of the color channel and the size of the saturation channel. After preprocessing, the size of the brightness channel is twice the size of the color channel and the size of the saturation channel.

[0109] As described above, depending on the format of the video file from which the first image originates, the first color encoding format of the first image may be a YUV format or an RGB format. For different color encoding formats, different target models may be used to improve the dynamic range of the first image.

[0110] The following describes in detail the processing of the first image in two color coding formats.

[0111] #1, the first color encoding format is YUV format.

[0112] When the first image is in YUV format, the first image includes a Y component, a U component, and a V component, that is, data of a brightness channel, data of a color channel, and data of a saturation channel. Since the first image in YUV format is preprocessed and then input into the target model, in one example, the preprocessing may be used to increase the resolution of the first image or reduce the resolution of the first image. When the resolution is reduced, the data belonging to the brightness channel may be fully sampled, and the data belonging to the color channel and the data belonging to the saturation channel may be downsampled.

[0113] Accordingly, the image to be input may include a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and the size of the first sampled image is larger than that of the second sampled image and the third sampled image.

[0114] Wherein, when the image to be input includes a first sampled image, a second sampled image, and a third sampled image, since the sizes of the first sampled image, the second sampled image, and the third sampled image are different, the method of processing the first image in YUV format using the target model may include two methods:

[0115] In one method A, when the target model (target model M1-2 described in subsequent embodiments) processes the first sampling image, the second sampling image and the third sampling image, the three are treated as feature maps of different feature channels. This method B can reduce the complexity of the target model and is beneficial to the development of the target model.

[0116] In another method B, the target model (target model M1-1 described in subsequent embodiments) can use corresponding sub-models to process the first sampling image, the second sampling image and the third sampling image respectively, and then splice the images output after the respective processing to obtain the image to be transcoded. This method A can reduce the computational complexity of the target model for image processing and improve processing efficiency.

[0117] Specifically, refer to Figure 4a and Figure 4b As shown, Figure 4aA schematic diagram of the processing flow of method A is shown. Figure 4b FIG. 4 shows a schematic diagram of the steps of processing an input image using a target model. Figure 4a and Figure 4b As shown, the following steps may be specifically included:

[0118] Step S401: stitching the second sampled image and the third sampled image to obtain a stitched image having the same size as the first sampled image;

[0119] Among them, the format of the image input to the target model can be YUV422 format. In this case, the size of the first sampled image of the brightness channel is W×H, the second sampled image belonging to the color channel and the third sampled image belonging to the saturation channel are W / 2×H respectively; then the second sampled image and the third sampled image can be spliced ​​in the width dimension, so that the size of the spliced ​​image is W×H, which is the same as the size of the first sampled image.

[0120] Step S402: inputting the first sampled image and the stitched image into a target model, where the target model is used to increase the dynamic ranges of the first sampled image and the stitched image, respectively.

[0121] Since the first sampled image and the stitched image have the same size, the target model can treat the first sampled image and the stitched image as two images in different feature spaces for parallel processing, such as parallel convolution to extract features, and then reconstruct the extracted feature maps in parallel to obtain the predicted images corresponding to the first sampled image and the stitched image, such as Figure 4a As shown, it includes prediction image 1, prediction image 2 and prediction image 3. In practice, the prediction image exists in the target model in the form of data, such as a data matrix.

[0122] Among them, the predicted image 1 can be the data of the brightness channel, the predicted image 2 can be the data of the chrominance channel, and the predicted image 3 can be the data of the saturation channel.

[0123] Step S403: obtaining an image to be transcoded based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

[0124] Since the target model does not change the resolution of the input image when processing it, the size of the predicted image corresponding to the first sampled image is W×H, and the size of the predicted image corresponding to the spliced ​​image is also W×H. Since the spliced ​​image is obtained by splicing the second sampled image and the third sampled image in the width dimension, the predicted images 2 and 3 corresponding to the spliced ​​image can be fused with the predicted image 1 corresponding to the first sampled image to form Figure 1The image shown is then used to obtain the image to be transcoded.

[0125] By adopting the processing method of this embodiment, since the target model only needs to process the first sampled image and the stitched image in parallel to obtain the image to be transcoded, its model structure can be designed to be relatively simple, thereby reducing the software and hardware resources occupied by the target model and improving operating efficiency.

[0126] Specifically, refer to Figure 5a and Figure 5b As shown, Figure 5a FIG. 4 shows a schematic diagram of the processing flow of mode B. Figure 5b FIG. 4 shows a schematic diagram of the steps of processing an input image using a target model. Figure 5a As shown, the target model includes a first sub-model and a second sub-model, wherein the first sub-model is used to process a first sampled image of a brightness channel, and the second sub-model is used to process a second sampled image of a color channel and a third sampled image belonging to a saturation channel.

[0127] like Figure 5a and Figure 5b As shown, the following steps may be specifically included:

[0128] Step S501: using a first sub-model to increase the dynamic range of a first sampled image to obtain a first predicted image.

[0129] Step S502: using the second sub-model to increase the dynamic range of the second sampled image and the third sampled image, to obtain the first predicted image and the second predicted image corresponding to the second sampled image, and the third predicted image corresponding to the third sampled image.

[0130] Step S503: obtaining an image to be transcoded based on the first predicted image, the second predicted image and the third predicted image.

[0131] In this embodiment, the format of the image input to the target model can be YUV422 format. In this case, the size of the first sampled image of the brightness channel is W×H, and the size of the second sampled image belonging to the color channel and the third sampled image belonging to the saturation channel are W / 2×H respectively. Since the sizes of the first sampled image and the second sampled image are different, and also different from the size of the third sampled image, while the sizes of the third sampled image and the second sampled image are the same, the first sampled image can be input into the first sub-model, and the dynamic range of the first sampled image is processed by the first sub-model. Also, the second sampled image and the third sampled image are input into the second sub-model, and the dynamic range of the second sampled image is processed by the second sub-model, so as to obtain the first predicted image output by the first sub-model, and the second predicted image and the third predicted image output by the second sub-model.

[0132] Among them, the sizes of the second predicted image and the third predicted image output by the second sub-model are W / 2×H respectively, and the size of the first predicted image is W×H. In practice, the second predicted image and the third predicted image can be spliced ​​in the width dimension and then fused with the first predicted image to obtain the image to be transcoded.

[0133] It should be noted that the formats of the to-be-transcoded images obtained by the above two processing methods can both be in YUV422 format.

[0134] By adopting the processing method of this embodiment, since the target model can process the first sampling image, the second sampling image and the third sampling image of different sizes respectively through the first sub-model and the second sub-model, compared with the solution of method A, the amount of data processed by each sub-model when processing each sampling image in parallel is reduced, thereby improving the calculation speed and thus improving the processing efficiency of each sub-model.

[0135] #2, the first color encoding format is RGB format.

[0136] In the case where the first color encoding format is an RGB format, the first image is an RGB image, and the values ​​of each pixel in the preprocessed image to be input can be in the range of 0 to 1. In this case, the image to be input can be directly input into the target model, and the target model can extract features from the data of the three RGB color channels of the input image, process them, and reconstruct them into a high dynamic range image to be transcoded. Specifically, please refer to the subsequent processing example of the target model.

[0137] The structure of the target model adapted to the RGB format may be the same as the structure of the target model adapted to the YUV format in method A.

[0138] It should be noted that, when the input image to the target model is in RGB format and the RGB value of each pixel is normalized to a range between 0 and 1, the image to be transcoded output by the target model is also in RGB format, and the RGB value of each pixel is normalized to a range between 0 and 1.

[0139] In some embodiments, when the target model processes the first image, in order to improve the accuracy of the target model's processing of the first image, retain the feature information of the first image while increasing the dynamic range, when extracting image features, the target model can extract global, large area local, small area local, low-frequency area and other features of the image feature domain under receptive fields of different sizes, thereby extracting features with high expression levels for reconstructing the dynamic range.

[0140] Specifically, refer to Figure 6 As shown, a schematic diagram of the steps of processing the first image by the target model is shown, such as Figure 6 As shown, the following steps may be specifically included:

[0141] Step S601: extract features from the first image to obtain an initial feature map.

[0142] Step S602: extracting features from the initial feature map under receptive fields of different sizes to obtain multiple feature maps;

[0143] Among them, different feature maps correspond to receptive fields of different sizes.

[0144] Step S603: performing feature fusion on multiple feature maps to obtain a first fused feature map.

[0145] Step S604: reconstructing the features of the first fused feature map to obtain an image to be transcoded.

[0146] In this embodiment, the first image may be convolved to extract image features from the first image to obtain an initial feature map. Specifically, a 3x3 convolution layer may be used for convolution, and the feature channel may be expanded from 3 channels to nf channels (nf=64, 48, 32 or other channels).

[0147] Next, feature extraction of different scales is performed on the initial feature map. Different scales can be understood as receptive fields of different sizes, wherein the convolution kernel sizes used for receptive fields of different sizes are different. By performing feature extraction of different scales on the initial feature map, subtle features, local detail features, and global features in the first image can be fully extracted.

[0148] In one example, three or more receptive fields of different sizes can be used to extract features from the initial features, thereby gradually transitioning from subtle details to global features to extract features on the first image in each transition process, thereby fully extracting features from the first image.

[0149] Next, feature fusion can be performed on multiple feature maps. When performing feature fusion, multiple feature maps can be first spliced ​​in the channel dimension, and then the spliced ​​feature maps are convolved using a 1×1 convolution kernel to obtain a first fused feature map. As a result, the first fused feature map can fuse the features extracted under different receptive fields, thereby enhancing the expression of the first fused feature map to the first image.

[0150] Afterwards, feature reconstruction is performed on the first fused feature map, wherein the reconstruction of the first fused feature map may refer to reconstructing each pixel in the first fused feature map under a high dynamic range, so as to obtain the image to be transcoded.

[0151] It should be noted that, no matter it is a target model for YUV format or a target model for RGB format, the processing process of the target model on the first image can include the above-mentioned steps S601 to S604. The difference lies in that the model parameters used for feature extraction of the first image in different first color coding formats are different, that is, the model parameters of the target models for different first color coding formats may be different.

[0152] By adopting the above processing method, features of the first image can be extracted at different scales to obtain image features extracted under different receptive fields, and then subtle features, local detail features and global features of the first image can be extracted, thereby enhancing the expression of the first image by the first fusion feature map. As a result, the reconstructed transcoded image has high quality, can fully reflect the image features of the first image, and can ensure a high dynamic range, thereby improving the quality of subsequent HDR videos.

[0153] In some examples, the target model can also extract low-frequency area features from the first image, that is, features of low-frequency information. Specifically, after extracting features from the first image to obtain an initial feature map, features belonging to low-frequency information in the initial feature map can be extracted based on a preset feature map to obtain a low-frequency feature map; and, under the target receptive field, feature extraction is performed on the low-frequency feature map to obtain a guided filtering feature map.

[0154] Correspondingly, after obtaining the first fused feature map, the guided filter feature map and the first fused feature map can be fused to obtain a second fused feature map. When performing feature reconstruction, the second fused feature map can be reconstructed to obtain the image to be transcoded.

[0155] The target receptive field is a receptive field other than the largest receptive field among the receptive fields of different sizes, and specifically, may be the smallest receptive field.

[0156] The preset feature map may refer to a feature map of a guide image. Specifically, the guide image may be the first image itself, a different version of the first image, or an image completely different from the first image. As with the filtering operation. When extracting features from the initial feature map, the preset feature map will affect the statistics of a certain area in the corresponding spatial neighborhood of the initial feature map, so that the features of the area set in the initial feature map can be retained. For example, the features of the edge are retained.

[0157] Among them, the initial feature map can be guided filtered by using a guided filter to obtain a guided filtered feature map.

[0158] For example, it is assumed that the preset feature map is the feature map of the guide image I, the input is the first image p, and the input is the image q (guided filter feature map). The guide image I is set in advance according to the specific application, and can also be directly taken as the input image p. For the i-th pixel in the output image q, its calculation method can be expressed as the following formula (2):

[0159]

[0160] In formula (2), i and j are pixel labels, W ij is the filter kernel function, which is defined by the following formula (3):

[0161]

[0162] In formula (3), w k is the kth kernel function window, |w| is the number of pixels in the window, μ k and is the mean and variance of the guided image I within the window, and ε is the smoothing factor.

[0163] By using guided filtering to extract features from the initial feature map, effects such as noise reduction, detail smoothing, HDR compression, cutout, defogging, and joint sampling can be achieved, thereby improving the quality of feature extraction of the first image and improving the quality of the image to be transcoded.

[0164] Among them, after obtaining the guided filtering feature map, the guided filtering feature map and the first fused feature map can be fused, so that the features extracted after denoising, detail smoothing, HDR compression, cutout, defogging or joint sampling of the first image are fused to the first fused feature map, thereby, the second fused feature map obtained after the fusion can also have the effects of denoising, detail smoothing, HDR compression, cutout, defogging or joint sampling.

[0165] In an optional fusion, the values ​​of each pixel in the guided filter feature map and the first fused feature map can be weighted and summed according to preset weights. For example, the guided filter feature map can correspond to a weight, and the first fused feature map can also correspond to a weight. Then, the two are weighted and summed to achieve the fusion of the two feature maps.

[0166] In this way, the features of the guided filter feature map and the first fused feature map can be fused in a targeted manner according to their respective importance, thereby improving the targetedness of image processing. For example, if the weight of the first fused feature map accounts for a large proportion, the focus of the representation should be considered on the degree of feature expression of the first image; if the weight of the guided filter feature map accounts for a large proportion, the focus of the representation should be considered on the clarity of feature expression of the first image (smoothing and denoising are all for improving clarity).

[0167] In an optional example of the present embodiment, when the guided filter feature map and the first fused feature map are fused, the values ​​of each pixel point in the guided filter feature map can be used as weights to update the values ​​of each pixel point in the first fused feature map. Specifically, the values ​​of each feature point in the guided filter feature map can be mapped to a target interval to obtain a mapped guided filter feature map, and based on the values ​​of each feature point in the mapped guided filter feature map, the values ​​of each feature point in the first fused feature map can be adjusted to obtain a second fused feature map; wherein the target interval ranges from 0 to 1, specifically, including values ​​of 0 and 1.

[0168] In this example, the values ​​of each feature point in the guided filtering feature map can be normalized so that the value range of its feature points is 0 to 1. In this way, the normalized value of each feature point in the guided filtering feature map can be multiplied by the value of each feature point in the first fused feature map to adjust the value of each feature point in the first fused feature map to obtain a second fused feature map.

[0169] In this way, each feature point in the first fused feature map can be adjusted according to the guided filtering feature map, so that the first fused feature map can be repaired using the guided filtering feature map, and then deep fusion between the guided filtering feature map and the first fused feature map can be achieved. The fused second fused feature map can be a feature map with better clarity and stronger feature expression.

[0170] After being processed by the target model, an image to be transcoded with a higher dynamic range can be obtained, wherein the image to be transcoded is also in the first color coding format. Since the second image input to the codec needs to conform to the second color coding format, the color coding format of the image to be transcoded needs to be reconverted so that the color coding format of the image to be transcoded is the second color coding format, so as to avoid the codec converting the color coding format of the image to be transcoded. As described above, the image to be transcoded can be processed into the second image based on the second color coding format and the first color coding format.

[0171] In an example of this embodiment, when the first color coding format and the second color coding format are the same, only the size of the image to be transcoded in each color channel needs to be adjusted; and when the first color coding format and the second color coding format are different, the color coding format of the image to be transcoded can be converted.

[0172] Specifically, when the first color coding format is the same as the second color coding format, in the color space of the second color coding format, the sizes of different channels of the image to be transcoded in the color space are adjusted to obtain the second image;

[0173] In the case that the first color coding format is different from the second color coding format, the color coding format of the image to be transcoded is converted into the second color coding format to obtain a second image.

[0174] Wherein, the second color encoding format may be a YUV format. Then, when the first color encoding format is a YUV format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be sampled to obtain a second image. For example, when the image to be transcoded is in a YUV444 format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be downsampled twice in the width dimension to obtain a second image in a YUV422 format. For another example, when the image to be transcoded is in a YUV420 format, the data belonging to the chroma channel and the saturation channel in the image to be transcoded may be upsampled twice in the height dimension to obtain a second image in a YUV422 format.

[0175] Of course, if the image to be transcoded is an image in the YUV422 format, there is no need to process the image to be transcoded, and the image can be directly input into the codec.

[0176] When the first color coding format is RGB, the color coding format of the image to be transcoded can be converted into a second color coding format to obtain a second image, wherein the format of the second image can be YUV422 format, which means that the image to be transcoded must first be converted into an image in a color space corresponding to YUV.

[0177] In a further example, the value of each pixel in the input image will be limited to the range of 0 to 1. At this time, the value of each pixel in the image to be transcoded will also be limited to the range of 0 to 1. In the image, whether it is in YUV format or RGB format, the value of each component in the color space is in a larger range. Therefore, it is necessary to restore the value of each pixel in the image to be transcoded to the range of each component in the color space.

[0178] Accordingly, when the image to be transcoded is processed into a second image based on the second color coding format and the first color coding format, the value of each pixel in the image to be transcoded can be inversely normalized and then mapped to the color space corresponding to the second color coding format to obtain a mapped image in the color space, and the value of each pixel in the mapped image can be converted into a pixel value under the second color coding format to obtain a second image.

[0179] Specifically, when the first color coding format is the YUV format, that is, the same as the second color coding format, it can be converted into the second image in the second color coding format by only performing inverse normalization on the values ​​of each pixel in the transcoded image.

[0180] When the first color coding format is RGB format, that is, different from the second color coding format, it is necessary to first inverse normalize the value of each pixel in the image to be transcoded, and then map it to the color space corresponding to the second color coding format to obtain a mapped image in the color space; then convert the value of each pixel in the mapped image into a pixel value under the second color coding format to obtain a second image.

[0181] Specifically, when the first color encoding format is an RGB format, the image to be transcoded is an image in the BT2020 color gamut, wherein the formula for normalizing the RGB values ​​in the BT2020 color gamut can be seen in the following formula (4):

[0182]

[0183]

[0184]

[0185] In formula (4), R2, G2 and B2 represent the normalized pixel values ​​corresponding to different color channels of the pixel in the BT2020 color gamut; the value range of R, G, and B is [0, 255].

[0186] According to formula (4), the RGB values ​​are inversely normalized in the BT2020 color gamut. For details, please refer to the following formula (5):

[0187]

[0188] In formula (5), R1, G1 and B1 respectively represent the inverse normalized pixel values ​​corresponding to different color channels of the pixel point in the BT2020 color gamut; R2, G2 and B2 respectively represent the normalized pixel values ​​corresponding to different color channels of the pixel point in the BT2020 color gamut.

[0189] Next, the image obtained after inverse normalization can be mapped to the color space corresponding to the second color encoding format. Specifically, the inverse normalized pixel value of each pixel point can be converted into a pixel value under the YUV color domain. Specifically, the conversion can be made by referring to the following formula (6):

[0190]

[0191] Among them, for the data in YUV format, it is stored in a narrow range, and the value range is Y∈[16×2 dep-8 , 235×2 dep-8 ],U∈[16×2 dep-8 , 240×2 dep-8 ], V∈[16×2 dep-8 , 240×2dep-8 ]. When HDR video is quantized to 10 bits, Y∈[64,940], U∈[64,960], V∈[64,960]; when YUV444 is converted to YcbCr, the following formula (7) can be used:

[0192]

[0193]

[0194]

[0195] According to the value range of YUV, we can get Yc∈[0,1], Cb∈[-0.5,0.5], Cr∈[-0.5,0.5]; then according to formula (7), taking dep as 10 bits as an example, 2 dep-8 =2 10-8 =2 2 =4, the pixel value of each pixel in the image to be transcoded in the YUV444 format can be obtained, which can be determined according to the following formula (8):

[0196] Y=(219*Yc+16)×2 dep-8

[0197] U=(224*Cb+128)×2 dep-8

[0198] V=(224*Cr+128)×2 dep-8 Formula (8)

[0199] Next, the obtained YUV444 format image can be converted into an image in YUV422 format. Specifically, the data of the chroma channel and the data of the saturation channel in the image are downsampled by 2 times in the width dimension, that is, the width W is downsampled to W / 2. After the conversion is completed, the sizes of the chroma channel data and the saturation channel data are both W / 2×H.

[0200] The image processing method of this embodiment is described below with reference to a specific example. Figure 7 As shown, a specific flow chart of this example is shown, such as Figure 7 As shown, specifically including:

[0201] S1: Obtain an input video, and decode a video frame in a first color coding format, that is, a first image, from the input video.

[0202] like Figure 7As shown, the format of the input video is first determined. If it is an SDR type video, the first image in RGB format is decoded. If it is a non-standard HDR type video, the first image in YUV format is decoded.

[0203] For the first image in YUV format, the conversion is performed according to the process of steps S2-S4, such as Figure 7 In the flow of the left branch, for the first image in RGB format, the conversion is performed according to the process of steps S5-S8, such as Figure 7 The process of the right branch in the middle.

[0204] S2: Determine whether the first image is in YUV422 format. If so, directly input the first image into the target model. If not, convert the first image into a YUV422 sampling format.

[0205] Among them, when the first image is in YUV420 format, the UV component of the first image is stretched (upsampled) by 2 times in the height dimension, that is, the height is H / 2 stretched to H. A public upsampling algorithm can be used, such as nearest interpolation (nearest), bilinear interpolation (bilinear), and bicubic interpolation (bicubic). After the conversion is completed, the size of the Y data is W×H, and the size of the U and V data is W / 2×H.

[0206] When the first image is in YUV444 format, the width dimension is downsampled by 2, that is, the width W is downsampled to W / 2, and a public downsampling algorithm can be used, such as nearest interpolation, bilinear interpolation, and bicubic interpolation. After the conversion is completed, the sizes of U and V data are both W / 2×H.

[0207] After the above preprocessing, a first sampled image of the first image in the Y component, a second sampled image of the first image in the U component, and a third sampled image of the first image in the V component can be obtained.

[0208] S31: Processing the first sampled image, the second sampled image and the third sampled image using the target model M1-1, where the target model M1-1 includes a first sub-model and a second sub-model.

[0209] Because the size of Y is different from that of U and V, the first sub-model is used to process the first sampling image, and the second sub-model is used to process the second sampling image and the third sampling image to obtain the processed first predicted image, second predicted image and third predicted image. Because the target model M1-1 does not change the resolution of the input and output, the output is still in the 4:2:2 sampling format under the YUV format.

[0210] S32: Process the first sampling image, the second sampling image and the third sampling image using the target model M1-2.

[0211] Because the size of Y is different from that of U and V, the sizes of U and V data are both w / 2×H. Therefore, the second sampling image and the third sampling image can be merged in the width dimension, and the two W / 2 can be merged into 1 W. At this time, the second sampling image and the third sampling image have the same size as the first sampling image, and are directly input into the target model M1-2. The target model M1-2 processes the spliced ​​image and the first sampling image as feature maps of two channels. The output processed image to be transcoded includes a predicted image corresponding to the spliced ​​image and a predicted image corresponding to the first sampling image. After the two predicted images are spliced, they are the image to be transcoded, that is, the sampling format is still 4:2:2 under the YUV format.

[0212] S33: The process of the target model M1-1 or the target model M1-2 processing the first sample image, the second sample image and the third sample image is as follows (the processing process of the first sample image, the second sample image and the third sample image is the same, so they are all replaced by the input image):

[0213] S331, extracting features from the input image to obtain an initial feature map;

[0214] S332, extracting features by performing local features of small areas, local features of large areas, and global features, including: extracting features from the initial feature map under receptive fields of different sizes to obtain multiple feature maps; wherein the convolution kernel sizes used for feature extraction under receptive fields of different sizes are different;

[0215] S333, extracting low-frequency information of the initial feature map using guided filtering, including:

[0216] Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; and, under the target receptive field, feature extraction is performed on the low-frequency feature map to obtain a guided filtering feature map; wherein the target receptive field is the receptive field of different sizes except the receptive field with the smallest size;

[0217] S334, performing feature fusion on the multiple feature maps to obtain a first fused feature map, and adjusting the values ​​of each feature point in the first fused feature map based on the guided filtering feature map to obtain a second fused feature map;

[0218] S335, reconstructing features of the second fused feature map to obtain an image to be transcoded.

[0219] S4: Obtain the image to be transcoded, which may also be an image in YUV422 format, wherein the second color coding format may be YUV422 format. In this case, the image to be transcoded does not need to be processed, and the image to be transcoded can be directly input into the codec and encoded into an HDR video.

[0220] For the first image in RGB format, the processing specifically includes the following steps:

[0221] S5: Normalize the RGB values ​​of each pixel in the first image. Specifically, the first image may be an image in the BT709 color gamut, and the normalization may be performed according to the following formula (1):

[0222] In formula (9), R_norm is the normalized value of the pixel in the red channel, G_norm is the normalized value of the pixel in the green channel, and B_norm is the normalized value of the pixel in the blue channel.

[0223] S6: Input the input image obtained in S5 to the target model M2, wherein the structure of the target model M2 is the same as that of the target model M1-1. The processing process is as follows:

[0224] S61, extracting features from the input image to obtain an initial feature map;

[0225] S62, extracting features by performing local features of small areas, local features of large areas, and global features, including: extracting features from the initial feature map under receptive fields of different sizes to obtain multiple feature maps; wherein the convolution kernel sizes used for feature extraction under receptive fields of different sizes are different;

[0226] S63, extracting low-frequency information of the initial feature map by using guided filtering, including:

[0227] Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; and, under the target receptive field, feature extraction is performed on the low-frequency feature map to obtain a guided filtering feature map; wherein the target receptive field is the receptive field of different sizes except the receptive field with the smallest size;

[0228] S64, performing feature fusion on the multiple feature maps to obtain a first fused feature map, and adjusting the values ​​of each feature point in the first fused feature map based on the guided filtering feature map to obtain a second fused feature map;

[0229] S65, reconstructing features of the second fused feature map to obtain an image to be transcoded, wherein the RGB value of each pixel in the image to be transcoded is still in the range of 0 to 1, but is already in the BT2020 color gamut.

[0230] S7, inverse normalize the pixel values ​​of the image to be transcoded in the BT2020 color gamut, which can be determined by referring to the above formula (5).

[0231] S8: Map the inverse normalized image to a color space in a YUV format to obtain a mapped image in a color space in a YUV format, wherein the pixel value is a YCbCr value, which can be specifically determined according to formula (6).

[0232] S9: Convert the value of each pixel point in the mapped image into a pixel value in a second color coding format to obtain a second image. Specifically, the conversion is performed according to formula (8).

[0233] The obtained second image is an image in the YUV444 format. In practice, its resolution (also known as data storage format or size) can be changed to obtain a second image in the YUV422 format for input to a codec.

[0234] S10: Input the second image obtained in step S9 into the codec to obtain an HDR video. In this case, it can be seen that the codec does not need to convert the image in RGB format into an image in YUV format, which can avoid the problem of low quality of the encoded HDR video caused by the codec not being able to convert the image in RGB format into YUV format under BT2020.

[0235] Based on the same inventive concept, the embodiment of the present disclosure also provides a method for obtaining a target model, referring to Figure 8 As shown, a schematic diagram of the steps of the method for obtaining the target model is shown, such as Figure 8 As shown, the following steps may be specifically included:

[0236] Step S801: Acquire a video sample and a plurality of first image samples in a first color coding format.

[0237] In this embodiment, the video sample may be a standard HDR video, and the plurality of first image samples may be derived from an SDR video sample or a non-standard HDR video sample. The video sample from which the plurality of first image samples are derived and the standard HDR video may be videos of the same picture content, for example, a high dynamic range video and a low dynamic range video shot for the same scene, wherein the plurality of first image samples are decoded from the low dynamic range video, and the image samples (second image samples) used as labels may be decoded from the high dynamic range video.

[0238] Among them, there may be multiple first image samples corresponding to different first color encoding formats. For example, there may be 100 first image samples for the RGB format; there may also be 100 first image samples for the YUV format. First image samples of different formats are used to train different target models.

[0239] Step S802: input a plurality of first image samples into the model to be trained to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples.

[0240] The model to be trained may be trained with multiple first image samples as training samples, and during the training process, the image samples to be transcoded output by the model to be trained may be obtained, and the color coding format of the image samples to be transcoded is the first color coding format. When the multiple first image samples are input into the model to be trained, they may be input in batches.

[0241] Step S803: Based on the first color coding format and the second color coding format, the image sample to be transcoded is processed into a converted image sample conforming to the second color coding format;

[0242] The converted image samples are used to input the codec to obtain high dynamic range video.

[0243] Since the image to be transcoded needs to be converted and then input into the codec to be encoded into an HDR video, in this embodiment, the conversion of the image samples to be transcoded can refer to the conversion processing of the image to be transcoded in the above-mentioned image processing method embodiment, which will not be repeated here.

[0244] Step S804: based on the target image sample, a plurality of second image samples are decoded from the video sample, and based on the target image sample and the second image sample, the parameters of the model to be trained are updated;

[0245] The target image sample is any one of the image sample to be transcoded, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the image to be transcoded into the converted image sample.

[0246] During the training process of the model to be trained, the parameters of the model to be trained need to be updated. In the present disclosure, since the video sample is an HDR video, it can be used as the supervision of the model to be trained. Since the image to be transcoded output by the model to be trained needs to be converted into a converted image, multiple intermediate image samples may be obtained in the conversion process. As described in the above image processing embodiment, when the image sample to be transcoded is in RGB format, the corresponding intermediate image samples will be obtained in sequence through formula (5), formula (6) and formula (8). Therefore, the loss can be calculated based on any intermediate image sample, or based on the image sample to be transcoded or the converted image sample.

[0247] Accordingly, when multiple second image samples are decoded from video samples, they can be decoded according to the format of the target image samples, and the decoded second image samples have the same format as the target image samples. The format here can include: consistency of color coding format and consistency of data storage format.

[0248] Step S805: Determine the model to be trained after multiple updates as a target model corresponding to the first color coding format.

[0249] Among them, the training model can be iteratively trained multiple times according to the above process of S802-S804 until the training end condition is met, and the training model obtained under the training end condition can be determined as the target model. The training end condition may include the convergence of the training model, the loss between the target image sample and the second image sample is less than the preset loss, and the training reaches a preset number of times. In practice, any of the above conditions can be selected.

[0250] According to the technical solution of the embodiment of the present disclosure, a target model matching the first color coding format is obtained by training using first image samples of different first color coding formats, so that the target model matches the first image samples it processes, so that images of different color coding formats can be converted into high-quality images to be transcoded by the target model; and according to any one of the image samples to be transcoded, intermediate image samples and transcoded image samples output by the model to be trained, in the process of training the target model, loss calculation can be performed based on the image output by the target model (first color coding format), or based on the image of any color coding format obtained during the process of converting the image output by the target model into an image of the second color coding format, or based on the image input to the codec (transcoded image sample). In this way, the processing capability of the target model for the input image can be suitable for the image input to the codec (in the second color coding format), thereby improving the quality of the image input to the codec, so that after the quality of the image input to the codec is improved, the quality of the HDR video converted from the images of different first color coding formats is also improved, solving the problem of uneven conversion quality of images of different formats.

[0251] As described in the above-mentioned embodiment of the image processing method, the image input to the target model can be an image obtained by preprocessing the first image according to the first color coding format. In this embodiment, the first image sample used as the training sample can also be an image obtained by preprocessing the original image according to the first color coding format. Thus, when the first color coding format is RGB format, the value of each pixel point of the first image sample is 0 to 1, including 0 and 1; when the first color coding format is YUV format, the data storage format of the first image sample is YUV422 format.

[0252] In this way, the amount of computation required for the model to be trained can be reduced, thus improving operational efficiency.

[0253] In some examples, the first color encoding format is RGB format, and the second color encoding format is YUV422 format. In this case, the color encoding format of the target image sample is RGB format, and the data storage format is that the RGB value in the target image sample ranges from 0 to 1;

[0254] Alternatively, the color encoding format of the target image sample is in YUV format, and the data storage format is in YUV444 format or YUV422 format.

[0255] As described above, when decoding the second image sample, the decoding can be performed according to the color coding format and data storage format of the target image sample, because the target image sample is the image sample to be transcoded, the intermediate image sample or the converted image sample output by the model to be trained. In the case where the first color coding format is the YUV format, since the first color coding format and the second color coding format are the same, the color coding format of the image sample to be transcoded is also the second color coding format, and no conversion is required, and the second image sample can be directly decoded from the video sample according to the YUV format.

[0256] When the first color encoding format is RGB format, since the first color encoding format and the second color encoding format are different, the color encoding format of the transcoded image sample is also RGB format. As mentioned above, when the RGB of each pixel in the first image sample input to the model to be trained is a normalized value, the RGB of each pixel in the image sample to be transcoded is also a normalized value. In practice, it is necessary to convert it into YUV422 format, and multiple intermediate image samples will be obtained. The multiple intermediate image samples include: image samples in RGB format but the RGB values ​​of each pixel are inverse normalized values, YcbCr image samples, and YUV444 format image samples.

[0257] Reference Fig. 9 and Fig.10 As shown in FIG. 1 , schematic diagrams of strategies for updating the parameters of the training model are shown respectively. Fig. 9 As shown, the video sample used as the label is a standard HDR video sample. In one example, the target image sample can be selected from the image sample to be transcoded, that is, its color encoding format is RGB format, and the RGB value of each pixel is in the range of 0 to 1. In this case, the second image sample can be decoded from the video sample used as the label in RGB format. Fig.11 FIG. 1 shows a schematic diagram of a process of decoding a second image sample, including the following process:

[0258] First, because the data sampling format of the labeled video sample (HDR video file) is YUV422, the first step is to convert the input video format to YUV444 sampling format. Specifically, the video frame in YUV422 format can be decoded from the video sample, and then the data of the chroma channel in the video frame is stretched (upsampled) by 2 times in the width dimension, and the data of the saturation channel is stretched (upsampled) by 2 times in the width dimension, that is, the width is stretched from W / 2 to W. After the conversion, the size of the data of the brightness channel, the data of the chroma channel, and the data of the saturation channel of the video frame is W×H.

[0259] Next, the converted YUV444 format video frames are stored in a narrow range, with a value range of Y∈[16×2 dep-8 , 235×2 dep-8 ],U∈[16×2 dep-8 , 240×2 dep-8 ], V∈[16×2 de p -8 , 240×2 dep-8 ]. When HDR video is quantized to 10 bits, Y∈[64,940], U∈[64,960], V∈[64,960]. Specifically, YUV444 is converted to YcbCr according to the following formula (9):

[0260]

[0261]

[0262]

[0263] In formula (10), according to the numerical range of YUV, we can obtain Yc∈[0,1], Cb∈[-0.5,0.5], Cr∈[-0.5,0.5].

[0264] After that, YcbCr is converted to RGB format, where YcbCr and RGB under the BT2020 color gamut satisfy the following relationship (10):

[0265] Yc=0.2627R+0.6780G+0.0593B

[0266]

[0267]

[0268] The reverse solution can obtain the converted RGB value. Specifically, RGB and YCbCr satisfy the following relationship (11):

[0269]

[0270] According to the value range of Yc∈[0, 1], Cb∈[-0.5, 0.5], Cr∈[-0.5, 0.5], we can obtain R∈[-0.7373, 1.7373], G∈[-0.36795, 1.36795], and B∈[-0.9407, 1.9407].

[0271] Therefore, through the relationship (11), an intermediate image sample that conforms to the RGB format under the BT2020 color gamut can be obtained, and then the RGB value of each pixel in the intermediate image sample is normalized. Specifically, it can be processed according to formula (4), thereby obtaining a second image sample that conforms to the RGB format and whose RGB value is a normalized value. It can then be based on the second image sample and the image sample to be transcoded to update the parameters of the model to be trained.

[0272] like Fig.10 As shown, the video sample used as a label is a standard HDR video sample. In some examples, since the image samples to be transcoded need to be converted into converted image samples and then input into the codec to be encoded as HDR video samples, the color encoding format of the target image samples can also be in YUV format, and the data storage format is in YUV444 format or YUV422 format.

[0273] When the target image sample is in YUV444 format or YUV422 format, the decoding process of the video sample used as a label can be avoided. For example, when the target image sample is in YUV444 format, it is only necessary to convert the input video format into the YUV444 sampling format. On the one hand, the decoding process is short, which can reduce unnecessary calculations and improve the efficiency of decoding the second image sample. On the other hand, the original image features can be retained as much as possible, so that when calculating the loss, the loss calculation accuracy can be improved, thereby improving the image processing accuracy of the model to be trained.

[0274] For another example, when the target image sample is in the YUV422 format, it is only necessary to decode the second image sample from the video sample, which reduces the calculation process and improves the efficiency of decoding the second image sample.

[0275] Preferably, the intermediate image samples in the YUV444 format may be used as the target image samples. In this case, the second image samples in the YUV444 format may be decoded from the video samples used as labels.

[0276] In some embodiments, as described in the above-mentioned embodiment of the image processing method, when the color encoding format of the input first image sample is in YUV format, there may be two target models with different structures to process the image in the input first color encoding format. In the process of training the target model, there are also two corresponding training methods, specifically, including training method C and training method D:

[0277] In the training method C, the first image sample can be first decomposed into an image of a Y component, an image of a U component, and an image of a V component; specifically, before inputting a plurality of first image samples into the model to be trained, specifically: a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel can be obtained from the first image sample; and the second sampled image sample and the third sampled image sample are spliced ​​to obtain a spliced ​​image with the same size as the first sampled image sample;

[0278] Accordingly, the first sampled image samples and the spliced ​​image samples may be input into the model to be trained, and based on the predicted image samples output by the model to be trained and corresponding to the first sampled image and the spliced ​​image samples respectively, the image samples to be transcoded may be obtained.

[0279] As described in the above-mentioned image processing method example, the first image sample is an image sample in YUV422 format. The second sampled image sample and the third sampled image sample can be spliced ​​in the width dimension to obtain a size of W×H. The size of the spliced ​​image sample is the same as the size of the first sampled image sample. The model to be trained can process the first sampled image sample and the spliced ​​image sample as image samples of different channels.

[0280] During the training process, the model to be trained can output the predicted image sample corresponding to the first sampled image sample and the predicted image sample corresponding to the spliced ​​image sample. The two predicted image samples can be fused to obtain the image sample to be transcoded. Fig. 9 and Fig.10 The process shown calculates the loss based on the second image sample and the target image sample (any one of the image sample to be transcoded, the intermediate image sample and the converted image sample), and updates the parameters of the model to be trained according to the loss value.

[0281] The adoption of technical solution C of this embodiment can reduce the complexity of the target model, be beneficial to the training and development of the target model, and reduce the model's occupancy of software and hardware resources.

[0282] In the training method D, a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel may be obtained from the first image sample;

[0283] Accordingly, the first sampling image sample can be input into the first sub-model, and the second sampling image sample and the third sampling image sample can be input into the second sub-model; and when the model to be trained is updated, the parameters of the first sub-model can be updated based on the predicted image samples and the second image samples corresponding to the first sampling image output by the first sub-model; and the parameters of the second sub-model can be updated based on the second image samples and the predicted image samples corresponding to the second sampling image samples and the third sampling image samples output by the second sub-model respectively.

[0284] As described in the above-mentioned image processing method example, the first image sample is an image sample in YUV422 format, the second sampled image sample and the third sampled image sample are half of the first sampled image sample in width dimension, and the model to be trained may include a first sub-model and a second sub-model to process sampled image samples of different sizes respectively. Specifically, the first sub-model is used to process the first sampled image sample, and the second sub-model is used to process the second sampled image sample and the third sampled image sample of the same size.

[0285] During the training process, the first sub-model can output the first predicted image sample corresponding to the first sampled image sample, the second sub-model can output the second predicted image sample corresponding to the second sampled image sample, and the third predicted image sample corresponding to the third sampled image sample; the first predicted image sample, the second predicted image sample and the third predicted image sample are data of different channels, and the three are fused to obtain the image sample to be transcoded. Then, according to the above Fig. 9 and Fig.10 The process shown calculates the loss based on the second image sample and the target image sample (any one of the image sample to be transcoded, the intermediate image sample and the converted image sample), and updates the parameters of the first sub-model and the parameters of the second sub-model according to the loss value.

[0286] It should be noted that, since the first sub-model and the second sub-model are included, during the training process, the parameters of the first sub-model and the parameters of the second sub-model need to be updated. Since the two process image data of different channels, they are updated in one model;

[0287] Sample data belonging to the first sub-model can be extracted from the target image samples and the second image samples, such as sample data corresponding to the brightness channel. In this way, sample data corresponding to the brightness channel are respectively extracted from the target image samples and the second image samples. Based on the two sample data, the loss corresponding to the first sub-model is calculated, and then the parameters of the first sub-model are updated according to the loss.

[0288] Similarly, sample data belonging to the second sub-model can be extracted from the target image samples and the second image samples, such as sample data corresponding to the hue and saturation channels. In this way, sample data corresponding to the bright hue and saturation channels are respectively extracted from the target image samples and the second image samples. Based on the two sample data, the loss corresponding to the second sub-model is calculated, and then the parameters of the second sub-model are updated according to the loss.

[0289] For example, taking the target image sample as an image sample in YUV444 format, sample data Y1 belonging to the Y channel can be extracted from the target image sample, and sample data Y1' of the Y channel can be extracted from the second image sample, and the parameters of the first sub-model can be updated according to the sample data Y1 and the sample data Y1'. Also, sample data U1 belonging to the U channel can be extracted from the target image sample, and sample data U1' of the U channel can be extracted from the second image sample. Similarly, sample data V1 belonging to the V channel can be extracted from the target image sample, and sample data V1' of the V channel can be extracted from the second image sample, and the parameters of the second sub-model can be updated according to the sample data U1 and the sample data U1', and the sample data V1 and the sample data V1'.

[0290] In this way, the updating accuracy of the training model can be improved.

[0291] By adopting the scheme of method D, the first sampling image samples, the second sampling image samples and the third sampling image samples of different sizes can be processed respectively by the first sub-model and the second sub-model. In this way, compared with the scheme of method C, the amount of data processed by each sub-model when processing each sampling image sample in parallel is reduced, thereby improving the calculation speed of the model and thus improving the processing efficiency of each sub-model.

[0292] Next, a method for obtaining a target model is described by way of example with a specific example. Figure 9-11 As shown, the specific process may include the following:

[0293] S1', obtaining a video sample used as a label and a video sample used as an input, wherein a second image sample can be decoded from the video sample used as the label, and a first image sample can be decoded from the video sample used as the input, wherein the first image sample is in a first color coding format;

[0294] Specifically, the process of decoding the first image sample from the video sample as input may be as follows:

[0295] A video frame is decoded from the input video sample, i.e., the standard HDR video sample. After that, the video frame is preprocessed corresponding to the first color encoding format. Specifically, for the video frame in the YUV format, its data storage format can be converted into the YUV422 format; for the video frame in the RGB format, the value of each pixel can be normalized, thereby reducing the amount of data storage.

[0296] Specifically, Fig.11 As shown, the process of decoding the second image sample from the standard HDR video sample may include the process of:

[0297] Process 1: A video frame in a second color coding format, that is, a video frame in a YUV format, can be decoded from the standard HDR video sample, and the data storage format of the video frame in the YUV format is transformed into a YUV422 format to obtain a second image sample SA in a YUV422 format;

[0298] Process 2: A video frame in a second color coding format, that is, a video frame in a YUV format, can be decoded from the video sample of the tag, and the data storage format of the video frame in the YUV format is transformed into a YUV444 format to obtain a second image sample SB in a YUV444 format;

[0299] Process 3: A video frame in the second color coding format, i.e., a YUV format video frame, can be decoded from the labeled video sample. The data storage format of the YUV format video frame is transformed into the YUV444 format. Then, the color coding format of the YUV444 format video frame is converted into the RGB format. The values ​​of each pixel in the obtained RGB format video frame are normalized to obtain a second image sample SC in the RGB format.

[0300] The above three types of second image samples may be used in updating the model to be trained, specifically, as described in subsequent steps.

[0301] S2', for the first image sample whose first color encoding format is YUV, there are the following training process A and training process B; wherein the training process A refers to steps S21'-S24', and the training process B refers to steps S25'-S29':

[0302] S21′: acquiring a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0303] S22′: splicing the second sampled image sample and the third sampled image sample to obtain a spliced ​​image with the same size as the first sampled image sample;

[0304] S23': inputting the first sampled image sample and the spliced ​​image sample into the model to be trained, and obtaining the image sample to be transcoded based on the predicted image samples output by the model to be trained and corresponding to the first sampled image and the spliced ​​image sample respectively; wherein the format of the image sample to be transcoded is YUV422 format;

[0305] S24': Calculate the loss using the image sample to be transcoded (and the target image sample is the image sample to be transcoded) and the second image sample SA (YUV422 format), update the parameters of the model to be trained according to the obtained loss value, and determine the model to be trained after multiple updates as the target model.

[0306] Training process B, the model to be trained in the training process B includes a first sub-model and a second sub-model, wherein:

[0307] S25′: acquiring a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0308] S26′: input the first sampled image sample into the first sub-model, and input the second sampled image sample and the third sampled image sample into the second sub-model;

[0309] S27′: determining a first loss value based on the second image sample SA and the predicted image sample output by the first sub-model and corresponding to the first sampled image sample, and updating the parameters of the first sub-model based on the first loss value;

[0310] S28′: determining a second loss value based on the second image sample SA and the predicted image samples output by the second sub-model corresponding to the second sampled image sample and the third sampled image sample, and updating the parameters of the second sub-model based on the second loss value;

[0311] S29': Calculate the overall loss value based on the first loss value and the second loss value. Specifically, the overall loss value can be calculated by weighted summation. Then, based on the overall loss value, fine-tune the parameters of the first sub-model and the second sub-model, and determine the model to be trained after multiple updates as the target model.

[0312] The target model can be used to process images input in YUV422 format to improve the dynamic range of images in YUV422 format.

[0313] S3', for a first image sample whose first color encoding format is RGB, has the following training process:

[0314] S31': inputting the first image sample into the model to be trained;

[0315] S32′: Obtain an image sample to be transcoded corresponding to the first image sample output by the model to be trained, wherein the image sample to be transcoded is in RGB format and is located in the BT2020 color space, and the value of each pixel in the image sample to be transcoded is 0 to 1;

[0316] S33': Transcode the image samples to be transcoded according to the YUV format, and obtain the following intermediate image samples in sequence:

[0317] Intermediate image sample 1: a sample for inverse normalizing the values ​​of each pixel in the image sample to be transcoded, wherein the RGB value of each pixel in the intermediate image sample 1 is within the range of the BT2020 color gamut;

[0318] Intermediate image sample 2: obtained by mapping the value of each pixel point of intermediate image sample 1 to the YUV color gamut. The value of each pixel point in intermediate image sample 2 is within the range of YCbCr.

[0319] Intermediate image sample 3: The resolution of intermediate image sample 2 is transformed to change its data storage format. The format of intermediate image sample 3 is YUV444.

[0320] Converted image samples: The resolution of the intermediate image sample 3 is converted again to change its data storage format. The format of the converted image sample is YUV442.

[0321] S34': Update the parameters of the model to be trained, including the following update methods:

[0322] Method 1: Fig. 9 As shown, based on the image sample to be transcoded and the second image sample SC, a third loss value of the model to be trained is calculated, and the parameters of the model to be trained are updated based on the third loss value;

[0323] Method 2: If Fig.10 As shown, based on the intermediate image sample 3 and the second image sample SB, a fourth loss value of the model to be trained is calculated, and the parameters of the model to be trained are updated based on the fourth loss value;

[0324] Method 3: Based on the converted image sample and the second image sample SA, a fifth loss value of the model to be trained is calculated, and the parameters of the model to be trained are updated based on the fifth loss value;

[0325] S35': The model to be trained obtained in step S34' and updated multiple times is used as the target model corresponding to the RGB format.

[0326] Among them, any of the above-mentioned parameter updating methods can obtain a model to be trained with higher accuracy, but when the above-mentioned method 2 is adopted, compared with method 1, since it uses the converted YUV format image and the second image sample SB to calculate the loss, the capability of the target model can match the HDR video to be converted, and the quality of the image to be transcoded can be improved; and when the above-mentioned method 2 is adopted, compared with method 3, since the process of converting from the YUV format image to the YUV444 format image is relatively simple, the efficiency of obtaining the second image sample can be improved, and since the YUV444 format image contains more image features, the accuracy of the loss calculation can be improved, thereby improving the accuracy of the target model.

[0327] Based on the same inventive concept, the present disclosure also provides an image processing system, referring to Fig.12 As shown, a schematic diagram of the framework structure of the system is shown, as Fig.12 As shown, it may specifically include: a plurality of target models, different target models corresponding to different first color coding formats;

[0328] An acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0329] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of the first images, and obtain a plurality of images to be transcoded that conform to the first color coding format; wherein the dynamic range of the images to be transcoded is higher than the dynamic range of the first images;

[0330] A post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0331] The codec module is configured to input the second image into a codec, so that the codec encodes a plurality of the second images into a high dynamic range video.

[0332] Among them, the system embodiment can refer to the description of the above-mentioned image processing method embodiment, and will not be repeated here.

[0333] By adopting the image processing system, different target models can be called for processing first images of different first color coding formats, and the images to be transcoded output by different target models can enter different post-processing processes, so that first images of different color coding formats can all be matched and processed by the target models to improve the quality of the images to be transcoded; and the codec module can directly input the converted second image that conforms to the second color coding format into the codec, so that the codec directly encodes the second image into an HDR video, thereby eliminating the need for the codec to perform internal conversion on images that do not conform to the second color coding format. On the one hand, the workload of the codec is reduced and the image conversion efficiency is improved; on the other hand, the problem of low quality of the encoded HDR video caused by the codec's inability to convert some of the first color coding formats can be avoided, so that first images of different color coding formats can all be converted into high-quality HDR videos.

[0334] Next, the structure of the target model is described. In an optional example, no matter whether the first color encoding format is the YUV format or the RGB format, the target model may include the following structure, referring to Fig.13 As shown, a schematic diagram of the structure of a target model is shown, such as Fig.13 As shown, it may specifically include:

[0335] A first feature extraction branch, a plurality of second feature extraction branches connected to an output end of the first feature extraction branch, a first fusion module connected to an output end of the plurality of second feature extraction branches, and a reconstruction module connected to an output end of the first fusion module; wherein different second feature extraction branches correspond to receptive fields of different sizes;

[0336] The first feature extraction branch is used to extract features from an input image to obtain an initial feature map; the input image is an image to be input after the first image is preprocessed, and the value of each pixel in the image to be input satisfies a first preset condition, or, in a color space corresponding to the first color encoding format, the size of each color channel of the image to be input in the color space satisfies a second preset condition;

[0337] The second feature extraction branch is used to extract features from the initial feature map under the corresponding receptive field to obtain a feature map;

[0338] A first fusion module, used for performing feature fusion on multiple feature maps to obtain a first fused feature map;

[0339] The reconstruction module is used to reconstruct the features of the first fused feature map to obtain an image to be transcoded.

[0340] In this embodiment, the image to be input (or the first image) can be input into the first feature extraction branch. The first feature extraction branch can be a 3x3 convolution layer for performing convolution processing on the input image.

[0341] Since different second feature extraction branches among the multiple second feature extraction branches correspond to receptive fields of different sizes, different second feature extraction branches can perform convolution processing of different scales. It can be understood that a second feature extraction branch with a larger receptive field can correspond to a larger convolution kernel, and a second feature extraction branch with a smaller receptive field can correspond to a smaller convolution kernel.

[0342] Among them, the first fusion module may include a splicing unit and a fusion unit, the splicing unit is used to splice the feature maps output by multiple second feature extraction branches, and the fusion unit can be used to perform 1×1 convolution on the spliced ​​feature maps.

[0343] Among them, the reconstruction module can be used to reconstruct the features of the obtained first fusion feature map, so as to output a predicted image.

[0344] In a specific implementation of this example, refer to Fig.14 As shown, a schematic diagram of the structure of the second target model is shown, such as Fig.14 As shown, the target model may further include:

[0345] A third feature extraction branch connected to the output end of the first feature extraction branch, and a second fusion module connected to the output end of the first fusion module and the output end of the fourth feature extraction branch, and an input end of the reconstruction module connected to the output end of the second fusion module; wherein:

[0346] The third feature extraction branch is used to extract the features belonging to the low-frequency information in the initial feature map by using a preset feature map to obtain a low-frequency feature map;

[0347] The fourth feature extraction branch in the second feature extraction branch is further used to extract features from the low-frequency feature map to obtain a guided filtering feature map; the receptive field corresponding to the fourth feature extraction branch is not the largest among the multiple second feature extraction branches;

[0348] A second fusion module is used to fuse the guided filtering feature map and the first fused feature map to obtain a second fused feature map;

[0349] The reconstruction module is specifically used to reconstruct the features of the second fused feature map to obtain the image to be transcoded.

[0350] In some examples, the second fusion module is specifically used to map the value of each pixel point in the guided filter feature map to a target interval to obtain a mapped guided filter feature map; and based on the value of each pixel point in the mapped guided filter feature map, adjust the value of each pixel point in the first fused feature map to obtain a second fused feature map; wherein the target interval ranges from 0 to 1.

[0351] The processing process can be described with reference to the above-mentioned embodiment of the image processing method, and will not be described in detail here.

[0352] Among them, the first color coding format includes YUV format and RGB format. When the first color coding format is YUV format, as described in the embodiments of the above-mentioned image processing method and the target model acquisition method, there may be target models with different structures. In this embodiment, the target model corresponding to the YUV format may include a first sub-model and a second sub-model, wherein the structure of the first sub-model is the same as the structure of the second sub-model, and both include: a first feature extraction branch, multiple second feature extraction branches, the first fusion module, and a reconstruction module.

[0353] In this example, although the target model includes two branches (the first sub-model and the second sub-model), the structure of each branch can be the same, that is, both have reference Fig.14 The model structure shown can enable the image belonging to the Y component, the image belonging to the U component and the image belonging to the V component in the first image to be processed through independent sub-models, thereby improving the processing accuracy and quality.

[0354] Of course, in another example, the target model corresponding to the YUV format may be a model having one branch, that is, a model such as Fig.14 The comprehensive model shown in this case needs to splice the data belonging to the U component (chrominance channel) and the V component (saturation channel) in the first image to obtain data of the same size as the data of the Y component (brightness channel). In this way, the spliced ​​image obtained after splicing has the same size as the image of the brightness channel, so that the target model can process the spliced ​​image and the image of the Y component as images of different channels in parallel, thereby reducing the complexity of the target model and the operating resources occupied by the target model, thereby improving processing efficiency.

[0355] Reference Figures 15a-15c As shown, the structural schematic diagrams of the second feature extraction branches with different sizes of receptive fields are shown respectively, Figures 15a-15c As shown, the plurality of second feature extraction branches include at least one second feature extraction branch of the first type and a second feature extraction branch of the second type. Figures 15a-15cIn the above figure, conv3×3 represents a convolutional layer with a kernel size of 3×3, FC represents a fully connected layer, and the symbol represents feature fusion, symbol represents feature addition, GMP represents global maximum pooling; Relu represents the activation function layer, and conv3-d2 represents a convolutional layer with a convolution kernel size of 3×3 and a stride of 2.

[0356] The first type of second feature extraction branch includes a plurality of first feature extraction units connected in series, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels; the smaller the receptive field size of the second feature extraction branch, the smaller the convolution kernel size of the first convolution unit and the second convolution unit;

[0357] Among them, the second feature extraction branch of the second type includes: a plurality of third feature extraction units connected in series in sequence, and a fifth convolution unit, an activation unit and a pooling unit are connected in series in sequence in the third feature extraction unit.

[0358] like Fig.15a As shown, a schematic diagram of the structure of the second feature extraction branch L1 corresponding to the first receptive field in the second feature extraction branch of the first type is shown, Fig.15a As shown, the second feature extraction branch corresponding to the first receptive field includes: a plurality of first feature extraction units connected in series in sequence, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels.

[0359] in, Fig.15a The second feature extraction branch shown can be called the second feature extraction branch L1, L1 includes first feature extraction units connected in series, each first feature extraction unit includes a first convolution unit and a second convolution unit M3, wherein the first convolution unit can be a 3×3 convolution layer, the second convolution unit M3 includes two first convolution branches in parallel, and the branch L1-1 in the two first convolution branches includes an M1×1 layer, an activation function ReLu layer, an M1×1 layer, an activation function ReLu layer and an M1×1 layer connected in series.

[0360] The branch L1-2 in the two first convolution branches includes an M3×3 layer, an activation function ReLu layer, an M3×3 layer, an activation function ReLu layer and an M3×3 layer connected in series.

[0361] Among them, the second convolution unit also includes a splicing subunit connected to the output ends of the two first convolution branches, a 3×3 convolution layer Conv3×3, a GMP (global maximum pooling) layer, and two fully connected layers FC connected to the GMP (global maximum pooling) layer, and a fusion subunit connected to the output end of a fully connected layer FC and branch L1-1, as shown in the figure As shown in the figure, the fusion subunit connected to the output end of another fully connected layer and branch L1-2, and the feature addition layer connected to the output ends of the two fusion subunits at the same time, as shown in the figure shown.

[0362] Among them, the structures of M1×1 layer, M3×3 layer and M5×5 layer can refer to Fig.16 As shown in the figure, the structures of the three are roughly the same, including a convolution layer, an activation function ReLu layer, a 1×1 convolution layer and a Sigmoid layer and a residual branch connected in series. The residual branch is used to fuse the features output by the activation function ReLu layer with the features output by the Sigmoid layer. The difference is that the sizes of the convolution kernels used in the first convolution layer in the M1×1 layer, the M3×3 layer and the M5×5 layer are different. The size of the convolution kernel used in the first convolution layer in the M1×1 layer is 1×1, the size of the convolution kernel used in the first convolution layer in the M3×3 layer is 3×3, and the size of the convolution kernel used in the first convolution layer in the M3×3 layer is 3×3, and the step size is 2.

[0363] like Fig.15b As shown, a schematic diagram of the structure of the second feature extraction branch corresponding to the second receptive field in the second feature extraction branch of the first type is shown, such as Fig.15b As shown, the second feature extraction branch corresponding to the second receptive field includes: a plurality of first feature extraction units connected in series in sequence, the two first feature extraction units include a first convolution unit and a second convolution unit M5, the second convolution unit M5 includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels.

[0364] in, Fig.15b The second feature extraction branch shown can be called the second feature extraction branch L2, L2 includes first feature extraction units connected in series, each first feature extraction unit includes a first convolution unit and a second convolution unit M5, wherein the first convolution unit can be a 3×3 convolution layer with a step size of 2, the second convolution unit M5 includes two first convolution branches in parallel, and the branch L2-1 in the two first convolution branches includes an M3×3 layer, an activation function ReLu layer, an M3×3 layer, an activation function ReLu layer and an M3×3 layer connected in series.

[0365] The branch L2-2 in the two first convolution branches includes an M5×5 layer, an activation function ReLu layer, an M5×5 layer, an activation function ReLu layer and an M5×5 layer connected in series.

[0366] Among them, the second convolution unit M5 also includes a splicing subunit connected to the output ends of the two first convolution branches, a 3×3 convolution layer, a GMP (global maximum pooling) layer, and two fully connected layers FC connected to the GMP (global maximum pooling) layer, and a fusion subunit connected to the output end of a fully connected layer and the branch L2-1, as shown in the figure. As shown in the figure, the fusion subunit connected to the output end of another fully connected layer and branch L2-2, and the feature addition layer connected to the output ends of the two fusion subunits at the same time, as shown in the figure The structure of the M5×5 layer can be seen in detail. Fig.16 As shown, no further description is given here.

[0367] Among them, the convolution scale of the second feature extraction branch L1 is smaller than the convolution scale corresponding to the second feature extraction branch L2. Therefore, the size of the first receptive field is smaller than the size of the second receptive field. The second feature extraction branch under the first receptive field can be used to extract subtle features in the initial feature map, and the second feature extraction branch under the second receptive field can be used to extract local detail features in the initial feature map.

[0368] like Fig.15c As shown, a structural schematic diagram of the second type of second feature extraction branch is shown, including: a plurality of second feature extraction units connected in series, the second feature extraction unit including a third convolution unit, an activation unit and a pooling unit connected in series; wherein, connected in series with the last second feature extraction unit is a fourth convolution unit, a fusion subunit connected to the fourth convolution unit, an activation function layer ReLu connected to the fusion subunit, and a convolution layer connected to the activation function layer.

[0369] The third feature extraction unit sequentially connects the fourth convolution unit, the activation unit ReLu and the pooling unit. The structure may include a convolution layer, an activation function layer ReLu, a maximum pooling layer and a normalization layer sequentially connected in series.

[0370] Among them, the second feature extraction branch of the second type corresponds to the third receptive field, the size of the third receptive field is larger than the size of the second receptive field, and the second feature extraction branch under the third receptive field can be used to extract global features in the initial feature map.

[0371] The following describes the entire process from acquiring the target model to processing the first image to be processed using the target model and converting it into an HDR video in combination with a specific example:

[0372] First, according to steps S21'-S29', a target model M1 corresponding to the YUV format is trained, wherein the target model M1 includes a target model M1-1 and a target model M1-2; the target model M1-1 includes a first sub-model and a second sub-model, which is trained through steps S25'-S29', and the target model M1-2 is a comprehensive model, which is trained through steps S21'-S24' and has a branch, and the structure of the branch can refer to Fig.14 The structure shown in the figure can be used. Similarly, the structure of the first sub-model and the second sub-model can refer to Fig.14 The structure shown is sufficient.

[0373] And, according to the above Figure 9-11 The process shown is used to train a target model M2 corresponding to the RGB format, wherein the process of training the target model M2 can refer to the above steps S31'-S35'. In this example, in order to ensure the accuracy of the target model M2, when updating the parameters of the model to be trained, the selected target image samples can be image samples in the YUV444 format, and the second image samples need to be decoded from the video samples used as labels in the YUV444 format to obtain multiple second image samples SB, and then the parameters of the model to be trained are updated based on the second image samples SB and the image samples in the YUV444 format (intermediate image samples 3).

[0374] Taking the video to be processed as an SDR video as an example, the following process is performed:

[0375] A first image conforming to the RGB format is decoded from the SDR video, wherein the parameters of the SDR video should at least conform to the following parameters: sampling (4:2:0 or 4:2:2 or 4:4:4), quantization (8 bit or 10 bit or 12 bit), color gamut (BT.470 or BT.601 or BT.709), and no high dynamic range.

[0376] Afterwards, the SDR video needs to be converted into an HDR video, the RGB value of each pixel in the decoded multiple first images is normalized, and then the normalized first image is input into the target model M2 to obtain the image to be transcoded output by the target model M2, wherein the image to be transcoded is an image under the BT2020 color gamut, and the RGB value of each pixel included therein ranges from 0 to 1;

[0377] Next, the image to be transcoded is subjected to inverse normalization, YUV color coding format conversion, and data storage format conversion in sequence to obtain a second image in YUV422 format;

[0378] Afterwards, the second image is sent to the codec to obtain an HDR video, wherein the parameters of the HDR video should at least meet the following parameters: sampling (4:2:2), quantization (10bit), color gamut (BT.2020, high dynamic range, in line with the HLG standard.

[0379] Among them, taking the video to be processed as a video in a non-standard HDR video format as an example, the video is not a broadcasting video, and may be a non-professional HDR video such as self-media or the Internet, but a video that conforms to the dynamic range of HLG, the main process is as follows:

[0380] Decoding a first image in a YUV format from a video in a non-standard HDR video format, and converting a data storage format of the first image into a first image in a YUV422 format;

[0381] Afterwards, the second sampled image belonging to the U component and the third sampled image belonging to the V component in the first image in the YUV422 format are spliced ​​to obtain a spliced ​​image, and then the spliced ​​image and the first sampled image belonging to the Y component are input into the target model M1-2 to obtain an image to be transcoded, which is in the YUV422 format;

[0382] Alternatively, the second sampled image belonging to the U component and the third sampled image belonging to the V component in the first image in YUV422 format are input into the second sub-model in the target model M1-1, and the first sampled image belonging to the Y component is input into the first sub-model in the target model M1-1, and the image to be transcoded is obtained according to the outputs of the first sub-model and the second sub-model, and the image to be transcoded is in YUV422 format.

[0383] Next, the image to be transcoded is directly input into the codec to obtain an HDR video with a dynamic range that conforms to the HLG standard. The parameters of the HDR video should at least meet the following parameters: sampling (4:2:2), quantization (10bit or 12bit), color gamut (BT.2020, high dynamic range, in line with the HLG standard.

[0384] In this case, the target model can adjust the high dynamic range / color of the non-standard HDR video so that the HDR video that does not meet the requirements can be output as a HDR video that meets the requirements through the target model.

[0385] Based on the same inventive concept, the present disclosure also provides an image processing device, such as Fig.12 As shown, the device comprises:

[0386] An acquisition module, configured to acquire a plurality of first images to be processed, wherein the first images are in a first color coding format;

[0387] a scheduling module, configured to call a target model matching the first color coding format, process the plurality of the first images, and obtain a plurality of images to be transcoded in the first color coding format; wherein the dynamic range of the images to be transcoded is higher than the dynamic range of the first images;

[0388] A post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the color coding format of the second image complies with the second color coding format;

[0389] The codec module is configured to input the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

[0390] Optionally, the device further comprises:

[0391] A first determining module, configured to determine a preprocessing corresponding to the first color coding format, wherein different first color coding formats correspond to different preprocessing;

[0392] A preprocessing module, configured to perform the preprocessing on the first image to obtain an image to be input; wherein the amount of calculation corresponding to the image to be input is smaller than the amount of calculation corresponding to the first image, or the resolution corresponding to the image to be input is larger than the resolution corresponding to the first image;

[0393] The step of using the target model to process the first image to obtain the image to be transcoded includes: using the target model to process the image to be input to obtain the image to be transcoded.

[0394] Optionally, the preprocessing module includes:

[0395] a first processing unit, configured to sample data belonging to different color channels in the first image respectively, so as to reduce or increase a resolution of the first image, when the first color encoding format is a YUV format;

[0396] The second processing unit is used to normalize the RGB value of each pixel in the first image when the first color encoding format is RGB format, so that the RGB value of each pixel in the image to be input is in the range of 0 to 1.

[0397] Optionally, the value range of each pixel in the image to be input is 0 to 1; or, in the color space corresponding to the first color encoding format, the size of the first channel in the image to be input is twice the size of the second channel and the size of the third channel.

[0398] Optionally, the first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and the size of the first sampled image is larger than that of the second sampled image and the third sampled image; the step of processing the image to be input using the target model includes:

[0399] splicing the second sampling image and the third sampling image to obtain a spliced ​​image with the same size as the first sampling image;

[0400] Inputting the first sampled image and the stitched image into the target model, wherein the target model is used to increase the dynamic range of the first sampled image and the stitched image respectively;

[0401] The image to be transcoded is obtained based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

[0402] Optionally, the first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and the size of the first sampled image is larger than that of the second sampled image and the third sampled image; the target model includes a first sub-model and a second sub-model, and the step of processing the image to be input using the target model includes:

[0403] Using the first sub-model to increase the dynamic range of the first sampled image, to obtain a first predicted image;

[0404] Using the second sub-model to increase the dynamic range of the second sampled image and the third sampled image, to obtain a second predicted image corresponding to the first predicted image and the second sampled image, and a third predicted image corresponding to the third sampled image;

[0405] The image to be transcoded is obtained based on the first predicted image, the second predicted image and the third predicted image.

[0406] Optionally, the post-processing module comprises:

[0407] a third processing unit, configured to, when the first color coding format is the same as the second color coding format, adjust the sizes of different channels of the image to be transcoded in the color space of the second color coding format to obtain a second image;

[0408] The fourth processing unit is configured to convert the color coding format of the image to be transcoded into the second color coding format to obtain the second image when the first color coding format is different from the second color coding format.

[0409] Optionally, the value of each pixel in the input image ranges from 0 to 1; and the post-processing module includes:

[0410] an inverse normalization unit, configured to inverse normalize the value of each pixel in the image to be transcoded, and then map the value to a color space corresponding to the second color encoding format to obtain a mapped image in the color space;

[0411] The conversion unit is used to convert the value of each pixel in the mapping image into a pixel value in the second color coding format to obtain the second image.

[0412] Optionally, the step of processing the first image by the target model includes:

[0413] Performing feature extraction on the first image to obtain an initial feature map;

[0414] Performing feature extraction on the initial feature map under receptive fields of different sizes respectively to obtain multiple feature maps; wherein different feature maps correspond to receptive fields of different sizes;

[0415] Performing feature fusion on the plurality of feature maps to obtain a first fused feature map;

[0416] Perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

[0417] Optionally, the pair of devices further comprises:

[0418] A low-frequency feature extraction module, used to extract features belonging to low-frequency information in the initial feature map based on a preset feature map to obtain a low-frequency feature map;

[0419] A multi-scale feature extraction module, used to extract features from the low-frequency feature map under a target receptive field to obtain a guided filtering feature map; wherein the target receptive field is a receptive field of different sizes except for the receptive field with the largest size;

[0420] A fusion processing module, used for fusing the guided filtering feature map and the first fused feature map to obtain a second fused feature map;

[0421] The step of performing feature reconstruction on the first fused feature map to obtain the image to be transcoded includes: performing feature reconstruction on the second fused feature map to obtain the image to be transcoded.

[0422] Optionally, the fusion processing module includes:

[0423] A mapping unit, used for mapping the value of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map, wherein the target interval ranges from 0 to 1;

[0424] An adjustment unit is used to adjust the values ​​of each feature point in the first fused feature map based on the values ​​of each feature point in the mapped guided filtering feature map to obtain the second fused feature map.

[0425] Based on the same inventive concept, the present disclosure also provides a device for acquiring a target model, referring to Fig.17 As shown, a schematic diagram of a framework of a device for acquiring a target model is shown, and the device includes:

[0426] A sample acquisition module, used for acquiring a video sample and a plurality of first image samples in a first color coding format;

[0427] An input module, used for inputting a plurality of the first image samples into a model to be trained, to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples;

[0428] A sample processing module, configured to process the image samples to be transcoded into converted image samples conforming to the second color coding format based on the first color coding format and the second color coding format; wherein the converted image samples are input into a codec to obtain a high dynamic range video;

[0429] an updating module, configured to decode a plurality of second image samples from the video sample based on a target image sample, and update the parameters of the model to be trained based on the target image sample and the second image sample; wherein the target image sample is any one of the image sample to be transcoded, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the image to be transcoded into the converted image sample;

[0430] The acquisition module is used to determine the model to be trained after multiple updates as a target model corresponding to the first color coding format.

[0431] Optionally, the sample processing module includes:

[0432] A decoding unit, configured to decode a plurality of video frames from the video sample;

[0433] The conversion unit is used to convert the plurality of video frames based on the color coding format and the data storage format of the target image samples to obtain the plurality of second image samples.

[0434] Optionally, the first color encoding format is an RGB format, and the second color encoding format is a YUV422 format; wherein:

[0435] The color encoding format of the target image sample is the RGB format, and the data storage format is that the RGB value in the target image sample ranges from 0 to 1;

[0436] Alternatively, the color encoding format of the target image sample is the YUV format, and the data storage format is the YUV444 format or the YUV422 format.

[0437] Optionally, when the first color encoding format is RGB format, the value of each pixel of the first image sample is 0-1; when the first color encoding format is YUV format, the data storage format of the first image sample is YUV422 format.

[0438] Optionally, the device further comprises:

[0439] An extraction module, used to obtain a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0440] a splicing processing module, used for splicing the second sampled image sample and the third sampled image sample to obtain a spliced ​​image with the same size as the first sampled image sample;

[0441] The input module is specifically used to input the first sampling image samples and the spliced ​​image samples into the model to be trained, and obtain the image samples to be transcoded based on the predicted image samples output by the model to be trained and corresponding to the first sampling image and the spliced ​​image samples respectively.

[0442] Optionally, the data storage format of the first image samples is a YUV422 format, the model to be trained includes a first sub-model and a second sub-model, and before inputting the plurality of first image samples into the model to be trained, the device further includes:

[0443] An extraction module, used to obtain a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample;

[0444] The input module is specifically used to: input the first sampled image sample into the first sub-model, and input the second sampled image sample and the third sampled image sample into the second sub-model;

[0445] The updating module is specifically used to: update the parameters of the first sub-model based on the predicted image samples corresponding to the first sampling image and the second image samples output by the first sub-model; and update the parameters of the second sub-model based on the second image samples and the predicted image samples corresponding to the second sampling image samples and the third sampling image samples output by the second sub-model respectively.

[0446] The embodiments of the present disclosure also provide an electronic device, in which a computer program stored enables a processor to execute the image processing method or the target model acquisition method as described in the embodiments of the present disclosure.

[0447] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program that enables a processor to execute the image processing method or the target model acquisition method as described in the embodiments of the present disclosure.

[0448] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0449] Those skilled in the art will appreciate that the embodiments of the disclosed embodiments may be provided as methods, devices, or computer program products. Therefore, the disclosed embodiments may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the disclosed embodiments may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0450] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0451] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0452] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0453] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present disclosure.

[0454] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0455] The above is a detailed introduction to an image processing method, system, target model acquisition method, device, equipment and medium provided by the present disclosure. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea; at the same time, for general technical personnel in the field, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a plurality of first images to be processed, wherein the first images are in a first color coding format; Invoking a target model that matches the first color coding format, processing the plurality of the first images, and obtaining a plurality of images to be transcoded that conform to the first color coding format; wherein a dynamic range of the images to be transcoded is higher than a dynamic range of the first images; Based on a second color coding format and the first color coding format, processing the image to be transcoded into a second image, the second image conforming to the second color coding format; The plurality of second images are input into a codec so that the codec encodes the plurality of second images into a high dynamic range video.

2. The method according to claim 1, characterized in that: Before calling the target model matching the first color coding format and processing the plurality of first images, the method further comprises: determining a preprocessing corresponding to the first color coding format, wherein different first color coding formats correspond to different preprocessing; Performing the preprocessing on the first image to obtain an image to be input; wherein the amount of calculation corresponding to the image to be input is smaller than the amount of calculation corresponding to the first image, or the resolution corresponding to the image to be input is larger than the resolution corresponding to the first image; The calling of the target model matching the first color coding format to process the plurality of the first images comprises: The target model is used to process the plurality of images to be input, so as to obtain the plurality of images to be transcoded.

3. The method according to claim 2, characterized in that The preprocessing of the first image includes: When the first color encoding format is a YUV format, sampling data belonging to different color channels in the first image respectively to reduce or increase the resolution of the first image; When the first color encoding format is an RGB format, the RGB value of each pixel in the first image is normalized so that the RGB value of each pixel in the image to be input is in the range of 0 to 1.

4. The method according to claim 2, characterized in that: The value range of each pixel in the image to be input is 0 to 1; or, in the color space corresponding to the first color encoding format, the size belonging to the first channel in the image to be input is different from the size belonging to the second channel and the size belonging to the third channel; wherein the size belonging to the second channel is the same as the size belonging to the third channel.

5. The method according to any one of claims 2 to 4, characterized in that: The first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and a size of the first sampled image is larger than sizes of the second sampled image and the third sampled image; The step of processing the plurality of images to be input by using the target model to obtain the plurality of images to be transcoded includes: Splicing the second sampling image and the third sampling image to obtain a spliced ​​image; wherein the size of the spliced ​​image is the same as the size of the first sampling image; Inputting the first sampled image and the stitched image into the target model, wherein the target model is used to respectively improve the dynamic range of the first sampled image and the stitched image; The image to be transcoded is obtained based on the predicted images output by the target model and corresponding to the first sampled image and the spliced ​​image respectively.

6. The method according to claim 2, characterized in that The first color encoding format is a YUV format, the image to be input includes a first sampled image belonging to a brightness channel, a second sampled image belonging to a color channel, and a third sampled image belonging to a saturation channel, and a size of the first sampled image is larger than sizes of the second sampled image and the third sampled image; The target model includes a first sub-model and a second sub-model, and the using the target model to process the plurality of images to be input to obtain the plurality of images to be transcoded includes: Using the first sub-model to improve the dynamic range of the first sampled image, to obtain a first predicted image; Using the second sub-model to improve the dynamic range of the second sampled image and the third sampled image, to obtain a second predicted image corresponding to the second sampled image and a third predicted image corresponding to the third sampled image; The image to be transcoded is obtained based on the first predicted image, the second predicted image and the third predicted image.

7. The method according to any one of claims 1 to 6, characterized in that: The step of processing the to-be-transcoded image into a second image based on the second color coding format and the first color coding format includes: When the first color coding format is the same as the second color coding format, in a color space of the second color coding format, adjusting the sizes of different channels of the image to be transcoded in the color space to obtain a second image; In a case where the first color coding format is different from the second color coding format, the color coding format of the image to be transcoded is converted into the second color coding format to obtain the second image.

8. The method according to claim 2, characterized in that: The value of each pixel in the image to be input is in the range of 0 to 1; and processing the image to be transcoded into a second image based on the second color coding format and the first color coding format includes: After inverse normalization, the values ​​of each pixel in the image to be transcoded are mapped to a color space corresponding to the second color encoding format to obtain a mapped image in the color space; The value of each pixel in the mapping image is converted into a pixel value in the second color coding format to obtain the second image.

9. The method according to any one of claims 1 to 8, characterized in that: When the target model processes the first image, it includes: Performing feature extraction on the first image to obtain an initial feature map; Performing feature extraction on the initial feature map under receptive fields of different sizes respectively to obtain multiple feature maps; wherein different feature maps correspond to receptive fields of different sizes; Performing feature fusion on the plurality of feature maps to obtain a first fused feature map; Perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

10. The method according to claim 9, characterized in that After extracting features from the first image to obtain an initial feature map, the method further includes: Based on the preset feature map, feature extraction is performed on the initial feature map to obtain a low-frequency feature map; Under the target receptive field, feature extraction is performed on the low-frequency feature map to obtain a guided filtering feature map; wherein the target receptive field is a receptive field of different sizes except for the receptive field with the largest size; After obtaining the first fusion feature map, the method further includes: Fusing the guided filtering feature map and the first fused feature map to obtain a second fused feature map; The step of reconstructing the first fused feature map to obtain the image to be transcoded includes: Perform feature reconstruction on the second fused feature map to obtain the image to be transcoded.

11. The method according to claim 9, characterized in that The step of fusing the guided filtering feature map and the first fused feature map to obtain a second fused feature map includes: Mapping the values ​​of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map, wherein the target interval ranges from 0 to 1; Based on the values ​​of each feature point in the mapped guided filtering feature map, the values ​​of each feature point in the first fused feature map are adjusted to obtain the second fused feature map.

12. A method for acquiring a target model, characterized in that: The method comprises: Acquire a video sample and a plurality of first image samples in a first color encoding format; Inputting a plurality of the first image samples into a model to be trained to obtain image samples to be transcoded output by the model to be trained; the dynamic range of the image samples to be transcoded is higher than the dynamic range of the first image samples; Based on the first color coding format and the second color coding format, the image samples to be transcoded are processed into converted image samples conforming to the second color coding format; wherein the converted image samples are used to input into a codec to obtain a high dynamic range video; Based on the target image sample, a plurality of second image samples are decoded from the video sample, and based on the target image sample and the second image sample, the parameters of the to-be-trained model are updated; wherein the target image sample is any one of the to-be-transcoded image sample, the intermediate image sample and the converted image sample, and the intermediate image sample is an image obtained in the process of converting the to-be-transcoded image into the converted image sample; The model to be trained after multiple updates is determined as the target model corresponding to the first color coding format.

13. The acquisition method according to claim 12, characterized in that: The step of decoding a plurality of second image samples from the video sample based on the target image sample comprises: Decoding a plurality of video frames from the video samples; Based on the color coding format and data storage format of the target image samples, the plurality of video frames are converted and processed respectively to obtain a plurality of the second image samples.

14. The acquisition method according to claim 12, characterized in that: The first color encoding format is RGB format, and the second color encoding format is YUV422 format; wherein: The color encoding format of the target image sample is the RGB format, and the range of the RGB value in the target image sample is 0 to 1; Alternatively, the color encoding format of the target image sample is the YUV format, and the data storage format is the YUV444 format or the YUV422 format.

15. The acquisition method according to claim 12, characterized in that: When the first color encoding format is RGB format, the value of each pixel of the first image sample is 0-1; when the first color encoding format is YUV format, the data storage format of the first image sample is YUV422 format.

16. The acquisition method according to claim 12, characterized in that: The data storage format of the first image samples is a YUV422 format. Before inputting the plurality of first image samples into the model to be trained, the method further includes: Acquire a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample; Splicing the second sampled image sample and the third sampled image sample to obtain a spliced ​​image with the same size as the first sampled image sample; Inputting a plurality of the first image samples into a model to be trained to obtain image samples to be transcoded output by the model to be trained, comprising: The first sampling image samples and the spliced ​​image samples are input into the model to be trained, and the image samples to be transcoded are obtained based on the predicted image samples output by the model to be trained and corresponding to the first sampling image and the spliced ​​image samples respectively.

17. The acquisition method according to claim 12, characterized in that: The data storage format of the first image samples is a YUV422 format, the model to be trained includes a first sub-model and a second sub-model, and before inputting the plurality of first image samples into the model to be trained, the method further includes: Acquire a first sampled image sample belonging to a brightness channel, a second sampled image sample belonging to a color channel, and a third sampled image sample belonging to a saturation channel from the first image sample; The step of inputting the plurality of first image samples into the model to be trained, obtaining the image samples to be transcoded output by the model to be trained, and updating the parameters of the model to be trained based on the target image samples and the second image samples, comprises: Inputting the first sampled image sample into the first sub-model, and inputting the second sampled image sample and the third sampled image sample into the second sub-model; Based on the second image sample and the predicted image sample output by the first sub-model and corresponding to the first sampled image, updating the parameters of the first sub-model; Based on the second image samples and the predicted image samples output by the second sub-model and corresponding to the second sampled image samples and the third sampled image samples respectively, the parameters of the second sub-model are updated.

18. An image processing system, characterized in that: The system is configured with a plurality of target models, and different target models correspond to different first color coding formats; the system comprises: An acquisition module, used for acquiring a plurality of first images to be processed, wherein the first images are in a first color coding format; a scheduling module, configured to call a target model matching the first color coding format, process the plurality of the first images, and obtain a plurality of images to be transcoded that conform to the first color coding format; wherein the dynamic range of the images to be transcoded is higher than the dynamic range of the first images; A post-processing module, configured to process the image to be transcoded into a second image based on a second color coding format and the first color coding format, wherein the second image complies with the second color coding format; The codec module is configured to input the plurality of second images into a codec, so that the codec encodes the plurality of second images into a high dynamic range video.

19. The system according to claim 18, characterized in that The target model comprises: a first feature extraction branch, a plurality of second feature extraction branches connected to the output end of the first feature extraction branch, a first fusion module connected to the output ends of the plurality of second feature extraction branches at the same time, and a reconstruction module connected to the output end of the first fusion module; wherein different second feature extraction branches correspond to receptive fields of different sizes; The first feature extraction branch is used to extract features from the first image to obtain an initial feature map; The second feature extraction branch is used to perform feature extraction on the initial feature map under the corresponding receptive field to obtain a feature map; The first fusion module is used to perform feature fusion on the plurality of feature maps to obtain a first fused feature map; The reconstruction module is used to perform feature reconstruction on the first fused feature map to obtain the image to be transcoded.

20. The system according to claim 19, characterized in that The target model further includes: a third feature extraction branch connected to the output end of the first feature extraction branch, and a second fusion module connected to the output end of the output end of the first fusion module, and the input end of the reconstruction module is connected to the output end of the second fusion module; wherein: The third feature extraction branch is used to perform feature extraction on the initial feature map using a preset feature map to obtain a low-frequency feature map; The fourth feature extraction branch in the second feature extraction branch is further used to extract features from the low-frequency feature map to obtain a guided filtering feature map; the receptive field corresponding to the fourth feature extraction branch is the receptive field except for the receptive field with the largest size; The second fusion module is used to fuse the guided filtering feature map and the first fused feature map to obtain a second fused feature map; The reconstruction module is specifically used to perform feature reconstruction on the second fused feature map to obtain the image to be transcoded.

21. The system according to claim 20, characterized in that The second fusion module is specifically used to map the values ​​of each feature point in the guided filtering feature map to a target interval to obtain a mapped guided filtering feature map; and based on the values ​​of each feature point in the mapped guided filtering feature map, adjust the values ​​of each feature point in the first fused feature map to obtain the second fused feature map; The target interval ranges from 0 to 1.

22. The system according to claim 21, characterized in that The first color encoding format includes a YUV format, and the target model corresponding to the YUV format includes a first sub-model and a second sub-model, wherein; The structure of the first sub-model is the same as that of the second sub-model, and both include: the first feature extraction branch, a plurality of the second feature extraction branches, the first fusion module and the reconstruction module.

23. The system according to claim 21, characterized in that The plurality of second feature extraction branches include at least one second feature extraction branch of the first type and a second feature extraction branch of the second type; the receptive field of the second feature extraction branch of the second type is larger than the receptive field of the at least one second feature extraction branch of the first type; The second feature extraction branch of the first type includes a plurality of first feature extraction units connected in series, the first feature extraction unit includes a first convolution unit and a second convolution unit, the second convolution unit includes two first convolution branches connected in parallel, and different first convolution branches correspond to different convolution kernels; wherein, the smaller the receptive field size of the second feature extraction branch, the smaller the convolution kernel size of the first convolution unit and the second convolution unit; The second feature extraction branch of the second type includes: a plurality of second feature extraction units connected in series in sequence, wherein the second feature extraction unit includes a third convolution unit, an activation unit and a pooling unit connected in series in sequence.

24. An electronic device, characterized in that: include: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the image processing method as described in any one of claims 1 to 11 or the target model acquisition method as described in any one of claims 12 to 17 is implemented.

25. A computer-readable storage medium, characterized in that: The computer program stored therein enables a processor to implement the image processing method described in any one of claims 1 to 11, or the target model acquisition method described in any one of claims 12 to 17 when executed.

Images