Video processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a video processing method and device, equipment, a storage medium and a program product. According to one embodiment of the invention, the method comprises the following steps: detecting memory colors in multiple frames of initial HDR images contained in an HDR video in response to an obtained HDR video to be processed; in response to the memory color detected in any initial HDR image of the multiple frames of initial HDR images, performing color reconstruction on the memory color in any initial HDR image based on a predetermined color mapping relationship; and generating a target HDR video corresponding to the to-be-processed HDR video based on each frame of HDR image subjected to color reconstruction in the multiple frames of initial HDR images and each frame of initial HDR image which is not subjected to color reconstruction in the multiple frames of initial HDR images. It can be ensured that the color reconstruction result conforms to the inherent cognitive habit of the user for the memory color, and the requirements of different display scenes are met.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to a video processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of terminal technology, more and more electronic devices such as mobile phones and tablets are able to support the display of HDR (High-Dynamic Range) video. HDR video has advantages such as higher dynamic range, richer color performance, and higher definition. As the number of HDR content in mainstream video apps increases, HDR video display is gradually becoming the mainstream video playback method.

[0003] In related technologies, when displaying HDR video, it is usually necessary to adjust hue and saturation to ensure the display effect. However, for colors such as skin tone, sky blue, and grass green, people's perceptions are usually stable and deeply ingrained in their memory, thus these colors are categorized as "memory colors." Therefore, ensuring that the various memory colors displayed in HDR video conform to users' inherent cognitive habits has become one of the urgent technical problems to be solved. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides a video processing method, apparatus, device, storage medium, and program product to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, a video processing method is provided, the method comprising:

[0006] In response to acquiring the high dynamic range (HDR) video to be processed, a memory color is detected in the multiple initial HDR images contained in the HDR video;

[0007] In response to detecting the memory color in any of the multiple initial HDR images, color reconstruction is performed on the memory color in the any initial HDR image based on a predetermined color mapping relationship, wherein the color mapping relationship includes a color mapping relationship constructed based on the subjective evaluation results of the memory color in sample images of different display scenes by the human eye;

[0008] Based on the color-reconstructed HDR images of each frame in the multi-frame initial HDR images and the remaining unreconstructed initial HDR images, a target HDR video corresponding to the HDR video to be processed is generated.

[0009] In some embodiments, detecting the memory color in the multiple initial HDR images contained in the HDR video includes:

[0010] The HDR video is subjected to frame extraction to obtain an initial HDR image for each frame contained in the HDR video;

[0011] The memory color is detected in each frame of the initial HDR image.

[0012] In some embodiments, the color mapping relationship includes a high-dimensional color lookup table (LUT), which includes nine columns of data, of which three columns are input RGB value data, three columns are the RGB mean data of a preset area surrounding the input RGB value, and three columns are the target RGB value data corresponding to the input RGB value.

[0013] The step of reconstructing the memory color in any initial HDR image based on a pre-determined color mapping relationship includes:

[0014] Determine the RGB mean value of a preset region surrounding the memory color in any initial HDR image;

[0015] Based on the current RGB value of the memory color and the average RGB value of its surrounding preset area, the high-dimensional LUT is searched to obtain the target RGB value of the memory color;

[0016] Reconstruct the current RGB value of the memory color in any initial HDR image to the target RGB value of the memory color.

[0017] In some embodiments, the method further includes pre-determining the high-dimensional LUT based on the following:

[0018] Obtain sample images under different display scenarios, wherein the memory color is present in the sample images;

[0019] The color information of the memory color in the sample image is adjusted to obtain the adjusted image;

[0020] The memory color in the adjusted image is subjectively evaluated using the human eye to obtain an evaluation score;

[0021] In response to the evaluation score being greater than or equal to a set score threshold, the adjusted image is used as the target image of the sample image;

[0022] The high-dimensional LUT is formed by generating three columns of input RGB value data based on the first RGB value of the memory color in the sample image, generating three columns of RGB mean data based on the RGB mean of the preset area surrounding the memory color in the sample image, and generating three columns of target RGB value data based on the second RGB value of the memory color in the target image.

[0023] In some embodiments, the method further includes:

[0024] The initial HDR image is input into a pre-trained color prediction network to obtain the probability of the correct display of the memory color in the initial HDR image.

[0025] In response to the display correctness probability being greater than or equal to a set probability threshold, the operation of reconstructing the memory color in any initial HDR image based on a predetermined color mapping relationship is performed.

[0026] In some embodiments, the method further includes pre-training the color prediction network based on the following:

[0027] The sample image and its target image are respectively input into the color prediction network to be trained to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image. The first network prediction probability and the second network prediction probability are respectively used to characterize the display correctness probability of the memory color in the sample image and the display correctness probability of the memory color in the target image.

[0028] Obtain the first actual predicted probability of the sample image and the second actual predicted probability of the target image;

[0029] Based on the first difference between the first network predicted probability and the first actual predicted probability, and the second difference between the second network predicted probability and the second actual predicted probability, a target loss function is constructed.

[0030] The parameters of the color prediction network are optimized based on the objective loss function until a well-trained color prediction network is obtained.

[0031] In some embodiments, the color mapping relationship further includes a one-dimensional color lookup table (LUT) for each of the three RGB channels. Each channel's one-dimensional LUT includes two columns of data, one column being the input color value data for the channel and the other column being the target color value data for the channel.

[0032] The step of reconstructing the memory color in any initial HDR image based on a pre-determined color mapping relationship includes:

[0033] In response to the display correctness probability being less than the set probability threshold, the memory color in any initial HDR image is subjected to RGB three-channel pixel value separation to obtain the R, G, and B three-channel color values ​​of the memory color in any initial HDR image;

[0034] Based on the one-dimensional LUTs of each of the three RGB channels, single-channel color mapping is performed on the three R, G, and B channels respectively to obtain the single-channel mapped HDR image corresponding to any initial HDR image;

[0035] Based on the high-dimensional LUT, color reconstruction is performed on the single-channel mapped HDR image.

[0036] In some embodiments, the method further includes pre-determining a one-dimensional LUT for each of the three RGB channels based on the following:

[0037] The input color value of each of the three RGB channels is input into the pre-trained sub-network of each channel to obtain the target color value of each channel;

[0038] Based on the input color value and the corresponding target color value of each channel, a one-dimensional LUT is formed for each channel.

[0039] In some embodiments, the method further includes pre-training a sub-network for each of the RGB three channels based on the following:

[0040] The RGB three-channel pixel values ​​of the sample image and its target image are separated respectively to obtain the RGB three-channel grayscale images of the sample image and its target image respectively;

[0041] The RGB three-channel grayscale images of the sample image are input into the sub-network to be trained for each channel to obtain the image of each channel;

[0042] Based on the difference between the image of each channel and the corresponding channel image of the target image, a loss function is constructed for each channel;

[0043] The parameters of the sub-network for each channel are optimized based on the loss function of each channel until a trained sub-network for each channel is obtained.

[0044] In some embodiments, generating the target HDR video corresponding to the HDR video to be processed based on the color-reconstructed HDR images of each frame in the multi-frame initial HDR images and the remaining unreconstructed initial HDR images includes:

[0045] Each frame of HDR image after color reconstruction and each frame of initial HDR image without color reconstruction are encoded to obtain encoded multi-frame video frames.

[0046] The multiple video frames are combined into the target HDR video.

[0047] According to a second aspect of the present disclosure, a video processing apparatus is provided, the apparatus comprising:

[0048] The color detection module is used to detect memory colors in the multiple initial HDR images contained in the HDR video in response to acquiring the high dynamic range HDR video to be processed;

[0049] A color reconstruction module is configured to, in response to detecting the memory color in any of the multiple initial HDR images, reconstruct the memory color in any of the initial HDR images based on a predetermined color mapping relationship, wherein the color mapping relationship includes a color mapping relationship constructed based on the subjective evaluation results of the memory color in sample images of different display scenes by the human eye;

[0050] The video generation module is used to generate a target HDR video corresponding to the HDR video to be processed based on the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction from the multiple initial HDR images.

[0051] In some embodiments, the color detection module includes:

[0052] A frame extraction processing unit is used to perform frame extraction processing on the HDR video to obtain an initial HDR image for each frame contained in the HDR video.

[0053] A color detection unit is used to detect the memory color in each frame of the initial HDR image.

[0054] In some embodiments, the color mapping relationship includes a high-dimensional color lookup table (LUT), which includes nine columns of data, of which three columns are input RGB value data, three columns are the RGB mean data of a preset area surrounding the input RGB value, and three columns are the target RGB value data corresponding to the input RGB value.

[0055] The color reconstruction module includes a first reconstruction unit;

[0056] The first reconstruction unit is used for:

[0057] Determine the RGB mean value of a preset region surrounding the memory color in any initial HDR image;

[0058] Based on the current RGB value of the memory color and the average RGB value of its surrounding preset area, the high-dimensional LUT is searched to obtain the target RGB value of the memory color;

[0059] Reconstruct the current RGB value of the memory color in any initial HDR image to the target RGB value of the memory color.

[0060] In some embodiments, the apparatus further includes a high-dimensional LUT determination module;

[0061] The high-dimensional LUT determination module includes:

[0062] A sample image acquisition unit is used to acquire sample images under different display scenarios, wherein the memory color is present in the sample images;

[0063] An image color adjustment unit is used to adjust the color information of the memory color in the sample image to obtain an adjusted image;

[0064] An image color evaluation unit is used to subjectively evaluate the memory color in the adjusted image using the human eye, and obtain an evaluation score;

[0065] A target image determination unit is configured to, in response to the evaluation score being greater than or equal to a set score threshold, use the adjusted image as the target image of the sample image;

[0066] A high-dimensional LUT determination unit is used to form three columns of input RGB value data of the high-dimensional LUT based on the first RGB value of the memory color in the sample image, form three columns of RGB mean data of the high-dimensional LUT based on the RGB mean of a preset area around the memory color in the sample image, and form three columns of target RGB value data of the high-dimensional LUT based on the second RGB value of the memory color in the target image.

[0067] In some embodiments, the apparatus further includes:

[0068] A color display prediction module is used to input any initial HDR image into a pre-trained color prediction network to obtain the probability of correct display of the memory color in any initial HDR image.

[0069] The color reconstruction module is also used to perform the operation of reconstructing the memory color in any initial HDR image based on a predetermined color mapping relationship in response to the display correctness probability being greater than or equal to a set probability threshold.

[0070] In some embodiments, the apparatus further includes a prediction network training module;

[0071] The prediction network training module includes:

[0072] The first probability acquisition unit is used to input the sample image and its target image into the color prediction network to be trained, respectively, to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image. The first network prediction probability and the second network prediction probability are used to characterize the display correctness probability of the memory color in the sample image and the display correctness probability of the memory color in the target image, respectively.

[0073] The second probability acquisition unit is used to acquire the first actual predicted probability of the sample image and the second actual predicted probability of the target image;

[0074] The loss function construction unit is used to construct a target loss function based on a first difference between the first network prediction probability and the first actual prediction probability, and a second difference between the second network prediction probability and the second actual prediction probability.

[0075] The prediction network acquisition unit is used to optimize the parameters of the color prediction network based on the target loss function until a trained color prediction network is obtained.

[0076] In some embodiments, the color mapping relationship further includes a one-dimensional color lookup table (LUT) for each of the three RGB channels. Each channel's one-dimensional LUT includes two columns of data, one column being the input color value data for the channel and the other column being the target color value data for the channel.

[0077] The color reconstruction module also includes a second reconstruction unit;

[0078] The second reconstruction unit is used for:

[0079] In response to the display correctness probability being less than the set probability threshold, the memory color in any initial HDR image is subjected to RGB three-channel pixel value separation to obtain the R, G, and B three-channel color values ​​of the memory color in any initial HDR image;

[0080] Based on the one-dimensional LUTs of each of the three RGB channels, single-channel color mapping is performed on the three R, G, and B channels respectively to obtain the single-channel mapped HDR image corresponding to any initial HDR image;

[0081] The first reconstruction unit is also used to perform color reconstruction on the single-channel mapped HDR image based on the high-dimensional LUT.

[0082] In some embodiments, the apparatus further includes a one-dimensional LUT determination module;

[0083] The one-dimensional LUT determination module includes:

[0084] The target color acquisition unit is used to input the input color value of each channel in the RGB three channels into the pre-trained sub-network of each channel to obtain the target color value of each channel;

[0085] A one-dimensional LUT determination unit is used to form a one-dimensional LUT for each channel based on the input color value and the corresponding target color value of each channel.

[0086] In some embodiments, the apparatus further includes a sub-network training module;

[0087] The sub-network training module includes:

[0088] The grayscale image acquisition unit is used to separate the RGB three-channel pixel values ​​of the sample image and its target image respectively, so as to obtain the RGB three-channel grayscale images of the sample image and its target image respectively.

[0089] The channel image acquisition unit is used to input the RGB three-channel grayscale image of the sample image into the training sub-network corresponding to each channel to obtain the image of each channel;

[0090] The channel function construction unit is used to construct a loss function for each channel based on the difference between the image of each channel and the corresponding channel image of the target image;

[0091] The sub-network acquisition unit is used to optimize the parameters of the sub-network for each channel based on the loss function of each channel until a trained sub-network for each channel is obtained.

[0092] In some embodiments, the video generation module includes:

[0093] An image encoding unit is used to encode each frame of HDR image after color reconstruction and each frame of initial HDR image without color reconstruction to obtain encoded multi-frame video frames.

[0094] A video generation unit is used to synthesize the multiple video frames into the target HDR video.

[0095] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0096] Processor and memory used to store computer programs;

[0097] The processor is configured to implement the video processing method described in any one of the first aspects when executing the computer program.

[0098] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video processing method described in any one of the first aspects above.

[0099] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video processing method described in any one of the first aspects above.

[0100] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0101] This disclosure reconstructs the colors in the HDR video to be processed based on a predetermined color mapping relationship. This color mapping relationship is constructed based on the subjective evaluation results of the memory colors in sample images of different display scenarios by the human eye. Therefore, it can ensure that the color reconstruction results can conform to the user's inherent cognitive habits of memory colors and adapt to the needs of different display scenarios, thereby improving the effect of subsequent playback of the target HDR video and bringing a better visual experience to the user.

[0102] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0103] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0104] Figure 1A This is a flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure;

[0105] Figure 1B This is a schematic diagram illustrating an application scenario of a video processing method according to an exemplary embodiment of the present disclosure;

[0106] Figure 2 This is a flowchart illustrating how to detect memory colors in multiple initial HDR images contained in the HDR video, according to an exemplary embodiment of the present disclosure;

[0107] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to reconstruct the memory color in any initial HDR image based on a predetermined color mapping relationship;

[0108] Figure 4A This is a flowchart illustrating how to determine the high-dimensional LUT according to an exemplary embodiment of the present disclosure;

[0109] Figure 4B This is a schematic diagram illustrating the process of an expert system scoring an image according to an exemplary embodiment of the present disclosure;

[0110] Figure 5A This is a flowchart illustrating a video processing method according to another exemplary embodiment of the present disclosure;

[0111] Figure 5B This is a logical schematic diagram illustrating a video processing method according to yet another exemplary embodiment of the present disclosure;

[0112] Figure 6 This is a flowchart illustrating how to train the color prediction network according to an exemplary embodiment of the present disclosure;

[0113] Figure 7 This is a flowchart illustrating how to determine the one-dimensional LUT for each of the three RGB channels according to an exemplary embodiment of this disclosure;

[0114] Figure 8A This is a flowchart illustrating how to train a sub-network for each channel of the RGB three-channel array according to an exemplary embodiment of this disclosure;

[0115] Figure 8B This is a schematic diagram illustrating the sub-networks of each of the three RGB channels according to an exemplary embodiment of this disclosure;

[0116] Figure 9 This is a flowchart illustrating how to generate a target HDR video corresponding to the HDR video to be processed, according to an exemplary embodiment of this disclosure;

[0117] Figure 10 This is a block diagram illustrating a video processing apparatus according to an exemplary embodiment of the present disclosure;

[0118] Figure 11 This is a block diagram illustrating yet another video processing apparatus according to an exemplary embodiment of the present disclosure;

[0119] Figure 12 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0120] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0121] With the development of terminal technology, more and more electronic devices such as mobile phones and tablets are able to support the display of HDR (High-Dynamic Range) video. HDR video has advantages such as higher dynamic range, richer color performance, and higher definition. As the number of HDR content in mainstream video apps increases, HDR video display is gradually becoming the mainstream video playback method.

[0122] However, colors such as skin tone, sky blue, and grass green fall within the range of memory colors. The human eye and brain typically have a relatively fixed perception of these colors; that is, the recognition of memory colors follows certain rules and has become a fixed habit. Therefore, when displaying HDR video, it is necessary to reconstruct the colors of memory colors such as skin tone to ensure that the displayed memory colors conform to the user's inherent cognitive habits.

[0123] In related technologies, when displaying colors in different display scenarios, it is usually necessary to adjust the hue and saturation to match the current display scenario. However, the 3DLUT or PCC (Post Color Correction) color matching schemes in current electronic devices map a certain color to a fixed target color, which cannot adapt to the needs of different display scenarios and affects the user's visual experience.

[0124] In view of the above, this disclosure provides the following video processing methods, apparatus, devices, storage media, and program products to address the aforementioned drawbacks in the related technologies.

[0125] Figure 1A This is a flowchart illustrating a video processing method according to an exemplary embodiment. The method of this embodiment can be executed by a video processing device, which can be configured in an electronic device, such as a server, workstation, personal computer, mobile terminal (such as mobile phone, tablet computer, etc.), wearable device (such as glasses, watch, etc.).

[0126] For example, Figure 1B This is a schematic diagram illustrating an application scenario of a video processing method according to an exemplary embodiment of this disclosure. For example... Figure 1BAs shown, this application scenario involves a mobile terminal 101 and a server 102. The video processing method in this embodiment can be executed by the server 102, and then the processed target HDR video can be sent to the mobile terminal 101 for display via the network (i.e., the mobile terminal 101 acts as a display device, and the server 102 acts as an electronic device for processing HDR video); or, the video processing method in this embodiment can be executed by the mobile terminal 101, and the mobile terminal 101 can display the processed target HDR video (i.e., the mobile terminal 101 can act as both an electronic device for processing SDR video and a display device).

[0127] like Figure 1A As shown, the method includes the following steps S101-S103:

[0128] In step S101, in response to acquiring the high dynamic range (HDR) video to be processed, a memory color is detected in the multiple initial HDR images contained in the HDR video.

[0129] In this embodiment, when the electronic device acquires a high dynamic range (HDR) video to be processed, it can detect the memory color in the multiple initial HDR images contained in the HDR video.

[0130] For example, the high dynamic range (HDR) video to be processed mentioned above may include, but is not limited to, the HDR video that the electronic device is currently playing.

[0131] Among them, the aforementioned memory color can be at least one of the colors that belong to the memory category, such as skin color, sky blue, and grass green.

[0132] In some embodiments, the method for detecting memory color in the multiple initial HDR images contained in the HDR video described above can be found in the following: Figure 2 The embodiments shown will not be described in detail here.

[0133] In step S102, in response to detecting the memory color in any of the initial HDR images of the multi-frame initial HDR images, color reconstruction is performed on the memory color in the any of the initial HDR images based on a predetermined color mapping relationship.

[0134] In this embodiment, when the memory color is detected in any of the initial HDR images of the multiple initial HDR images, the memory color in any of the initial HDR images can be reconstructed based on a predetermined color mapping relationship.

[0135] The aforementioned color mapping relationship may include a color mapping relationship constructed based on the subjective evaluation results of the memorized colors in sample images of different display scenarios by the human eye.

[0136] For example, the aforementioned human eye can be the eyes of one or more experts in a pre-defined expert system. Exemplarily, this expert system could include a system composed of multiple real experts used to evaluate image quality. Therefore, the aforementioned expert system can subjectively measure the color accuracy of an image based on human visual perception.

[0137] In some embodiments, the method of reconstructing the memory color in any initial HDR image based on a predetermined color mapping relationship can be found in the following: Figure 3 The embodiments shown will not be described in detail here.

[0138] In step S103, a target HDR video corresponding to the HDR video to be processed is generated based on the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction in the multi-frame initial HDR images.

[0139] In this embodiment, after color reconstruction of the memory color in any initial HDR image based on a predetermined color mapping relationship, a target HDR video corresponding to the HDR video to be processed can be generated based on each HDR image that has undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction.

[0140] It is worth noting that the above method of generating target HDR video from HDR image can refer to video synthesis schemes in related technologies, and this embodiment does not limit it.

[0141] In other embodiments, the above-described method for generating target HDR video from HDR images may also be referred to the following: Figure 9 The embodiments shown will not be described in detail here.

[0142] As described above, the method of this embodiment reconstructs the memory colors in the HDR video to be processed based on a predetermined color mapping relationship. This color mapping relationship is constructed based on the subjective evaluation results of the memory colors in sample images of different display scenarios by the human eye. Therefore, it can ensure that the color reconstruction results can conform to the user's inherent cognitive habits of memory colors and adapt to the needs of different display scenarios, thereby improving the effect of subsequent playback of the target HDR video and bringing a better visual experience to the user.

[0143] Figure 2 This is a flowchart illustrating how to detect memory colors in multiple initial HDR images contained in an HDR video according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to detect memory colors in multiple initial HDR images contained in an HDR video as an example for illustrative explanation.

[0144] like Figure 2 As shown, the detection of memory color in the multiple initial HDR images contained in the HDR video in step S101 above may include the following steps S201-S202:

[0145] In step S201, the HDR video is subjected to frame extraction processing to obtain an initial HDR image for each frame contained in the HDR video.

[0146] In this embodiment, after obtaining the high dynamic range (HDR) video to be processed, the initial HDR image of each frame in the HDR video can be extracted frame by frame, thereby obtaining the multiple initial HDR images contained in the HDR video.

[0147] In step S202, the memory color is detected in each frame of the initial HDR image.

[0148] In this embodiment, after performing frame extraction on the HDR video to obtain each initial HDR image contained in the HDR video, the memory color can be detected in each initial HDR image.

[0149] For example, before detecting the memory color in the initial HDR image, the color gamut range of each memory color can be determined. For instance, for the green of grass green, a hue range of green and certain saturation and brightness thresholds can be defined. Based on this, the initial HDR image can be converted from the RGB space to other color spaces that are more suitable for describing and separating colors, such as CIELAB, CIELUV, or HSL (hue, saturation, brightness). Then, each pixel in the space-converted image can be traversed, and its color value can be compared with the memory color range to determine whether it belongs to a memory color.

[0150] It is worth noting that the above method of spatial transformation of the initial HDR image and subsequent memory color detection is only for illustrative purposes. Other detection methods can be selected as needed in application, and this embodiment does not limit this.

[0151] As described above, this embodiment obtains an initial HDR image for each frame of the HDR video by performing frame extraction processing on the HDR video, and detects the memory color in each initial HDR image. This enables accurate detection of the memory color in multiple initial HDR images contained in the HDR video, and allows subsequent color reconstruction of images containing the memory color based on a predetermined color mapping relationship. This ensures that the color reconstruction results conform to the user's inherent cognitive habits regarding memory colors and adapt to the needs of different display scenarios, thereby improving the effect of subsequent playback of the target HDR video and bringing a better visual experience to the user.

[0152] Figure 3 This is a flowchart illustrating how to reconstruct the memory color in any initial HDR image based on a predetermined color mapping relationship, according to an exemplary embodiment of the present disclosure. This embodiment is an exemplary description based on the above embodiment, using the example of how to reconstruct the memory color in any initial HDR image based on a predetermined color mapping relationship.

[0153] The color mapping relationship includes a high-dimensional color lookup table (LUT). The high-dimensional LUT comprises nine columns of data: three columns are input RGB value data, three columns are the RGB mean value data of a preset area surrounding the input RGB value, and three columns are the target RGB value data corresponding to the input RGB value. For example, the above high-dimensional LUT can be represented as shown in Table 1 below:

[0154] Table 1

[0155]

[0156]

[0157] As shown in Table 1 above, the three columns of data below RGB_ori are the input RGB data of the high-dimensional LUT, the three columns of data below RGB_loc are the RGB mean data of the preset area around the input RGB value, and the three columns of data below RGB_tgt are the target RGB value data corresponding to the input RGB value.

[0158] The method for determining the aforementioned high-dimensional LUT can be found in the following... Figure 4A The embodiments shown will not be described in detail here.

[0159] like Figure 3 As shown, the color reconstruction of the memory color in any initial HDR image based on a predetermined color mapping relationship in step S102 above may include the following steps S301-S303:

[0160] In step S301, the average RGB value of the preset region surrounding the memory color in any initial HDR image is determined.

[0161] In this embodiment, the RGB values ​​of each pixel in a preset area surrounding the pixel containing the memory color in any initial HDR image can be obtained, and then the average RGB value of the preset area surrounding it can be calculated.

[0162] It is worth noting that the size of the aforementioned surrounding preset area can be set based on the size of any initial HDR image. For example, if the HDR image is a 2k (i.e., 1920*1080) image, the size of the aforementioned surrounding preset area can be set to 128*128.

[0163] In step S302, the high-dimensional LUT is searched based on the current RGB value of the memory color and the average RGB value of the surrounding preset area to obtain the target RGB value of the memory color.

[0164] In this embodiment, after determining the RGB mean value of the preset area surrounding the memory color in any initial HDR image, the high-dimensional LUT can be searched based on the current RGB value of the memory color and the RGB mean value of the preset area surrounding it to obtain the target RGB value of the memory color.

[0165] It is worth noting that the current RGB value mentioned above can be the initial RGB value of any of the initial HDR images mentioned above, or it can be the RGB value of any of the initial HDR images mentioned above after being transformed by 1DLUT for each of the three RGB channels.

[0166] For example, after obtaining the above RGB average value, the corresponding target RGB value can be found by looking up the high-dimensional LUT shown in Table 1 based on the current RGB value of the memory color and the RGB average value of the surrounding preset area.

[0167] It's worth noting that, under normal circumstances, the target RGB value corresponding to the current RGB value and the RGB mean of its surrounding preset region can be directly found in the high-dimensional LUT shown in Table 1. However, if a corresponding target RGB value cannot be directly found in the high-dimensional LUT, the two closest values ​​can be found for interpolation calculation.

[0168] In step S303, the current RGB value of the memory color in any initial HDR image is reconstructed as the target RGB value of the memory color.

[0169] In this embodiment, after obtaining the target RGB value of the memory color, the current RGB value of the memory color in any initial HDR image can be reconstructed as the target RGB value of the memory color, thus realizing the color reconstruction of the memory color in any initial HDR image.

[0170] As described above, this embodiment can reconstruct the memory color in any initial HDR image, thereby generating a target HDR video corresponding to the HDR video to be processed from subsequent HDR images based on color reconstruction and the remaining initial HDR images that have not undergone color reconstruction. This ensures that the color reconstruction result conforms to the user's inherent cognitive habits regarding the memory color and adapts to the needs of different display scenarios, thereby improving the effect of subsequent playback of the target HDR video and bringing a better visual experience to the user.

[0171] Figure 4A This is a flowchart illustrating how to determine the high-dimensional LUT according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the high-dimensional LUT as an example for illustrative explanation.

[0172] like Figure 4A As shown, the video processing method of this embodiment may further include determining the high-dimensional LUT based on the following steps S401-S406:

[0173] In step S401, sample images under different display scenarios are acquired.

[0174] In this embodiment, when the high-dimensional LUT is determined, sample images pic_ori under different display scenarios can be obtained.

[0175] The sample image contains at least one of the memory colors, such as skin tone, sky blue, and grass green.

[0176] In some embodiments, different display scenarios may include the following two categories:

[0177] 1) Display environment, such as under the starry sky at night, outdoors on a sunny day, outdoors on a rainy day, dusk, twilight, etc.

[0178] 2) Display background, such as solid color background, natural scenery, building background, etc.

[0179] In other words, assuming there are 5 display environments and 3 display backgrounds, there can be 5*3=15 different display scenarios. Consequently, in a high-dimensional LUT, there can be 15 rows of mapping data corresponding to the same memory color (i.e., each row of data corresponds to a display scenario).

[0180] It should be noted that the types and quantities of the different display scenarios mentioned above are for illustrative purposes only, and this embodiment does not limit them.

[0181] In step S402, the color information of the memory color in the sample image is adjusted to obtain the adjusted image.

[0182] In this embodiment, after obtaining sample images under different display scenarios, the color information of the memorized colors in each sample image can be adjusted using software such as Photoshop to obtain the adjusted image. As an example, the current sample image can be converted from RGB space to HSV space, and then at least one of the hue, saturation, and brightness of the converted image can be fine-tuned (for example, the HSV value of the image can be adjusted in steps of 1).

[0183] In step S403, the memory color in the adjusted image is subjectively evaluated using the human eye to obtain an evaluation score.

[0184] In this embodiment, the aforementioned human eye can be the eye of a real expert in a preset expert system. For example, the expert system includes multiple real experts. Each expert can subjectively evaluate the display effect of the memory color in the adjusted image based on their own human visual perception and obtain their own score. Then, the average of the scores of the multiple experts can be used as the final evaluation score.

[0185] For example, Figure 4B This is a schematic diagram illustrating the process of an expert system scoring an image according to an exemplary embodiment of this disclosure. Figure 4B As shown, after converting the sample image to HSV space and fine-tuning the HSV values ​​of the converted image using methods such as PS, it can be sent to an expert system for scoring. For example, this expert system includes experts 1 to 5, each of whom can subjectively evaluate the display effect of the memory color in the adjusted image. The score can be set from 0 to 10; a higher score indicates that the memory color display effect in the current scene better meets the subjective perception of the human eye. Furthermore, the evaluation score of the adjusted image is obtained by calculating the average score of the experts.

[0186] In step S404, it is determined whether the evaluation score is greater than or equal to a set score threshold: if yes, then step S405 is executed; if no, then step S402 is returned.

[0187] In this embodiment, after obtaining the evaluation score of the image, the evaluation score can be compared with a set score threshold. If the score is greater than or equal to the set score threshold, step S405 is executed. Conversely, if the score is less than the set score threshold, the process can return to step S402, that is, fine-tune the HSV value of the image again.

[0188] In step S405, the adjusted image is used as the target image of the sample image.

[0189] In this embodiment, the currently adjusted image can be used as the target image pic_tgt of the sample image, and the process can be switched to the next image for fine-tuning and expert system scoring. That is, let n = n + 1, until n is the total number of sample images (i.e., all sample images in the sample image set have been processed), where n is the sequence number of the currently processed image.

[0190] Through the above steps S401 to S405, sample images input for different display scenarios can be transformed into target images with better display effects, that is, the same color can be better displayed in different scenarios.

[0191] In step S406, three columns of input RGB value data of the high-dimensional LUT are formed based on the first RGB value of the memory color in the sample image, three columns of RGB mean data of the high-dimensional LUT are formed based on the RGB mean of the preset area surrounding the memory color in the sample image, and three columns of target RGB value data of the high-dimensional LUT are formed based on the second RGB value of the memory color in the target image.

[0192] Taking skin color as an example, assuming the RGB value of skin color is (209, 190, 183), the first three columns of the mapping relationship data of different display scenarios corresponding to skin color in the high-dimensional LUT to be generated can all be this RGB value, named RGB_ori (that is, the three columns of data on the left in Table 1 above).

[0193] Then, we can iterate through all sample images pic_ori, calculate the mean of RGB values ​​of a preset area of ​​size 128*128 centered on the memory color pixel, and then use this RGB mean as the middle three columns in the above high-dimensional LUT, named RGB_loc (i.e., the middle three columns of data in Table 1 above).

[0194] Furthermore, we can iterate through all the target images pic_tgt and extract the RGB values ​​of the memory color in all the images. This value is the optimal value of the memory color in the current display scene. Therefore, we can use this RGB value as the last three columns in the above high-dimensional LUT and name it RGB_tgt (that is, the three columns of data on the right in Table 1 above).

[0195] As described above, this embodiment establishes a high-dimensional LUT for the display target of the same memory color in different display scenarios. In each scenario, the pixel values ​​of the preset area around the memory color in the HDR content are considered, and different target values ​​are formed according to the output of the expert system. This method can be applied to any scenario, ensuring the diversity of memory color display in any scenario and conforming to the user's inherent cognitive habits.

[0196] Figure 5AThis is a flowchart illustrating a video processing method according to another exemplary embodiment of the present disclosure; the method of this embodiment can be executed by a video processing device, which can be configured in an electronic device, such as a server, workstation, personal computer, mobile terminal (such as mobile phone, tablet computer, etc.), wearable device (such as glasses, watch, etc.).

[0197] like Figure 5A As shown, the method includes the following steps S501-S507:

[0198] In step S501, in response to acquiring the high dynamic range HDR video to be processed, a memory color is detected in the multiple initial HDR images contained in the HDR video;

[0199] In step S502, in response to detecting the memory color in any of the initial HDR images of the multi-frame initial HDR images, the any initial HDR image is input into a pre-trained color prediction network to obtain the display correctness probability of the memory color in the any initial HDR image;

[0200] In step S503, it is determined whether the probability of display correctness is greater than or equal to a set probability threshold: if yes, then step S504 is executed; if no, then step S505 is executed.

[0201] In step S504, color reconstruction is performed on the memory colors in the image based on a pre-determined high-dimensional LUT;

[0202] like Figure 5A As shown, the image in this step can be any initial HDR image from step S502, or it can be the single-channel mapped HDR image obtained in step S506.

[0203] In step S505, the memory color in any initial HDR image is separated into RGB three-channel pixel values;

[0204] In step S506, based on the one-dimensional LUT of each of the RGB three channels, single-channel color mapping is performed on the R, G, and B channels respectively to obtain the single-channel mapped HDR image corresponding to any initial HDR image;

[0205] In step S507, a target HDR video corresponding to the HDR video to be processed is generated based on the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction in the multi-frame initial HDR images.

[0206] The explanations and descriptions of steps S501 and S507 can be found above. Figure 1ASteps S101 and S103 in the illustrated embodiment will not be described in detail here.

[0207] For example, Figure 5B This is a logical schematic diagram illustrating a video processing method according to another exemplary embodiment of the present disclosure; as shown below. Figure 5B As shown, in step S51, the electronic device can acquire the HDR video to be played; in step S52, after acquiring the HDR video to be played, the HDR can be processed by frame extraction; in step S53, it can detect whether there is a memory color (such as skin color) in the currently extracted video frame image; in step S54, when the memory color is detected in the current video frame image, the video frame image can be input into the color prediction network to obtain the probability of the correct display of the memory color in the frame image. Then, when the probability is ≥0.7 (a preset probability threshold, which can be set according to requirements), it means that the display of the memory color in the frame image meets the requirements in the current display scene, so step S56 can be executed; in step S54... In step S56, a high-dimensional LUT is searched based on the RGB values ​​of the memory color in the current video frame image and the RGB mean of the preset area around the memory color, so as to reconstruct the memory color according to the found target RGB values; in step S55, when the probability is <0.7, it means that the display of the memory color in the frame image in the current display scene does not conform to the user's inherent cognitive habits (i.e., there is a large deviation from the memory color in the user's mind). At this time, it is necessary to first perform RGB three-channel color mapping separately through 1DLUTs of each of the RGB three channels to ensure accurate conversion of pixel nodes, and then perform color reconstruction through a high-dimensional LUT on the converted image to realize the correction of the memory color in the current video frame image in the current display scene.

[0208] As described above, this embodiment performs color reconstruction of the memory colors in video images simultaneously across multiple dimensions. This includes using 1DLUTs for each of the RGB channels to map the RGB channels of the video image separately, ensuring accurate conversion of each pixel node, and using a preset high-dimensional LUT to correct the overall color display of the video image. This ensures more accurate color conversion and achieves a more accurate display effect that conforms to human visual perception.

[0209] Figure 6 This is a flowchart illustrating how to train the color prediction network according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and uses the training of the color prediction network as an example for illustrative explanation.

[0210] like Figure 6 As shown, the video processing method of this embodiment may further include training the color prediction network based on the following steps S601-S604:

[0211] In step S601, the sample image and its target image are respectively input into the color prediction network to be trained to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image.

[0212] In this embodiment, when training the color prediction network, the sample image pic_ori and the target image pic_tgt can be input into the color prediction network to be trained to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image.

[0213] The first network prediction probability can be used to characterize the probability of the memory color being displayed correctly in the sample image; while the second network prediction probability can be used to characterize the probability of the memory color being displayed correctly in the target image.

[0214] It is worth noting that the aforementioned color prediction network can analyze the overall content of the input image and can abstractly represent the pixel colors surrounding the memory color in the image to form features.

[0215] For example, the output of the color prediction network is 0 to 1, representing the probability that the image correctly displays the memory color in the current display scene.

[0216] In step S602, the first actual prediction probability of the sample image and the second actual prediction probability of the target image are obtained.

[0217] In this embodiment, while obtaining the first network prediction probability of the sample image and the second network prediction probability of the target image, the first actual prediction probability of the sample image and the second actual prediction probability of the target image can also be obtained.

[0218] For example, the first and second actual predicted probabilities can be obtained through manual evaluation (e.g., evaluation through an expert system), and this embodiment does not limit this.

[0219] In step S603, a target loss function is constructed based on the first difference between the first network predicted probability and the first actual predicted probability, and the second difference between the second network predicted probability and the second actual predicted probability.

[0220] In this embodiment, after obtaining the first network prediction probability of the sample image, the second network prediction probability of the target image, the first actual prediction probability of the sample image, and the second actual prediction probability of the target image, a target loss function can be constructed based on the first difference between the first network prediction probability and the first actual prediction probability, and the second difference between the second network prediction probability and the second actual prediction probability.

[0221] It is worth noting that the method of constructing the target loss function based on the first and second differences mentioned above can refer to the method of constructing a loss function based on the difference between the predicted value and the true value in related technologies, such as Mean Squared Error (MSE), etc. This embodiment does not limit this method.

[0222] In step S604, the parameters of the color prediction network are optimized based on the target loss function until a trained color prediction network is obtained.

[0223] In this embodiment, after constructing the target loss function, the parameters of the color prediction network can be optimized based on the target loss function. That is, by adjusting the network parameters to minimize the function value of the target loss function, until satisfactory network parameters are obtained, the trained color prediction network can be obtained.

[0224] As described above, this embodiment inputs the sample image and its target image into the color prediction network to be trained, respectively, to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image. It also obtains the first actual prediction probability of the sample image and the second actual prediction probability of the target image. Based on the first difference between the first network prediction probability and the first actual prediction probability, and the second difference between the second network prediction probability and the second actual prediction probability, a target loss function is constructed. The parameters of the color prediction network are then optimized based on the target loss function until a well-trained color prediction network is obtained. This allows for accurate training of the color prediction network, enabling subsequent acquisition of the correctness probability of displaying the memorized color in an HDR image based on the trained color prediction network.

[0225] Figure 7 This is a flowchart illustrating how to determine the one-dimensional LUT for each of the three RGB channels according to an exemplary embodiment of this disclosure; this embodiment is based on the above embodiment and takes the determination of the one-dimensional LUT for each of the three RGB channels as an example for illustrative explanation.

[0226] like Figure 7As shown, the video processing method of this embodiment may further include determining the one-dimensional LUT for each of the three RGB channels based on the following steps S701-S702:

[0227] In step S701, the input color value of each channel in the RGB three channels is input into the pre-trained sub-network of each channel to obtain the target color value of each channel.

[0228] In this embodiment, when the one-dimensional LUT of each of the three RGB channels is determined, the input color value of each channel in the three RGB channels can be obtained, and then the input color value of each channel in the three RGB channels can be input into the pre-trained sub-network of each channel to obtain the target color value of each channel.

[0229] The RGB three-channel sub-networks mentioned above can be three simple fully connected layer networks. The input of each channel's fully connected layer network can be the color input value of that channel, and the output can be the target color value of that channel.

[0230] In some embodiments, the RGB values ​​of each point in the sample image can be separated into RGB three-channel pixel values ​​to obtain the input color value of each of the RGB three channels. For example, assuming the RGB value of a point in the sample image is (100, 150, 200), after separating its RGB three-channel pixel values, we can obtain 100, 150, and 200, which are the input color values ​​of the R, G, and B channels respectively. Based on this, the input color values ​​of the R, G, and B channels of each point in the sample image are input into the respective sub-networks of the R, G, and B channels to obtain the output results of the three sub-networks, i.e., the target color values ​​of the R, G, and B channels respectively.

[0231] In other embodiments, the training method of the above-mentioned sub-network can be referred to below. Figure 8A The embodiments shown will not be described in detail here.

[0232] In step S702, a one-dimensional LUT for each channel is formed based on the input color value and the corresponding target color value of each channel.

[0233] In this embodiment, after obtaining the target color value of each channel, a one-dimensional LUT for each channel can be formed based on the input color value and the corresponding target color value of each channel.

[0234] As described above, this embodiment obtains the target color value of each channel by inputting the input color value of each of the three RGB channels into the pre-trained sub-network of each channel. Based on the input color value and the corresponding target color value of each channel, a one-dimensional LUT is formed for each channel. This allows for accurate determination of the one-dimensional LUT for each of the three RGB channels. Subsequently, single-channel color mapping of the R, G, and B channels of the HDR image can be performed based on the one-dimensional LUT for each of the three RGB channels. This ensures accurate conversion of each pixel node in the HDR image, thereby achieving a more accurate display effect that conforms to human visual perception.

[0235] Figure 8A This is a flowchart illustrating how to train a sub-network for each channel in the RGB three-channel configuration according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes the training of a sub-network for each channel in the RGB three-channel configuration as an example for illustrative explanation.

[0236] like Figure 8A As shown, the video processing method in this embodiment may further include training a sub-network for each channel of the RGB three channels based on the following steps S801-S804:

[0237] In step S801, the RGB three-channel pixel values ​​of the sample image and the target image are separated respectively to obtain the RGB three-channel grayscale images of the sample image and the target image respectively.

[0238] In this embodiment, when training the sub-network of each channel in the RGB three channels, a pair of sample images pic_ori and their target images pic_tgt can be obtained. The target image of the sample image is an image that displays the same content as the sample image, but whose display effect (including the effect of displaying memory colors) is more in line with the visual characteristics of the human eye.

[0239] It is worth noting that the methods for obtaining the paired sample image pic_ori and its target image pic_tgt can be found in the above description. Figure 4A The embodiment shown may be used, or other acquisition methods may be used in related technologies, and this embodiment does not limit the scope of the invention.

[0240] Based on this, the RGB three-channel pixel values ​​of the sample image and the target image can be separated to obtain the RGB three-channel grayscale image of the sample image and the RGB three-channel grayscale image of the target image.

[0241] For example, assuming the pixel value of a certain point in a sample image is (100, 150, 200), it can be separated into pixel values ​​of 100, 150, and 200 through the RGB three-channel pixel values, and the corresponding single-channel image can be generated. The one-to-one correspondence between the corresponding image data can be shown in Table 2 below:

[0242] Table 2

[0243] Sample image: Img_src Target image: Img_tgt R channel image of the sample image: Img_src_R R channel image of the target image: Img_tgt_R G-channel image of the sample image: Img_src_G G-channel image of the target image: Img_tgt_G B-channel image of the sample image: Img_src_B B-channel image of the target image: Img_tgt_B

[0244] In step S802, the RGB three-channel grayscale image of the sample image is input into the sub-network to be trained for each channel to obtain the image of each channel.

[0245] In this embodiment, after obtaining the RGB three-channel grayscale images of the sample image and the target image respectively, the RGB three-channel grayscale images of the sample image can be input into the training sub-network corresponding to each channel to obtain the image of each channel.

[0246] For example, Figure 8B This is a schematic diagram illustrating the sub-networks of each of the three RGB channels according to an exemplary embodiment of this disclosure; as shown below. Figure 8B As shown, the three networks from top to bottom are the training sub-network netR corresponding to the R channel (the input of this network is Rx_in, the input is Rx_out, x = 1, 2, ..., 4913), the training sub-network netG corresponding to the G channel (the input of this network is Gx_in, the input is Gx_out, x = 1, 2, ..., 4913), and the training sub-network netB corresponding to the B channel (the input of this network is Bx_in, the input is Bx_out, x = 1, 2, ..., 4913).

[0247] Taking a sample image as an example, after obtaining the grayscale images of the R, G, and B channels of the sample image, the grayscale images of the R channel, G channel, and B channel can be input into the training subnetwork netR corresponding to the R channel, the training subnetwork netG corresponding to the G channel, and the training subnetwork netB corresponding to the B channel, respectively, so as to obtain the image Img_gen_R1 of the R channel, the image Img_gen_G1 of the G channel, and the image Img_gen_B1 of the B channel.

[0248] Similarly, the grayscale images of the R, G, and B channels of the target image can also be input into the sub-network to be trained for each of the above channels to obtain the image of each channel, such as the image of the R channel Img_gen_R2, the image of the G channel Img_gen_G2, and the image of the B channel Img_gen_B2.

[0249] In step S803, a loss function for each channel is constructed based on the difference between the image of each channel and the corresponding channel image of the target image.

[0250] In this embodiment, after inputting the RGB three-channel grayscale image of the sample image into the training sub-network corresponding to each channel to obtain the image of each channel, the loss function of each channel can be constructed based on the difference between the image of each channel and the corresponding channel image of the target image.

[0251] For example, after obtaining the RGB three-channel images Img_gen_R1, Img_gen_G1, and Img_gen_B1 of the sample image, and the RGB three-channel images Img_gen_R2, Img_gen_G2, and Img_gen_B2 of the target image, the root mean square error between Img_gen_R1 and Img_gen_R2 can be calculated, and then the loss function lossR of the sub-network netR to be trained corresponding to the R channel can be constructed. Similarly, the root mean square error between Img_gen_G1 and Img_gen_G2 can be calculated, and then the loss function lossG of the sub-network net G to be trained corresponding to the G channel can be constructed. And the root mean square error between Img_gen_B1 and Img_gen_B2 can be calculated, and then the loss function lossB of the sub-network net B to be trained corresponding to the B channel can be constructed.

[0252] In step S804, the parameters of the sub-network for each channel are optimized based on the loss function of each channel until a trained sub-network for each channel is obtained.

[0253] In this embodiment, after constructing the loss function for each channel in the RGB three-channel architecture, the parameters of the sub-network for each channel can be optimized based on the loss function of each channel. That is, by adjusting the parameters of the sub-network for each channel to minimize the function value of the loss function for each channel, until satisfactory network parameters are obtained, the trained sub-network for each channel can be obtained.

[0254] In other embodiments, the loss functions loss R, loss G, and loss B of the three channels can be weighted and combined to form a comprehensive loss function, where the weights can be automatically obtained during network learning. For example, if the red component accounts for a large proportion of an image, the red pixel value can be the focus of optimization, i.e., the weight corresponding to loss_R can be increased accordingly; the same applies to green and blue. If the distribution ratio of red, green, and blue is relatively equal, the weights corresponding to loss_R, loss_G, and loss_B can be made equal. In other embodiments, the overall display color of the image can also be adjusted and corrected using 3DLUT.

[0255] As described above, this embodiment separates the RGB three-channel pixel values ​​of the sample image and its target image respectively to obtain the RGB three-channel grayscale images of the sample image and its target image. The RGB three-channel grayscale images of the sample image are then input into the sub-network to be trained for each channel to obtain the image of each channel. Then, based on the difference between the image of each channel and the corresponding channel image of the target image, a loss function for each channel is constructed. The parameters of the sub-network of each channel are then optimized based on the loss function of each channel until the trained sub-network of each channel is obtained. This can accurately train the sub-network of each channel in the RGB three channels, thus laying an accurate foundation for subsequently determining the one-dimensional LUT of each of the RGB three channels based on the trained sub-network of each channel in the RGB three channels.

[0256] Figure 9 This is a flowchart illustrating how to generate a target HDR video corresponding to the HDR video to be processed, according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to generate a target HDR video corresponding to the HDR video to be processed as an example for illustrative explanation.

[0257] like Figure 9 As shown, the step S103 above, which generates the target HDR video corresponding to the HDR video to be processed based on the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction, may include the following steps S901-S902:

[0258] In step S901, each frame of HDR image after color reconstruction and each frame of initial HDR image without color reconstruction are encoded to obtain encoded multi-frame video frames.

[0259] In this embodiment, when generating the target HDR video, the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images of each frame that have not undergone color reconstruction can be encoded according to a preset encoding method. The preset encoding method can be set according to the actual needs of the scene, such as at least one of intra-frame compression, inter-frame compression and entropy encoding. This embodiment does not limit this.

[0260] In step S902, the multiple video frames are combined into the target HDR video.

[0261] In this embodiment, after obtaining the encoded multi-frame video, the multi-frame video can be synthesized into the target HDR video. For example, the encoded multi-frame video can be encapsulated into a specified video container format (such as MP4, AVI, or MOV). It is worth noting that this container can contain not only video streams but also audio streams, subtitles, and other metadata. Based on this, the synthesized video file can be output, and then read and played by a media player.

[0262] As described above, this embodiment encodes each frame of the HDR image after color reconstruction and each frame of the initial HDR image before color reconstruction to obtain encoded multi-frame video frames, and then synthesizes the multi-frame video frames into the target HDR video. This can accurately generate the target HDR video corresponding to the HDR video to be processed. Since the color reconstruction process conforms to the user's inherent cognitive habits of memorized colors and can adapt to the needs of different display scenarios, it can improve the effect of subsequent playback of the target HDR video and bring a better visual experience to the user.

[0263] Figure 10 This is a block diagram illustrating a video processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as glasses, watches, etc.). Figure 10 As shown, the device may include: an image extraction module 110, an image conversion module 120, and a video generation module 130, wherein:

[0264] Image extraction module 110 is used to extract each frame of SDR image from the SDR video in response to acquiring the standard dynamic range SDR video to be processed;

[0265] Image conversion module 120 is used to perform brightness conversion and color gamut conversion on each frame of SDR image based on the display information of the display device to obtain multiple frames of high dynamic range HDR images, wherein the display device includes a device for displaying HDR video;

[0266] The video generation module 130 is used to generate an HDR video corresponding to the SDR video based on the multi-frame HDR images.

[0267] As described above, the device in this embodiment reconstructs the memory colors in the HDR video to be processed based on a predetermined color mapping relationship. This color mapping relationship is constructed based on the subjective evaluation results of the memory colors in sample images of different display scenarios by the human eye. Therefore, it can ensure that the color reconstruction results can conform to the user's inherent cognitive habits of memory colors and adapt to the needs of different display scenarios, thereby improving the effect of subsequent playback of the target HDR video and bringing a better visual experience to the user.

[0268] Figure 11 This is a block diagram illustrating yet another video processing apparatus according to an exemplary embodiment of the present disclosure;

[0269] The device in this embodiment can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (e.g., mobile phones, tablets), wearable devices (e.g., glasses, watches), etc. The image extraction module 210, image conversion module 220, and video generation module 230 are the same as those described above. Figure 10 The image extraction module 110, image conversion module 120, and video generation module 130 in the illustrated embodiment have the same functions, which will not be described in detail here.

[0270] like Figure 11 As shown, the color detection module 210 may include:

[0271] The frame extraction processing unit 211 is used to perform frame extraction processing on the HDR video to obtain an initial HDR image for each frame contained in the HDR video.

[0272] Color detection unit 212 is used to detect the memory color in each frame of the initial HDR image.

[0273] In some embodiments, the color mapping relationship described above includes a high-dimensional color lookup table (LUT), which includes nine columns of data, of which three columns are input RGB value data, three columns are the RGB mean value data of a preset area surrounding the input RGB value, and three columns are the target RGB value data corresponding to the input RGB value.

[0274] Furthermore, the color reconstruction module 220 may include a first reconstruction unit 221;

[0275] The first reconstruction unit 221 is used for:

[0276] Determine the RGB mean value of a preset region surrounding the memory color in any initial HDR image;

[0277] Based on the current RGB value of the memory color and the average RGB value of its surrounding preset area, the high-dimensional LUT is searched to obtain the target RGB value of the memory color;

[0278] Reconstruct the current RGB value of the memory color in any initial HDR image to the target RGB value of the memory color.

[0279] In some embodiments, the above-described apparatus may further include a high-dimensional LUT determination module 240;

[0280] The high-dimensional LUT determination module 240 may include:

[0281] The sample image acquisition unit 241 is used to acquire sample images under different display scenarios, wherein the memory color is present in the sample images;

[0282] Image color adjustment unit 242 is used to adjust the color information of the memory color in the sample image to obtain the adjusted image;

[0283] The image color evaluation unit 243 is used to subjectively evaluate the memory color in the adjusted image using the human eye, and obtain an evaluation score;

[0284] The target image determination unit 244 is configured to, in response to the evaluation score being greater than or equal to a set score threshold, use the adjusted image as the target image of the sample image;

[0285] The high-dimensional LUT determination unit 245 is used to form three columns of input RGB value data of the high-dimensional LUT based on the first RGB value of the memory color in the sample image, form three columns of RGB mean data of the high-dimensional LUT based on the RGB mean of a preset area around the memory color in the sample image, and form three columns of target RGB value data of the high-dimensional LUT based on the second RGB value of the memory color in the target image.

[0286] In some embodiments, the above-described apparatus may further include:

[0287] The color display prediction module 250 is used to input any initial HDR image into a pre-trained color prediction network to obtain the probability of correct display of the memory color in any initial HDR image.

[0288] Furthermore, the color reconstruction module 220 can also be used to perform the operation of reconstructing the memory color in any initial HDR image based on a predetermined color mapping relationship in response to the display correctness probability being greater than or equal to a set probability threshold.

[0289] In some embodiments, the above apparatus may further include a prediction network training module 260;

[0290] The prediction network training module 260 may include:

[0291] The first probability acquisition unit 261 is used to input the sample image and its target image into the color prediction network to be trained, respectively, to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image. The first network prediction probability and the second network prediction probability are used to characterize the display correctness probability of the memory color in the sample image and the display correctness probability of the memory color in the target image, respectively.

[0292] The second probability acquisition unit 262 is used to acquire the first actual prediction probability of the sample image and the second actual prediction probability of the target image;

[0293] The loss function construction unit 263 is used to construct a target loss function based on a first difference between the first network prediction probability and the first actual prediction probability, and a second difference between the second network prediction probability and the second actual prediction probability.

[0294] The prediction network acquisition unit 264 is used to optimize the parameters of the color prediction network based on the target loss function until a trained color prediction network is obtained.

[0295] In some embodiments, the above color mapping relationship also includes a one-dimensional color lookup table (LUT) for each of the three RGB channels. Each channel's one-dimensional LUT includes two columns of data, one column being the input color value data for the channel and the other column being the target color value data for the channel.

[0296] Furthermore, the color reconstruction module 220 may also include a second reconstruction unit 222;

[0297] The second reconstruction unit 222 can be used for:

[0298] In response to the display correctness probability being less than the set probability threshold, the memory color in any initial HDR image is subjected to RGB three-channel pixel value separation to obtain the R, G, and B three-channel color values ​​of the memory color in any initial HDR image;

[0299] Based on the one-dimensional LUTs of each of the three RGB channels, single-channel color mapping is performed on the three R, G, and B channels respectively to obtain the single-channel mapped HDR image corresponding to any initial HDR image;

[0300] Furthermore, the first reconstruction unit 221 can also be used to perform color reconstruction on the single-channel mapped HDR image based on the high-dimensional LUT.

[0301] In some embodiments, the above-described apparatus may further include a one-dimensional LUT determination module 270;

[0302] The one-dimensional LUT determination module 270 may include:

[0303] The target color acquisition unit 271 is used to input the input color value of each channel in the RGB three channels into the pre-trained sub-network of each channel to obtain the target color value of each channel;

[0304] The one-dimensional LUT determination unit 272 is used to form a one-dimensional LUT for each channel based on the input color value and the corresponding target color value of each channel.

[0305] In some embodiments, the above-described apparatus may further include a sub-network training module 280;

[0306] Subnetwork training module 280 may include:

[0307] The grayscale image acquisition unit 281 is used to perform RGB three-channel pixel value separation on the sample image and its target image respectively to obtain the RGB three-channel grayscale images of the sample image and its target image respectively.

[0308] The channel image acquisition unit 282 is used to input the RGB three-channel grayscale image of the sample image into the training sub-network corresponding to each channel to obtain the image of each channel;

[0309] The channel function construction unit 283 is used to construct a loss function for each channel based on the difference between the image of each channel and the corresponding channel image of the target image;

[0310] The sub-network acquisition unit 284 is used to optimize the parameters of the sub-network of each channel based on the loss function of each channel until the trained sub-network of each channel is obtained.

[0311] In some embodiments, the video generation module 230 described above may include:

[0312] Image encoding unit 231 is used to encode each frame of HDR image after color reconstruction and each frame of initial HDR image without color reconstruction to obtain encoded multi-frame video frames.

[0313] The video generation unit 232 is used to synthesize the multiple video frames into the target HDR video.

[0314] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0315] Figure 12 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0316] Reference Figure 12 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0317] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the video processing method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0318] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0319] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0320] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0321] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0322] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0323] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0324] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0325] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the video processing method described above.

[0326] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to complete the video processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0327] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 920 of the device 900 to perform the video processing method described above.

[0328] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0329] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: In response to acquiring the high dynamic range (HDR) video to be processed, a memory color is detected in the multiple initial HDR images contained in the HDR video; In response to detecting the memory color in any of the multiple initial HDR images, color reconstruction is performed on the memory color in the any initial HDR image based on a predetermined color mapping relationship, wherein the color mapping relationship includes a color mapping relationship constructed based on the subjective evaluation results of the memory color in sample images of different display scenes by the human eye; Based on the color-reconstructed HDR images of each frame in the multi-frame initial HDR images and the remaining unreconstructed initial HDR images, a target HDR video corresponding to the HDR video to be processed is generated.

2. The method according to claim 1, characterized in that, The detection of memory color in the multiple initial HDR images contained in the HDR video includes: The HDR video is subjected to frame extraction to obtain an initial HDR image for each frame contained in the HDR video; The memory color is detected in each frame of the initial HDR image.

3. The method according to claim 1, characterized in that, The color mapping relationship includes a high-dimensional color lookup table (LUT), which includes nine columns of data: three columns are input RGB value data, three columns are the RGB mean data of a preset area surrounding the input RGB value, and three columns are the target RGB value data corresponding to the input RGB value. The step of reconstructing the memory color in any initial HDR image based on a pre-determined color mapping relationship includes: Determine the RGB mean value of a preset region surrounding the memory color in any initial HDR image; Based on the current RGB value of the memory color and the average RGB value of its surrounding preset area, the high-dimensional LUT is searched to obtain the target RGB value of the memory color; Reconstruct the current RGB value of the memory color in any initial HDR image to the target RGB value of the memory color.

4. The method according to claim 3, characterized in that, The method further includes pre-determining the high-dimensional LUT based on the following: Obtain sample images under different display scenarios, wherein the memory color is present in the sample images; The color information of the memory color in the sample image is adjusted to obtain the adjusted image; The memory color in the adjusted image is subjectively evaluated using the human eye to obtain an evaluation score; In response to the evaluation score being greater than or equal to a set score threshold, the adjusted image is used as the target image of the sample image; The high-dimensional LUT is formed by generating three columns of input RGB value data based on the first RGB value of the memory color in the sample image, generating three columns of RGB mean data based on the RGB mean of the preset area surrounding the memory color in the sample image, and generating three columns of target RGB value data based on the second RGB value of the memory color in the target image.

5. The method according to claim 4, characterized in that, The method further includes: The initial HDR image is input into a pre-trained color prediction network to obtain the probability of the correct display of the memory color in the initial HDR image. In response to the display correctness probability being greater than or equal to a set probability threshold, the operation of reconstructing the memory color in any initial HDR image based on a predetermined color mapping relationship is performed.

6. The method according to claim 5, characterized in that, The method also includes pre-training the color prediction network based on the following: The sample image and its target image are respectively input into the color prediction network to be trained to obtain the first network prediction probability of the sample image and the second network prediction probability of the target image. The first network prediction probability and the second network prediction probability are respectively used to characterize the display correctness probability of the memory color in the sample image and the display correctness probability of the memory color in the target image. Obtain the first actual predicted probability of the sample image and the second actual predicted probability of the target image; Based on the first difference between the first network predicted probability and the first actual predicted probability, and the second difference between the second network predicted probability and the second actual predicted probability, a target loss function is constructed. The parameters of the color prediction network are optimized based on the objective loss function until a well-trained color prediction network is obtained.

7. The method according to claim 5, characterized in that, The color mapping relationship also includes a one-dimensional color lookup table (LUT) for each of the three RGB channels. Each channel's one-dimensional LUT includes two columns of data: one column is the input color value data for the channel, and the other column is the target color value data for the channel. The step of reconstructing the memory color in any initial HDR image based on a pre-determined color mapping relationship includes: In response to the display correctness probability being less than the set probability threshold, the memory color in any initial HDR image is subjected to RGB three-channel pixel value separation to obtain the R, G, and B three-channel color values ​​of the memory color in any initial HDR image; Based on the one-dimensional LUTs of each of the three RGB channels, single-channel color mapping is performed on the three R, G, and B channels respectively to obtain the single-channel mapped HDR image corresponding to any initial HDR image; Based on the high-dimensional LUT, color reconstruction is performed on the single-channel mapped HDR image.

8. The method according to claim 7, characterized in that, The method also includes pre-determining a one-dimensional LUT for each of the three RGB channels based on the following: The input color value of each of the three RGB channels is input into the pre-trained sub-network of each channel to obtain the target color value of each channel; Based on the input color value and the corresponding target color value of each channel, a one-dimensional LUT is formed for each channel.

9. The method according to claim 8, characterized in that, The method also includes pre-training a sub-network for each of the RGB three channels based on the following: The RGB three-channel pixel values ​​of the sample image and its target image are separated respectively to obtain the RGB three-channel grayscale images of the sample image and its target image respectively; The RGB three-channel grayscale images of the sample image are input into the sub-network to be trained for each channel to obtain the image of each channel; Based on the difference between the image of each channel and the corresponding channel image of the target image, a loss function is constructed for each channel; The parameters of the sub-network for each channel are optimized based on the loss function of each channel until a trained sub-network for each channel is obtained.

10. The method according to claim 1, characterized in that, The step of generating the target HDR video corresponding to the HDR video to be processed based on the color-reconstructed HDR images of each frame from the multiple initial HDR images and the remaining initial HDR images without color reconstruction includes: Each frame of HDR image after color reconstruction and each frame of initial HDR image without color reconstruction are encoded to obtain encoded multi-frame video frames. The multiple video frames are combined into the target HDR video.

11. A video processing apparatus, characterized in that, The device includes: The color detection module is used to detect memory colors in the multiple initial HDR images contained in the HDR video in response to acquiring the high dynamic range HDR video to be processed; A color reconstruction module is configured to, in response to detecting the memory color in any of the multiple initial HDR images, reconstruct the memory color in any of the initial HDR images based on a predetermined color mapping relationship, wherein the color mapping relationship includes a color mapping relationship constructed based on the subjective evaluation results of the memory color in sample images of different display scenes by the human eye; The video generation module is used to generate a target HDR video corresponding to the HDR video to be processed based on the HDR images of each frame that have undergone color reconstruction and the remaining initial HDR images that have not undergone color reconstruction from the multiple initial HDR images.

12. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the video processing method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the video processing method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video processing method according to any one of claims 1 to 10.

Images