Image conversion method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses an image conversion method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: obtaining a standard dynamic range image and a reference image thereof, and determining the RGB color value of the standard dynamic range image; calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image; obtaining the fusion feature correlation according to the internal feature correlation and the feature correlation, and determining the weight value of each three-dimensional lookup table according to the fusion feature correlation; searching for the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, performing weighted calculation on the initial RGB color value and the weight value to obtain the target RGB color value; and generating the high dynamic range image corresponding to the standard dynamic range image by using the target RGB color value. The scheme can effectively improve the quality of the converted image in the conversion process from SDR data to HDR data, and guarantee the user experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image conversion method, as well as an image conversion device, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, most existing videos are SDR (Standard Dynamic Range) videos, meaning their maximum brightness is 100 nits and their color gamut is sRGB, with a dynamic range far below the monitor's specifications. To fully utilize the monitor's performance and improve the user's viewing experience, SDR videos can be converted to HDR (High-Dynamic Range) videos. However, this conversion process often results in abnormal exposure of the converted image.

[0003] Therefore, how to effectively improve the image quality and enhance the user experience during the conversion from SDR to HDR data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide an image conversion method that can effectively improve the quality of the converted image and ensure user experience during the conversion process from SDR data to HDR data. Another purpose of this application is to provide an image conversion device, electronic device, and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, this application discloses an image conversion method, comprising:

[0006] Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image;

[0007] Calculate the internal feature correlation of the standard dynamic range image, and calculate the feature correlation between the standard dynamic range image and the reference image;

[0008] The fused feature correlation is calculated based on the internal feature correlation and the feature correlation, and the weight value of each three-dimensional lookup table is determined based on the fused feature correlation.

[0009] The initial RGB color value corresponding to the RGB color value is found in each of the three-dimensional lookup tables, and the target RGB color value is obtained by weighted calculation using each of the initial RGB color values ​​and each of the weight values.

[0010] A high dynamic range image corresponding to the standard dynamic range image is generated using the target RGB color values.

[0011] Optionally, acquiring a standard dynamic range image and a reference image of the standard dynamic range image includes:

[0012] Obtain a standard dynamic range video, perform frame segmentation on the standard dynamic range video, and obtain a standard dynamic range image sequence;

[0013] For any target standard dynamic range image in the standard dynamic range image sequence, one or more other standard dynamic range images other than the target standard dynamic range image are selected in the standard dynamic range image sequence as reference images for the target standard dynamic range image.

[0014] Accordingly, after generating the high dynamic range image corresponding to the standard dynamic range image using the target RGB color values, the method further includes:

[0015] A high dynamic range video corresponding to the standard dynamic range video is generated using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence.

[0016] Optionally, the standard dynamic range video is subjected to frame-by-frame processing to obtain a standard dynamic range image sequence, including:

[0017] The standard dynamic range video is segmented into scenes to obtain standard dynamic range image sequences corresponding to each scene.

[0018] Accordingly, selecting one or more other standard dynamic range images, excluding the target standard dynamic range image, within the standard dynamic range image sequence as a reference image for the target standard dynamic range image includes:

[0019] Within the standard dynamic range image sequence, a first preset number of other standard dynamic range images preceding the target standard dynamic range image and a second preset number of other standard dynamic range images following the target standard dynamic range image are selected as reference images for the target standard dynamic range image.

[0020] Optionally, before obtaining the target RGB color value by performing a weighted calculation using each of the initial RGB color values ​​and each of the weight values, the method further includes:

[0021] For an RGB color value for which no corresponding initial RGB color value can be found in the three-dimensional lookup table, an interpolation calculation is performed on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value.

[0022] Optionally, before calculating the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image, the method further includes:

[0023] The standard dynamic range image and the reference image are registered using an optical flow estimation algorithm to align the associated feature points between the standard dynamic range image and the reference image.

[0024] Optionally, calculating the internal feature correlation of the standard dynamic range image includes:

[0025] The internal feature correlation of the standard dynamic range image is calculated using a self-attention mechanism.

[0026] Optionally, calculating the feature correlation between the standard dynamic range image and the reference image includes:

[0027] The feature correlation between the standard dynamic range image and the reference image is calculated using a cross-attention mechanism.

[0028] Secondly, this application also discloses an image conversion apparatus, comprising:

[0029] The acquisition module is used to acquire a standard dynamic range image and a reference image of the standard dynamic range image, and to determine the RGB color values ​​of the standard dynamic range image;

[0030] The calculation module is used to calculate the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image;

[0031] The determination module is used to calculate the fused feature correlation based on the internal feature correlation and the feature correlation, and to determine the weight value of each three-dimensional lookup table based on the fused feature correlation.

[0032] The lookup module is used to look up the initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and to perform a weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain the target RGB color value;

[0033] The generation module is used to generate a high dynamic range image corresponding to the standard dynamic range image using the target RGB color values.

[0034] Thirdly, this application also discloses an electronic device, comprising:

[0035] Memory, used to store computer programs;

[0036] A processor for executing the computer program to implement any of the image conversion methods described above.

[0037] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image conversion methods described above.

[0038] This application provides an image conversion method, comprising: acquiring a standard dynamic range image and a reference image of the standard dynamic range image, and determining the RGB color values ​​of the standard dynamic range image; calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image; calculating a fusion feature correlation based on the internal feature correlation and the feature correlation, and determining the weight values ​​of each three-dimensional lookup table based on the fusion feature correlation; searching for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and performing a weighted calculation using each initial RGB color value and each weight value to obtain a target RGB color value; and generating a high dynamic range image corresponding to the standard dynamic range image using the target RGB color value.

[0039] By applying the technical solution provided in this application, for a standard dynamic range image that needs to be converted into a high dynamic range image, the relevant feature information of its corresponding reference image can be obtained to achieve the conversion from a standard dynamic range image to a high dynamic range image. First, multiple three-dimensional lookup tables are pre-created. Each three-dimensional lookup table records the mapping relationship between the RGB color values ​​of the standard dynamic range image and the RGB color values ​​of the high dynamic range image. Each three-dimensional lookup table can be used to obtain the RGB color value of the high dynamic range image corresponding to the RGB color value of the standard dynamic range image. Second, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are fused to determine the weight of each three-dimensional lookup table. Finally, based on the found RGB color values ​​of each high dynamic range image (i.e., the initial RGB color values ​​mentioned above) and each weight value, the final RGB color value of the high dynamic range image (i.e., the target RGB color value mentioned above) is obtained through weight calculation, thereby obtaining the high dynamic range image corresponding to the standard dynamic range image and completing the conversion from a standard dynamic range image to a high dynamic range image. Therefore, this technical solution can recover the image quality of the current standard dynamic range image by learning the feature information of relevant reference images, effectively improving the image quality after conversion during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0040] The image conversion device, electronic device, and computer-readable storage medium provided in this application also have the above-mentioned technical effects, and will not be described in detail here. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0042] Figure 1 A schematic flowchart of an image conversion method provided in this application;

[0043] Figure 2 A flowchart illustrating a video conversion method provided in this application;

[0044] Figure 3 An image conversion flowchart of an image conversion model provided in this application;

[0045] Figure 4 A schematic diagram of the structure of an image conversion device provided in this application;

[0046] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0047] The core of this application is to provide an image conversion method that can effectively improve the quality of the converted image and ensure user experience during the conversion process from SDR data to HDR data. Another core aspect of this application is to provide an image conversion device, electronic device, and computer-readable storage medium, all of which have the aforementioned beneficial effects.

[0048] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0049] This application provides an image conversion method.

[0050] Please refer to Figure 1 , Figure 1This is a schematic flowchart of an image conversion method provided in this application, which may include the following steps S101 to S105.

[0051] S101: Obtain the standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image.

[0052] This step aims to acquire a standard dynamic range image (hereinafter referred to as SDR image) and its reference image, as well as determine the RGB color values ​​of the SDR image. The RGB color values ​​include the R, G, and B channel colors. After acquiring the SDR image, it can be converted to RGB format, thereby determining the RGB color values ​​of the SDR image. The reference image is used to learn feature information from the SDR image. It can be an image with the same scene as the SDR image, for example, a neighboring frame in the SDR video containing the SDR image.

[0053] S102: Calculate the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image.

[0054] This step aims to achieve image feature learning, including the internal feature correlation of the SDR image itself and the feature correlation between the SDR image and its reference image. By learning its own feature information and the feature information of its reference image, the conversion from SDR image to HDR image is achieved, so as to effectively ensure the image quality during the conversion process.

[0055] Calculating the internal feature correlation of a standard dynamic range image may include: using a self-attention mechanism to calculate the internal feature correlation of a standard dynamic range image; calculating the feature correlation between a standard dynamic range image and a reference image may include: using a cross-attention mechanism to calculate the feature correlation between a standard dynamic range image and a reference image.

[0056] In one embodiment of this application, before calculating the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image, the method may further include: registering the standard dynamic range image and the reference image using an optical flow estimation algorithm to align the associated feature points between the standard dynamic range image and the reference image.

[0057] To further ensure the quality of the converted image, image registration can be performed on the SDR image and its reference image before feature learning. This achieves spatial alignment of associated feature points between the images, effectively reducing the difficulty of subsequent feature learning. In practice, the Lucas-Kanade optical flow estimation algorithm can be used.

[0058] S103: Calculate the fused feature correlation based on the internal feature correlation and feature correlation, and determine the weight value of each three-dimensional lookup table based on the fused feature correlation.

[0059] This step aims to achieve the fusion processing of learned features to further determine the weight values ​​of each 3D look-up table. Specifically, multiple 3D look-up tables (3D LUTs) can be pre-created. Each 3D look-up table is used to record the mapping relationship between a set of RGB color values ​​of SDR images and RGB color values ​​of HDR images. Through each 3D look-up table, the RGB color value of an HDR image corresponding to the RGB color value of an SDR image can be obtained, that is, as described in S104, "finding the initial RGB color value corresponding to the RGB color value in each 3D look-up table".

[0060] In one possible implementation, multiple predefined masks (a type of binary function) can be used to non-linearly transform each 3D lookup table, mapping the input of the 3D lookup table to different brightness ranges, thus enabling enhancement for different brightness levels. For example, when using a 16×16×16 3D LUT, due to precision limitations, storing the mapping results of the entire brightness space in each 3D LUT may not be effective. Therefore, one 3D LUT can store only the transformation results of the dark areas, another 3D LUT can store only the transformation results of the bright areas, and so on, with different 3D LUTs storing transformation results for different areas.

[0061] Furthermore, both internal feature correlation and feature correlation can be vectors representing feature correlation. By inputting these vectors into the neural network layer for processing, a new set of vectors can be obtained, which is a set of vectors used to represent the fused feature correlation. Each value in this new set of vectors corresponds to a weight value of a three-dimensional lookup table.

[0062] Understandably, the purpose of using multiple 3D lookup tables is to increase generalization ability, because a single 3D lookup table can only represent one mapping relationship. Different types of SDR images often require multiple different types of 3D lookup tables to achieve the best conversion effect (for example, SDR images of different styles and scenes generally require different 3D lookup tables). Therefore, multiple 3D lookup tables can be used. Furthermore, a set of weight values ​​determined based on the correlation of fused features is used to combine each 3D lookup table to obtain the final result (such as the weight calculation in S104).

[0063] S104: Find the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, and perform weighted calculation using each initial RGB color value and each weight value to obtain the target RGB color value.

[0064] As described above, each three-dimensional lookup table is used to record the mapping relationship between the RGB color values ​​of a set of SDR images and the RGB color values ​​of HDR images. Based on each three-dimensional lookup table, an initial HDR image RGB color value corresponding to the RGB color value of the SDR image can be obtained, that is, the aforementioned initial RGB color value. Combined with the weight value of the corresponding three-dimensional lookup table, the final HDR image RGB color value, that is, the aforementioned target RGB color value, can be calculated.

[0065] In one embodiment of this application, before obtaining the target RGB color value by weighted calculation using each initial RGB color value and each weight value, the method may further include: for RGB color values ​​for which the corresponding initial RGB color value cannot be found in the three-dimensional lookup table, interpolation calculation is performed on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value. In specific implementation, methods such as trilinear interpolation or tetrahedral interpolation can be used.

[0066] S105: Generate a high dynamic range image corresponding to the standard dynamic range image using the target RGB color values.

[0067] This step aims to generate the HDR image corresponding to the SDR image. That is, after obtaining the target RGB color value, the HDR image corresponding to the SDR image can be directly generated using the target RGB color value.

[0068] As can be seen, the image conversion method provided in this application, for a standard dynamic range image that needs to be converted into a high dynamic range image, can obtain the relevant feature information of its corresponding reference image to achieve the conversion from a standard dynamic range image to a high dynamic range image. First, multiple three-dimensional lookup tables are pre-created. Each three-dimensional lookup table is used to record the mapping relationship between the RGB color values ​​of the standard dynamic range image and the RGB color values ​​of the high dynamic range image. Through each three-dimensional lookup table, the RGB color value of the high dynamic range image corresponding to the RGB color value of the standard dynamic range image can be obtained. Second, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are fused to determine the weight of each three-dimensional lookup table. Finally, the final RGB color value of the high dynamic range image (i.e., the target RGB color value) is obtained by weight calculation based on the found RGB color values ​​of each high dynamic range image (i.e., the initial RGB color values ​​mentioned above) and each weight value, thereby obtaining the high dynamic range image corresponding to the standard dynamic range image and completing the conversion from a standard dynamic range image to a high dynamic range image. Therefore, this technical solution can recover the image quality of the current standard dynamic range image by learning the feature information of relevant reference images, effectively improving the image quality after conversion during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0069] In one embodiment of this application, obtaining a standard dynamic range image and a reference image of the standard dynamic range image may include: obtaining a standard dynamic range video, performing frame-by-frame processing on the standard dynamic range video to obtain a standard dynamic range image sequence; for any target standard dynamic range image in the standard dynamic range image sequence, selecting one or more other standard dynamic range images other than the target standard dynamic range image within the standard dynamic range image sequence as a reference image of the target standard dynamic range image.

[0070] Accordingly, after generating a high dynamic range image corresponding to a standard dynamic range image using the target RGB color values, it may also include: generating a high dynamic range video corresponding to a standard dynamic range video using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence.

[0071] The image conversion method provided in this application embodiment is also applicable to video conversion processes, that is, converting SDR video to HDR video. In this process, steps S101 to S105 described above can be executed sequentially for each video frame in the complete video. Obviously, for any SDR image in the SDR video, a reference image can also be selected and determined in the same SDR video to effectively ensure the image quality after conversion.

[0072] The process of performing frame-by-frame processing on standard dynamic range video to obtain standard dynamic range image sequences may include: performing scene segmentation on standard dynamic range video to obtain standard dynamic range image sequences corresponding to each scene.

[0073] Accordingly, selecting one or more other standard dynamic range images besides the target standard dynamic range image within the standard dynamic range image sequence as reference images for the target standard dynamic range image may include: selecting a first preset number of other standard dynamic range images before the target standard dynamic range image and a second preset number of other standard dynamic range images after the target standard dynamic range image within the standard dynamic range image sequence as reference images for the target standard dynamic range image.

[0074] This application provides a method for determining a reference image for an SDR image, which involves selecting as many other SDR images as possible that are in the same scene as the current SDR image as reference images. This is because images in the same scene have a high degree of similarity, which helps to achieve more accurate feature learning, more accurate image conversion, and ensure the quality of the converted image.

[0075] In one possible implementation, scene segmentation of SDR video can be achieved by comparing the brightness changes between any two adjacent SDR frames. That is, when the brightness change between any two adjacent SDR frames is higher than a certain threshold, it can be considered that these two adjacent SDR frames belong to different scenes, thereby achieving scene segmentation of SDR video.

[0076] In another possible implementation, the Canny edge detection operator can be directly used to achieve scene segmentation in SDR video. The Canny operator detects edge contours in an image, outputting a binary image (0 represents non-edge, 1 represents edge). Based on this, for any two adjacent SDR frames, the interpolation between the binary images obtained after passing through the Canny operator can be calculated. If the interpolation value is greater than a certain threshold, the edge contours are considered to be significantly different, meaning the two adjacent SDR frames are considered to belong to different scenes, thus achieving scene segmentation in SDR video.

[0077] Based on the above embodiments, please refer to Figure 2 , Figure 2 A flowchart illustrating a video conversion method provided in this application is shown below:

[0078] 1. Video decapsulation and decoding: Decapsulate and decode the SDR video file according to its encapsulation and encoding formats to obtain continuous RGB format video frame images (SDR images).

[0079] 2. Segmenting Video Frames Based on Scene: The SDR video is segmented into several groups of consecutive video frames, each group representing a scene. This can be achieved by directly calculating the brightness change between adjacent video frames and segmenting when the brightness change exceeds a certain threshold; alternatively, the Canny edge detection operator can be used, and segmentation can be performed when the edge contour differences are too large.

[0080] 3. Model Training: Please refer to... Figure 3 , Figure 3 This is a flowchart of an image conversion model provided in this application. Based on this image conversion model, the above steps S101 to S105 can be implemented.

[0081] (1) Initialize all model parameters and initialize all 3D LUTs.

[0082] (2) Process all frames of the current scene sequentially. For any current frame, sample several frames forward and backward as reference frames (reference images). Random sampling or fixed-distance sampling can be used.

[0083] (3) Use the Lucas-Kanade optical flow estimation algorithm to register the current frame and the reference frame to align the associated feature points between the current frame and the reference frame.

[0084] (4) The current frame and all its reference frames are fed into the neural network to learn the corresponding feature vectors.

[0085] (5) The correlation of features in the feature vector of the current frame can be learned using a self-attention mechanism.

[0086] (6) The correlation between the feature vector of the current frame and the feature vectors of each reference frame can be learned using the cross attention mechanism.

[0087] (7) A set of vectors representing feature correlations can be obtained through (5) and (6). These vectors (integrating feature correlations) are fed into the neural network layer for transformation to obtain a new set of vectors. Each value in this new set of vectors corresponds to a weight of a predefined 3D LUT. Each 3D LUT is non-linearly transformed using different predefined masks to map the input values ​​of the 3D LUT to a specified brightness range, thereby enabling enhancement for different brightness ranges.

[0088] (8) Query all 3D LUTs respectively to obtain a set of initial HDR image RGB color values. If the query result is not in the corresponding 3D LUT, interpolation processing can be performed, such as trilinear interpolation or tetrahedral interpolation.

[0089] (9) Combine the 3D LUT output values ​​in (8) and the weight values ​​in (7) to calculate the final mapping value, which is the final HDR image RGB color value.

[0090] (10) Calculate the difference between the mapped value and the true value through the loss function, and update the parameters through backpropagation. The loss function can be either MSE or MAE until the final image conversion model is obtained.

[0091] 4. Model inference: Based on the image conversion model trained in step 3, the execution process is as follows (1) to (9) in step 3.

[0092] 5. Post-processing: Smooth the transition frames between scenes as needed to effectively maintain natural scene transitions.

[0093] As can be seen, the image conversion method provided in this application, for a standard dynamic range image that needs to be converted into a high dynamic range image, can obtain the relevant feature information of its corresponding reference image to achieve the conversion from a standard dynamic range image to a high dynamic range image. First, multiple three-dimensional lookup tables are pre-created. Each three-dimensional lookup table is used to record the mapping relationship between the RGB color values ​​of the standard dynamic range image and the RGB color values ​​of the high dynamic range image. Through each three-dimensional lookup table, the RGB color value of the high dynamic range image corresponding to the RGB color value of the standard dynamic range image can be obtained. Second, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are fused to determine the weight of each three-dimensional lookup table. Finally, the final RGB color value of the high dynamic range image (i.e., the target RGB color value) is obtained by weight calculation based on the found RGB color values ​​of each high dynamic range image (i.e., the initial RGB color values ​​mentioned above) and each weight value, thereby obtaining the high dynamic range image corresponding to the standard dynamic range image and completing the conversion from a standard dynamic range image to a high dynamic range image. Therefore, this technical solution can recover the image quality of the current standard dynamic range image by learning the feature information of relevant reference images, effectively improving the image quality after conversion during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0094] This application provides an image conversion device.

[0095] Please refer to Figure 4 , Figure 4 This is a schematic diagram of an image conversion apparatus provided in this application. The image conversion apparatus may include:

[0096] Module 1 is used to acquire a standard dynamic range image and a reference image of the standard dynamic range image, and to determine the RGB color values ​​of the standard dynamic range image.

[0097] Calculation module 2 is used to calculate the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image;

[0098] Module 3 is used to calculate the fused feature correlation based on the internal feature correlation and feature correlation, and to determine the weight value of each three-dimensional lookup table based on the fused feature correlation.

[0099] The lookup module 4 is used to find the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, and to perform a weighted calculation using each initial RGB color value and each weight value to obtain the target RGB color value;

[0100] Generation module 5 is used to generate a high dynamic range image corresponding to a standard dynamic range image using the target RGB color values.

[0101] As can be seen, the image conversion apparatus provided in this application embodiment can obtain the relevant feature information of the corresponding reference image to realize the conversion from a standard dynamic range image to a high dynamic range image for a standard dynamic range image that needs to be converted into a high dynamic range image. First, multiple three-dimensional lookup tables are pre-created. Each three-dimensional lookup table is used to record the mapping relationship between the RGB color values ​​of the standard dynamic range image and the RGB color values ​​of the high dynamic range image. Through each three-dimensional lookup table, the RGB color value of the high dynamic range image corresponding to the RGB color value of the standard dynamic range image can be obtained. Second, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are fused to determine the weight of each three-dimensional lookup table. Finally, the final RGB color value of the high dynamic range image (i.e., the target RGB color value) is obtained by weight calculation based on the found RGB color values ​​of each high dynamic range image (i.e., the initial RGB color values ​​mentioned above) and each weight value, thereby obtaining the high dynamic range image corresponding to the standard dynamic range image and completing the conversion from a standard dynamic range image to a high dynamic range image. Therefore, this technical solution can recover the image quality of the current standard dynamic range image by learning the feature information of relevant reference images, effectively improving the image quality after conversion during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0102] In one embodiment of this application, the acquisition module 1 may include:

[0103] The framing unit is used to acquire standard dynamic range video, perform framing processing on the standard dynamic range video, and obtain a standard dynamic range image sequence.

[0104] The selection unit is used to select one or more other standard dynamic range images (excluding the target standard dynamic range image) within the standard dynamic range image sequence for any target standard dynamic range image in the standard dynamic range image sequence, and use them as reference images for the target standard dynamic range image.

[0105] Accordingly, the generation module 5 can also be used to generate a high dynamic range video corresponding to a standard dynamic range video using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence after generating a high dynamic range image corresponding to a standard dynamic range image using the target RGB color values.

[0106] In one embodiment of this application, the above-mentioned framing unit can be specifically used to perform scene segmentation on standard dynamic range video to obtain standard dynamic range image sequences corresponding to each scene;

[0107] Accordingly, the selection unit can be specifically used to select, within the standard dynamic range image sequence, a first preset number of other standard dynamic range images before the target standard dynamic range image and a second preset number of other standard dynamic range images after the target standard dynamic range image, as reference images for the target standard dynamic range image.

[0108] In one embodiment of this application, the search module 4 can also be used to perform interpolation calculations on the RGB color values ​​to obtain the initial RGB color values ​​corresponding to the RGB color values ​​before the target RGB color value is obtained by weighted calculation using each initial RGB color value and each weight value.

[0109] In one embodiment of this application, the image conversion apparatus may further include a registration module, which is used to register the standard dynamic range image and the reference image using an optical flow estimation algorithm before calculating the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image, so as to align the associated feature points between the standard dynamic range image and the reference image.

[0110] In one embodiment of this application, the above-mentioned calculation module 2 can be specifically used to calculate the internal feature correlation of a standard dynamic range image using a self-attention mechanism.

[0111] In one embodiment of this application, the above-mentioned calculation module 2 can be specifically used to calculate the feature correlation between the standard dynamic range image and the reference image using a cross-attention mechanism.

[0112] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0113] This application provides an electronic device.

[0114] Please refer to Figure 5 , Figure 5 This application provides a schematic diagram of the structure of an electronic device, which may include:

[0115] Memory, used to store computer programs;

[0116] A processor, used to execute computer programs, can implement the steps of any of the image conversion methods described above.

[0117] like Figure 5 The diagram shows the structural composition of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0118] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0119] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the image conversion method.

[0120] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0121] Obtain the standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image;

[0122] Calculate the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image;

[0123] The fused feature correlation is calculated based on the internal feature correlation and feature correlation, and the weight value of each three-dimensional lookup table is determined based on the fused feature correlation.

[0124] The initial RGB color value corresponding to the RGB color value is found in each 3D lookup table, and the target RGB color value is obtained by weighted calculation using each initial RGB color value and each weight value.

[0125] Generate a high dynamic range image corresponding to a standard dynamic range image using the target RGB color values.

[0126] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0127] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0128] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 5 More or fewer components as shown, or combinations of certain components.

[0129] This application provides a computer-readable storage medium.

[0130] The computer-readable storage medium provided in this application embodiment stores a computer program, which, when executed by a processor, can implement the steps of any of the image conversion methods described above.

[0131] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] For a description of the computer-readable storage medium provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0134] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0136] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. An image conversion method, characterized in that, include: Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image; The internal feature correlation of the standard dynamic range image is calculated using a self-attention mechanism, and the feature correlation between the standard dynamic range image and the reference image is calculated using a cross-attention mechanism; both the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image are vectors representing feature correlation. Based on the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image, a fusion feature correlation is calculated using a neural network, and the weight values ​​of each three-dimensional lookup table are determined based on the fusion feature correlation. The fusion feature correlation is a set of vectors used to represent the fusion feature correlation, and each value in the fusion feature correlation vector corresponds to a weight value of one of the three-dimensional lookup tables. The initial RGB color value corresponding to the RGB color value is found in each of the three-dimensional lookup tables, and the target RGB color value is obtained by weighted calculation using each of the initial RGB color values ​​and each of the weight values. A high dynamic range image corresponding to the standard dynamic range image is generated using the target RGB color values; The process of acquiring a standard dynamic range image and a reference image for the standard dynamic range image includes: acquiring a standard dynamic range video; performing scene segmentation on the standard dynamic range video to obtain a sequence of standard dynamic range images corresponding to each scene; and for any target standard dynamic range image in the sequence of standard dynamic range images, selecting a first preset number of other standard dynamic range images before the target standard dynamic range image and a second preset number of other standard dynamic range images after the target standard dynamic range image as reference images for the target standard dynamic range image. Before obtaining the target RGB color value by performing weighted calculation using each of the initial RGB color values ​​and each of the weight values, the method further includes: for RGB color values ​​for which the corresponding initial RGB color value cannot be found in the three-dimensional lookup table, performing interpolation calculation on the RGB color values ​​to obtain the initial RGB color value corresponding to the RGB color value.

2. The image conversion method according to claim 1, characterized in that, After generating the high dynamic range image corresponding to the standard dynamic range image using the target RGB color values, the method further includes: A high dynamic range video corresponding to the standard dynamic range video is generated using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence.

3. The image conversion method according to claim 1, characterized in that, Before calculating the internal feature correlation of the standard dynamic range image using a self-attention mechanism and the feature correlation between the standard dynamic range image and the reference image using a cross-attention mechanism, the method further includes: The standard dynamic range image and the reference image are registered using an optical flow estimation algorithm to align the associated feature points between the standard dynamic range image and the reference image.

4. An image conversion device, characterized in that, include: The acquisition module is used to acquire a standard dynamic range image and a reference image of the standard dynamic range image, and to determine the RGB color values ​​of the standard dynamic range image; The calculation module is used to calculate the internal feature correlation of the standard dynamic range image using a self-attention mechanism, and to calculate the feature correlation between the standard dynamic range image and the reference image using a cross-attention mechanism; the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image are both vectors representing feature correlation; The determination module is used to calculate the fusion feature correlation using a neural network based on the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and the reference image, and to determine the weight value of each three-dimensional lookup table based on the fusion feature correlation; the fusion feature correlation is a set of vectors used to represent the fusion feature correlation, and each value in the fusion feature correlation vector corresponds to a weight value of one of the three-dimensional lookup tables. The lookup module is used to look up the initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and to perform a weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain the target RGB color value; The generation module is used to generate a high dynamic range image corresponding to the standard dynamic range image using the target RGB color values; Specifically, the acquisition module is used to acquire standard dynamic range video, perform scene segmentation on the standard dynamic range video, and obtain standard dynamic range image sequences corresponding to each scene; for any target standard dynamic range image in the standard dynamic range image sequence, within the standard dynamic range image sequence, select a first preset number of other standard dynamic range images before the target standard dynamic range image and a second preset number of other standard dynamic range images after the target standard dynamic range image as reference images for the target standard dynamic range image; The lookup module is further configured to perform interpolation calculations on the RGB color values ​​to obtain the initial RGB color values ​​corresponding to the RGB color values ​​before performing weighted calculations using the initial RGB color values ​​and the weight values ​​to obtain the target RGB color values.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image conversion method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image conversion method as described in any one of claims 1 to 3.

Images