Image conversion method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image conversion method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a standard dynamic range image and a reference image, and determining the RGB color value of the standard dynamic range image; calculating internal feature correlation of the standard dynamic range image, and calculating feature correlation between the standard dynamic range image and the reference image; calculating fusion feature correlation according to the internal feature correlation and the feature correlation, and determining a weight value of each three-dimensional lookup table according to the fusion feature correlation; searching an initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, and performing weighted calculation by using each initial RGB color value and each weight value to obtain a target RGB color value; and generating a high dynamic range image corresponding to the standard dynamic range image by using the target RGB color value. According to the scheme, the quality of the converted image can be effectively improved in the conversion process from the SDR data to the HDR data, and the user experience is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image conversion method, and also to an image conversion device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Currently, most of the existing videos are SDR (Standard Dynamic Range) videos, that is, the maximum brightness is 100 nits, the color gamut is sRGB, and the dynamic range is far lower than the display indicators. In order to fully utilize the performance of the display and improve the user's sensory experience, SDR videos can be converted to HDR (High-Dynamic Range) videos. However, during this conversion process, there is a problem of abnormal exposure of the converted picture.

[0003] Therefore, how to effectively improve the quality of the converted image and improve the user experience during the conversion process from SDR data to HDR data is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] The purpose of this application is to provide an image conversion method, which can effectively improve the quality of the converted image and ensure user experience during the conversion process from SDR data to HDR data; another purpose of this application is to provide an image conversion device, an electronic device and a computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present application discloses an image conversion method, comprising:

[0006] Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image;

[0007] Calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image;

[0008] Calculating a fused feature correlation according to the internal feature correlation and the feature correlation, and determining a weight value of each three-dimensional lookup table according to the fused feature correlation;

[0009] Searching for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and performing weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain a target RGB color value;

[0010] A high dynamic range image corresponding to the standard dynamic range image is generated using the target RGB color values.

[0011] Optionally, acquiring a standard dynamic range image and a reference image of the standard dynamic range image includes:

[0012] Acquire a standard dynamic range video, perform frame processing on the standard dynamic range video, and obtain a standard dynamic range image sequence;

[0013] For any target standard dynamic range image in the standard dynamic range image sequence, selecting one or more other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images for the target standard dynamic range image;

[0014] Accordingly, after generating the high dynamic range image corresponding to the standard dynamic range image using the target RGB color value, the method further includes:

[0015] The high dynamic range video corresponding to the standard dynamic range video is generated by using the high dynamic range image corresponding to each of the standard dynamic range images in the standard dynamic range image sequence.

[0016] Optionally, performing frame processing on the standard dynamic range video to obtain a standard dynamic range image sequence includes:

[0017] Performing scene segmentation on the standard dynamic range video to obtain a standard dynamic range image sequence corresponding to each scene;

[0018] Accordingly, selecting one or more frames of other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images of the target standard dynamic range image includes:

[0019] In the standard dynamic range image sequence, a first preset number of frames of other standard dynamic range images before the target standard dynamic range image and a second preset number of frames of other standard dynamic range images after the target standard dynamic range image are selected as reference images for the target standard dynamic range image.

[0020] Optionally, before performing weighted calculation using the initial RGB color values ​​and the weight values ​​to obtain the target RGB color value, the method further includes:

[0021] For the RGB color value whose corresponding initial RGB color value cannot be found in the three-dimensional lookup table, an interpolation calculation is performed on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value.

[0022] Optionally, before calculating the internal feature correlation of the standard dynamic range image and calculating the feature correlation between the standard dynamic range image and the reference image, the method further includes:

[0023] The standard dynamic range image and the reference image are registered using an optical flow estimation algorithm to align associated feature points between the standard dynamic range image and the reference image.

[0024] Optionally, calculating the internal feature correlation of the standard dynamic range image includes:

[0025] The internal feature correlation of the standard dynamic range image is calculated using a self-attention mechanism.

[0026] Optionally, calculating the feature correlation between the standard dynamic range image and the reference image includes:

[0027] The feature correlation between the standard dynamic range image and the reference image is calculated using a cross-attention mechanism.

[0028] In a second aspect, the present application further discloses an image conversion device, comprising:

[0029] An acquisition module, used for acquiring a standard dynamic range image and a reference image of the standard dynamic range image, and determining RGB color values ​​of the standard dynamic range image;

[0030] A calculation module, used for calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image;

[0031] A determination module, configured to calculate a fused feature correlation according to the internal feature correlation and the feature correlation, and determine a weight value of each three-dimensional lookup table according to the fused feature correlation;

[0032] A search module, used to search for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and perform weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain a target RGB color value;

[0033] A generating module is used to generate a high dynamic range image corresponding to the standard dynamic range image by using the target RGB color value.

[0034] In a third aspect, the present application further discloses an electronic device, comprising:

[0035] Memory for storing computer programs;

[0036] A processor is used to implement the steps of any one of the image conversion methods described above when executing the computer program.

[0037] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the image conversion methods described above are implemented.

[0038] The present application provides an image conversion method, comprising: acquiring a standard dynamic range image and a reference image of the standard dynamic range image, and determining the RGB color value of the standard dynamic range image; calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image; calculating a fusion feature correlation according to the internal feature correlation and the feature correlation, and determining the weight value of each three-dimensional lookup table according to the fusion feature correlation; searching for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and performing weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain a target RGB color value; and generating a high dynamic range image corresponding to the standard dynamic range image using the target RGB color value.

[0039] By applying the technical solution provided in the present application, for a standard dynamic range image that needs to be converted into a high dynamic range image, the relevant feature information of its corresponding reference image can be obtained to realize the conversion of the standard dynamic range image to the high dynamic range image. First, multiple three-dimensional lookup tables are created in advance, each of which is used to record the mapping relationship between the RGB color value of the standard dynamic range image and the RGB color value of the high dynamic range image. Through each three-dimensional lookup table, a high dynamic range image RGB color value corresponding to the RGB color value of the standard dynamic range image can be obtained; secondly, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are integrated to determine the weight of each three-dimensional lookup table; finally, the final high dynamic range image RGB color value (i.e., the target RGB color value) is obtained by weight calculation based on each high dynamic range image RGB color value (i.e., the above-mentioned initial RGB color value) and each weight value, thereby obtaining the high dynamic range image corresponding to the standard dynamic range image, and completing the conversion of the standard dynamic range image to the high dynamic range image. It can be seen that this technical solution can restore the image quality of the current standard dynamic range image by learning the feature information of the relevant reference image, thereby effectively improving the quality of the converted image during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0040] An image conversion device, electronic device and computer-readable storage medium provided in this application also have the above-mentioned technical effects, which will not be described in detail in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0042] Figure 1 A flowchart of an image conversion method provided by this application;

[0043] Figure 2 A schematic diagram of a video conversion method provided by this application;

[0044] Figure 3 An image conversion flow chart of an image conversion model provided in this application;

[0045] Figure 4 A schematic diagram of the structure of an image conversion device provided by the present application;

[0046] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0047] The core of this application is to provide an image conversion method, which can effectively improve the quality of the converted image and ensure user experience during the conversion process from SDR data to HDR data; another core of this application is to provide an image conversion device, an electronic device and a computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0048] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] An embodiment of the present application provides an image conversion method.

[0050] Please refer to Figure 1 , Figure 1This is a flowchart of an image conversion method provided in the present application. The image conversion method may include the following S101 to S105.

[0051] S101: Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine RGB color values ​​of the standard dynamic range image.

[0052] This step aims to achieve the acquisition of standard dynamic range images (hereinafter referred to as SDR images) and their reference images, as well as the determination of the RGB color values ​​of the SDR images. The RGB color values ​​include the color values ​​of the R channel, the G channel, and the B channel. After the SDR image is obtained, it can be converted into RGB format, thereby determining the RGB color values ​​of the SDR image. The reference image is an image for the SDR image to learn feature information, which can be an image of the same scene as the SDR image, for example, it can be an adjacent frame in the SDR video where the SDR image is located.

[0053] S102: Calculate the internal feature correlation of the standard dynamic range image, and calculate the feature correlation between the standard dynamic range image and the reference image.

[0054] This step aims to achieve image feature learning, including the internal feature correlation of the SDR image itself and the feature correlation between the SDR image and its reference image. By learning its own feature information and the feature information of its reference image, the conversion of the SDR image to the HDR image is achieved to effectively ensure the image quality during the conversion process.

[0055] Among them, calculating the internal feature correlation of the standard dynamic range image may include: calculating the internal feature correlation of the standard dynamic range image using a self-attention mechanism; calculating the feature correlation between the standard dynamic range image and a reference image may include: calculating the feature correlation between the standard dynamic range image and the reference image using a cross-attention mechanism.

[0056] In one embodiment of the present application, before calculating the internal feature correlation of the standard dynamic range image and calculating the feature correlation between the standard dynamic range image and the reference image, it may also include: using an optical flow estimation algorithm to align the standard dynamic range image and the reference image to align the associated feature points between the standard dynamic range image and the reference image.

[0057] To further ensure the quality of the converted image, the SDR image and its reference image can be registered before feature learning to achieve spatial alignment of the associated feature points between the images, thereby effectively reducing the difficulty of the subsequent feature learning process. In the specific implementation process, the Lucas–Kanade optical flow estimation algorithm can be used.

[0058] S103: Calculate the fused feature correlation according to the internal feature correlation and the feature correlation, and determine the weight value of each three-dimensional lookup table according to the fused feature correlation.

[0059] This step aims to achieve the fusion processing of learning features to further determine the weight values ​​of each three-dimensional lookup table. Specifically, multiple three-dimensional lookup tables (3D LUT, 3D Look-Up-Table) can be created in advance, each of which is used to record the mapping relationship between a set of RGB color values ​​of SDR images and the RGB color values ​​of HDR images. Through each three-dimensional lookup table, an HDR image RGB color value corresponding to an SDR image RGB color value can be obtained, that is, "searching for the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table" as described in S104.

[0060] In a possible implementation, a plurality of different predefined masks (a binary function) can be used to perform nonlinear changes on each three-dimensional lookup table, so as to map the input of the three-dimensional lookup table to different brightness ranges, so as to enhance different brightness. For example, when using a 16×16×16 3D LUT, due to precision limitations, the mapping result of each 3D LUT saving the entire brightness space may not be good, so one 3D LUT can only save the conversion result of the dark area, and another 3D LUT can only save the conversion result of the bright area, and so on, different 3D LUTs save the conversion results of different areas.

[0061] Furthermore, both internal feature correlation and feature correlation can be vectors representing feature correlation. By inputting these vectors into the neural network layer for processing, a new set of vectors can be obtained, that is, a set of vectors used to represent the correlation of fused features. Each value in this new set of vectors corresponds to a weight value of a three-dimensional lookup table.

[0062] It is understandable that the purpose of using multiple three-dimensional lookup tables is to increase the generalization ability, because a three-dimensional lookup table can only represent one mapping relationship, and different types of SDR images often require multiple different types of three-dimensional lookup tables to achieve the best conversion effect (for example, SDR images of different styles and scenes generally require different three-dimensional lookup tables). Therefore, multiple three-dimensional lookup tables can be used. Furthermore, a set of weight values ​​determined based on the correlation of the fused features is used to combine each three-dimensional lookup table to obtain the final result (such as the weight calculation in S104).

[0063] S104: searching for the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, and performing weighted calculation using each initial RGB color value and each weight value to obtain a target RGB color value.

[0064] As mentioned above, each three-dimensional lookup table is used to record the mapping relationship between a set of RGB color values ​​of SDR images and the RGB color values ​​of HDR images. Based on each three-dimensional lookup table, an initial HDR image RGB color value corresponding to the SDR image RGB color value can be obtained, that is, the above-mentioned initial RGB color value. Combined with the weight value of the corresponding three-dimensional lookup table, the final HDR image RGB color value, that is, the above-mentioned target RGB color value, can be calculated.

[0065] In one embodiment of the present application, before performing weighted calculation using each initial RGB color value and each weight value to obtain the target RGB color value, the method may further include: for an RGB color value for which the corresponding initial RGB color value cannot be found in the three-dimensional lookup table, performing interpolation calculation on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value. In the specific implementation process, methods such as trilinear interpolation or tetrahedral interpolation may be used.

[0066] S105: Generate a high dynamic range image corresponding to the standard dynamic range image using the target RGB color values.

[0067] This step aims to achieve the final generation of the HDR image corresponding to the SDR image, that is, after obtaining the target RGB color value, the HDR image corresponding to the SDR image can be directly generated using the target RGB color value.

[0068] It can be seen that the image conversion method provided in the embodiment of the present application can obtain the relevant feature information of the corresponding reference image for the standard dynamic range image that needs to be converted into a high dynamic range image to realize the conversion of the standard dynamic range image to the high dynamic range image. First, multiple three-dimensional lookup tables are created in advance, each of which is used to record the mapping relationship between the RGB color value of the standard dynamic range image and the RGB color value of the high dynamic range image. Through each three-dimensional lookup table, a high dynamic range image RGB color value corresponding to the RGB color value of the standard dynamic range image can be obtained; secondly, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are integrated to determine the weight of each three-dimensional lookup table; finally, the final high dynamic range image RGB color value (that is, the above-mentioned target RGB color value) is obtained by weight calculation based on each high dynamic range image RGB color value (that is, the above-mentioned initial RGB color value) and each weight value, so as to obtain the high dynamic range image corresponding to the standard dynamic range image, and complete the conversion of the standard dynamic range image to the high dynamic range image. It can be seen that this technical solution can restore the image quality of the current standard dynamic range image by learning the feature information of the relevant reference image, thereby effectively improving the quality of the converted image during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0069] In one embodiment of the present application, acquiring a standard dynamic range image and a reference image of the standard dynamic range image may include: acquiring a standard dynamic range video, performing frame processing on the standard dynamic range video, and obtaining a standard dynamic range image sequence; for any target standard dynamic range image in the standard dynamic range image sequence, selecting one or more frames of other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images of the target standard dynamic range image;

[0070] Correspondingly, after generating a high dynamic range image corresponding to the standard dynamic range image using the target RGB color value, the method may further include: generating a high dynamic range video corresponding to the standard dynamic range video using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence.

[0071] The image conversion method provided in the embodiment of the present application is also applicable to the video conversion process, that is, converting SDR video into HDR video. In this process, it can be achieved by sequentially executing the above steps S101 to S105 for each video frame in the complete video. Obviously, for the reference image of any SDR image in the SDR video, it can also be selected and determined in the SDR video to effectively ensure the image effect after conversion.

[0072] The step of performing frame processing on the standard dynamic range video to obtain a standard dynamic range image sequence may include: performing scene segmentation on the standard dynamic range video to obtain a standard dynamic range image sequence corresponding to each scene;

[0073] Accordingly, selecting one or more frames of other standard dynamic range images other than the target standard dynamic range image in the standard dynamic range image sequence as reference images for the target standard dynamic range image may include: selecting, in the standard dynamic range image sequence, a first preset number of frames of other standard dynamic range images before the target standard dynamic range image, and a second preset number of frames of other standard dynamic range images after the target standard dynamic range image, as reference images for the target standard dynamic range image.

[0074] An embodiment of the present application provides a method for determining a reference image of an SDR image, namely, ensuring as much as possible that multiple SDR images of the same scene as the current SDR image are selected as reference images of the current SDR image. Because images in the same scene have high similarity, this helps to achieve more accurate feature learning, achieve more accurate image conversion, and ensure the quality of the converted image.

[0075] In one possible implementation, scene segmentation of SDR video can be achieved by comparing the brightness changes between any adjacent frames of SDR images. That is, when the brightness change between any adjacent frames of SDR images is higher than a certain threshold, it can be considered that the two adjacent frames of SDR images belong to different scenes, thereby achieving scene segmentation of SDR video.

[0076] In another possible implementation, the canny edge detection operator can be directly used to implement the scene segmentation of the SDR video. It can be understood that the canny operator is used to detect edge contours in an image, and the output result is a binary image (0 represents non-edge, 1 represents edge). Based on this, for any two adjacent SDR image frames, the interpolation of the binary images obtained after the two images are respectively processed by the canny operator can be calculated. If the interpolation is greater than a certain threshold, it can be considered that the edge contours are quite different, that is, it is considered that the two adjacent SDR image frames belong to different scenes, thereby achieving the scene segmentation of the SDR video.

[0077] Based on the above embodiments, please refer to Figure 2 , Figure 2 A schematic diagram of a video conversion method provided in this application, the implementation process is as follows:

[0078] 1. Video decapsulation and decoding: Decapsulate and decode according to the encapsulation format and encoding format of the SDR video file to obtain continuous RGB format video frame images (SDR images).

[0079] 2. Segment video frames according to scenes: SDR videos are segmented into several groups of continuous video frames according to scenes, each group of video frames represents a scene. The brightness change of adjacent video frames can be directly calculated, and segmentation is performed when the brightness change value exceeds a certain threshold; or the canny edge detection operator can be used, and segmentation is performed when the edge contour difference is too large.

[0080] 3. Model training: please refer to Figure 3 , Figure 3 This is an image conversion flow chart of an image conversion model provided in the present application. The above S101 to S105 can be implemented based on the image conversion model.

[0081] (1) Initialize all model parameters and all 3D LUTs.

[0082] (2) Process all frames of the current scene in sequence. For any current frame, sample several frames forward and backward as reference frames (reference images). Sampling can be random or fixed distance.

[0083] (3) Use the Lucas–Kanade optical flow estimation algorithm to align the current frame and the reference frame to align the associated feature points between the current frame and the reference frame.

[0084] (4) The current frame and all its reference frames are sent to the neural network respectively to learn the corresponding feature vectors.

[0085] (5) Learning the correlation of features in the feature vector of the current frame can be achieved using the self-attention mechanism.

[0086] (6) Learning the correlation between the feature vector of the current frame and the feature vectors of each reference frame can be achieved using the cross-attention mechanism.

[0087] (7) Through (5) and (6), a set of vectors representing feature correlations can be obtained. These vectors (fused feature correlations) are sent to the neural network layer for transformation to obtain a set of new vectors. Each value in this set of new vectors corresponds to a predefined 3D LUT weight. Each 3D LUT uses a predefined different mask for nonlinear transformation, which is used to map the input value of the 3D LUT to a specified brightness range, so that enhancement can be achieved for different brightness ranges.

[0088] (8) All 3D LUTs are queried respectively to obtain a set of initial HDR image RGB color values. If the queried value is not in the corresponding 3D LUT, interpolation processing can be performed, such as trilinear interpolation or tetrahedral interpolation.

[0089] (9) The final mapping value, i.e., the final HDR image RGB color value, is calculated by combining the 3D LUT output values ​​in (8) and the weight values ​​in (7).

[0090] (10) The difference between the mapped value and the true value is calculated through the loss function, and the parameters are updated through back propagation. The loss function can choose MSE or MAE until the final image conversion model is obtained.

[0091] 4. Model reasoning: Based on the image conversion model trained in step 3, the execution process is as follows (1) to (9) in step 3.

[0092] 5. Post-processing: Smooth the transition frames between scenes as needed to effectively keep the switching between scenes natural.

[0093] It can be seen that the image conversion method provided in the embodiment of the present application can obtain the relevant feature information of the corresponding reference image for the standard dynamic range image that needs to be converted into a high dynamic range image to realize the conversion of the standard dynamic range image to the high dynamic range image. First, multiple three-dimensional lookup tables are created in advance, each of which is used to record the mapping relationship between the RGB color value of the standard dynamic range image and the RGB color value of the high dynamic range image. Through each three-dimensional lookup table, a high dynamic range image RGB color value corresponding to the RGB color value of the standard dynamic range image can be obtained; secondly, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are integrated to determine the weight of each three-dimensional lookup table; finally, the final high dynamic range image RGB color value (that is, the above-mentioned target RGB color value) is obtained by weight calculation based on each high dynamic range image RGB color value (that is, the above-mentioned initial RGB color value) and each weight value, so as to obtain the high dynamic range image corresponding to the standard dynamic range image, and complete the conversion of the standard dynamic range image to the high dynamic range image. It can be seen that this technical solution can restore the image quality of the current standard dynamic range image by learning the feature information of the relevant reference image, thereby effectively improving the quality of the converted image during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0094] An embodiment of the present application provides an image conversion device.

[0095] Please refer to Figure 4 , Figure 4 This is a structural schematic diagram of an image conversion device provided by the present application, and the image conversion device may include:

[0096] An acquisition module 1 is used to acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color value of the standard dynamic range image;

[0097] A calculation module 2 is used to calculate the internal feature correlation of the standard dynamic range image and calculate the feature correlation between the standard dynamic range image and the reference image;

[0098] Determination module 3, used for calculating fusion feature correlation according to internal feature correlation and feature correlation, and determining weight values ​​of each three-dimensional lookup table according to the fusion feature correlation;

[0099] A search module 4 is used to search for an initial RGB color value corresponding to the RGB color value in each three-dimensional search table, and perform weighted calculation using each initial RGB color value and each weight value to obtain a target RGB color value;

[0100] The generating module 5 is used to generate a high dynamic range image corresponding to the standard dynamic range image by using the target RGB color value.

[0101] It can be seen that the image conversion device provided in the embodiment of the present application can obtain the relevant feature information of the corresponding reference image for the standard dynamic range image that needs to be converted into the high dynamic range image to realize the conversion of the standard dynamic range image to the high dynamic range image. First, multiple three-dimensional lookup tables are created in advance, each of which is used to record the mapping relationship between the RGB color value of the standard dynamic range image and the RGB color value of the high dynamic range image. Through each three-dimensional lookup table, a high dynamic range image RGB color value corresponding to the RGB color value of the standard dynamic range image can be obtained; secondly, the internal feature correlation of the standard dynamic range image and the feature correlation between the standard dynamic range image and its reference image are integrated to determine the weight of each three-dimensional lookup table; finally, the final high dynamic range image RGB color value (that is, the target RGB color value) is obtained by weight calculation based on each high dynamic range image RGB color value (that is, the above-mentioned initial RGB color value) and each weight value, so as to obtain the high dynamic range image corresponding to the standard dynamic range image, and complete the conversion of the standard dynamic range image to the high dynamic range image. It can be seen that this technical solution can restore the image quality of the current standard dynamic range image by learning the feature information of the relevant reference image, thereby effectively improving the quality of the converted image during the conversion process from SDR data to HDR data, and further ensuring the user experience.

[0102] In one embodiment of the present application, the acquisition module 1 may include:

[0103] A frame division unit is used to obtain a standard dynamic range video, perform frame division processing on the standard dynamic range video, and obtain a standard dynamic range image sequence;

[0104] A selection unit is used for selecting, for any target standard dynamic range image in the standard dynamic range image sequence, one or more frames of other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images of the target standard dynamic range image;

[0105] Correspondingly, the above-mentioned generation module 5 can also be used to generate a high dynamic range video corresponding to the standard dynamic range video using the high dynamic range images corresponding to each standard dynamic range image in the standard dynamic range image sequence after generating the high dynamic range image corresponding to the standard dynamic range image using the target RGB color value.

[0106] In one embodiment of the present application, the above-mentioned frame segmentation unit can be specifically used to perform scene segmentation on the standard dynamic range video to obtain a standard dynamic range image sequence corresponding to each scene;

[0107] Accordingly, the above-mentioned selection unit can be specifically used to select a first preset number of frames of other standard dynamic range images before the target standard dynamic range image and a second preset number of frames of other standard dynamic range images after the target standard dynamic range image within the standard dynamic range image sequence as reference images of the target standard dynamic range image.

[0108] In one embodiment of the present application, the above-mentioned search module 4 can also be used to perform interpolation calculation on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value for which the corresponding initial RGB color value cannot be found in the three-dimensional lookup table before performing weighted calculation using each initial RGB color value and each weight value to obtain the target RGB color value.

[0109] In one embodiment of the present application, the image conversion device may also include a registration module for registering the standard dynamic range image with the reference image using an optical flow estimation algorithm before calculating the internal feature correlation of the standard dynamic range image and calculating the feature correlation between the standard dynamic range image and the reference image, so as to align the associated feature points between the standard dynamic range image and the reference image.

[0110] In one embodiment of the present application, the above-mentioned calculation module 2 can be specifically used to calculate the internal feature correlation of the standard dynamic range image using the self-attention mechanism.

[0111] In one embodiment of the present application, the above-mentioned calculation module 2 can be specifically used to calculate the feature correlation between the standard dynamic range image and the reference image using a cross-attention mechanism.

[0112] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0113] An embodiment of the present application provides an electronic device.

[0114] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application, and the electronic device may include:

[0115] Memory for storing computer programs;

[0116] A processor, which is used to implement the steps of any one of the above-mentioned image conversion methods when executing a computer program.

[0117] like Figure 5 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0118] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0119] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the image conversion method.

[0120] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0121] Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine RGB color values ​​of the standard dynamic range image;

[0122] calculating internal feature correlation of the standard dynamic range image, and calculating feature correlation between the standard dynamic range image and a reference image;

[0123] The fusion feature correlation is calculated based on the internal feature correlation and the feature correlation, and the weight value of each three-dimensional lookup table is determined based on the fusion feature correlation;

[0124] Searching for the initial RGB color value corresponding to the RGB color value in each three-dimensional lookup table, and performing weighted calculation using each initial RGB color value and each weight value to obtain a target RGB color value;

[0125] Generates a high dynamic range image corresponding to the standard dynamic range image using the target RGB color values.

[0126] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0127] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0128] Of course, it should be noted that Figure 5 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0129] An embodiment of the present application provides a computer-readable storage medium.

[0130] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned image conversion methods can be implemented.

[0131] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0132] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0133] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0136] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. An image conversion method, characterized in that: include: Acquire a standard dynamic range image and a reference image of the standard dynamic range image, and determine the RGB color values ​​of the standard dynamic range image; Calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image; Calculating a fused feature correlation according to the internal feature correlation and the feature correlation, and determining a weight value of each three-dimensional lookup table according to the fused feature correlation; Searching for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and performing weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain a target RGB color value; A high dynamic range image corresponding to the standard dynamic range image is generated using the target RGB color values.

2. The image conversion method according to claim 1, characterized in that: Acquiring a standard dynamic range image and a reference image of the standard dynamic range image, comprising: Acquire a standard dynamic range video, perform frame processing on the standard dynamic range video, and obtain a standard dynamic range image sequence; For any target standard dynamic range image in the standard dynamic range image sequence, selecting one or more other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images for the target standard dynamic range image; Accordingly, after generating the high dynamic range image corresponding to the standard dynamic range image using the target RGB color value, the method further includes: The high dynamic range video corresponding to the standard dynamic range video is generated by using the high dynamic range image corresponding to each of the standard dynamic range images in the standard dynamic range image sequence.

3. The image conversion method according to claim 2, characterized in that: The standard dynamic range video is subjected to frame processing to obtain a standard dynamic range image sequence, including: Performing scene segmentation on the standard dynamic range video to obtain a standard dynamic range image sequence corresponding to each scene; Accordingly, selecting one or more frames of other standard dynamic range images except the target standard dynamic range image in the standard dynamic range image sequence as reference images of the target standard dynamic range image includes: In the standard dynamic range image sequence, a first preset number of frames of other standard dynamic range images before the target standard dynamic range image and a second preset number of frames of other standard dynamic range images after the target standard dynamic range image are selected as reference images for the target standard dynamic range image.

4. The image conversion method according to any one of claims 1 to 3, characterized in that: Before performing weighted calculation using the initial RGB color values ​​and the weight values ​​to obtain the target RGB color value, the method further includes: For the RGB color value whose corresponding initial RGB color value cannot be found in the three-dimensional lookup table, an interpolation calculation is performed on the RGB color value to obtain the initial RGB color value corresponding to the RGB color value.

5. The image conversion method according to claim 1, characterized in that: Before calculating the internal feature correlation of the standard dynamic range image and calculating the feature correlation between the standard dynamic range image and the reference image, the method further includes: The standard dynamic range image and the reference image are registered using an optical flow estimation algorithm to align associated feature points between the standard dynamic range image and the reference image.

6. The image conversion method according to claim 1, characterized in that: Calculating the internal feature correlation of the standard dynamic range image, including: The internal feature correlation of the standard dynamic range image is calculated using a self-attention mechanism.

7. The image conversion method according to claim 1, characterized in that: Calculating the feature correlation between the standard dynamic range image and the reference image, comprising: The feature correlation between the standard dynamic range image and the reference image is calculated using a cross-attention mechanism.

8. An image conversion device, characterized in that: include: An acquisition module, used for acquiring a standard dynamic range image and a reference image of the standard dynamic range image, and determining RGB color values ​​of the standard dynamic range image; A calculation module, used for calculating the internal feature correlation of the standard dynamic range image, and calculating the feature correlation between the standard dynamic range image and the reference image; A determination module, configured to calculate a fused feature correlation according to the internal feature correlation and the feature correlation, and determine a weight value of each three-dimensional lookup table according to the fused feature correlation; A search module, used to search for an initial RGB color value corresponding to the RGB color value in each of the three-dimensional lookup tables, and perform weighted calculation using each of the initial RGB color values ​​and each of the weight values ​​to obtain a target RGB color value; A generating module is used to generate a high dynamic range image corresponding to the standard dynamic range image by using the target RGB color value.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image conversion method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image conversion method according to any one of claims 1 to 7 are implemented.

Images