Training method, inverse tone mapping method and device of deep convolutional neural network

Abstract

The application provides a deep convolutional neural network training method, an inverse tone mapping method and device, computer equipment and a storage medium, and belongs to the fields of image technology and deep learning. In the training process, multiple adjacent sample image frames are used as training data, the inverse tone mapping capability of the sample image frames on a global and local basis can be learned, and the correlation between the image frames in time can also be learned, so that a deep convolutional neural network capable of better and more effective video quality improvement is obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a training method, inverse tone mapping method, apparatus, computer device and storage medium for a deep convolutional neural network. Background Technology

[0002] With the increasing popularity of video applications, people have higher and higher requirements for video quality, and the demand for high dynamic range (HDR) video is even stronger. Dynamic range is an evaluation indicator of video quality; the higher the dynamic range, the better the video quality. However, due to the limitations of device performance, a large number of videos are only standard dynamic range videos. Therefore, the technology that can convert standard dynamic range videos into HDR videos has broad prospects.

[0003] Currently, converting standard dynamic range (HDR) video into high dynamic range (HMR) video mainly involves using a convolutional neural network to extract color feature information and dynamic range information from the standard HDR video, performing color conversion and dynamic range expansion on the standard HDR video, and finally obtaining the converted HMR video.

[0004] However, the above method only processes a single frame of the video. It uses the color feature information and dynamic range information of the standard dynamic range video extracted from a single frame to perform color conversion and dynamic range expansion on the standard dynamic range video. This may result in deviations, leading to poor quality of the obtained high dynamic range video. Summary of the Invention

[0005] This application provides a training method for a deep convolutional neural network, an inverse tone mapping method, an apparatus, a computer device, and a storage medium, which can effectively improve video quality. The technical solution is as follows:

[0006] On the one hand, a training method for a deep convolutional neural network is provided, the method comprising:

[0007] Acquire multiple sample image frames of a sample video, the multiple sample image frames including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame;

[0008] The multiple sample image frames are input into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame. The color features of the multiple sample image frames are processed to obtain the temporal inverse tone mapping parameters of the multiple sample image frames. Based on the spatial local inverse tone mapping parameters, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple sample image frames, the first sample image frame is processed to output the generated image frame of the first sample image frame.

[0009] Based on the reference sample image frame of the first sample image frame and the generated image frame, an image loss is determined, and the model parameters of the deep convolutional neural network are adjusted based on the image loss.

[0010] Wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

[0011] On another front, an inverse tone mapping method is provided, the method comprising:

[0012] Acquire multiple image frames of the video to be processed, the multiple image frames including a first image frame and multiple second image frames adjacent to the first image frame;

[0013] The multiple image frames are input into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first image frame. The color features of the multiple image frames are processed to obtain the temporal inverse tone mapping parameters of the multiple image frames. Based on the spatial local inverse tone mapping parameters, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple image frames, the first image frame is processed to output the generated image frame of the first image frame.

[0014] In another aspect, a training apparatus for a deep convolutional neural network is provided, the apparatus comprising:

[0015] The sample acquisition unit is configured to acquire multiple sample image frames of the sample video, the multiple sample image frames including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame;

[0016] The mapping parameter acquisition unit is configured to input the plurality of sample image frames into a deep convolutional neural network, and the deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, and processes the color features of the plurality of sample image frames to obtain the temporal inverse tone mapping parameters of the plurality of sample image frames.

[0017] The image frame processing unit is configured to process the first sample image frame based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the plurality of sample image frames, and output the generated image frame of the first sample image frame.

[0018] The adjustment unit is configured to determine the image loss based on the reference sample image frame of the first sample image frame and the generated image frame, and adjust the model parameters of the deep convolutional neural network based on the image loss;

[0019] Wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

[0020] In some embodiments, the network processing unit includes:

[0021] The color feature extraction subunit is configured to extract features from the multiple sample image frames by multiple color extraction modules in the deep convolutional neural network to obtain the color features of each sample image frame.

[0022] The single-frame processing subunit is configured to have the spatial local inverse tone mapping module and the single-frame global inverse tone mapping module in the deep convolutional neural network process the color features of the first sample image frame respectively to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame.

[0023] The multi-frame processing subunit is configured to process the color features of the multiple sample image frames by the temporal inverse tone mapping module in the deep convolutional neural network to obtain the temporal inverse tone mapping parameters of the multiple sample image frames.

[0024] In some embodiments, the image frame processing unit includes:

[0025] The mapping feature acquisition subunit is configured to have the feature inverse tone mapping module in the deep convolutional neural network process the first sample image frame based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple sample image frames to obtain the mapping features of the first sample image frame.

[0026] The generation subunit is configured to output a generated image frame of the first sample image frame based on the mapping features of the first sample image frame.

[0027] In some embodiments, the mapping feature acquisition subunit is configured to map the input features of the mapping module based on the temporal inverse tone mapping parameters of the plurality of sample image frames by the mapping module in the deep convolutional neural network to obtain a first feature, and to process the first feature at least once based on the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame to obtain the mapping feature of the first sample image frame.

[0028] In some embodiments, the apparatus further includes:

[0029] The fusion unit is configured to fuse the multiple sample image frames by the fusion module in the deep convolutional neural network to obtain the fusion features of the multiple sample image frames;

[0030] The mapping feature acquisition subunit is configured to have the feature inverse tone mapping module in the deep convolutional neural network process the fusion features of the multiple sample image frames based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple sample image frames to obtain the mapping features of the first sample image frame.

[0031] In some embodiments, the fusion unit is configured to obtain translation parameters of the plurality of sample image frames based on the fusion module in the deep convolutional neural network, and fuse the plurality of sample image frames using the translation parameters to obtain fusion features of the plurality of sample image frames, wherein the translation parameters are used to align the plurality of sample image frames.

[0032] In some embodiments, the apparatus further includes:

[0033] The residual acquisition unit is configured to process the fusion features of the plurality of sample image frames by the residual module in the deep convolutional neural network to obtain the residuals of the plurality of sample image frames;

[0034] The generation subunit is configured to generate a generated image frame of the first sample image frame based on the fusion features of the plurality of sample image frames and the mapping features of the first sample image frame.

[0035] Furthermore, an inverse tone mapping apparatus is provided, the apparatus comprising:

[0036] The acquisition unit is configured to acquire multiple image frames of the video to be processed, the multiple image frames including a first image frame and multiple second image frames adjacent to the first image frame;

[0037] The image frame processing unit is configured to input the plurality of image frames into a deep convolutional neural network, whereby the deep convolutional neural network processes the color features of the first image frame to obtain spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame, processes the color features of the plurality of image frames to obtain temporal inverse tone mapping parameters of the plurality of image frames, and processes the first image frame based on the spatial local inverse tone mapping parameters, single-frame global inverse tone mapping parameters, and temporal inverse tone mapping parameters of the plurality of image frames to output a generated image frame of the first image frame.

[0038] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the operations performed by the deep convolutional neural network training method or the inverse tone mapping method in the embodiments of this application.

[0039] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform operations as performed by the deep convolutional neural network training method or the inverse tone mapping method in the embodiments of this application.

[0040] On the other hand, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the deep convolutional neural network training method or inverse tone mapping method provided in the various optional implementations described above. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of an implementation environment provided according to an embodiment of this application;

[0043] Figure 2 This is a flowchart of a training method for a deep convolutional neural network provided in an embodiment of this application;

[0044] Figure 3 This is a structural diagram of a deep convolutional neural network according to an embodiment of this application;

[0045] Figure 4 This is a flowchart of another training method for a deep convolutional neural network provided in the embodiments of this application;

[0046] Figure 5 This is a flowchart of an inverse tone mapping method provided in an embodiment of this application;

[0047] Figure 6 This is a flowchart of an inverse tone mapping method provided in an embodiment of this application;

[0048] Figure 7 This is an application diagram illustrating an inverse tone mapping method according to an embodiment of this application;

[0049] Figure 8 This is a block diagram of a training device for a deep convolutional neural network according to an embodiment of this application;

[0050] Figure 9 This is a block diagram of an inverse tone mapping device provided according to an embodiment of this application;

[0051] Figure 10 This is a comparison diagram of an image after applying the inverse tone mapping method according to an embodiment of this application, and an image in other states.

[0052] Figure 11 This is a comparison diagram of the effects of an inverse tone mapping method provided in the embodiments of this application and other methods on the restoration of overexposed images;

[0053] Figure 12 This is a structural block diagram of a computer device 1200 provided according to an embodiment of this application;

[0054] Figure 13This is a schematic diagram of the structure of a server according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0057] In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0058] In this application, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0059] It should be noted that all data involved in this application (including but not limited to data used for analysis, stored data, and displayed data) has been authorized by the user or by all parties in full, and the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the videos involved in this application were all obtained with full authorization.

[0060] The deep convolutional neural network training method and inverse tone mapping method provided in this application can be executed by a computer device. In some embodiments, the computer device is a terminal and a server. The following describes the implementation environment of the method provided in this application, using the computer device as a terminal and a server as an example. See [link to relevant documentation]. Figure 1 The implementation environment includes terminal 101 and server 102.

[0061] Terminal 101 and server 102 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this.

[0062] In some embodiments, terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited thereto. Terminal 101 has an application installed and running that supports video playback.

[0063] In some embodiments, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), big data, and artificial intelligence platforms. Server 102 is used to provide background services for applications that support video playback. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0064] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0065] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can be any network, including but not limited to LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including HTML (Hypertext Markup Language), XML (Extensible Markup Language), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as SSL (Secure Socket Layer), TLS (Transport Layer Security), VPN (Virtual Private Network), and IPsec (Internet Protocol Security) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0066] To achieve inverse tone mapping, this application provides a training method for a deep convolutional neural network to train a deep convolutional neural network with good inverse tone mapping capabilities. Before training, a training set is generated automatically. In some embodiments, the process of generating the training set includes the following steps.

[0067] Step 1: Acquire the raw video signal. For example, use a camera capable of capturing RAW video to acquire the raw RAW video signal.

[0068] Step 2 involves generating multiple video pairs based on the original video signal. Each video pair includes a sample video and a reference video for that sample video. The sample video is an SDR video, and the reference video is an HDR video; both are obtained by XRAW conversion of the original RAW video signal using video editing software. It can be seen that the sample video and the reference video are identical in content, but the HDR video has a greater dynamic range than the SDR video.

[0069] Step 3 involves filtering multiple video pairs. For example, a no-reference video quality assessment algorithm is used to score SDR videos, and video pairs containing SDR videos with scores below a quality threshold are filtered out to improve sample quality.

[0070] Step 4 involves scene segmentation of the sample videos in the remaining video pairs to determine multiple sets of image-video frame pairs belonging to various scenes. That is, each set of image-video frame pairs corresponds to one scene, and the reference video can be segmented according to the scene segmentation results of the sample videos. In some embodiments, the scene segmentation is based on calculating the structural similarity and / or histogram similarity of adjacent video frames.

[0071] Step 5: For each scene, extract several consecutive image frames from the corresponding image-video frame pair as a training set. In some embodiments, these several consecutive image frames are 10 consecutive image frames.

[0072] Figure 2 This is a flowchart illustrating a training method for a deep convolutional neural network provided in an embodiment of this application, such as... Figure 2 As shown, in this embodiment of the application, the execution subject is a server as an example for explanation.

[0073] 201. Obtain multiple sample image frames of the sample video, including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame.

[0074] Here, "adjacent" refers to image frames being adjacent in time. The multiple second sample image frames refer to adjacent image frames located to the left and right of the first image frame, and the number of these second sample image frames is an integer greater than or equal to 2. For example, the multiple sample image frames of this video may include 7 image frames, where one image frame is the first image frame, and the remaining 6 image frames include 3 adjacent sample image frames located to the left of the first image frame and 3 adjacent sample image frames located to the right of the first image frame.

[0075] Of course, the training set can be sorted according to the temporal order of the image frames. Accordingly, multiple sample image frames can be directly obtained from the training set in sequence each time.

[0076] 202. Input multiple sample image frames into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame. Process the color features of multiple sample image frames to obtain the temporal inverse tone mapping parameters of multiple sample image frames. Based on the spatial local inverse tone mapping parameters, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of multiple sample image frames, process the first sample image frame and output the generated image frame of the first sample image frame.

[0077] 203. Based on the reference sample image frame and the generated image frame of the first sample image frame, determine the image loss, and adjust the model parameters of the deep convolutional neural network based on the image loss. The reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

[0078] The reference sample image frame is the image frame in the HDR video that corresponds to the first sample image frame, such as an image frame with the same timestamp. The reference sample image frame can be understood as a version with higher image quality than the first sample image frame. Therefore, based on the difference between the reference sample image frame and the generated image frame, the training effect can be quantified, and based on the image loss of this iteration, it can be determined whether to continue iterative training.

[0079] In the training process described above, multiple adjacent sample image frames were used as training data. This enabled the learning of the inverse tone mapping capabilities of the sample image frames in both global and local contexts. Furthermore, it also enabled the learning of the temporal correlation between image frames, thus resulting in a deep convolutional neural network that can better and more effectively improve video quality.

[0080] Figure 3 This is a structural diagram of a deep convolutional neural network provided according to an embodiment of this application, such as... Figure 3As shown, this deep convolutional neural network includes multiple color extraction modules 3011-301N, a spatial local inverse tone mapping module 302, a single-frame global inverse tone mapping module 303, a temporal inverse tone mapping module 304, a fusion module 305, a feature inverse tone mapping module 306, and a residual module 307. It should be noted that the training process for the above network can include multiple iterations to achieve the training objective. Below, in conjunction with... Figure 3 Network structure and Figure 4 The flowchart shown illustrates one iteration of a training method for a deep convolutional neural network provided in this application embodiment.

[0081] 401. The server acquires multiple sample image frames of the sample video, including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame.

[0082] Step 401 is the same as step 201.

[0083] 402. The server extracts features from multiple sample image frames using multiple color extraction modules 3011 to 301N in a deep convolutional neural network to obtain the color features of each sample image frame.

[0084] In this embodiment of the application, the deep convolutional neural network has N color extraction modules 3011 to 301N, where N is an integer greater than 1. The above step 202 includes: inputting the multiple sample image frames into the deep convolutional neural network, where the first color extraction module 3011 in the deep convolutional neural network extracts features from the multiple sample image frames to obtain the first color features of the multiple sample image frames; inputting the first color features of the multiple sample image frames into the second color extraction module 3012 in the deep convolutional neural network, where the second color extraction module 3012 in the deep convolutional neural network extracts features from the first color features of the multiple sample image frames to obtain the second color features of the multiple sample image frames; and so on, until the (N-1)th color features of the multiple sample image frames are input into the Nth color extraction module 301N in the deep convolutional neural network, where the Nth color extraction module 301N in the deep convolutional neural network extracts features from the (N-1)th color features of the multiple sample image frames to obtain the color features of the multiple sample image frames.

[0085] Each color extraction module 3011 to 301N includes a convolutional layer, a pooling layer, an activation layer, and a normalization layer. Accordingly, in step 202, each color extraction module 3011 to 301N sequentially performs a convolution operation, a pooling operation, a LeaklyReLU activation operation, and an InstanceNorm normalization operation. The parameters for the convolution and pooling operations in each color extraction module 3011 to 301N may not be the same.

[0086] For example, taking N=4 as an example, the parameters of the convolutional layer of the first color extraction module 3011 are (3, 16, 1, 1, 0), and the parameters of the pooling layer are (2, 2, 0); the parameters of the convolutional layer of the second color extraction module 3012 are (16, 32, 1, 1, 0), and the parameters of the pooling layer are (2, 2, 0); the parameters of the convolutional layer of the third color extraction module 3013 are (32, 64, 1, 1, 0), and the parameters of the pooling layer are (2, 2, 0); the parameters of the convolutional layer of the fourth color extraction module 3014 are (64, 128, 1, 1, 0), and the parameters of the pooling layer are (2, 2, 0).

[0087] The parameters of the convolutional layer are (input channels, output channels, kernel size, stride, padding), and the parameters of the pooling layer are (window size, stride, padding).

[0088] 403. The server processes the color features of the first sample image frame through the spatial local inverse tone mapping module 302 and the single-frame global inverse tone mapping module 303 in the deep convolutional neural network to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame.

[0089] The spatial local inverse tone mapping module 302 includes an upsampling layer, a convolutional layer, an average pooling layer, and an activation layer. Accordingly, in step 403, upsampling, convolution, average pooling, and LeaklyReLU activation operations are sequentially performed on the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters of the first sample image frame. These spatial local inverse tone mapping parameters are used to represent the features of pixels in each patch of the first sample image frame.

[0090] The single-frame global inverse tone mapping module 303 includes a convolutional layer, an average pooling layer, and a convolutional layer. Accordingly, in step 403, the color features of the first sample image frame are sequentially subjected to convolution, average pooling, and convolution operations to obtain the single-frame global inverse tone mapping parameters of the first sample image frame. The single-frame global inverse tone mapping parameters are used for the overall features of the pixels in the first sample image frame.

[0091] In some embodiments, the parameters of the convolutional layer in step 403 are all (128, 128, 1, 1, 0), and the parameters of the average pooling layer are all (1, 1, 0).

[0092] Since the spatial local inverse tone mapping module and the single-frame global inverse tone mapping module in the deep convolutional neural network are both trained based on sample images, the two modules have the ability to predict the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the image based on the input.

[0093] 404. The server processes the color features of multiple sample image frames through the temporal inverse tone mapping module 304 in the deep convolutional neural network to obtain the temporal inverse tone mapping parameters of multiple sample image frames.

[0094] The temporal inverse tone mapping module 304 includes a convolutional layer, an average pooling layer, and a convolutional layer. Accordingly, in step 404, the color features of the multiple sample image frames are sequentially subjected to convolutional, average pooling, and convolutional operations to obtain the temporal inverse tone mapping parameters.

[0095] In some embodiments, the convolutional layers in step 404 are all (128, 128, 1, 1, 0), and the parameters of the average pooling layer are (1, 1, 0).

[0096] Since the temporal inverse tone mapping module in a deep convolutional neural network is trained based on sample images, the aforementioned temporal inverse tone mapping module has the ability to predict the temporal inverse tone mapping parameters of an image based on the input.

[0097] It should be noted that the steps of calculating the spatial local inverse tone mapping parameters, the single-frame global inverse tone mapping parameters in step 403, and the steps of calculating the temporal inverse tone mapping parameters in step 404 are parallel and do not have a sequential order.

[0098] 405. The server fuses multiple sample image frames through the fusion module 305 in the deep convolutional neural network to obtain the fusion features of the multiple sample image frames.

[0099] The fusion module in the deep convolutional neural network obtains translation parameters of multiple sample image frames based on multiple sample image frames, and fuses the multiple sample image frames using the translation parameters to obtain the fusion features of the multiple sample image frames. The translation parameters are used to align the multiple sample image frames.

[0100] The fusion module 305 includes a first convolutional layer, a second convolutional layer, a first activation layer, a third convolutional layer, a second activation layer, a transposed convolutional layer, a third activation layer, and a fourth convolutional layer. Accordingly, in step 405, the first convolutional operation, the second convolutional operation, the first ReLU activation operation, the third convolutional operation, the second ReLU activation operation, the transposed convolutional operation, the third ReLU activation operation, and the fourth convolutional operation are sequentially performed on the multiple sample image frames to obtain translation parameters. These translation parameters are used to align the multiple sample image frames and can indicate the displacement change of a certain pixel in different image frames. The multiple sample image frames are fused using these translation parameters to obtain the fusion features of the multiple sample image frames.

[0101] In some embodiments, the parameters of the first convolutional layer are (21, 32, 3, 2, 1), the parameters of the second convolutional layer are (32, 32, 17, 1, 8), the parameters of the third convolutional layer are (32, 32, 17, 1, 8), the parameters of the transposed convolutional layer are (32, 32, 3, 2, 1), and the parameters of the fourth convolutional layer are (32, 64, 3, 2, 1).

[0102] It should be noted that the steps of extracting color features in step 402 and extracting fusion features in step 405 can be parallel and do not have a specific order.

[0103] By fusing multiple adjacent image frames, the pixel changes between each image frame can be referenced, resulting in a smoother and more natural transition in the image effect.

[0104] 406. The server processes the fusion features of multiple sample image frames based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of multiple sample image frames through the feature inverse tone mapping module 306 in the deep convolutional neural network, and obtains the mapping features of the first sample image frame.

[0105] In some embodiments, the mapping module in the deep convolutional neural network maps the input features of the mapping module based on the temporal inverse tone mapping parameters of multiple sample image frames to obtain a first feature. Based on the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, the first feature is processed at least once to obtain the mapping features of the first sample image frame.

[0106] The feature inverse tone mapping module 306 includes a first convolutional layer, a temporal feature inverse tone mapping layer, a first activation layer, a second convolutional layer, a first single-frame global feature inverse tone mapping layer, a first spatial local feature inverse tone mapping layer, a second activation layer, a third convolutional layer, a second single-frame global feature inverse tone mapping layer, a second spatial local feature inverse tone mapping layer, a third activation layer, a fourth convolutional layer, a third single-frame global feature inverse tone mapping layer, and a third spatial local feature inverse tone mapping layer. Correspondingly, in step 406, the fused feature is sequentially subjected to a first convolutional operation to obtain the first mapped feature. The temporal inverse tone mapping parameters output by the temporal inverse tone mapping module 304 are multiplied by the first mapping feature to perform a temporal feature inverse tone mapping operation, resulting in a second mapping feature. A first ReLU activation operation and a second convolution operation are then performed on the second mapping feature to obtain a third mapping feature. The single-frame global inverse tone mapping parameters output by the single-frame global inverse tone mapping module 303 are multiplied by the third mapping feature to perform a first single-frame global feature inverse tone mapping operation, resulting in a fourth mapping feature. The spatial local inverse tone mapping parameters output by the spatial local inverse tone mapping module 302 are multiplied by the fourth... The mapping features are multiplied by a dot to perform the first spatial local feature inverse tone mapping operation, resulting in the fifth mapping feature. The fifth mapping feature is then subjected to a second ReLU activation operation and a third convolution operation to obtain the sixth mapping feature. The single-frame global inverse tone mapping parameters output by the single-frame global inverse tone mapping module 303 are multiplied by the sixth mapping feature to perform the second single-frame global feature inverse tone mapping operation, resulting in the seventh mapping feature. The spatial local inverse tone mapping parameters output by the spatial local inverse tone mapping module 302 are then multiplied by the seventh mapping feature to perform the second spatial local feature inverse tone mapping operation, resulting in the eighth mapping feature. The eighth mapping feature is then subjected to a third ReLU activation operation and a fourth convolution operation to obtain the ninth mapping feature. The single-frame global inverse tone mapping parameters output by the single-frame global inverse tone mapping module 303 are then multiplied by the ninth mapping feature to perform the third single-frame global feature inverse tone mapping operation, resulting in the tenth mapping feature. The spatial local inverse tone mapping parameters output by the spatial local inverse tone mapping module 302 are then multiplied by the tenth mapping feature to perform the third spatial local feature inverse tone mapping operation, thus obtaining the mapping features of the first sample image frame.

[0107] The parameters of the first convolutional layer are (64, 32, 1, 1, 0), and the parameters of the second, third, and fourth convolutional layers are all (32, 32, 1, 1, 0).

[0108] In the above process, the fused features are first mapped based on the above temporal feature inverse tone mapping operation to obtain the second mapped features. Then, the ReLU activation operation, convolution operation, single-frame global feature inverse tone mapping operation and spatial local feature inverse tone mapping operation are executed three times to obtain a more accurate mapping feature of the first sample image frame.

[0109] It should be noted that the aforementioned temporal feature inverse tone mapping parameters, single-frame global feature inverse tone mapping parameters, and spatial local feature inverse tone mapping parameters can be processed by fully connected layers before being input into the feature inverse tone mapping module 306 for further processing.

[0110] 407. The server processes the fusion features of multiple sample image frames through the residual module 307 in the deep convolutional neural network to obtain the residuals of multiple sample image frames.

[0111] In this embodiment of the application, the deep convolutional neural network has multiple residual modules 3071 to 307M, where M is an integer greater than 1. The above step 407 includes: inputting the fusion features of the multiple sample image frames into the first residual module in the deep convolutional neural network, processing the fusion features by the first residual module in the deep convolutional neural network to obtain a first residual, inputting the first residual into the second residual module in the deep convolutional neural network, processing the first residual by the second residual module in the deep convolutional neural network to obtain a second residual, and so on, until the (M-1)th residual is input into the Mth residual module in the deep convolutional neural network, and processing the (M-1)th residual by the Mth residual module in the deep convolutional neural network to obtain the residuals of the multiple sample image frames.

[0112] Each residual module 3071 to 307M includes a convolutional layer, a first activation layer, a depthwise separable convolutional layer, a second activation layer, and a residual connection layer. Accordingly, in step 407, each residual module sequentially performs a convolution operation, a first ReLU activation operation, a depthwise separable convolution operation, a second ReLU activation operation, and a residual connection operation. The parameters for the convolution operation are the same for each residual module.

[0113] For example, taking M=4 as an example, the parameters of the convolutional layer of the first residual module, the second residual module, the third residual module, and the fourth residual module are the same, which are (32, 32, 3, 1, 1).

[0114] Before the residual modules 3071 to 307M in this deep convolutional neural network process the fused features of the first image, the fused features are input into the fused feature processing module for preliminary processing. The fused feature processing module includes a first convolutional layer, an activation layer, and a second convolutional layer. Accordingly, the fused feature processing module sequentially performs the first convolution operation, the LeaklyReLU activation operation, and the second convolution operation. The parameters of the first convolutional layer are (64, 32, 3, 1, 1), and the parameters of the second convolutional layer are (32, 32, 3, 1, 1).

[0115] 408. Based on the mapping features of the first sample image frame and the residuals of the multiple sample image frames, the server outputs the generated image frame of the first sample image frame.

[0116] The generated image is the sum of the residuals of the multiple sample image frames and the mapping features of the first sample image.

[0117] In some embodiments, the steps of multi-frame fusion and residual extraction described above may be omitted. Instead, the mapping features of the first sample image frame may be directly output as the generated image frame. Alternatively, the multi-frame fusion described above may be omitted. Instead, the residual may be extracted based solely on the first sample image frame, and then the generated image frame may be generated based on the residual and mapping features.

[0118] 409. The server determines the image loss based on the reference sample image frame and the generated image frame of the first sample image frame, and adjusts the model parameters of the deep convolutional neural network based on the image loss; wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

[0119] In some embodiments, step 409 includes: determining an image loss MSE_loss value based on the reference sample image of the first sample image frame and the generated image frame; iteratively updating the model parameters of the deep convolutional neural network using the Adam optimizer according to the image loss, until the mean of the image loss MSE_loss value converges, then terminating the iterative update; and outputting the obtained deep convolutional neural network as a trained deep convolutional neural network. Here, the image loss is used to represent the difference between the reference sample image of the first sample image frame and the generated image frame.

[0120] In this embodiment of the application, by building a deep convolutional neural network, training the deep convolutional neural network, and adjusting the model parameters of the deep convolutional neural network, preparations are made for the smooth execution of the inverse tone mapping method, ensuring the high quality of the output high dynamic range video.

[0121] Figure 5This is a flowchart of an inverse tone mapping method provided in an embodiment of this application. See also... Figure 5 The following explanation will take the method executed by the server as an example.

[0122] 501. Obtain multiple image frames of the video to be processed, including a first image frame and multiple second image frames adjacent to the first image frame.

[0123] In some embodiments, the video to be processed is a standard dynamic range video. In some embodiments, the video to be processed may be uploaded by a user or specified by a user on the platform; this application does not restrict the method of obtaining the video to be processed.

[0124] 502. Input multiple image frames into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first image frame. Process the color features of multiple image frames to obtain the temporal inverse tone mapping parameters of multiple image frames. Based on the spatial local inverse tone mapping parameters, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of multiple image frames, process the first image frame and output the generated image frame of the first image frame.

[0125] In this embodiment, color feature information and dynamic range information of the video to be processed are extracted based on multiple adjacent image frames. Based on the color feature information and dynamic range information extracted from multiple adjacent image frames, the video can be subjected to more accurate color conversion and dynamic range expansion, thereby improving the quality of the obtained high dynamic range video.

[0126] The above Figure 5 This is a brief explanation of the inverse tone mapping method. Figure 6 This is a flowchart of an inverse tone mapping method provided in an embodiment of this application. See also... Figure 6 The following explanation will take the method executed by the server as an example.

[0127] 601. The server acquires multiple image frames of the video, including a first image frame and multiple second image frames adjacent to the first image frame.

[0128] The first image frame refers to the image frame to be processed in the video. The multiple second image frames refer to the adjacent image frames of the image frame to be processed. That is, each time the image frame to be processed is acquired, the adjacent image frames located to the left and right of the image frame are acquired. The number of the second image frames is an integer greater than or equal to 1. For example, the multiple images in the video may include 7 image frames, of which one image frame is the image frame to be processed in the video, and the remaining 6 image frames include the 3 adjacent image frames located to the left of the image frame to be processed and the 3 adjacent image frames located to the right of the image frame to be processed.

[0129] 602. The server extracts features from multiple image frames using multiple color extraction modules 3011 to 301N in a deep convolutional neural network to obtain the color features of each image frame.

[0130] 603. The server processes the color features of the first image frame through the spatial local inverse tone mapping module 302 and the single-frame global inverse tone mapping module 303 in the deep convolutional neural network to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first image frame.

[0131] 604. The server processes the color features of multiple image frames through the temporal inverse tone mapping module 304 in the deep convolutional neural network to obtain the temporal inverse tone mapping parameters of multiple image frames.

[0132] 605. The server fuses multiple image frames using the fusion module 305 in the deep convolutional neural network to obtain the fused features of the multiple image frames.

[0133] 606. The server uses the feature inverse tone mapping module 306 in the deep convolutional neural network to process the fusion features of multiple image frames based on the spatial local inverse tone mapping parameters of the first image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of multiple image frames to obtain the mapping features of the first image frame.

[0134] 607. The server processes the fusion features of multiple image frames through the residual module 307 in the deep convolutional neural network to obtain the residuals of multiple image frames.

[0135] 608. The server outputs the generated image frame of the first image frame based on the mapping features of the first image frame and the residuals of the multiple image frames.

[0136] Steps 601 to 608 are the same as steps 401 to 408 above.

[0137] This approach combines deep learning with video inverse tone mapping technology. A trained convolutional neural network model extracts multi-frame video information to generate a high dynamic range (HDR) video with a wider color gamut than the original video. By extracting multi-frame information, the HDR video is reconstructed more accurately. First, temporal inverse tone mapping parameters are estimated using multi-frame video information in the temporal dimension, ensuring continuity between video frames and preventing flicker. Intra-frame global and local feature inverse tone mapping further enhances local colors in a more refined manner. Furthermore, the fusion features of multiple frames are utilized to further improve the visual quality of the HDR video.

[0138] Figure 7 This is an application diagram of an inverse tone mapping method provided in an embodiment of this application.

[0139] In some embodiments, see Figure 7 In Figure (a), the computer program corresponding to the inverse tone mapping method can be stored on a server. The user uploads the video to be processed to the server or specifies the video to be processed on the platform. After the server receives or recognizes the video to be processed, it uses the inverse tone mapping method provided in this application to convert the video to be processed into a high dynamic range video. The server encodes the obtained high dynamic range video and distributes the encoded high dynamic range video to the terminal. The terminal downloads the encoded high dynamic range video, decodes the high dynamic range video, and plays it on the terminal.

[0140] In some embodiments, see Figure 7 In Figure (b), the computer program corresponding to the inverse tone mapping method can be stored on a terminal, such as a player. The player on the terminal decodes the video to be processed and then uses the inverse tone mapping method provided in this application to convert the video to be processed into a high dynamic range video for playback.

[0141] Figure 8 This is a block diagram of a training apparatus for a deep convolutional neural network according to an embodiment of this application. The apparatus is used to execute the steps of the training method for the deep convolutional neural network described above, see [link to relevant documentation]. Figure 8 The device includes:

[0142] The sample acquisition unit 801 is configured to acquire multiple sample image frames of a sample video, the multiple sample image frames including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame;

[0143] The mapping parameter acquisition unit 802 is configured to input the plurality of sample image frames into a deep convolutional neural network, and the deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, and processes the color features of the plurality of sample image frames to obtain the temporal inverse tone mapping parameters of the plurality of sample image frames.

[0144] The image frame processing unit 803 is configured to process the first sample image frame based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the plurality of sample image frames, and output the generated image frame of the first sample image frame.

[0145] The adjustment unit 804 is configured to determine image loss based on the reference sample image frame of the first sample image frame and the generated image frame, and adjust the model parameters of the deep convolutional neural network based on the image loss;

[0146] Wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

[0147] In some embodiments, the network processing unit includes:

[0148] The color feature extraction subunit is configured to extract features from the multiple sample image frames by multiple color extraction modules in the deep convolutional neural network to obtain the color features of each sample image frame.

[0149] The single-frame processing subunit is configured to have the spatial local inverse tone mapping module and the single-frame global inverse tone mapping module in the deep convolutional neural network process the color features of the first sample image frame respectively to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame.

[0150] The multi-frame processing subunit is configured to process the color features of the multiple sample image frames by the temporal inverse tone mapping module in the deep convolutional neural network to obtain the temporal inverse tone mapping parameters of the multiple sample image frames.

[0151] In some embodiments, the image frame processing unit includes:

[0152] The mapping feature acquisition subunit is configured to have the feature inverse tone mapping module in the deep convolutional neural network process the first sample image frame based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple sample image frames to obtain the mapping features of the first sample image frame.

[0153] The generation subunit is configured to output a generated image frame of the first sample image frame based on the mapping features of the first sample image frame.

[0154] In some embodiments, the mapping feature acquisition subunit is configured to map the input features of the mapping module based on the temporal inverse tone mapping parameters of the plurality of sample image frames by the mapping module in the deep convolutional neural network to obtain a first feature, and to process the first feature at least once based on the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame to obtain the mapping feature of the first sample image frame.

[0155] In some embodiments, the apparatus further includes:

[0156] The fusion unit is configured to fuse the multiple sample image frames by the fusion module in the deep convolutional neural network to obtain the fusion features of the multiple sample image frames;

[0157] The mapping feature acquisition subunit is configured to have the feature inverse tone mapping module in the deep convolutional neural network process the fusion features of the multiple sample image frames based on the spatial local inverse tone mapping parameters of the first sample image frame, the single-frame global inverse tone mapping parameters, and the temporal inverse tone mapping parameters of the multiple sample image frames to obtain the mapping features of the first sample image frame.

[0158] In some embodiments, the fusion unit is configured to obtain translation parameters of the plurality of sample image frames based on the fusion module in the deep convolutional neural network, and fuse the plurality of sample image frames using the translation parameters to obtain fusion features of the plurality of sample image frames, wherein the translation parameters are used to align the plurality of sample image frames.

[0159] In some embodiments, the apparatus further includes:

[0160] The residual acquisition unit is configured to process the fusion features of the plurality of sample image frames by the residual module in the deep convolutional neural network to obtain the residuals of the plurality of sample image frames;

[0161] The generation subunit is configured to generate a generated image frame of the first sample image frame based on the fusion features of the plurality of sample image frames and the mapping features of the first sample image frame.

[0162] It should be noted that the deep convolutional neural network training device provided in the above embodiments is only illustrated by the division of the above functional modules during deep convolutional neural network training. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the deep convolutional neural network training device and the deep convolutional neural network training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0163] Figure 9 This is a block diagram of an inverse tone mapping apparatus according to an embodiment of this application. The apparatus is used to perform the steps of the above-described inverse tone mapping method, see [link to relevant documentation]. Figure 9 The device includes:

[0164] The acquisition unit 901 is configured to acquire multiple image frames of the video to be processed, the multiple image frames including a first image frame and multiple second image frames adjacent to the first image frame;

[0165] The image frame processing unit 902 is configured to input the plurality of image frames into a deep convolutional neural network, whereby the deep convolutional neural network processes the color features of the first image frame to obtain spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame, processes the color features of the plurality of image frames to obtain temporal inverse tone mapping parameters of the plurality of image frames, and processes the first image frame based on the spatial local inverse tone mapping parameters, single-frame global inverse tone mapping parameters, and temporal inverse tone mapping parameters of the plurality of image frames to output a generated image frame of the first image frame.

[0166] It should be noted that the inverse tone mapping device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the inverse tone mapping device and the inverse tone mapping method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0167] In some embodiments, a simulation experiment was conducted on the inverse tone mapping method. The hardware platform for the simulation experiment was: an Intel i7-7280 CPU with a main frequency of 3.60GHz, 16GB of memory, and an Nvidia GeForce 2080Ti GPU with 11GB of video memory. The software platform for the simulation experiment was: Ubuntu 16.04 operating system, Python 3.6, and PyTorch 1.0.

[0168] For example, a video dataset is obtained by processing videos captured by a Canon camera. This video dataset includes 50 sample videos and high dynamic range video pairs containing rich natural texture information. The 50 sample videos and high dynamic range video pairs are segmented into 54,302 video segments. Based on these 54,302 video segments, multiple sample images and reference sample images of these multiple sample images are obtained as the image training set for training a deep convolutional neural network.

[0169] The inverse tone mapping method and other methods provided in the embodiments of this application were used to test a video to be processed and a high dynamic range video pair, which were converted from the video captured in the application. Four evaluation indicators were used to evaluate the inverse tone mapping method and other methods provided in the embodiments of this application. The test results are shown in Table 1.

[0170] Table 1

[0171]

[0172] Among them, Params, PSNR (Peak Signal-to-Noise Ratio), SR-SIM, and ΔEIT P are all evaluation metrics for the above methods. The smaller the values ​​of Params and ΔEIT P, the better the method performance; the larger the values ​​of PSNR and SR-SIM, the better the method performance.

[0173] The inverse tone mapping method provided in this application embodiment has significant improvements in multiple evaluation metrics compared to other methods in the table.

[0174] Figure 10 This is a comparison image of the image effect after applying the inverse tone mapping method according to an embodiment of this application. The images to be compared include: SDR video image, video image after applying HDRTVNET, video image obtained by applying the inverse tone mapping method of this application, and real HDR video image.

[0175] As shown in the figure Figure 10In the left and right images, the road surface texture in the SDR video is blurry. However, in the video processed by HDRTVNET, the real HDR video, and the video obtained by applying the inverse tone mapping method of this application, it can be seen from the details that the originally blurry road surface texture has been largely restored. In other words, the visual quality of the image has been greatly improved.

[0176] Furthermore, it can be seen that the video image obtained by applying the inverse tone mapping method of this application has more vibrant colors, higher saturation, and is closer to the image of a real high dynamic range video. That is, the inverse tone mapping method provided in the embodiments of this application has advantages in both color restoration and brightness restoration.

[0177] Figure 11 This is another image effect comparison diagram provided according to an embodiment of this application, wherein the images to be compared include video images before and after applying the inverse tone mapping method of this application. Figure 11 The first image from the left in the first row is the original video frame, which is an overexposed image. The second image from the left in the first row is a magnified view of details from the original video frame. The first image from the right in the first row is the processed video frame, and the second image from the right in the first row is a magnified view of details from the processed video frame. It can be seen that the inverse tone mapping method provided in this application can remove artifacts from overexposed images, showing a significant advantage in the restoration of overexposed images. Furthermore, Figure 11 The second row is another example of a video, with the video frames arranged in the same way as the first row. It also demonstrates the significant advantages of the inverse tone mapping method provided in this application embodiment in the restoration of overexposed images.

[0178] Through the above simulation experiments, it can be seen that the inverse tone mapping method provided in this application uses a deep convolutional neural network to process the video to be processed, accurately converting the video into a high dynamic range video, making the colors of the video more vivid, the saturation higher, and closer to the real world, thereby improving the quality of the high dynamic range video.

[0179] In the embodiments of this application, the computer device can be configured as a terminal or a server. When the computer device is configured as a terminal, the terminal can act as the execution subject to implement the technical solutions provided in the embodiments of this application. When the computer device is configured as a server, the server can act as the execution subject to implement the technical solutions provided in the embodiments of this application. Alternatively, the technical solutions provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this.

[0180] Figure 12This is a structural block diagram of a computer device 1200 according to an embodiment of this application. The computer device 1200 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 1200 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0181] Typically, computer device 1200 includes a processor 1201 and a memory 1202.

[0182] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0183] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one computer program, which is executed by the processor 1201 to implement the deep convolutional neural network training method or inverse tone mapping method provided in the method embodiments of this application.

[0184] In some embodiments, the computer device 1200 may also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1204, a display screen 1205, a camera assembly 1209, an audio circuit 1207, and a power supply 1208.

[0185] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0186] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0187] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of computer device 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of computer device 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of computer device 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0188] The camera assembly 1209 is used to acquire images or videos. In some embodiments, the camera assembly 1209 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1209 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0189] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1201 for processing, or input to the radio frequency circuit 1204 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the computer device 1200. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0190] Power supply 1208 is used to supply power to the various components in computer device 1200. Power supply 1208 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1208 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0191] In some embodiments, the computer device 1200 further includes one or more sensors 1209. The one or more sensors 1209 include, but are not limited to, an accelerometer 1210, a gyroscope 1211, a pressure sensor 1212, an optical sensor 1213, and a proximity sensor 1214.

[0192] Accelerometer 1210 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 1200. For example, accelerometer 1210 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1201 can control display screen 1205 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1210. Accelerometer 1210 can also be used for games or for acquiring user motion data.

[0193] The gyroscope sensor 1211 can detect the orientation and rotation angle of the computer device 1200. The gyroscope sensor 1211 can work in conjunction with the accelerometer sensor 1210 to acquire 3D motion data from the user on the computer device 1200. Based on the data acquired by the gyroscope sensor 1211, the processor 1201 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0194] The pressure sensor 1212 can be disposed on the side bezel of the computer device 1200 and / or on the lower layer of the display screen 1205. When the pressure sensor 1212 is disposed on the side bezel of the computer device 1200, it can detect the user's grip signal on the computer device 1200, and the processor 1201 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1212. When the pressure sensor 1212 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0195] Optical sensor 1213 is used to collect ambient light intensity. In one embodiment, processor 1201 can control the display brightness of display screen 1205 based on the ambient light intensity collected by optical sensor 1213. Specifically, when the ambient light intensity is high, the display brightness of display screen 1205 is increased; when the ambient light intensity is low, the display brightness of display screen 1205 is decreased. In another embodiment, processor 1201 can also dynamically adjust the shooting parameters of camera assembly 1209 based on the ambient light intensity collected by optical sensor 1213.

[0196] The proximity sensor 1214, also known as a distance sensor, is typically installed on the front panel of the computer device 1200. The proximity sensor 1214 is used to detect the distance between the user and the front of the computer device 1200. In one embodiment, when the proximity sensor 1214 detects that the distance between the user and the front of the computer device 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from a screen-on state to a screen-off state; when the proximity sensor 1214 detects that the distance between the user and the front of the computer device 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from a screen-off state to a screen-on state.

[0197] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on the computer device 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0198] Figure 13 This is a schematic diagram of a server structure according to an embodiment of this application. The server 1300 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memory 1302 stores at least one computer program, which is loaded and executed by the processor 1301 to implement the deep convolutional neural network training method or inverse tone mapping method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0199] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the methods described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0200] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0201] This application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the deep convolutional neural network training method or inverse tone mapping method provided in the various optional implementations described above.

[0202] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0203] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a deep convolutional neural network, characterized in that, The method includes: Acquire multiple sample image frames of a sample video, the multiple sample image frames including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame; The plurality of sample image frames are input into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame. The color features of the plurality of sample image frames are processed to obtain the temporal inverse tone mapping parameters of the plurality of sample image frames. The feature inverse tone mapping module in the deep convolutional neural network maps the input features of the feature inverse tone mapping module based on the temporal inverse tone mapping parameters of the plurality of sample image frames to obtain a first feature. Based on the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, the first feature is processed at least once to obtain the mapping feature of the first sample image frame. Based on the mapping features of the first sample image frame, the generated image frame of the first sample image frame is output; Based on the reference sample image frame of the first sample image frame and the generated image frame, an image loss is determined, and the model parameters of the deep convolutional neural network are adjusted based on the image loss. Wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

2. The training method for a deep convolutional neural network according to claim 1, characterized in that, The process of processing the color features of the first sample image frame using the deep convolutional neural network to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, and processing the color features of the plurality of sample image frames to obtain the temporal inverse tone mapping parameters of the plurality of sample image frames, includes: The multiple color extraction modules in the deep convolutional neural network perform feature extraction on the multiple sample image frames to obtain the color features of each sample image frame; The spatial local inverse tone mapping module and the single-frame global inverse tone mapping module in the deep convolutional neural network respectively process the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame. The temporal inverse tone mapping module in the deep convolutional neural network processes the color features of the multiple sample image frames to obtain the temporal inverse tone mapping parameters of the multiple sample image frames.

3. The training method for a deep convolutional neural network according to claim 1, characterized in that, The method further includes: The fusion module in the deep convolutional neural network fuses the multiple sample image frames to obtain the fusion features of the multiple sample image frames, and the fusion features are the input features of the feature inverse tone mapping module.

4. The method according to claim 3, characterized in that, The fusion module in the deep convolutional neural network fuses the multiple sample image frames to obtain the fused features of the multiple sample image frames, including: The fusion module in the deep convolutional neural network obtains translation parameters of the multiple sample image frames based on the multiple sample image frames, and fuses the multiple sample image frames using the translation parameters to obtain the fusion features of the multiple sample image frames. The translation parameters are used to align the multiple sample image frames.

5. The training method for a deep convolutional neural network according to claim 3, characterized in that, The method further includes: The residual module in the deep convolutional neural network processes the fusion features of the multiple sample image frames to obtain the residuals of the multiple sample image frames; The step of generating an image frame based on the mapping features of the first sample image frame includes: Based on the residuals of the multiple sample image frames and the mapping features of the first sample image frame, a generated image frame of the first sample image frame is generated.

6. An inverse tone mapping method, characterized in that, The method includes: Acquire multiple image frames of the video to be processed, the multiple image frames including a first image frame and multiple second image frames adjacent to the first image frame; The plurality of image frames are input into a deep convolutional neural network. The deep convolutional neural network processes the color features of the first image frame to obtain spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame. The color features of the plurality of image frames are processed to obtain temporal inverse tone mapping parameters of the plurality of image frames. The feature inverse tone mapping module in the deep convolutional neural network maps the input features of the feature inverse tone mapping module based on the temporal inverse tone mapping parameters of the plurality of image frames to obtain a first feature. Based on the spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame, the first feature is processed at least once to obtain the mapping features of the first image frame. Based on the mapping features of the first image frame, the generated image frame of the first image frame is output.

7. A training device for a deep convolutional neural network, characterized in that, The device includes: The sample acquisition unit is configured to acquire multiple sample image frames of the sample video, the multiple sample image frames including a first sample image frame and multiple second sample image frames adjacent to the first sample image frame; The mapping parameter acquisition unit is configured to input the plurality of sample image frames into a deep convolutional neural network, and the deep convolutional neural network processes the color features of the first sample image frame to obtain the spatial local inverse tone mapping parameters and the single-frame global inverse tone mapping parameters of the first sample image frame, and processes the color features of the plurality of sample image frames to obtain the temporal inverse tone mapping parameters of the plurality of sample image frames. The image frame processing unit is configured to, based on the spatial local inverse tone mapping parameters, single-frame global inverse tone mapping parameters, and temporal inverse tone mapping parameters of the first sample image frame, map the input features of the feature inverse tone mapping module in the deep convolutional neural network based on the temporal inverse tone mapping parameters of the multiple sample image frames to obtain a first feature; process the first feature at least once based on the spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first sample image frame to obtain the mapped features of the first sample image frame; and output the generated image frame of the first sample image frame based on the mapped features of the first sample image frame. The adjustment unit is configured to determine the image loss based on the reference sample image frame of the first sample image frame and the generated image frame, and adjust the model parameters of the deep convolutional neural network based on the image loss; Wherein, the reference sample image frame of the first sample image frame has the same image frame content as the first sample image frame, and the dynamic range of the reference sample image frame is greater than that of the first sample image frame.

8. An inverse tone mapping device, characterized in that, The device includes: The acquisition unit is configured to acquire multiple image frames of the video to be processed, the multiple image frames including a first image frame and multiple second image frames adjacent to the first image frame; An image frame processing unit is configured to input the plurality of image frames into a deep convolutional neural network, whereby the deep convolutional neural network processes the color features of the first image frame to obtain spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame; processes the color features of the plurality of image frames to obtain temporal inverse tone mapping parameters of the plurality of image frames; a feature inverse tone mapping module in the deep convolutional neural network maps the input features of the feature inverse tone mapping module based on the temporal inverse tone mapping parameters of the plurality of image frames to obtain a first feature; processes the first feature at least once based on the spatial local inverse tone mapping parameters and single-frame global inverse tone mapping parameters of the first image frame to obtain the mapping features of the first image frame; and outputs a generated image frame of the first image frame based on the mapping features of the first image frame.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed as the training method of the deep convolutional neural network according to any one of claims 1 to 5 or the inverse tone mapping method according to claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one computer program for executing the training method of the deep convolutional neural network according to any one of claims 1 to 5 or the inverse tone mapping method according to claim 6.

Images