High dynamic range video reconstruction method and system based on hybrid expert system

Abstract

The invention discloses a high dynamic range video reconstruction method and system based on a hybrid expert system. The method comprises the following steps: acquiring an SDR video to be processed and decoding the SDR video into a continuous SDR image frame sequence; degeneration features used for representing insufficient image quality are extracted frame by frame from the SDR image frame sequence through a degeneration coding network, and the degeneration features are coded; generating a probabilistic weight vector corresponding to each SDR image in the SDR image frame sequence through a routing network based on the degradation codes output by the degradation coding network; on the basis of the probabilistic weight vector, feature enhancement and feature weighted fusion after enhancement are carried out on SDR images in the SDR image frame sequence frame by frame by integrating a video frame enhancement network of a plurality of parallel expert sub-networks, and a preliminary HDR video frame sequence is formed; performing highlight area repairing on the preliminary HDR video frame sequence frame by frame through a highlight area repairing network to obtain a target HDR video frame sequence; and coding and packaging the target HDR video frame sequence to obtain a reconstructed HDR video file.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and more specifically, to a high dynamic range video reconstruction method and system based on a hybrid expert system. Background Technology

[0002] High dynamic range (HDR) video reconstruction technology aims to recover HDR video content with a higher brightness range, richer color levels and clearer details from standard dynamic range (SDR) video, which is of great significance for improving the visual experience.

[0003] Currently, most methods in this field are typically trained on a single degradation model, constructing a fixed mapping from SDR to HDR to perform video conversion. However, the fundamental limitation of these methods lies in their insufficient generalization ability. Because video content in real-world applications varies greatly, and lighting conditions are complex and diverse, a single degradation model cannot cover all scenarios. This leads to a significant decrease in reconstruction quality and unstable performance when the model encounters videos with distributions different from the training data.

[0004] Traditional methods, such as those based on luminance mapping curves or local filtering, also face challenges in terms of reconstruction quality. These methods often lack a deep understanding and modeling of image semantic information, regional luminance distribution, and complex texture features. As a result, reconstructed HDR videos are prone to problems such as insufficient color reproduction, loss or overexposure of highlight details, and blurred details in shadows. Especially when processing videos with complex scenes or non-standard exposures, existing methods struggle to achieve a good balance between color fidelity, detail retention, and overall visual quality.

[0005] In summary, existing technologies have failed to effectively resolve the contradiction between model adaptability and the requirements for high-quality reconstruction. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a high dynamic range video reconstruction method and system based on a hybrid expert system. It can adaptively enhance the input SDR video and reconstruct high dynamic range (HDR) video with rich detail.

[0007] The objective of this invention is achieved through the following technical solutions.

[0008] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0009] Some embodiments of this application propose a high dynamic range video reconstruction method and system based on a hybrid expert system to solve the technical problems mentioned in the background section above.

[0010] As a first aspect of this application, some embodiments of this application provide a high dynamic range video reconstruction method based on a hybrid expert system, comprising the following steps: S1. Acquire the SDR video to be processed and decode it into a continuous sequence of SDR image frames; S2. Degradation features that characterize insufficient image quality are extracted frame by frame from the SDR image frame sequence through a degradation coding network and then encoded. S3. Based on the degradation coding output by the degradation coding network, a probabilistic weight vector corresponding to each SDR image in the SDR image frame sequence is generated through the routing network. S4. Based on probabilistic weight vectors, a video frame enhancement network integrating multiple parallel expert sub-networks is used to enhance the features of different scenes in the SDR image frame sequence frame by frame and perform weighted fusion of the enhanced features to form a preliminary HDR video frame sequence. S5. The highlight regions of the HDR video frames in the initial HDR video frame sequence are repaired frame by frame using a highlight region repair network to obtain the target HDR video frame sequence. S6. Encode and encapsulate the target HDR video frame sequence to obtain the reconstructed HDR video file.

[0011] Furthermore, degenerate coding networks include: The adjustment layer is configured to adjust the SDR images in the SDR image frame sequence to a preset size; A convolutional feature extraction network, consisting of at least one convolutional block connected sequentially, is configured to extract degradation features from an SDR image and output a two-dimensional feature map. The global average pooling layer is configured to perform global average pooling on the two-dimensional feature map to obtain an aggregated global feature vector. The feature-encoding fully connected network is configured to map aggregated global feature vectors to a preset low-dimensional space and output degenerate codes.

[0012] Furthermore, the routing network includes: The first processing unit includes a first fully connected layer and a first activation function layer; The first fully connected layer is configured to perform a linear transformation on the degenerate encoding; the first activation function layer is a ReLU activation function, configured to perform non-linear activation on the linearly transformed features and output an intermediate feature vector; The second processing unit includes a second fully connected layer and a second activation function layer; The second fully connected layer is configured to map the intermediate feature vectors to multiple initial weights equal to the number of expert subnetworks; the second activation function layer is a Softmax activation function, configured to normalize the multiple initial weights and output a probabilistic weight vector.

[0013] Furthermore, the video frame enhancement network adopts a U-Net-based architecture, including: The initial feature extraction network is configured to extract basic features from the input SDR image frames; The encoder network is configured to downsample and abstract basic features, and output an encoded feature map. The hybrid expert bottleneck network consists of multiple parallel expert subnetworks, which are configured to adaptively enhance the encoded feature maps by using multiple expert subnetworks to enhance features of different scenarios, and to use the probabilistic weight vector corresponding to the SDR image as a gating parameter to weight and fuse the enhanced features output by each expert subnetwork to output the bottleneck layer features. The decoder network is configured to upsample and reconstruct the features from the bottleneck layer. The HDR reconstruction output network is configured to map the features output by the decoder network to initial HDR video frames.

[0014] Furthermore, the expert sub-network includes: The exposure estimation module is configured to generate an exposure map of the same size as the encoded feature map and with values ​​in the range [0, 1]. ; The dual-branch residual processing module includes two independent branches, each containing at least one residual block, which are configured to extract highlight region features from the encoded feature map. and features of dark areas ; The exposure-guided fusion module is configured to follow the formula Calculate and output enhanced features .

[0015] Furthermore, the highlight area restoration network includes: The mask generation unit is configured to generate a specular mask by comparing the luminance channel of an initial HDR video frame with a preset luminance threshold. The region separation unit is configured to separate the highlight region image from the initial HDR video frame using a highlight mask; The highlight generation network is configured to perform detail restoration and enhancement on the highlight region image and output the restored highlight image. The image synthesis unit is configured to synthesize the restored highlight image with the non-highlight areas in the initial HDR video frame to output the target HDR video frame.

[0016] Furthermore, the highlight region restoration network also includes a highlight discrimination network; The highlight discrimination network is configured to process the highlight regions of the input target HDR video frame and the highlight regions of the real HDR image separately, and output the discrimination results. The adversarial loss is calculated based on the discrimination results, and the parameters of the specular generation network are optimized using the adversarial loss.

[0017] Furthermore, it is characterized by, The specular generation network includes: At least one sequentially connected convolutional block, each convolutional block comprising a 3×3 convolutional layer and a ReLU activation function layer; An output layer consists of a 3×3 convolutional layer and a sigmoid activation function layer; The specular discrimination network includes: The feature extraction layer includes a convolutional layer with a kernel size of 3×3 and a stride of 1; The deep feature discrimination module includes at least one downsampled convolutional block, each downsampled convolutional block including a convolutional layer with a kernel size of 3×3 and a stride of 2 and a LeakyReLU activation function layer; The output layer consists of a convolutional layer, a fully connected layer, and a sigmoid activation function layer.

[0018] As a second aspect of this application, some embodiments of this application provide a high dynamic range video reconstruction system based on a hybrid expert system, including: The data preprocessing module is configured to acquire and decode the SDR video to be processed, and obtain a continuous sequence of SDR image frames; The degradation coding module is configured to process the SDR image frame sequence frame by frame, extract degradation features that characterize insufficient image quality, and encode them into degradation codes. The routing module is configured to generate probabilistic weight vectors corresponding to each SDR image in the SDR image frame sequence based on the degradation coding. The video frame enhancement module is configured to use probabilistic weight vectors to perform feature enhancement on SDR images for different scenes and weighted fusion of the enhanced features through multiple integrated expert sub-networks to generate a preliminary HDR video frame sequence. The highlight restoration module is configured to restore the highlight areas of HDR video frames in the initial HDR video frame sequence to generate the target HDR video frame sequence. The video output module is configured to encode and encapsulate the target HDR video frame sequence into a reconstructed HDR video file.

[0019] As a third aspect of this application, some embodiments of this application provide an electronic device characterized by including a processor and a memory; The memory stores computer programs; The processor is used to execute computer programs to implement the high dynamic range video reconstruction method based on a hybrid expert system as described in any one of claims 1 to 8.

[0020] Compared with the prior art, the advantages of this invention are: (1) This invention introduces a degradation coding network that can learn multiple degradation features, a routing network that can make dynamic decisions, and a video frame enhancement network that integrates multiple expert sub-networks to achieve adaptive enhancement of input SDR video. This overcomes the problem of insufficient generalization ability caused by a single degradation model. It can automatically adapt to changes in different scenes and lighting conditions, and improve the reconstruction stability and generalization ability on diverse video content. (2) This invention extracts multi-level degradation features from SDR images by constructing a convolutional feature extraction network containing convolutional blocks, which can accurately analyze and characterize the deficiencies of images in terms of brightness, color, and detail; (3) The present invention constructs a routing network by using a first processing unit containing a fully connected layer and a ReLU activation function and a second processing unit containing a fully connected layer and a Softmax activation function. The gating parameters are dynamically generated according to the degradation features of each frame of the image, thereby quantifying and controlling the weights of different experts in the network fusion. Each expert sub-network generates an exposure map and uses double-branch residual processing to extract the bright and dark features respectively, and finally guides adaptive fusion. This design can process the bright and dark areas in the image more precisely, thereby alleviating the problems of blurred details in the dark area and easy overexposure of the highlights. (4) By introducing a highlight discrimination network and conducting adversarial training with the highlight generation network, the generated highlight content is constrained to be consistent with the real data in terms of distribution, making the generated highlight area more detailed and visually more natural and closer to the real HDR effect, thus improving the realism and consistency of highlight restoration. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the overall process of a high dynamic range video reconstruction method based on a hybrid expert system in one embodiment of the present invention. Figure 2 This is a flowchart illustrating a high dynamic range video reconstruction method based on a hybrid expert system in one embodiment of the present invention. Figure 3 This is a structural diagram of the degradation coding module in one embodiment of the present invention; Figure 4 This is a structural diagram of a video frame enhancement module according to an embodiment of the present invention; Figure 5 This is a high dynamic range video reconstruction system based on a hybrid expert system in one embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] like Figures 1 to 5 As shown, this embodiment of the invention provides a high dynamic range video reconstruction method and system based on a hybrid expert system. It includes the following steps: S1. Data Input and Preprocessing: Acquire the standard dynamic range (SDR) video to be processed and decode it into a continuous sequence of SDR image frames.

[0024] Specifically, Standard Dynamic Range (SDR) video is a continuous video data format, typically in common formats such as MP4, AVI, and MOV. SDR video contains compressed and encoded video and audio streams, as well as other metadata. The recorded luminance and color information is located within the Standard Dynamic Range (SDR) range.

[0025] The standard dynamic range video to be processed is preprocessed using a video decoding tool (such as FFmpeg) to convert it into static image data that can be processed frame-by-frame by the subsequent neural network model. The preprocessing operations are as follows: First, the encapsulation and encoding formats of the standard dynamic range video to be processed are analyzed, and the compressed video stream is decoded into continuous, uncompressed raw SDR image frame data.

[0026] Then, following the original timestamp order of the standard dynamic range video to be processed, each decoded SDR image frame is output as an independent still image file, and numbered sequentially to preserve and clarify its temporal context.

[0027] Finally, the extracted SDR image frames are stored uniformly in a static image format (such as PNG) to avoid introducing additional quality loss.

[0028] Through the above preprocessing operations, a continuous sequence of SDR image frames arranged in chronological order is obtained. ,in For frame number ( Each frame in an SDR image frame sequence is a three-channel (RGB) digital image with the same spatial resolution as the original video, and pixel values ​​typically range from 0 to 255. The SDR image frame sequence serves as direct input data for all subsequent image processing steps.

[0029] S2. Degradation Feature Extraction: Based on the SDR image frame sequence output from step S1. Degradation feature analysis is performed to analyze and extract degradation features that characterize insufficient image quality. Degradation features are extracted for each frame of SDR image and encoded into degradation codes.

[0030] In this embodiment, a degradation coding network is used to analyze the deficiencies or degradation information of each frame of SDR image in the SDR image frame sequence in terms of image quality such as brightness, color and detail, and compress this information into a low-dimensional feature vector, namely degradation coding.

[0031] Specifically, the degenerate coding network is a sequentially executed convolutional neural network, including adjustment layers, convolutional feature extraction networks, global average pooling layers, and fully connected feature encoding networks. The data processing flow of the degenerate coding network is as follows: For the input SDR image frame sequence The adjustment layer uniformly scales the spatial dimensions of each frame of SDR image to a fixed standard size (256×256 pixels in this embodiment) to accommodate the processing of subsequent convolutional layers.

[0032] Convolutional feature extraction network is composed of It consists of several consecutive convolutional blocks. Each convolutional block includes a convolutional layer with a kernel size of 3×3 and a stride of 1, an instance normalization layer, and a ReLU activation function layer. In this embodiment, .

[0033] Convolutional feature extraction networks (CFRs) are used to extract multi-level degradation features from images. The SDR image, scaled by an adjustment layer, is fed into the CFR and passes sequentially through its layers. Each convolutional block, through the combined action of convolutional layers, instance normalization layers, and ReLU activation function layers, progressively extracts deep features from the SDR image.

[0034] Specifically, the convolutional feature extraction network performs layer-by-layer convolution and nonlinear transformation on the input SDR image to extract degradation features from the SDR image from shallow to deep, and finally outputs a set of two-dimensional feature maps. The two-dimensional feature map can be understood as a stack of multiple two-dimensional matrices, where each matrix is ​​called a "channel", representing the response intensity in the image space of a certain degradation feature (such as texture loss and color deviation) learned by the network.

[0035] The global average pooling layer calculates the average value of all pixels in each channel of the two-dimensional feature map, thereby compressing each channel into a single scalar value; and concatenates the scalar values ​​of all channels in order to obtain a one-dimensional aggregated global feature vector, which aggregates the global information of all feature channels.

[0036] Feature-encoding fully connected networks are composed of It consists of several fully connected layers (alternating between linear layers and ReLU activation functions) used to perform nonlinear transformations and dimensional mappings on the aggregated global feature vector, ultimately outputting a low-dimensional feature vector with a fixed preset dimension, forming the final degenerate code. In this embodiment, =2.

[0037] In one specific embodiment, the SDR image frame sequence Each frame of SDR image is input into the degradation coding network, processed sequentially through the layers described above, and finally outputs a feature vector representing the degradation information of that frame, i.e., the degradation code. The process can be expressed as follows: ; in, This represents the functional mapping of a degenerate coding network.

[0038] By analyzing SDR image frame sequences The above processing of each frame yields the degradation code corresponding to each frame of the SDR image. Together, they constitute the degenerate coding sequence.

[0039] In this embodiment, the preset dimension of the degenerate coding network is 16, therefore the output degenerate code... It is a 1×16 dimension vector, and the degenerate encoding sequence will be used as the input to the routing network in step S3.

[0040] This step, through multi-level convolution extraction and global average pooling, can comprehensively capture the degradation features of the image in terms of brightness, color, and texture, providing accurate scene representation for subsequent adaptive enhancement.

[0041] S3. Gating parameter generation: Generate a set of gating parameters for fusing multiple expert networks based on the degenerate coding.

[0042] This step aims to analyze the degradation coding generated for each SDR image frame in step S2, generate the gating parameters corresponding to that SDR image frame, and use them to quantitatively specify the contribution weights of each expert subnetwork in the subsequent video frame enhancement network during fusion, thereby realizing an enhancement strategy that is adaptive to the current frame scene.

[0043] This step involves a routing network. The network is a lightweight multilayer perceptron, whose structure consists of two sequentially connected processing units: The first processing unit consists of a fully connected layer (linear layer) and a ReLU activation function in sequence. The first processing unit receives the degenerate code of the input. First, the fully connected layer performs a linear transformation on the degenerate code, and then performs non-linear activation through the ReLU function to output a set of intermediate feature vectors.

[0044] The second processing unit consists of a fully connected layer (linear layer) and a Softmax activation function in sequence. The second processing unit receives the intermediate feature vector output by the first processing unit, and its fully connected layer maps the features to a dimension equal to the number of expert network members (4 in this embodiment), outputting a set of initial weight values. Subsequently, the Softmax function normalizes these initial weight values ​​to ensure that their sum is 1, thus forming a probability distribution.

[0045] In one specific embodiment, the degradation code is... The input routing network is processed sequentially through the first and second linear layers, ultimately outputting a set of parameters characterizing the weights of each expert network, i.e., the gating parameters. Therefore, the gating parameters corresponding to each frame of the SDR image are obtained. The process can be expressed as follows: ; in, This represents the functional mapping of the routing network.

[0046] In this embodiment, the output gating parameters It is a 1×4 probabilistic weight vector. Each element in the probabilistic weight vector corresponds to an expert subnetwork, and its value represents the contribution weight of that expert in the network fusion output. This gating parameter is the control signal for the video frame enhancement network in step S4.

[0047] In this step, the routing network dynamically generates a weight distribution based on the degradation features of each frame of image, achieving adaptive scheduling for different scenes and enabling the model to flexibly cope with diverse content.

[0048] S4. Multi-expert fusion video frame enhancement: Using a video frame enhancement network integrated with a hybrid expert system, the SDR image frame sequence is enhanced according to the gating parameters generated in step S3. Adaptive enhancement is performed to generate initial high dynamic range (HDR) video frames.

[0049] Specifically, based on the SDR image frame sequence output in step S1 And step S3 is the SDR image frame sequence. Gating parameters generated for each frame of SDR image By adaptively fusing the enhancement capabilities of multiple expert networks using the gating parameters generated in step S3, the SDR image frame sequence is processed. Convert a sequence of HDR video frames into an initial enhanced version with significant improvements in brightness, color, and detail. .

[0050] Specifically, this step uses a video frame enhancement network. The video frame enhancement network is an integrated neural network that employs an encoder-decoder architecture with a U-Net backbone and integrates a hybrid expert system at its bottleneck layer. The video frame enhancement network is divided into five sequentially connected functional sub-modules in the order of the data flow: an initial feature extraction network, an encoder network, a hybrid expert bottleneck network, a decoder network, and an HDR reconstruction output network.

[0051] The initial feature extraction network consists of a convolutional layer used to extract basic features from the input frame.

[0052] An encoder network is a network consisting of multiple layers of residual blocks and downsampling blocks. It is used to perform deep abstraction and spatial downsampling of features. Through layer-by-layer downsampling, it extracts deep abstract features and expands the receptive field of the model.

[0053] The Hybrid Expert Bottleneck Network (BottomNeck) comprises multiple parallel expert subnetworks, each specializing in feature enhancement for a different scenario, and is controlled by gating parameters. Control its output fusion.

[0054] The decoder network is a network consisting of multiple layers of residual blocks and upsampling blocks. It is used to recover the spatial size of the feature map and fuse details. By upsampling layer by layer, the spatial size of the feature map is recovered, and shallow features from the encoder are fused to reconstruct a high-resolution result.

[0055] The HDR reconstruction output network consists of a convolutional layer and a sigmoid activation function, which is responsible for mapping the high-dimensional features output by the decoder network to the final HDR image.

[0056] In one specific embodiment, for an SDR image frame sequence For each SDR image and its corresponding gating parameters, the video frame enhancement network performs the following processing flow: Video Frame Enhancement Network Received input SDR image frame sequence and gating parameters .

[0057] First, the initial feature extraction network performs a convolution operation on the input SDR image frame using a 3×3 convolutional layer, with the number of channels set to [value missing]. The output is a basic feature map, the size of which is consistent with the size of the input SDR image frame sequence.

[0058] The basic feature map is input into the encoder network.

[0059] The encoder network has the following number of layers: Each floor consists of Each residual block consists of a residual block and a downsampling block. Each residual block contains a 3×3 convolutional layer, a ReLU activation layer, and an instance normalization layer. Each downsampling block contains a 3×3 convolutional layer and an UnPixelShuffle layer.

[0060] In this example, , .

[0061] After being processed by each layer in the encoder network, the basic feature map's size is compressed to half its original size, and the number of channels doubles. The encoder network outputs a deeply abstracted and compressed encoded feature map.

[0062] In this embodiment, the hybrid expert bottleneck network comprises four structurally identical expert subnetworks. Each expert subnetwork includes: The exposure estimation module is configured to generate an exposure map with the same size as the encoded feature map and a numerical range of [0, 1]. The dual-branch residual processing module includes two independent branches, each containing at least one residual block, which are respectively configured to extract bright region features and dark region features from the encoded feature map. The exposure-guided fusion module is used to calculate and output the output features of the expert subnetwork.

[0063] In one specific embodiment, the encoded feature maps are simultaneously input into four structurally identical expert subnetworks of the hybrid expert bottleneck network. Each expert subnetwork independently performs the following operations: An exposure map with the same size as the input encoded feature map and whose values ​​are normalized to [0, 1] is generated using a light quantum network (such as two 3×3 convolutional layers). The exposure map is used to reflect the brightness distribution of the current frame of the SDR image.

[0064] The encoded feature maps are processed by inputting them into two independent branches: a bright branch and a dark branch. Each branch contains... Each residual block has two branches that focus on modeling the features of bright and dark regions, respectively, and the outputs are the bright region features. and dark features In this example, =2.

[0065] Using exposure images As weights, the outputs of the two branches and Weighted fusion is performed to obtain the enhanced features of this expert subnetwork. , represented as: ; Using gating parameters The enhanced features of the four expert subnetworks are weighted and fused to obtain the fused bottleneck layer features. The process is represented as follows: ; in, This represents the index of the expert subnetwork. Indicates the first The weights corresponding to each expert subnetwork Indicates the first Enhanced features of the output of each expert subnetwork.

[0066] Bottleneck layer features Input decoder network.

[0067] The decoder network is symmetrical to the encoder network. The decoder network has the following number of layers: Each floor consists of The bottleneck layer consists of a residual block and an upsampling block. Each residual block contains two 3×3 convolutional layers, a ReLU activation layer, and an instance normalization layer. Each upsampling block contains a 3×3 convolutional layer and a PixelShuffle layer. (Bottleneck layer features) With each decoder layer, the feature map size doubles, while the number of channels is halved. The decoder network ultimately outputs a refined and resized decoded feature map.

[0068] In this example, , .

[0069] The decoded feature map is input into the HDR reconstruction output network.

[0070] The HDR reconstruction output network maps the number of channels in the decoded feature map to 3 (corresponding to RGB three channels) through a 3×3 convolutional layer, and then normalizes the pixel values ​​to the [0, 1] interval through a Sigmoid activation function, outputting the preliminary high dynamic range (HDR) video frame corresponding to the SDR image. This is achieved by processing the input SDR image frame sequence. The above processing of each frame ultimately yields a preliminary HDR video frame sequence. .

[0071] In this step, the hybrid expert bottleneck network processes different scene features in parallel through multiple expert subnetworks and combines them with gating weights, which significantly improves the model's adaptability to complex lighting and content changes.

[0072] S5. Highlight Area Restoration: Re-processing the initial HDR video frame sequence. The highlight areas of each frame are specifically enhanced to generate the target HDR video frame sequence.

[0073] This step uses a highlight restoration network to repair the initial HDR video frame sequence generated in step S4. This will help to remove any overexposed or detail-lost highlight areas from the image, thus further enhancing the visual quality and detail of the picture.

[0074] Specifically, the highlight region restoration network includes a mask generation unit, a region separation unit, a highlight generation network, an image synthesis unit, and a highlight discrimination network. The highlight generation network and the highlight content discrimination network form a generative adversarial structure (GAN), which, through a generative-discriminative game learning approach, enables the network to more accurately and naturally restore the texture, lighting levels, and local structure of highlight regions.

[0075] The highlight generation network is used to repair content, while the highlight content discrimination network is used to improve the authenticity of the generated content.

[0076] Mask generation unit: The mask generation unit is configured to generate a specular mask by comparing the luminance channel of an initial HDR video frame with a preset luminance threshold.

[0077] Specifically, the mask generation unit calculates the luminance channel of the initial HDR video frame, performs binarization segmentation using a preset luminance threshold (set to 0.95 in this embodiment), and outputs a highlight mask. Among them, specular mask This is used to identify areas in an image whose brightness exceeds a set threshold: areas with brightness greater than the threshold are marked as 1, and areas with brightness less than the threshold are marked as 0. That is, areas with a mask value of 1 are considered highlight areas, and areas with a mask value of 0 are considered non-highlight areas.

[0078] Region separation unit: The region separation unit is configured to separate the highlight region image from the initial HDR video frame using a highlight mask.

[0079] Specifically, the region separation unit performs element-wise multiplication between the initial HDR video frame and the highlight mask to extract the highlight regions in the image, forming a highlight region image. .

[0080] Specular Generation Network: The highlight generation network is configured to perform detail restoration and enhancement on the highlight region image and output the restored highlight image.

[0081] In this embodiment, the specular generation network It is a neural network composed of multiple convolutional networks, including a specular input fusion layer, a convolutional enhancement subnetwork, and a specular response output layer.

[0082] (1) Convolutional Enhancement Subnetwork: It consists of multiple convolutional blocks stacked sequentially. Each convolutional block contains a 3×3 convolutional layer (stride of 1) and a ReLU activation function layer, which are used for multi-level feature extraction and enhancement of the highlight region.

[0083] Specifically, the convolutional enhancement subnetwork in this embodiment consists of six convolutional blocks stacked sequentially.

[0084] (2) Specular response output layer: It consists of a 3×3 convolutional layer (stride of 1) and a sigmoid activation function. The specular response output layer maps the enhanced features back to the image space and outputs the repaired result with pixel values ​​normalized to [0, 1].

[0085] Specifically, the highlight area image Input specular generation network The feature map is extracted and enhanced by passing through multiple convolutional blocks of the convolutional enhancement sub-network. The feature map is then compressed to 3 channels by the highlight response output layer through a convolutional layer with a kernel size of 3×3 and a stride of 1. The Sigmoid activation function is then used to constrain the numerical range to the interval [0, 1], and the output pixel values ​​are normalized to the repaired highlight image of [0, 1].

[0086] Finally, the restored highlight image is output by the highlight response output layer.

[0087] Image synthesis unit: The image synthesis unit is configured to synthesize the restored highlight image with the non-highlight areas in the initial HDR video frame to output the target HDR video frame.

[0088] Specifically, the restored highlight image and the frame The non-highlight regions in the image are weighted and synthesized to output the final optimized target HDR video frame. Its mathematical expression is: ; in, This represents the data processing procedure of the specular generation network.

[0089] By analyzing the initial HDR video frame sequence The above processing of each frame ultimately yields the optimized target HDR video frame sequence. .

[0090] More specifically, to improve the realism of specular generation, a specular discrimination network is introduced. Conduct adversarial training. Highlight content discrimination network. It is a deep neural network composed of multi-layer convolutional structures and non-linear activation functions, used to determine whether the input highlight region image comes from the real data distribution.

[0091] In one specific embodiment, the specular content discrimination network include: Feature extraction layer: A convolutional layer with a kernel size of 3×3 and a stride of 1 is used to perform initial feature extraction on the input highlight region image and output shallow features.

[0092] Deep feature discrimination module: composed of One (in this embodiment) The downsampling convolutional blocks are stacked sequentially. Each convolutional block contains a convolutional layer with a kernel size of 3×3 and a stride of 2, and a LeakyReLU activation function layer, which is used to progressively extract deeper and more discriminative features.

[0093] The discriminant output layer consists of a 3×3 convolutional layer with a stride of 1, a fully connected layer, and a sigmoid activation function layer, which outputs the probability that the input image is a true highlight region.

[0094] In a specific embodiment, the highlight restoration adversarial training process is as follows: During the adversarial training phase, the specular inpainting network involved in this step needs to be trained on a dataset containing pairwise data. In this dataset, each input SDR image corresponds to a real HDR image used as a quality standard, denoted as . (in (Indicates the frame number). This step utilizes... To provide a monitoring signal for highlight area restoration.

[0095] Specifically, true HDR images The data was obtained by segmenting HDR video frames from the dataset, resulting in three-channel RGB image data with values ​​ranging from 0 to 1. The specular mask M was obtained in the same way as the M obtained in the specular generation network described above.

[0096] The generated target frame image is multiplied element-wise with its corresponding specular mask M, retaining only the highlight areas in the image; similarly, the real HDR image is multiplied... Multiply by the specular mask M to extract the real specular region. The output is two specular region images: ( )and( The value ranges from 0 to 1.

[0097] The generated final frame Highlight areas ( ) and the highlight area of ​​the real image ( (Input to highlight content discrimination network) By calculating and backpropagating adversarial losses, the specular generation network is continuously optimized. The parameters make the output highlight content visually indistinguishable from the real data.

[0098] Specifically, the target HDR video frame sequence The highlight areas and their corresponding highlight areas in the real HDR dataset are sequentially processed by the feature extraction layer of the highlight content discrimination network and... Each downsampled convolutional block has several sub-blocks. The feature extraction layer extracts features from the input image to obtain shallow features, which are then sequentially fed into each downsampled convolutional block. Each downsampled convolutional block consists of a convolutional layer with a kernel size of 3 and a stride of 2, and a LeakyReLU activation layer with a negative half-axis slope of 0.2. In each downsampled convolutional block, the size of the input features is gradually halved to obtain the final discrimination probability. Using discriminant probability and true probability Calculate the adversarial loss (where the true probability is...) (Based on degenerate labels from the dataset), the parameters of the highlight generation network are continuously optimized to make it closer to the highlight distribution of the real data.

[0099] This step utilizes a modular highlight restoration network to achieve precise detection, separation, restoration, and fusion of overexposed areas. Combined with an adversarial training mechanism, it significantly enhances the richness and realism of highlight details, effectively avoiding the detail loss and visual unnaturalness common in traditional methods.

[0100] S6. Video Sequence Output: Output the optimized target HDR video frame sequence from step S5. Perform video encoding and packaging to generate standard, storable, and distributable HDR video files.

[0101] Specifically, the target HDR video frame sequence Arrange them in chronological order, so that their frame rate, frame number, and playback sequence match the SDR image frame sequence. Maintain consistency to ensure that the final HDR video file is temporally coherent. This results in an organized, encoding-ready frame sequence.

[0102] Subsequently, the frame sequence is compressed using a video encoding module. In this embodiment, this module is implemented based on the FFmpeg toolchain and HEVC (H.265) encoder, and performs an encoding process conforming to the HDR10 standard format, specifically including: Color and Luminance Formatting: The input frame is converted to the BT.2020 color gamut, luminance is encoded using the PQ (ST 2084) transfer function, and set to 10-bit color depth to preserve high dynamic range information; Video compression coding: The HEVC encoder is used to perform standard compression operations on the formatted frame sequence, including block partitioning, motion estimation, transform, quantization and entropy coding, to achieve frame-level and inter-frame compression and output a compressed video stream that conforms to the HDR10 standard.

[0103] Finally, the encoded video stream is encapsulated into a common multimedia container format. This embodiment uses the MPEG-4 Part 14 (MP4) container format to ensure compatibility with common playback devices and HDR10 decoders. The final HDR video file is then output. The HDR video file contains the reconstructed high dynamic range video content and can be displayed on playback devices that support the HDR10 standard.

[0104] In one specific embodiment, the present invention provides a high dynamic range video reconstruction system based on a hybrid expert system, comprising: The data preprocessing module is configured to acquire and decode the SDR video to be processed, and obtain a continuous sequence of SDR image frames; The degradation coding module is configured to process the SDR image frame sequence frame by frame, extract degradation features and encode them into degradation codes; The routing module is configured to generate a probabilistic weight vector corresponding to each SDR image in the SDR image frame sequence based on the degradation encoding, as a gating parameter; The video frame enhancement module is configured to use the gating parameters to adaptively enhance SDR image frames through a video frame enhancement network integrating multiple expert subnetworks to generate a preliminary HDR video frame sequence. The highlight restoration module is configured to restore the highlight areas of the initial HDR video frame sequence to generate a target HDR video frame sequence. The video output module is configured to encode and encapsulate the target HDR video frame sequence into a reconstructed HDR video file.

[0105] Those skilled in the art will understand that all or part of the steps of the various methods in the embodiments of the present invention can be executed by hardware related to program instructions. The above-described program can be stored in a computer-readable storage medium, including but not limited to read-only memory, random access memory, magnetic disk, optical disk, or other media capable of storing program code.

[0106] The invention and its embodiments have been described above illustratively. This description is not restrictive, and the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A high dynamic range video reconstruction method based on a hybrid expert system, comprising the following steps: S1. Acquire the SDR video to be processed and decode it into a continuous sequence of SDR image frames; S2. Degradation features that characterize insufficient image quality are extracted frame by frame from the SDR image frame sequence using a degradation coding network and then encoded. S3. Based on the degradation coding output by the degradation coding network, a probabilistic weight vector corresponding to each SDR image in the SDR image frame sequence is generated through the routing network. S4. Based on the probabilistic weight vector, a video frame enhancement network integrating multiple parallel expert sub-networks is used to enhance the features of different scenes and perform weighted fusion of the enhanced features on each frame of the SDR image in the SDR image frame sequence, thereby forming a preliminary HDR video frame sequence. S5. Perform highlight region restoration on each frame of the HDR video frame in the preliminary HDR video frame sequence using a highlight region restoration network to obtain the target HDR video frame sequence. S6. Encode and encapsulate the target HDR video frame sequence to obtain the reconstructed HDR video file.

2. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 1, characterized in that, The degenerate coding network includes: The adjustment layer is configured to adjust the SDR images in the SDR image frame sequence to a preset size; A convolutional feature extraction network, consisting of at least one convolutional block connected sequentially, is configured to extract degradation features from an SDR image and output a two-dimensional feature map. A global average pooling layer is configured to perform global average pooling on the two-dimensional feature map to obtain an aggregated global feature vector; The feature-encoding fully connected network is configured to map the aggregated global feature vector to a preset low-dimensional space and output the degenerate code.

3. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 1, characterized in that, The routing network includes: The first processing unit includes a first fully connected layer and a first activation function layer; The first fully connected layer is configured to perform a linear transformation on the degenerate encoding; the first activation function layer is a ReLU activation function, configured to perform non-linear activation on the linearly transformed features and output an intermediate feature vector; The second processing unit includes a second fully connected layer and a second activation function layer; The second fully connected layer is configured to map the intermediate feature vector to a plurality of initial weights equal to the number of expert subnetworks; the second activation function layer is a Softmax activation function, configured to normalize the plurality of initial weights and output a probabilistic weight vector.

4. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 1, characterized in that, The video frame enhancement network adopts a U-Net-based architecture, including: The initial feature extraction network is configured to extract basic features from the input SDR image frames; An encoder network is configured to downsample and abstract the basic features, and output an encoded feature map. The hybrid expert bottleneck network comprises multiple parallel expert subnetworks and is configured to adaptively enhance the encoded feature map by using multiple expert subnetworks to enhance features of different scenarios, and to use the probabilistic weight vector corresponding to the SDR image as a gate parameter to weight and fuse the enhanced features output by each expert subnetwork to output bottleneck layer features. The decoder network is configured to upsample and reconstruct the features of the bottleneck layer. An HDR reconstruction output network is configured to map features output by the decoder network to the initial HDR video frames.

5. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 4, characterized in that, The expert subnetwork includes: The exposure estimation module is configured to generate an exposure map of the same size as the encoded feature map and with values ​​in the range [0, 1]. ; The dual-branch residual processing module includes two independent branches, each containing at least one residual block, which are respectively configured to extract highlight region features from the encoded feature map. and features of dark areas ; The exposure-guided fusion module is configured to follow the formula Calculate and output enhanced features .

6. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 1, characterized in that, The highlight area restoration network includes: The mask generation unit is configured to generate a highlight mask by comparing the luminance channel of the initial HDR video frame with a preset luminance threshold. The region separation unit is configured to separate the highlight region image from the initial HDR video frame using the highlight mask; A highlight generation network is configured to perform detail restoration and enhancement on the highlight region image and output a restored highlight image. The image synthesis unit is configured to synthesize the restored highlight image with the non-highlight areas in the initial HDR video frame to output the target HDR video frame.

7. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 6, characterized in that, The highlight region restoration network also includes a highlight discrimination network; The highlight discrimination network is configured to process the highlight regions of the input target HDR video frame and the highlight regions of the real HDR image respectively, and output the discrimination result. The adversarial loss is calculated based on the discrimination result, and the parameters of the specular generation network are optimized using the adversarial loss.

8. The high dynamic range video reconstruction method based on a hybrid expert system according to claim 7, characterized in that, The specular generation network includes: At least one sequentially connected convolutional block, each of the convolutional blocks comprising a 3×3 convolutional layer and a ReLU activation function layer; An output layer consists of a 3×3 convolutional layer and a sigmoid activation function layer; The specular discrimination network includes: The feature extraction layer includes a convolutional layer with a kernel size of 3×3 and a stride of 1; The deep feature discrimination module includes at least one downsampled convolutional block, each of which includes a convolutional layer with a kernel size of 3×3 and a stride of 2 and a LeakyReLU activation function layer; The output layer consists of a convolutional layer, a fully connected layer, and a sigmoid activation function layer.

9. A high dynamic range video reconstruction system based on a hybrid expert system, characterized in that, include: The data preprocessing module is configured to acquire and decode the SDR video to be processed, and obtain a continuous sequence of SDR image frames; The degradation coding module is configured to process the SDR image frame sequence frame by frame, extract degradation features used to characterize insufficient image quality, and encode them as degradation codes. The routing module is configured to generate a probabilistic weight vector corresponding to each SDR image in the SDR image frame sequence based on the degradation encoding. The video frame enhancement module is configured to use the probabilistic weight vector to perform feature enhancement on the SDR image for different scenes and weighted fusion of the enhanced features through multiple integrated expert sub-networks to generate a preliminary HDR video frame sequence. The highlight restoration module is configured to restore the highlight areas of the HDR video frames in the initial HDR video frame sequence to generate a target HDR video frame sequence. The video output module is configured to encode and encapsulate the target HDR video frame sequence into a reconstructed HDR video file.

10. An electronic device, characterized in that, Including processor and memory; The memory stores computer programs; The processor is used to execute the computer program to implement the high dynamic range video reconstruction method based on a hybrid expert system as described in any one of claims 1 to 8.

Images