DIBR synthesized view optimization method based on distortion anisotropy perception and quality regularization

Abstract

The present application relates to the technical field of image quality optimization, and particularly relates to a DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization, comprising the following steps: S1: acquiring a DIBR synthesized view and a reference view thereof as training samples; S2: performing multi-scale difference modeling from a pixel dimension, an edge dimension and a texture dimension to generate a continuous distortion intensity distribution map; S3: inputting the DIBR synthesized view into an image inpainting model to output a corresponding optimized image; S4: calculating a comprehensive training loss comprising a consistency constraint and a perception quality regularization constraint; S5: optimizing learnable parameters of the image inpainting model based on the comprehensive training loss; S6: iteratively training until the model converges or a maximum number of iterations is reached; and S7: inputting a DIBR synthesized view to be optimized into the trained image inpainting model to output a corresponding optimized image. The present application improves pixel fidelity, structural consistency and subjective visual quality of the DIBR synthesized view.

Description

Technical Field

[0001] This invention relates to the field of image quality optimization technology, specifically to a DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization. Background Technology

[0002] With the rapid development of immersive media and 3D video system technologies, capturing images from all viewpoints is impractical due to limitations such as equipment cost and transmission pressure. Virtual view compositing has emerged as a cost-effective alternative, generating a composite view from any target perspective using a limited number of reference viewpoints. Depth-Image-Based Rendering (DIBR) technology, as one of the core methods for generating composite views, has been widely used in practical engineering. However, during the view rendering process, DIBR technology inevitably produces several typical distortions due to factors such as discontinuous depth information, changes in occlusion relationships, and sampling errors. These include voids caused by background filling, artifacts formed by geometric misalignment, and edge stretching distortion. These distortions severely degrade the visual presentation quality of the composite image, thereby affecting the performance of subsequent encoding, transmission, and terminal display. Therefore, constructing targeted image quality optimization schemes in the back-end processing of composite views has become a key technical requirement in current 3D video processing systems.

[0003] In recent years, image enhancement and other optimization methods have been extensively studied. However, unlike ordinary images, DIBR (Distributed Image Backed Block) composite views exhibit significant heterogeneous distortion characteristics. This means that multiple distortion modes, such as edge artifacts, geometric stretching, and texture degradation, may coexist within the same image, and their spatial distribution is typically sparse. This makes optimizing DIBR composite views quite challenging. Meanwhile, while there is considerable research on quality assessment models for composite viewpoint images, most studies only focus on simple judgments of image quality. Given that the ultimate goal of image and video processing systems is to optimize the end-user experience, utilizing the feedback information provided by quality assessment models to guide and drive image quality optimization, forming a closed-loop mechanism of "evaluation-feedback-optimization," represents a technical approach with clear engineering value and application prospects. However, current research in this direction is still insufficient.

[0004] In summary, how to propose a DIBR composite view content optimization method with distortion heterogeneous perception and quality regularization has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization, which can integrate multi-scale difference modeling of pixel intensity, edge structure and high-frequency texture, generate continuous distortion intensity distribution maps to quantify the spatial distribution characteristics of heterogeneous distortion in the DIBR synthesized view, and combine an image inpainting network that integrates residual structure and attention mechanism with a comprehensive training loss that includes consistency constraints and dual no-reference quality evaluation perception regularization to achieve distortion repair of the DIBR synthesized view, thereby improving the pixel fidelity, structural consistency and subjective visual quality of the DIBR synthesized view.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization includes:

[0008] S1: Obtain the DIBR composite view used as the training sample and its corresponding reference view;

[0009] S2: Based on the DIBR composite view and reference view, multi-scale difference modeling is performed from the pixel dimension, edge dimension and texture dimension to generate a continuous distortion intensity distribution map;

[0010] S3: Input the DIBR composite view into the image restoration model and output the corresponding optimized image;

[0011] S4: Based on the optimized image and the corresponding continuous distortion intensity distribution map and reference view, calculate the comprehensive training loss including consistency constraints and perceptual quality regularization constraints;

[0012] S5: Optimize the learnable parameters of the image inpainting model based on comprehensive training loss;

[0013] S6: Repeat steps S1 to S5 to iteratively train the image restoration model until the model converges or reaches the maximum number of iterations, and obtain the trained image restoration model.

[0014] S7: Input the DIBR synthesized view to be optimized into the trained image inpainting model, and output the corresponding optimized image.

[0015] Preferably, step S2 specifically includes the following processing steps:

[0016] S201: Obtain DIBR Composite View and reference view ;

[0017] S202: Through Norm calculation DIBR composite view and reference view The offset in color or brightness yields a pixel intensity difference map. ;

[0018] The calculation formula is:

[0019] ;

[0020] In the formula: Represents pixel intensity difference map In pixel coordinates The pixel intensity difference value at that location; , These represent DIBR composite views. and reference view In pixel coordinates Pixel intensity value at;

[0021] S203: Calculate the DIBR composite view using the Sobel operator and reference view The gradient difference map is obtained by analyzing the edge fractures and structural misalignments. ;

[0022] The calculation formula is:

[0023] ;

[0024] In the formula: Gradient difference plot In pixel coordinates The gradient difference value at that location; , They represent , Sobel gradient operator in the direction; , DIBR composite view (separate tables) In pixel coordinates along and Gradient of direction; , These represent reference views. In pixel coordinates along and Gradient of direction;

[0025] S204: Calculate the DIBR composite view using the Laplace operator and reference view The high-frequency difference map is obtained by analyzing the texture loss and high-frequency information changes caused by geometric faults. ;

[0026] High frequency differences The calculation formula is:

[0027] ;

[0028] In the formula: High-frequency difference graph In pixel coordinates High-frequency difference values ​​at the location; Represents the Laplace operator; , These represent DIBR composite views. and reference view In pixel coordinates High-frequency information at the location;

[0029] S205: Pixel intensity difference map Gradient difference plot High-frequency difference graph Weighted fusion is performed to obtain the initial distortion intensity map. ;

[0030] The calculation formula is:

[0031] ;

[0032] In the formula: Represents the initial distortion intensity map In pixel coordinates The initial distortion intensity value at the location; , , They are respectively , , The weighting coefficients, and satisfying ;

[0033] S206: Robustly normalize the initial distortion intensity map to obtain the continuous distortion intensity distribution map. ;

[0034] The calculation formula is:

[0035] ;

[0036] In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; This represents the 99th percentile; min(∙) is the minimum value function.

[0037] Preferably, in step S3, the image restoration model processing steps include:

[0038] S301: Use the DIBR composite view as input to the image restoration model;

[0039] S302: The encoder extracts and downsamples the DIBR synthesized view layer by layer to obtain a multi-scale deep semantic feature map;

[0040] S303: The image resolution of the multi-scale deep semantic feature map is gradually restored through the upsampling operation of the decoder, and the shallow texture features of the encoder and the deep semantic features of the decoder are fused by skip connections to obtain the preliminary restored feature map.

[0041] S304: Input the initially recovered feature map into the residual branch, obtain the three-channel repaired residual through convolution mapping, and predict the repaired residual map. ;

[0042] S305: Input the initially recovered feature map into the attention branch, obtain the single-channel attention response through convolution mapping, and map it to [0,1] through the Sigmoid function to predict the attention map. ;

[0043] S306: Combine DIBR composite view with repaired residual map Attention map The images are fused to obtain an optimized image. ;

[0044] The calculation formula is:

[0045] ;

[0046] In the formula: Indicates optimized image In pixel coordinates Pixel value at; , These represent the repaired residual map and attention map at pixel coordinates, respectively. The residual value and attention response value at the location; This indicates pixel-by-pixel multiplication.

[0047] Preferably, in step S4, the comprehensive training loss includes reconstruction and structural consistency constraint loss, low-frequency context consistency constraint, attention mechanism related loss, attention smoothing regularization loss, and double-no-reference quality evaluation perceptual quality regularization loss.

[0048] The calculation formula is:

[0049] ;

[0050] In the formula: Indicates overall training loss; , , , , These represent the reconstruction and structural consistency constraint loss, low-frequency context consistency constraint loss, attention mechanism related loss, attention smoothing regularization loss, and double-no-reference quality evaluation perceived quality regularization loss, respectively. , , , These represent the weight parameters for the corresponding loss terms.

[0051] Preferably, in step S4, the processing steps for calculating the reconstruction and structural consistency constraint loss include:

[0052] S401: Construct pixel weights using a continuous distortion intensity distribution map;

[0053] The formula is expressed as:

[0054] ;

[0055] In the formula: Represents pixel coordinates Pixel weights at the location; Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; A factor indicating the degree to which the weights of distorted regions are adjusted;

[0056] S402: Calculate the weighted reconstruction loss of the distorted region based on pixel weights;

[0057] The formula is expressed as:

[0058] ;

[0059] In the formula: Indicates the weighted reconstruction loss of the distorted area; This represents the total number of pixels in the image; {R,G,B} represents the color channel; , These represent optimized images. and reference view In pixel coordinates place The pixel values ​​of the color channels; where p represents the pixel position index in the image, and N=H×W represents the total number of pixel positions involved in the loss calculation;

[0060] S403: Calculate gradient consistency loss based on pixel weights;

[0061] The formula is expressed as:

[0062] ;

[0063] In the formula: , These represent the Sobel horizontal and vertical gradient operators, respectively. , These represent the optimized image and the reference view in pixel coordinates, respectively. The brightness component at that location;

[0064] S404: Calculate Laplacian consistency loss based on pixel weights;

[0065] The formula is expressed as:

[0066] ;

[0067] In the formula: Δ represents the effect applied to the luminance component. The Laplacian operator is used to constrain the consistency of local high-frequency textures and fault structures;

[0068] S405: The weighted reconstruction loss, gradient consistency loss and Laplace consistency loss of the distorted region are calculated to obtain the weighted reconstruction and structural consistency constraint loss of the distorted region.

[0069] The formula is expressed as:

[0070] ;

[0071] In the formula: , These are the weight coefficients for gradient consistency loss and Laplace consistency loss, respectively.

[0072] Preferably, in step S4, the low-frequency context consistency constraint is calculated using the following formula:

[0073] ;

[0074] In the formula: This represents the Gaussian low-pass operator; Represents the L1 norm; This indicates the number of elements involved in the calculation in the low-frequency difference tensor.

[0075] Preferably, in step S4, the attention mechanism-related loss is calculated using the following formula:

[0076] ;

[0077] In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; This indicates the attention map at pixel coordinates. Attention response value at the location.

[0078] Preferably, in step S4, the attention smoothing regularization loss is calculated using the following formula:

[0079] ;

[0080] In the formula: , These represent the number of adjacent pixel difference terms in the horizontal and vertical directions, respectively. Indicating attention graph Attention response value at the adjacent position on the right. Indicating attention graph Attention response value at the adjacent position below.

[0081] Preferably, in step S4, the optimized image is evaluated using two no-reference quality assessment models, the CONTIN-SIQA module and the LGGS-SIQA module, respectively, and the corresponding quality scores are output. and ;

[0082] The perceived quality regularization loss for dual-no-reference quality assessment is calculated using the following formula:

[0083] ;

[0084] In the formula: The preset quality improvement interval; This indicates the number of no-reference quality assessment models; For the first The normalized quality score output by a no-reference quality assessment model after adaptation. The values ​​are {1,2}, which correspond to the CONTIN-SIQA model and the LGGS-SIQA model, respectively. This is a function to find the maximum value. This represents the input image.

[0085] Compared with existing technologies, the DIBR synthesized view optimization method based on distortion heterogeneous sensing and quality regularization in this invention has the following advantages:

[0086] Current research on content optimization and quality restoration of DIBR composite views is still relatively scarce. Existing studies mostly focus on general image restoration, lacking systematic solutions specifically for the unique distortion problems caused by DIBR rendering, such as irregular holes, edge distortion, and artifacts. This invention addresses the inherent distortion characteristics of DIBR composite viewpoints by constructing a restoration and optimization framework driven by perceptual quality, filling the technical gap in dedicated optimization methods for this scenario and demonstrating good engineering applicability and practical feasibility.

[0087] This invention fully considers the unique characteristics of DIBR composite views, such as large hole areas and uneven distortion distribution, and breaks through the limitations of traditional single-dimensional constraints. It performs multi-scale difference modeling from pixel, edge, and texture dimensions, fits and generates a continuous distortion intensity distribution map; it can accurately quantify the distortion severity of different regions, providing a basis for subsequent regional differentiated repair, and avoiding the problems of insufficient repair or over-smoothing caused by traditional global uniform repair.

[0088] This invention designs an image inpainting network that integrates residual structure and attention mechanism. The residual structure can alleviate the gradient degradation problem of deep networks and ensure the effective transmission of deep features. The attention mechanism can automatically focus on the key distortion areas of DIBR and strengthen the contextual feature association modeling, ensuring the feature extraction capability of the network structure and the adaptive focusing capability of the distortion area, and greatly improving the restoration effect of large-scale hole areas.

[0089] This invention combines multi-dimensional constraints to jointly supervise network training, introducing distorted region weighted reconstruction, structural consistency constraints, low-frequency contextual consistency constraints, attention-supervised loss, and attention smoothing regularization. It also incorporates two no-reference quality evaluation models as perceptual regularization constraints. This ensures the continuous stability of image pixels, edge structures, low-frequency color distribution, and attention response, while also constraining the restoration quality from the perspective of human subjective perception. It effectively improves problems such as boundary abrupt changes, inconsistent color transitions, and texture artifacts, so that the restoration results simultaneously meet both objective indicators and subjective visual perception. Attached Figure Description

[0090] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0091] Figure 1 This is a network framework diagram for the DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization.

[0092] Figure 2The images are visualizations comparing the images before and after optimization with the reference images on the IRCCyN / IVC database test set. (a), (d), and (g) are the input images before optimization, (b), (e), and (h) are the result images after optimization, and (c), (f), and (i) are the corresponding reference images.

[0093] Figure 3 Visualizations of the optimization results on the IETR database: (a) and (d) are the input images before optimization, (b) and (e) are the result images after optimization, and (c) and (f) are the corresponding distortion interest region masks.

[0094] Figure 4 Visualization of the optimization results on the MCL-3D database: (a) and (d) are the input images before optimization, (b) and (e) are the result images after optimization, and (c) and (f) are the corresponding distortion interest region masks. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0096] The following detailed explanation illustrates the specific implementation methods:

[0097] Example:

[0098] This embodiment discloses a DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization.

[0099] like Figure 1 As shown, the DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization includes:

[0100] S1: Obtain the DIBR composite view used as the training sample and its corresponding reference view;

[0101] S2: Based on the DIBR composite view and reference view, multi-scale difference modeling is performed from the pixel dimension, edge dimension and texture dimension to generate a continuous distortion intensity distribution map;

[0102] S3: Input the DIBR composite view into the image restoration model and output the corresponding optimized image;

[0103] S4: Based on the optimized image and the corresponding continuous distortion intensity distribution map and reference view, calculate the comprehensive training loss including consistency constraints and perceptual quality regularization constraints;

[0104] S5: Optimize the learnable parameters of the image inpainting model based on comprehensive training loss;

[0105] S6: Repeat steps S1 to S5 to iteratively train the image restoration model until the model converges or reaches the maximum number of iterations, and obtain the trained image restoration model.

[0106] S7: Input the DIBR synthesized view to be optimized into the trained image inpainting model, and output the corresponding optimized image.

[0107] This invention proposes a content optimization scheme for DIBR synthesized views. The core of this scheme is to improve the quality of the synthesized view through multi-stage collaboration: First, the DIBR synthesized view to be optimized provides the processing object for the entire optimization process. Second, the multi-scale difference modeling and continuous distortion intensity distribution map constructed during model training accurately characterize the spatial heterogeneous distribution features of DIBR synthesis distortion, solving the problem of traditional methods struggling to locate sparse and irregular distortion regions, thus providing a basis for subsequent restoration. Third, the image inpainting network, which integrates residual structure and attention mechanisms, can learn regional differential restoration capabilities under the supervision of distortion distribution during training, effectively avoiding the technical problem of excessive smoothing in non-distortion regions and ensuring the structural integrity of the initially optimized image. Fourth, regularization constraints, combined with multi-dimensional constraints and a quality evaluation model, serve as perceptual regularization, effectively compensating for the mismatch between traditional pixel-level and structural-level losses and human subjective perception quality, achieving a balance between structural consistency and subjective quality in the restoration results. Fifth, the simplified design in the testing phase enhances the practicality and operability of this method, making it easier to promote and apply in practical engineering.

[0108] I. Continuous Distortion Intensity Distribution Map

[0109] In the specific implementation process, given an input DIBR distorted image I d and reference view I r (Available only during training), this invention constructs a local distortion metric from three complementary dimensions to characterize different types of distortion features: pixel intensity difference ( This reflects color / brightness shift; gradient differences ( Highlighting edge fractures and structural misalignment; high-frequency differences ( It is used to characterize high-frequency information changes caused by texture loss and geometric faults.

[0110] The above three types of differences are fused to form a continuous distortion intensity map D (D∈[0,1]). The larger the value in this intensity map, the more severe the distortion at the corresponding location. Its core function is to expand the representation of DIBR distortion regions from the traditional coarse-grained binary mask to a fine-grained continuous value representation, thereby more accurately depicting the degree of distortion at different locations in the image and providing fine and reliable guidance information for subsequent restoration operations.

[0111] Specifically, the processing steps for constructing a continuous distortion intensity distribution map include:

[0112] S201: Obtain DIBR Composite View and reference view ;

[0113] S202: Through Norm calculation DIBR composite view and reference view The offset in color or brightness yields a pixel intensity difference map. ;

[0114] The calculation formula is:

[0115] ;

[0116] In the formula: Represents pixel intensity difference map In pixel coordinates The pixel intensity difference value at that location; , These represent DIBR composite views. and reference view In pixel coordinates Pixel intensity value at; These are the normalization coefficients;

[0117] S203: Calculate the DIBR composite view using the Sobel operator and reference view The gradient difference map is obtained by analyzing the edge fractures and structural misalignments. ;

[0118] The calculation formula is:

[0119] ;

[0120] In the formula: Gradient difference plot In pixel coordinates The gradient difference value at that location; , They represent , Sobel gradient operator in the direction; , DIBR composite view (separate tables) In pixel coordinates along and Gradient of direction; , These represent reference views. In pixel coordinates along and Gradient of direction; These are the normalization coefficients;

[0121] S204: Calculate the DIBR composite view using the Laplace operator and reference view The high-frequency difference map is obtained by analyzing the texture loss and high-frequency information changes caused by geometric faults. ;

[0122] High frequency differences The calculation formula is:

[0123] ;

[0124] In the formula: High-frequency difference graph In pixel coordinates High-frequency difference values ​​at the location; Represents the Laplace operator; , These represent DIBR composite views. and reference view In pixel coordinates High-frequency information at the location; This is a normalization coefficient used to ensure that all three types of difference values ​​fall within the range of [0,1], achieving alignment in both dimension and numerical range.

[0125] S205: Pixel intensity difference map Gradient difference plot High-frequency difference graph Weighted fusion is performed to obtain the initial distortion intensity map. ;

[0126] The calculation formula is:

[0127] ;

[0128] In the formula: Represents the initial distortion intensity map In pixel coordinates The initial distortion intensity value at the location; , , They are respectively , , The weighting coefficients, and satisfying The values ​​are 0.3, 0.45, and 0.25, used to adjust the contribution ratio of the three types of differences in the initial distortion intensity.

[0129] S206: Robustly normalize the initial distortion intensity map to obtain the continuous distortion intensity distribution map. ;

[0130] The calculation formula is:

[0131] ;

[0132] In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; The 99th percentile is used to suppress the influence of extreme differences on the overall scale, and min(∙) is the minimum value function.

[0133] II. Image Restoration Model

[0134] In practice, the DIBR synthesized view serves as the actual input to the image inpainting model that integrates the residual structure and attention mechanism. The reference view and continuous distortion intensity distribution map are only used during the training phase to construct supervision signals and calculate the comprehensive training loss, and are not input into the encoder. This image inpainting model employs an encoder-decoder (U-Net) as the backbone network to extract multi-scale contextual information, adapting to the inpainting requirements of DIBR synthesized images. The encoder expands the receptive field through layer-by-layer downsampling, thereby capturing a wide range of contextual features of the image. The decoder restores image details through upsampling operations and skip connections, and integrates the shallow texture features extracted by the encoder. This U-Net structure is crucial for DIBR image inpainting because hole filling relies on the overall consistency of the surrounding color and structure, rather than simple local pixel copying.

[0135] To avoid oversmoothing caused by processing the entire image with the same intensity, this invention designs a dual-branch prediction structure at the network output: the residual branch is used to predict and repair the residual R, and the attention branch predicts the attention map A (A∈[0,1]).

[0136] Specifically, the processing steps of the image restoration model include:

[0137] S301: Use the DIBR composite view as input to the image restoration model;

[0138] S302: The encoder extracts and downsamples the DIBR synthesized view layer by layer to obtain a multi-scale deep semantic feature map;

[0139] It should be noted that during the training phase, a continuous distortion intensity distribution map is constructed based on the DIBR synthesized view and the reference view. This distribution map is used to construct the pixel weights of the distortion region and supervise the attention branch. The reference view is used to calculate the supervision loss such as reconstruction and structural consistency. Neither of them is used as the input to the encoder. Figure 1 The connection between the continuous distortion intensity distribution map and the inpainting network represents the training supervision relationship, not the input connection during the inference stage. During the inference stage, only the DIBR synthesized view to be optimized needs to be input into the trained image inpainting model, which internally predicts the inpainting residual map and attention map and generates the optimized image.

[0140] S303: The image resolution of the multi-scale deep semantic feature map is gradually restored through the upsampling operation of the decoder, and the shallow texture features of the encoder and the deep semantic features of the decoder are fused by the SkipConnection to obtain the preliminary restored feature map.

[0141] S304: Input the initially recovered feature map into the residual branch, obtain the three-channel repaired residual through convolution mapping, and predict the repaired residual map. ;

[0142] S305: Input the initially recovered feature map into the attention branch, obtain the single-channel attention response through convolution mapping, and map it to [0,1] through the Sigmoid function to predict the attention map. ;

[0143] S306: Fuse the DIBR composite view with the repaired residuals and attention map to obtain the optimized image. ;

[0144] The calculation formula is:

[0145] ;

[0146] In the formula: Indicates optimized image In pixel coordinates Pixel value at; , These represent the repaired residual map and attention map at pixel coordinates, respectively. The residual value and attention response value at the location; This represents pixel-wise multiplication. This fusion mechanism allows the attention map and residual branch to collaboratively participate in the final output generation. The mechanism is intuitive: for regions with minimal or no distortion, the network learns a smaller attention response, making the output as close as possible to the input image. This reduces unnecessary modifications to the original details and texture information; for high-distortion areas such as faults and holes, the network generates a large attention response, allowing residual compensation to play a full role and enhancing the local repair effect.

[0147] III. Loss Function

[0148] In practice, the above-mentioned structural design effectively represents and specifically repairs heterogeneous distortions in DIBR-synthesized images. However, relying on this network structure, only preliminary optimization of the repair effect can be achieved, and it is still difficult to simultaneously meet the core requirements of image structural continuity, natural color transitions, and subjective visual quality. Therefore, this invention will start from the loss function and introduce a quality-aware-driven multi-constraint loss design to more specifically constrain and guide network training, thereby obtaining more reliable repair results.

[0149] Specifically, the integrated training loss includes reconstruction and structural consistency constraint loss, low-frequency context consistency constraint, attention mechanism related loss, attention smoothing regularization loss, and double-no-reference quality evaluation perceptual quality regularization loss.

[0150] 1. Weighted reconstruction of distorted regions and structural consistency constraint loss

[0151] Since distorted regions in DIBR composite views are typically sparse and irregularly distributed, applying the same weights to all pixels can easily lead to the training loss being dominated by large, normal regions, resulting in insufficient repair of highly distorted areas such as faults, holes, and edge misalignments. Therefore, this invention designs a distorted region-weighted reconstruction and structural consistency constraint loss.

[0152] Specifically, the steps for calculating the weighted reconstruction of distorted regions and processing the structural consistency constraint loss include:

[0153] S401: Construct pixel weights using a continuous distortion intensity distribution map;

[0154] The formula is expressed as:

[0155] ;

[0156] In the formula: Represents pixel coordinates Pixel weights at the location; Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; This factor represents the degree to which the weights of distorted regions are adjusted. A larger value indicates a higher weight of highly distorted regions in the loss function, and a stronger focus of the network on repairing distorted regions. The value is 6.0. Adjust pixel weight only The spatial distribution does not replace the λ weights of each loss term in the total loss.

[0157] S402: Calculate the weighted reconstruction loss of the distorted region based on pixel weights;

[0158] The formula is expressed as:

[0159] ;

[0160] In the formula: Indicates the weighted reconstruction loss of the distorted area; This represents the total number of pixels in the image; {R,G,B} represents the color channel; , These represent optimized images. and reference view In pixel coordinates place Pixel values ​​of color channels; This is used to average the absolute errors of the RGB three channels and all pixel positions. Here, p represents the pixel position index in the image, and N = H × W represents the total number of pixel positions involved in the loss calculation.

[0161] While weighted reconstruction loss alone can constrain the consistency of pixel intensity between the restored image and the reference image, it is insufficient for addressing common issues in DIBR composite views such as edge breakage, contour misalignment, and local high-frequency degradation. Therefore, this invention further introduces gradient consistency loss and Laplacian consistency loss to enhance the consistency of the restored results in terms of edge structure and high-frequency details.

[0162] S403: Calculate gradient consistency loss based on pixel weights;

[0163] The formula is expressed as:

[0164] ;

[0165] In the formula: , These represent the Sobel horizontal and vertical gradient operators, respectively. , These represent the optimized image and the reference view in pixel coordinates, respectively. The brightness component at the location; the absolute difference of the gradient in the two directions is added together to constrain the consistency between the restored image and the reference image in terms of edge direction and structural contour.

[0166] S404: Calculate Laplacian consistency loss based on pixel weights;

[0167] The formula is expressed as:

[0168] ;

[0169] In the formula: Δ represents the effect applied to the luminance component. The Laplacian operator is used to constrain the consistency of local high-frequency textures and fault structures;

[0170] S405: The weighted reconstruction loss, gradient consistency loss and Laplace consistency loss of the distorted region are calculated to obtain the weighted reconstruction and structural consistency constraint loss of the distorted region.

[0171] The formula is expressed as:

[0172] ;

[0173] In the formula: , These are the weight coefficients for gradient consistency loss and Laplacian consistency loss, respectively. =0.35, =0.30.

[0174] 2. Low-frequency context consistency constraints

[0175] DIBR hole regions typically exhibit large-area defects, lacking directly referable texture and color information. Relying solely on pixel-level and gradient-level constraints can easily lead to color blocks, abrupt boundary changes, or inconsistencies between the filled region and its surrounding color within the holes. To enhance the overall consistency of color distribution, this invention introduces low-frequency contextual consistency constraints. Low-frequency components of the image are extracted using a low-pass operator, and consistency constraints are constructed in the low-frequency space, ensuring that the output image matches the reference image in the low-frequency domain.

[0176] Specifically, low-frequency context consistency constraints are calculated using the following formula:

[0177] ;

[0178] In the formula: This represents a Gaussian low-pass operator. In this embodiment, an 11×11 Gaussian kernel and standard deviation are used. , used to extract low-frequency components of an image; Represents the L1 norm; This indicates the number of elements involved in the calculation in the low-frequency difference tensor. This term is used to calculate the average pixel-wise absolute error between two low-frequency component images.

[0179] In this invention, low-frequency contextual consistency constraints emphasize the matching of large-scale brightness and color distribution, which helps to promote the natural propagation of color information from the periphery of the hole to the interior, thereby improving the problem of unnatural color transition in large hole areas.

[0180] 3. Loss related to attention mechanisms

[0181] To ensure that the attention branch truly focuses on the DIBR distortion region, this method uses the continuous distortion intensity map D as the attention supervision signal and applies a binary cross-entropy constraint to the attention map A.

[0182] Specifically, the loss related to the attention mechanism is calculated using the following formula:

[0183] ;

[0184] In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; This indicates the attention map at pixel coordinates. Attention response value at the location.

[0185] In this invention, the attention mechanism-related loss enables the predicted attention map to maintain consistency with the distortion intensity distribution, thereby improving the ability to locate the repaired region.

[0186] 4. Attention smoothing regularization loss

[0187] To avoid noisy or fragmented responses in the attention map, attention smoothing regularization is further introduced.

[0188] Specifically, the attention smoothing regularization loss is calculated using the following formula:

[0189] ;

[0190] In the formula: , These represent the number of adjacent pixel difference terms in the horizontal and vertical directions, respectively. Indicating attention graph Attention response value at the adjacent position on the right. Indicating attention graph Attention response value at the adjacent position below.

[0191] In this invention, the attention smoothing regularization loss makes the attention response of adjacent pixels smoother, which is beneficial to forming a continuous and stable repair region.

[0192] 5. Perceived quality regularization loss in dual-reference quality assessment

[0193] While reference supervision and structural consistency can improve objective metrics, the subjective quality of DIBR synthetic views often deviates from objective metrics. To explicitly introduce optimization objectives that conform to human visual perception, this invention introduces two no-reference quality assessment models for DIBR synthetic views: CONTIN-SIQA (from 2026 IEEE Transactions on Artificial Intelligence (TAI)) and LGGS-SIQA (2025 Displays), which output quality scores Q1 and Q2, respectively. During training, the parameters of the two no-reference quality assessment networks are frozen and used only as quality evaluators for regularization. To avoid training instability caused by directly maximizing scores, this invention employs a ranking loss.

[0194] Specifically, the optimized image is evaluated using two no-reference quality assessment models: the CONTIN-SIQA module and the LGGS-SIQA module, respectively, and the corresponding quality scores are output. and ;

[0195] The perceived quality regularization loss for dual-no-reference quality assessment is calculated using the following formula:

[0196] ;

[0197] In the formula: This is a preset quality improvement interval, used to control the minimum improvement in the quality of the repaired image relative to the input image. =0.03; This represents the number of no-reference quality assessment models, in this embodiment. =2; For the first The normalized quality score output by a no-reference quality assessment model after adaptation has a value range of [0,1], and a higher score indicates higher quality. The values ​​are {1,2}, which correspond to the CONTIN-SIQA model and the LGGS-SIQA model, respectively. This is a function for maximizing the value, used to implement constraints in the form of hinge loss; This represents the input image, i.e., the DIBR composite view to be optimized.

[0198] In summary, the formula for calculating the overall training loss is:

[0199] ;

[0200] In the formula: Indicates overall training loss; , , , , These represent the reconstruction and structural consistency constraint loss, low-frequency context consistency constraint loss, attention mechanism related loss, attention smoothing regularization loss, and double-no-reference quality evaluation perceived quality regularization loss, respectively. , , , These represent the weight parameters for the corresponding loss term. =0.10, =0.25, =0.05, =0.10.

[0201] This invention, through its multi-constraint collaborative optimization approach, can simultaneously consider pixel fidelity, structural continuity, low-frequency color naturalness, attention localization stability, and subjective perceived quality. Compared to methods relying solely on a single reconstruction loss or ordinary image enhancement networks, it is more suitable for handling spatially uneven and complex distortions in DIBR synthesized views. Furthermore, the quality ranking constraint of the dual-no-reference quality evaluation perceptual quality regularization loss has a relatively intuitive physical meaning: when the restored image already surpasses the input image in quality score and reaches a preset improvement margin, this loss term is 0, no longer driving the network to produce unnecessary changes and avoiding over-optimization; when the restored image has not yet reached the required quality score to surpass the input image, this term generates a gradient, guiding the network to improve factors affecting subjective quality such as tortuosity, texture distortion, and color inconsistency, thereby improving the human visual perception of the restored image.

[0202] IV. Experimental Instructions

[0203] To verify the effectiveness of the technical solution disclosed in this invention, experimental verification and result analysis were conducted on the proposed DIBR synthesized view content optimization method based on distortion heterogeneous perception and quality regularization. First, PSNR and SSIM were used to quantitatively evaluate the image quality before and after optimization to analyze the improvement in pixel fidelity and structural similarity. Second, module ablation and loss function ablation experiments were used to analyze the contribution of each component to performance improvement. Finally, combined with visualization results, the performance of the method in terms of edge continuity, hole mitigation, and texture naturalness was analyzed from a subjective visual perspective.

[0204] 1. Experimental details:

[0205] This experiment was implemented based on the PyTorch framework. Both training and testing were completed on a computing platform equipped with a 12th generation Intel Core i7-12700K processor (3.60 GHz, 12 cores), 64 GB of memory, and an NVIDIA RTX 3090 GPU (24 GB of video memory).

[0206] In each training iteration, the model first uses the input I... d With reference view I r A continuous distortion intensity map D is constructed, and then only the input image is fed into the distortion-aware inpainting network for forward propagation to obtain the inpainting result I. out Attention map A and residual R are used. Based on this, distortion-weighted reconstruction and structural consistency loss, low-frequency context consistency loss, attention supervision loss, attention smoothing regularization, and quality ranking loss based on dual no-reference quality assessment are calculated, and their weighted sum is used as the overall optimization objective. During backpropagation, only the parameters of the repair network are updated; the parameters of the two no-reference quality assessment models are kept frozen and used only to output quality scores and participate in loss calculation. This training strategy can stably introduce subjective quality information into the optimization process of the repair network without changing the parameters of the IQA model.

[0207] In terms of optimization settings, this experiment uses the Adam optimizer with an initial learning rate of 2×10⁻⁶. −4 The batch size is set to 4, the number of training rounds is 500, the training patch size is 384×384, and the random seed is set to 42.

[0208] 2. Method validity verification:

[0209] To verify the effectiveness of the proposed optimization method, the optimization performance was quantitatively evaluated by comparing the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the virtual viewpoint synthesized images before and after optimization. The result before optimization refers to the synthesized image directly generated using the basic rendering algorithm, while the result after optimization is the restored image processed by the method described in this chapter. Experiments were conducted on the IRCCyN / IVC virtual viewpoint synthesized image quality database. This database contains 12 sequences (3 scenes × 4 viewpoint positions), each sequence containing one reference image and its corresponding seven distorted synthesized images generated by different rendering algorithms, totaling 12 reference images and 84 distorted images, for a total of 96 images. To avoid images from the same sequence appearing simultaneously in both the training and test sets, this invention constructs the training and test sets by dividing them into sequences. Ten sequences are used as the training set, and the remaining two sequences (Sequence 11 and Sequence 12) are used as the test set to verify the model's restoration performance in unseen scenes.

[0210] Table 1 shows the comparison of PSNR and SSIM indices of the synthesized images before and after optimization for the test set sequences (Sequence 11 and Sequence 12). It can be seen that after optimization using the proposed method, both PSNR and SSIM indices for the two test scenarios are improved. For Sequence 11, PSNR increased from 22.44 dB to 24.75 dB, an improvement of 2.31 dB, and SSIM increased from 0.8044 to 0.8305, an improvement of 0.0261. For Sequence 12, PSNR increased from 22.66 dB to 25.52 dB, an improvement of 2.86 dB, and SSIM increased from 0.8063 to 0.8449, an improvement of 0.0386. Looking at the average performance of the test set, PSNR increased from 22.55 dB to 25.14 dB, an improvement of 2.59 dB; and SSIM increased from 0.8054 to 0.8377, an improvement of 0.0323. This result demonstrates that the optimization method proposed in this invention can improve the pixel-level fidelity and structural consistency of virtual viewpoint synthesized images to a certain extent.

[0211] Table 1 Comparison of PSNR and SSIM metrics before and after test set optimization.

[0212]

[0213] Based on the performance of the two test sequences, Sequence 12 shows a more significant improvement. Preliminary analysis of the test image content suggests that Sequence 12 contains more complex texture regions and geometric edges. The original rendering algorithm is more prone to producing noticeable distortion in such areas, thus the proposed method achieves a greater performance gain in this scene. Sequence 11 shows a slightly smaller improvement, which is related to its relatively simple scene structure and high original rendering quality. However, the optimized metrics still show a steady improvement, indicating that the proposed method maintains stable optimization results under different levels of complexity. Both untrained test sequences achieved relatively stable performance improvements without significant performance degradation, demonstrating that the proposed method has certain transferability and generalization potential in unknown scenarios.

[0214] Figure 2 Visualization results for three typical examples from the IRCCYN / IVC database test set are presented, with key distortion areas marked in red. The comparison shows that before optimization, the image exhibits edge stretching, local misalignment, or structural discontinuities in local structures such as plant outlines, human figures, books, and desktops. After processing with this method, the outline continuity and local structural integrity of the relevant areas are improved, while the main content of non-key distortion areas remains largely stable. This visual result is consistent with the overall improvement trend of PSNR and SSIM in Table 1, indicating that this method can alleviate local DIBR artifacts while maintaining overall image fidelity.

[0215] 3. The impact of different modules on repair performance:

[0216] To verify the function of each module in the proposed restoration model, module ablation experiments were conducted on the IRCCyN / IVC virtual viewpoint synthetic image database. Each ablation model used the same data partitioning method, training strategy, and optimization parameters as the complete model to ensure the comparability of experimental results. PSNR and SSIM were used as reconstruction performance metrics to compare the restoration effects of different module configurations on the test set.

[0217] Specifically, a baseline model containing only the basic encoder-decoder structure was first constructed. Then, a continuous distortion intensity representation module and an attention-guided inpainting mechanism were sequentially introduced onto this baseline to analyze the contribution of each module to the model performance. Experimental results for different models are shown in Table 2.

[0218] Table 2. Impact of different modules on repair performance

[0219]

[0220] As shown in Table 2, compared to the baseline, Model-1, after introducing continuous distortion intensity representation, saw improvements in PSNR and SSIM from 23.07 dB and 0.8102 to 23.35 dB and 0.8213, respectively. This indicates that the module can enhance the model's utilization of distortion region distribution information to a certain extent, thereby improving the overall restoration effect. Building upon this, the addition of the attention-guided restoration mechanism further improved Model-2's PSNR and SSIM to 25.14 dB and 0.8377, respectively, showing an improvement over Model-1. This indicates that the introduction of the attention-guided restoration mechanism further improved the overall restoration effect of the model and helped enhance its ability to restore key distortion regions. In summary, both continuous distortion intensity representation and the attention-guided restoration mechanism have a positive effect on improving model performance, and their combination can more effectively improve the restoration quality of DIBR synthesized views.

[0221] 4. Comparison of model repair performance under different combinations of loss functions:

[0222] To analyze the role of different optimization constraints in the proposed method, this invention conducted loss function ablation experiments based on the complete repair network structure. All experiments used the same data partitioning, training epochs, and optimization parameters as the complete model to ensure the fairness and comparability of the experimental results. The experiments used PSNR and SSIM as reconstruction performance metrics to compare the repair performance of the model on the test set under different combinations of loss functions.

[0223] Specifically, to ensure consistent training conditions for the attention branch, all three experimental groups maintained a fixed attention supervision loss L. att And attention smoothing regularization L tv Loss-1 employs a reconstruction weighted by distorted regions and a structural consistency constraint loss L. ws Loss-2 adds a low-frequency context consistency loss L to Loss-1. low Loss-3 adds a double-no-reference quality assessment perceived quality ranking loss L to Loss-2. q Therefore, the three sets of experiments only changed L ws L low and L q The experimental results of the model on the test set under different combinations of loss functions are shown in Table 3, while other training conditions remain unchanged.

[0224] Table 3 Comparison of model repair performance under different combinations of loss functions

[0225]

[0226] As shown in Table 3, the PSNR and SSIM of the model on the test set both show an upward trend as the loss constraints gradually increase. Compared with Loss-1, which only uses weighted reconstruction and structural consistency constraint loss, Loss-2, which adds low-frequency context consistency constraint, shows an improvement in reconstruction performance. This indicates that the constraint can enhance the model's utilization of low-frequency context information in large hole regions to a certain extent, thereby improving the overall restoration effect. Based on this, Loss-3, which introduces double-no-reference quality assessment perceptual quality regularization, obtained the best result, indicating that introducing quality-related constraints into the existing optimization framework helps to further improve the overall fidelity and structural consistency of the restoration results. Combined with subsequent visualization analysis, it also shows a certain positive effect on the visual restoration of locally distorted regions. In summary, both low-frequency context consistency constraint and double-no-reference quality assessment perceptual quality regularization have a positive effect on improving model performance, and the synergy of the three can achieve better DIBR synthesized view restoration results.

[0227] 5. Cross-database generalization experiment:

[0228] To examine the applicability of the proposed method under different data distributions, in addition to the above-mentioned same-database experiments, this invention conducted cross-database generalization experiments. Specifically, the model trained on the IRCCYN / IVC database was directly tested on the IETR and MCL-3D databases without any additional fine-tuning during the testing process. This experiment examines the transfer capability of the proposed method on unseen database samples, and analyzes its external generalization performance using quantitative indicators and visualization results. Figure 3A visualization of the optimization results on the IETR database. Figure 4 A visualization of the optimization results on the MCL-3D database.

[0229] During the test, DIBR synthesized views from two external databases were input into the trained image inpainting model to obtain optimized images and corresponding distortion region of interest maps. Without updating the model parameters, the PSNR and SSIM before and after optimization were calculated using reference images from each database, and typical examples were selected for visualization comparison.

[0230] Table 4. Comparison of SSIM and PSNR of images before and after optimization in cross-database tests.

[0231]

[0232] The evaluation metrics used are PSNR and SSIM, which measure the improvement in pixel fidelity and structural similarity, respectively. Furthermore, before-and-after optimization images and corresponding mask images of typical examples are presented to analyze the distortion-related regions of interest and recovery trends of the model on an external database.

[0233] Table 4 presents the cross-database test results of the proposed method on the IETR and MCL-3D databases. The results in the table are the average PSNR and SSIM of the entire sample in the corresponding test set.

[0234] Combine Table 4 and Figure 3 As can be seen, on the IETR database, the SSIM of the optimized image increased from 0.8060 to 0.8707, and the PSNR increased from 26.6475 dB to 27.3969 dB. Figure 3 In the example shown, the red box area mainly corresponds to the image boundary, the head of the person, and the outline of the person, which are prone to geometric misalignment. After optimization, the edge breakage and local misalignment in the relevant areas are alleviated, and the continuity of the outline is improved. Figure 3 (c) and Figure 3 The distortion concern region in (f) corresponds highly to the aforementioned easily distorted structures, indicating that the model can target key distortion locations.

[0235] On the MCL-3D database, the optimized image's SSIM improved from 0.8085 to 0.8695, and PSNR improved from 30.4722 dB to 30.9872 dB. Figure 4 It is evident that the balloon outline and thin lines, the boundaries of figures and foreground objects, and other areas exhibited significant stretching, bending, or structural discontinuity before optimization. After optimization, the relevant local structures became more stable. Figure 4 (c) and Figure 4The distortion concern area shown in (f) also mainly covers the aforementioned complex edges and geometrically varied locations. The quantitative results and visualization phenomena corroborate each other, indicating that this method can still improve overall fidelity and structural consistency under different scene content, depth error distribution, and rendering characteristics, and has a certain cross-database generalization ability.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization, characterized in that, include: S1: Obtain the DIBR composite view used as the training sample and its corresponding reference view; S2: Based on the DIBR composite view and reference view, multi-scale difference modeling is performed from the pixel dimension, edge dimension and texture dimension to generate a continuous distortion intensity distribution map; S3: Input the DIBR composite view into the image restoration model and output the corresponding optimized image; S4: Based on the optimized image and the corresponding continuous distortion intensity distribution map and reference view, calculate the comprehensive training loss including consistency constraints and perceptual quality regularization constraints; S5: Optimize the learnable parameters of the image inpainting model based on comprehensive training loss; S6: Repeat steps S1 to S5 to iteratively train the image restoration model until the model converges or reaches the maximum number of iterations, and obtain the trained image restoration model. S7: Input the DIBR synthesized view to be optimized into the trained image inpainting model, and output the corresponding optimized image.

2. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 1, characterized in that: Step S2 specifically includes the following processing steps: S201: Obtain DIBR Composite View and reference view ; S202: Through Norm calculation DIBR composite view and reference view The offset in color or brightness yields a pixel intensity difference map. ; The calculation formula is: ； In the formula: Represents pixel intensity difference map In pixel coordinates The pixel intensity difference value at that location; , These represent DIBR composite views. and reference view In pixel coordinates Pixel intensity value at; S203: Calculate the DIBR composite view using the Sobel operator and reference view The gradient difference map is obtained by analyzing the edge fractures and structural misalignments. ; The calculation formula is: ； In the formula: Gradient difference plot In pixel coordinates The gradient difference value at that location; , They represent , Sobel gradient operator in the direction; , DIBR composite view (separate tables) In pixel coordinates along and Gradient of direction; , These represent reference views. In pixel coordinates along and Gradient of direction; S204: Calculate the DIBR composite view using the Laplace operator and reference view The high-frequency difference map is obtained by analyzing the texture loss and high-frequency information changes caused by geometric faults. ; High frequency differences The calculation formula is: ； In the formula: High-frequency difference graph In pixel coordinates High-frequency difference values ​​at the location; Represents the Laplace operator; , These represent DIBR composite views. and reference view In pixel coordinates High-frequency information at the location; S205: Pixel intensity difference map Gradient difference plot High-frequency difference graph Weighted fusion is performed to obtain the initial distortion intensity map. ; The calculation formula is: ； In the formula: Represents the initial distortion intensity map In pixel coordinates The initial distortion intensity value at the location; , , They are respectively , , The weighting coefficients, and satisfying ; S206: Robustly normalize the initial distortion intensity map to obtain the continuous distortion intensity distribution map. ; The calculation formula is: ； In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; Represents the 99th percentile; min(∙) is the function that takes the minimum value.

3. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 1, characterized in that: In step S3, the image restoration model processing steps include: S301: Use the DIBR composite view as input to the image restoration model; S302: The encoder extracts and downsamples the DIBR synthesized view layer by layer to obtain a multi-scale deep semantic feature map; S303: The image resolution of the multi-scale deep semantic feature map is gradually restored through the upsampling operation of the decoder, and the shallow texture features of the encoder and the deep semantic features of the decoder are fused by skip connections to obtain the preliminary restored feature map. S304: Input the initially recovered feature map into the residual branch, obtain the three-channel repaired residual through convolution mapping, and predict the repaired residual map. ; S305: Input the initially recovered feature map into the attention branch, obtain the single-channel attention response through convolution mapping, and map it to [0,1] through the Sigmoid function to predict the attention map. ; S306: Combine DIBR composite view with repaired residual map Attention map The images are fused to obtain an optimized image. ; The calculation formula is: ； In the formula: Indicates optimized image In pixel coordinates Pixel value at; , These represent the repaired residual map and attention map at pixel coordinates, respectively. The residual value and attention response value at the location; This indicates pixel-by-pixel multiplication.

4. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 1, characterized in that: In step S4, the comprehensive training loss includes reconstruction and structural consistency constraint loss, low-frequency context consistency constraint, attention mechanism related loss, attention smoothing regularization loss, and double no-reference quality evaluation perceptual quality regularization loss; The calculation formula is: ； In the formula: Indicates overall training loss; , , , , These represent the reconstruction and structural consistency constraint loss, low-frequency context consistency constraint loss, attention mechanism related loss, attention smoothing regularization loss, and double-no-reference quality evaluation perceived quality regularization loss, respectively. , , , These represent the weight parameters for the corresponding loss terms.

5. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 4, characterized in that: Step S4, the steps for calculating the reconstruction and structural consistency constraint loss, include: S401: Construct pixel weights using a continuous distortion intensity distribution map; The formula is expressed as: ； In the formula: Represents pixel coordinates Pixel weights at the location; Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; A factor indicating the degree to which the weights of distorted regions are adjusted; S402: Calculate the weighted reconstruction loss of the distorted region based on pixel weights; The formula is expressed as: ； In the formula: Indicates the weighted reconstruction loss of the distorted area; This represents the total number of pixels in the image; {R,G,B} represents the color channel; , These represent optimized images. and reference view In pixel coordinates place The pixel values ​​of the color channels; where p represents the pixel position index in the image, and N=H×W represents the total number of pixel positions involved in the loss calculation; S403: Calculate gradient consistency loss based on pixel weights; The formula is expressed as: ； In the formula: , These represent the Sobel horizontal and vertical gradient operators, respectively. , These represent the optimized image and the reference view in pixel coordinates, respectively. The brightness component at that location; S404: Calculate Laplacian consistency loss based on pixel weights; The formula is expressed as: ； In the formula: Δ represents the effect applied to the luminance component. The Laplacian operator is used to constrain the consistency of local high-frequency textures and fault structures; S405: The weighted reconstruction loss, gradient consistency loss and Laplace consistency loss of the distorted region are calculated to obtain the weighted reconstruction and structural consistency constraint loss of the distorted region. The formula is expressed as: ； In the formula: , These are the weight coefficients for gradient consistency loss and Laplace consistency loss, respectively.

6. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 4, characterized in that: In step S4, the low-frequency context consistency constraint is calculated using the following formula: ； In the formula: This represents the Gaussian low-pass operator; Represents the L1 norm; This indicates the number of elements involved in the calculation in the low-frequency difference tensor.

7. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 4, characterized in that: In step S4, the loss related to the attention mechanism is calculated using the following formula: ； In the formula: Represents the distribution of continuous distortion intensity. In pixel coordinates The continuous distortion intensity value at the location; This indicates the attention map at pixel coordinates. Attention response value at the location.

8. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 4, characterized in that: In step S4, the attention smoothing regularization loss is calculated using the following formula: ； In the formula: , These represent the number of adjacent pixel difference terms in the horizontal and vertical directions, respectively. Indicating attention graph Attention response value at the adjacent position on the right. Indicating attention graph Attention response value at the adjacent position below.

9. The DIBR synthesized view optimization method based on distortion heterogeneous perception and quality regularization as described in claim 4, characterized in that: In step S4, the optimized image is evaluated using two no-reference quality assessment models, the CONTIN-SIQA module and the LGGS-SIQA module, respectively, and the corresponding quality scores are output. and ; The perceived quality regularization loss for dual-no-reference quality assessment is calculated using the following formula: ； In the formula: The preset quality improvement interval; This indicates the number of no-reference quality assessment models; For the first The normalized quality score output by a no-reference quality assessment model after adaptation. The values ​​are {1,2}, which correspond to the CONTIN-SIQA model and the LGGS-SIQA model, respectively. This is a function to find the maximum value. This represents the input image.

Images