Statistical feature analysis-based high-robustness virtual-real fusion production content compression damage detection method
Through the method based on statistical feature analysis, scene-independent compression damage features are extracted, and the problem of insufficient generalization ability of compression damage detection in virtual scenes is solved, efficient detection in virtual and real fusion scenarios is achieved, and training costs and labeling costs are reduced. And maintains the compatibility and interpretability of the model.
Patent Information
- Application Number
- CN202411006209.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
AI Technical Summary
The existing compression damage detection methods have insufficient generalization capabilities in cross-scenario and cross-model applications, making it difficult to effectively detect the degree of compression damage in virtual scenes, and lack interpretability during training, resulting in a degradation of the model's performance in virtual and real fusion scenarios. .
Using a method based on statistical feature analysis, scene-independent compression damage-related features are extracted, energy distribution and normalization coefficient are calculated through channel-by-channel Gaussian convolution and matrix quadratic analysis, and a lightweight compression damage detection model is constructed, using nature Scenario database trains the model to be suitable for virtual and real fusion scenarios.
It realizes highly robust detection of compression damage in virtual and real fusion scenarios, reduces training costs and labeling costs, maintains model compatibility and interpretability, and can effectively evaluate compression damage in virtual scenes, which is suitable for different depth visions. Model.
Smart Images

Figure CN119011842A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing and quality assessment, and in particular to a highly robust virtual-real fusion video compression damage detection method based on statistical feature analysis. Background Art
[0002] Compression coding technology can effectively reduce the transmission bandwidth and storage resources of video signals. However, the compression coding process uses different step lengths for quantization, which will inevitably introduce block effects, pseudo contours, blur and other damage, which will interfere with the overall quality of the picture to varying degrees and affect the user's viewing experience. Therefore, designing a picture quality prediction system that can automatically evaluate the degree of compression damage of a given video is important for assisting in optimizing the encoding and decoding parameters of the video center and ensuring the viewing experience of the majority of users.
[0003] On the other hand, virtual-reality fusion production seamlessly integrates virtual scenes and real-life scenes through new studio content production equipment and supporting systems such as green screens, LED screens, and rendering servers, providing people with a more novel and better visual experience. At this stage, virtual-reality fusion has been widely used in content production fields such as sports events, news, and entertainment variety shows.
[0004] Therefore, it is necessary to design a compression damage detection system for virtual-reality fusion production content, and expand the compression damage detection method originally oriented to traditional production (or natural scenes) to virtual-reality fusion scenes, so as to respond to the development needs of high-tech video technology and more comprehensively guarantee the viewing experience of the majority of users. However, this expansion of the application scope from natural scenes to virtual-reality fusion scenes faces many challenges:
[0005] First, there is the problem of the generalization ability of compression damage detection methods. Most existing compression damage assessment methods follow the design paradigm of pre-training-end-to-end fine-tuning, that is, using a model that has been pre-trained on other tasks such as semantic recognition as the backbone network, adjusting its feature output head, and then performing end-to-end fine-tuning on the corresponding video compression damage database. This end-to-end fine-tuning training paradigm actually treats the degree of compression damage as an additional semantic category, which lacks interpretability, which restricts its generalization ability across databases.
[0006] Second, there is the problem of differences in the characteristics of virtual scenes and natural scenes. The virtual parts of virtual-reality fusion content are mostly produced through computer rendering. The characteristics of the virtual scene itself (such as texture, smoothness, light distribution, color, etc.) are obviously different from the natural scene captured by the camera. Figure 1As shown. This difference results in that the compression damage degree detection model originally trained for natural scene databases through end-to-end fine-tuning cannot achieve good detection results for virtual scenes. The establishment of a database specifically for virtual-reality fusion content requires huge manpower and material resources, and with the continuous development of virtual-reality fusion production technology, the constructed database cannot fully reflect the characteristics of all virtual content. For example, the traditional virtual-reality fusion material built based on green screens cannot reflect the content characteristics of XR production based on LED large screens, and the materials designed by different engines, plug-ins and rendering methods such as UE4, UE5, Unity are also different.
[0007] Third, the compatibility and robustness of intelligent detection models. With the development of AI intelligence, deep visual models such as object recognition and image segmentation that process semantic information are often used in the production process of virtual-reality fusion. If the damage detection model is designed according to the end-to-end fine-tuning paradigm, the weight parameters originally oriented to tasks such as semantic recognition are gradually transferred to the image quality evaluation task in the process, and the process is irreversible, that is, the trained network no longer has good semantic recognition capabilities. This requires us to establish two backbone networks with different backbone parameters for damage detection and other semantic-related AI tasks respectively. It can be seen that this method does not have good compatibility, and it is difficult to use a unified backbone network to compatibly implement damage detection and other semantic-related intelligent tasks. In addition, in actual applications, the backbone network is often replaced or iterated to improve the accuracy of semantic-related intelligent tasks, such as replacing the VGG model with ResNet, ViT, Swin-Transformer, etc. Even if the model architecture is the same, its pre-trained parameters may be replaced from the Pytorch source to the HuggingFace source. At this time, if the above-mentioned end-to-end fine-tuning method is adopted, the quality evaluation model must be retrained every time the backbone network architecture or pre-training parameters are changed.
[0008] The present invention deeply analyzes the above problems and believes that the root cause of these limitations is that the judgment of compression damage requires quality-related features, while existing deep vision models mostly provide semantic-related features, and existing methods mostly transfer semantic features to quality features through black-box end-to-end fine-tuning methods, but this actually lacks interpretability, resulting in low generalization ability and difficulty in achieving quality damage detection across scenes (natural scenes, virtual scenes) and models (such as the ResNet model given by Pytorch and the ResNet model given by HuggingFace). An extreme case is as follows: if the images with severe compression damage in the training database are mostly fast-running "dogs", and the images with slight compression are mostly relatively static "cats", then after the above end-to-end fine-tuning training, the detection model tends to give lower scores to images belonging to the semantic category of "dog" and higher scores to images belonging to "cat", regardless of whether the actual compression degree of these images is high or low. Summary of the invention
[0009] Combined with the above analysis, the present invention provides a highly robust method for detecting compression damage of virtual-reality fusion content based on statistical feature analysis, which can robustly extract compression damage-related features for a given video image signal, and this feature extraction method is effective for both natural and virtual scenes. Based on this feature extraction method, the present invention uses a compression damage database of traditional natural scenes for training to construct a lightweight compression damage detection model. Since its features are scene-independent, the trained lightweight model achieves good prediction results for virtual-reality fusion content, and does not have obvious cross-scene performance limitations like traditional quality detection models.
[0010] A highly robust method for detecting compression damage of virtual-reality fusion production content based on statistical feature analysis comprises the following steps:
[0011] Step 1: Preprocess each frame of the video to be processed to obtain the scale feature S (i) Set {S (i)} and rescaling features gather According to the rearrangement feature set Calculate the covariance matrix E (i) Set {E (i)};
[0012] Step 2: Obtain the scale feature S through channel-by-channel Gaussian convolution (i) The neighborhood average features are rearranged to obtain the high-dimensional features of the rearranged neighborhood average Through matrix quadratic analysis and The energy difference is obtained by (i);
[0013] Step 3: Analysis by matrix quadratic form and The local deviation degree and local deviation direction of the deviation degree matrix are obtained and symbol matrix Sign (i) , combined with the energy distribution Eng calculated in step 2 (i) , get the normalization coefficient Norm (i) ;
[0014] Step 4: By analyzing the scale feature set {S (i)} in which two adjacent scale features S (i) and S (j) Energy difference, calculate the cross-level energy distribution Distr (i,j) ;
[0015] Step 5: The energy distribution Eng obtained in step 2 (i) , the normalization coefficient Norm obtained in step 3 (i) , the cross-level energy distribution Distr obtained in step 4 (i,j) Perform multi-level feature concatenation to obtain fused feature FT;
[0016] Step 6: Construct a prediction model, use the fusion feature FT obtained in step 5 as input, and use the natural scene training set to train the prediction model to obtain a compression damage degree prediction model;
[0017] Step 7: Input each frame of the video to be detected in the virtual-reality fusion scene into the compression damage degree prediction model to obtain the overall compression damage degree of the virtual-reality fusion video sequence.
[0018] In step 1, each frame of the video to be processed is preprocessed to obtain a set of scale features {S (i)} and the rearranged feature set at each scale Specifically include:
[0019] Each frame of the video to be processed is used as input and sent to the deep visual model to extract the deep visual features of each level. The deep visual features at the i-th level are recorded as the scale feature S (i) , Among them C i ,H i ,W i Respectively represent S (i) The number of channels, height, and width of the scale features at all levels constitute the scale feature set {S (i)};
[0020] The scale feature S (i) According to the sample size N i =Hi W i and dimension C i Rearrange and get the rearranged features The rearranged features at all levels constitute the rearranged feature set of each scale
[0021] In step 2, through matrix quadratic analysis and The energy difference is obtained by (I) , specifically including:
[0022] Through matrix quadratic calculation, we can respectively and Fusion to height H i , Width W i Two-dimensional array of and Where I represents the covariance matrix E (i) The unit matrix is equal in size. diag means taking the diagonal elements, and reshape means rearranging the vector after extracting the diagonal elements to a height of H. i , Width W i A two-dimensional array:
[0023]
[0024]
[0025] make Convolve it with a two-dimensional Gaussian convolution kernel to obtain the energy distribution
[0026] In step 3, through matrix quadratic analysis and The local deviation degree and local deviation direction of the deviation degree matrix are obtained and symbol matrix Sign (i) , combined with the energy distribution Eng calculated in step 2 (i) , get the normalization coefficient Norm (i) , specifically including:
[0027] The degree of deviation between each spatial position and its neighborhood mean is calculated by matrix quadratic calculation, which is recorded as the degree of deviation matrix
[0028]
[0029]
[0030] Then, the symbol matrix defining the degree of deviation Where sign() represents the sign function, that is, less than zero is -1, and greater than zero is +1:
[0031]
[0032] Sign (i) = reshape(Sign' (i) ,[H i ,W i ])
[0033] Finally, by:
[0034]
[0035] Get the normalized coefficient K is a constant.
[0036] In step 4, by analyzing the scale feature set {S (i)} in which two adjacent scale features S (i) and S (j) Energy difference, calculate the cross-level energy distribution Distr (i,j) , specifically including:
[0037] Through matrix quadratic calculation, the scale feature set {S (i)} in which two adjacent scale features S (i) and S (j) The second-order signal strength on all channels at each spatial position is recorded as energy distribution Distr (i) and Distr (j) ,in and Respectively by S (i) and S (j) Rearrange to get:
[0038]
[0039]
[0040] Calculate Distr (j) Relative to Distr (i) The degree of change is recorded as the cross-level energy distribution K is a constant:
[0041]
[0042] The series of features obtained by the present invention through steps 2 to 4 are obtained by analyzing the statistical characteristics of high-dimensional features. These features are independent of the scale feature S extracted by the original deep learning. (i) The distribution of S(i) When the extracted Eng (i) 、Norm (i) 、Distr (i,j) In addition, through experimental analysis, the above two-dimensional features are highly correlated with the degree of compression damage, that is, when the image damage is relatively minor, Eng (i) 、Norm (i) 、Distr (i,j) The contours of the two-dimensional features are similar to the original image content, but when the compression damage is severe, the contours of these two-dimensional features are destroyed, and an obvious irregular image block structure appears. In summary, the extracted features have high robustness and are highly correlated with the degree of compression damage.
[0043] Step 5: The energy distribution Eng obtained in step 2 (i) , the normalization coefficient Norm obtained in step 3 (i) , the cross-level energy distribution Distr obtained in step 4 (i,j) Perform multi-level feature concatenation to obtain fused feature FT;
[0044] Step 6: Construct a prediction model, use the fusion feature FT obtained in step 5 as input, and use the natural scene training set to train the prediction model to obtain a compression damage degree prediction model;
[0045] In step 7, the overall compression damage degree of the virtual-real fusion video sequence is calculated by the following formula:
[0046] The compression damage prediction model outputs the probability α that the compression damage is slight i and the probability of severe compression damage β i ,pass:
[0047]
[0048] Set all frames i The average value is s, and the overall compression damage degree s of the virtual-real fusion video sequence is obtained. i .
[0049] If the overall compression damage degree of the virtual-real fusion video sequence is s i If the overall compression damage degree of the virtual-real fusion video sequence is s i If the value is low, it means that the overall image quality of the video sequence to be tested is acceptable and can provide users with a good viewing experience, and no adjustment is required.
[0050] Compared with the existing quality evaluation model, the present invention has the following advantages:
[0051] 1. In terms of interpretability, traditional methods directly fine-tune deep features for semantic tasks and convert semantic features into compression damage related features in a "black box" manner. This method relies heavily on the comprehensiveness of the quality data set during the training process. If the training samples with mild / severe compression damage have obvious semantic differences, this black box learning paradigm may focus on related semantic features rather than the quality itself, which severely limits its generalization ability and makes it difficult to expand the compression damage detection model from natural scenes to virtual-reality fusion scenes. In contrast, this method explicitly characterizes the compression damage related characteristics in the deep learning framework based on the normalization coefficient, energy distribution, and cross-level energy distribution by analyzing the features of each level of the deep vision model and their statistical characteristics. This characterization method is independent of the semantic information in the deep vision model and is scene-independent. Therefore, the detection model originally trained in natural scenes can be directly transplanted to the virtual-reality fusion scene.
[0052] 2. In terms of compatibility, the existing compression algorithm detection method loses its semantic recognition capability during the end-to-end fine-tuning process, resulting in the inability to simultaneously perform semantic recognition and compression damage detection tasks with a shared backbone network, and the application scenario is single; while in the present invention, the original deep vision model and its weight parameters are not changed during the extraction of quality-related statistical characteristics and subsequent training. Therefore, the compression damage degree of a given content can be accurately evaluated without changing the backbone network parameters for semantic recognition tasks, making it possible to simultaneously predict the semantics and compression damage degree of an image.
[0053] 3. In terms of training cost and annotation cost, the existing methods require an end-to-end training paradigm, so the learnable parameters in the entire backbone network and feature head structure must be trained using sample labels. Large-scale learnable parameters mean that more labeled data is needed to effectively train them. In comparison, the backbone network parameters of this method are fixed and do not require training. The learnable parameters of the subsequent compression damage prediction module are relatively small, and its training cost and annotation cost are significantly lower than those of existing methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 Provide examples of differences in characteristics between virtual and natural scenes;
[0055] Figure 2 It is the overall flow chart of the present invention;
[0056] Figure 3 This is a statistical information extraction flow chart of the present invention, corresponding to steps 1.1 to 1.4 in the embodiment;
[0057] Figure 4 This is a multi-level feature stitching flow chart of the present invention, corresponding to step 1.5 in the embodiment;
[0058] Figure 5 Some screenshots of natural scene source materials constructed in the embodiment and used for training the compression damage prediction model are given;
[0059] Figure 6 A block diagram of the lightweight compression damage prediction model described in step 2.2 of the embodiment is given;
[0060] Figure 7 The following is a diagram showing the compression damage prediction effect of the embodiment on the content of the natural scene after training is completed;
[0061] Figure 8 An intuitive display of the extracted statistical characteristics (energy distribution, cross-level energy distribution) is given. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] The method of the present invention is as follows Figure 2 As shown, specifically including:
[0064] Step 1: Construct a model-independent, scene-independent, and highly robust compression damage feature extraction method: For the video to be processed, it is sent to the deep vision model frame by frame, and the deep vision features of each level that have not been activated by the RELU layer are stored. Based on this, statistical information such as energy distribution, normalization coefficient, and cross-level energy distribution at each scale are extracted, and the compression damage features of the corresponding video frame are obtained through feature splicing.
[0065] Step 1.1: Let the input video frame be x, and let the backbone network pre-trained on the semantic recognition task be f, whose parameters θ remain fixed in the subsequent quality feature extraction process. After x is fed into f, the local semantic features S output at each scale are extracted. (i) =f θ (x; i), where i represents the serial number of different scales, Among them C i ,H i ,W i Represents the semantic features S of the i-th layer respectively (i) The number of channels, height, and width. Figure 3As shown, the operations described in steps 1.2 to 1.4 are performed in sequence to extract each type of statistical information respectively.
[0066] Step 1.2: Extract S (i) Energy distribution (i) The energy distribution is used to characterize the intensity of the change between high-dimensional features at different spatial locations and their neighborhood high-dimensional features.
[0067] Step 1.2.1: Set each scale feature S (i) Rearrange to sample size N i =H i W i , dimension is C i The new feature of Seek Covariance matrix in each dimension
[0068]
[0069] in Indicates finding the mean value in the sample size dimension, x T Represents the transpose of the given matrix x.
[0070] Step 1.2.2: Set each scale feature S (i) Each channel of is convolved with a two-dimensional Gaussian convolution kernel, and then concatenated in the channel dimension to obtain a high-dimensional feature reflecting the neighborhood average, denoted as Through a similar rearrangement method as described in step 1.2.1, the neighborhood average high-dimensional feature after rearrangement is obtained, which is recorded as
[0071] Step 1.2.3: Use the covariance matrix to transform the high-dimensional Use the following formula to merge into height H i , Width W i A two-dimensional image of:
[0072]
[0073]
[0074] Different locations reflect The degree of fluctuation on all channels at a certain spatial position, where I represents and E (i) The purpose of the unit matrix is to prevent the occurrence of singular values when the matrix is inverted. The diag method means taking the diagonal elements. Convert to size H i Wi Then, it is rearranged into a height H i , Width W i of
[0075] Step 1.2.4: Use a method similar to 1.2.3 to convert the high-dimensional Use the following formula to merge into height H i , Width W i A two-dimensional image of:
[0076]
[0077]
[0078] Step 1.2.5: Pass Get a two-dimensional image that integrates all channel fluctuations and reflects high-dimensional features The degree of difference between different positions and their neighborhood mean. Then, a two-dimensional Gaussian convolution kernel is used for convolution to calculate the mean of the degree of difference at each position, which is recorded as the energy distribution Eng. (i) .
[0079] Step 1.3: Extract S (i) The normalization coefficient Norm (i) The normalization coefficient is derived from the simple normalization idea of "subtracting the mean and dividing by the standard deviation", but the innovation of this method is to extend it to high-dimensional space.
[0080] Step 1.3.1: Calculate the degree of deviation between the high-dimensional feature and the average vector of its neighborhood. According to the rearranged signal obtained in step 1.1 and the average signal of the neighborhood after rearrangement The degree of deviation is calculated by the following formula:
[0081]
[0082]
[0083] Similar to steps 1.2.3 and 1.2.4, this method can fuse the high-dimensional deviation degree into a high-dimensional i , Width W i A two-dimensional image.
[0084] Step 1.3.2: Define the coefficient signs at different positions. In traditional two-dimensional images, due to the subtraction of the mean, positive and negative normalized coefficients can be naturally obtained. For high-dimensional signals, the deviations obtained in step 1.2.1 are all positive values, which is not conducive to subsequent analysis. Therefore, the following method is used to define the positive and negative signs of the above deviations: Let the rearrangement feature at position t be s t, and the neighborhood mean of the corresponding position is recorded as f t , if the vector quadratic form (s t -f t ) T (E (i) +I) -1 (f t ) is a positive value, indicating that s t Protruding from f t , the deviation degree of the position is defined as positive; on the contrary, the deviation degree of the position is defined as negative. Therefore, the sign of the deviation degree of each position can be obtained by the following method (i) :
[0085]
[0086] Sign (i) = reshape(Sign' (i) ,[H i ,W i ])
[0087] Where sign() represents the sign function, that is, less than zero is -1, and greater than zero is +1.
[0088] Step 1.3.3: Energy distribution Eng (i) It actually reflects the "variance" of the spatial domain of high-dimensional features, and the degree of deviation It reflects the "deviation" of a given signal from its neighborhood mean, combined with the position symbol information Sign (i) , a two-dimensional, high-dimensional feature that integrates all channel fluctuations can be obtained as follows The normalization coefficient Norm (i) :
[0089]
[0090] The addition, subtraction, multiplication and division in the above steps are all performed pixel by pixel, where K is a constant to prevent the energy coefficient from being too small to cause numerical errors.
[0091] Step 1.4: Extract cross-level energy distribution Distr (i,j) The depth features of the same dimension extracted from two different levels in the deep vision model are respectively denoted as S (i) and S (j) , Since their dimensions are the same, we have: C i =C j ,H i =H j ,W i =W jThe core idea of this step is to use S (i) The covariance matrix of S (j) The energy distribution characteristics in the image are used to model the relationship between deep features at different levels.
[0092] Step 1.4.1: Distribute S by similar operations as described in step 1.2.1 (i) and S (j) Rearrange to and And calculate Covariance matrix in each dimension
[0093] Step 1.4.2: Similar to step 1.2.3, calculate The position of each airspace relative to E (i) Energy distribution characteristics of Distr (i) :
[0094]
[0095] At the same time, E (i) is the covariance matrix, calculate The position of each airspace relative to E (i) Energy distribution characteristics of Distr (j) :
[0096]
[0097] Step 1.4.3: Calculate Distr (j) Relative to Distr (i) The degree of change of is used to describe their mutual relationship and is recorded as the cross-level energy distribution Distr (i,j) :
[0098]
[0099] Where K is a constant to prevent numerical errors from increasing.
[0100] Step 1.5: Feature stitching. Figure 4 As shown, the above features extracted by each level i of the deep vision model are spliced into a whole to reflect the compression damage related features of the image.
[0101] Step 1.5.1: Fusion of statistical information such as energy distribution, normalization coefficient, and cross-level energy distribution at level i. For each type of feature extracted at level i, its dimension is changed from [H i ,W i ]Additionally increase to [1,H i ,Wi ], and concatenate them on the first channel. After this step, the size of the quality-related statistical information obtained on level i is [3,H i ,W i ], much smaller than its original size.
[0102] Step 1.5.2: Let the feature level with the smallest spatial size be L min , for the level i to be spliced, let For the three two-dimensional matrices extracted at level i, they are divided into non-overlapping vertical matrices Horizontal direction The size is [B i ,B i ] image blocks.
[0103] Step 1.5.3: Rearrange each image block into [1, B i B i ] dimension, the entire two-dimensional matrix is rearranged into Then, the three partitions on level i are rearranged to have dimensions of The two-dimensional matrix of is concatenated in the second dimension to obtain Dimensional concatenation matrix.
[0104] Step 1.5.4: Set the dimensions of each level to The feature matrix of is concatenated to get the dimension The fusion feature FT.
[0105] Step 2: Training of compression damage detection model on natural scenes. Using lossless natural scene videos as source sequences, a compression damage dataset is constructed by using different compression coding levels. Combined with the feature extraction method described in step 1, the extracted highly robust compression damage-related features are sent to a learning model that is smaller in scale than the existing deep vision model, and trained using a lightweight learning method.
[0106] Step 2.1: Construct a compressed damage dataset of natural scene content. By collecting open source datasets and shooting by yourself, N lossless materials are constructed. The cost of establishing a natural scene content database is much lower than producing virtual content one by one. Then, using compression coding standards, they are compressed to different degrees to obtain a slightly damaged video sequence set. and severe injury video sequence set
[0107] Step 2.2: Build a compression damage detection model that is lighter than the existing deep vision model. It can only contain a small number of convolutional layers or Transformer encoders. Its final output layer contains two units, which respectively reflect the probability of the content to be tested belonging to "severe compression damage" and "slight compression damage". The original deep vision model is denoted as f θ (·) and the smaller learning model is
[0108] Step 2.3: For the dataset and All video frames in the video are fed into f frame by frame θ (·) Get the features of each level, x represents the input content:
[0109] [S (1) ,...,S (L) ] = f θ (x)
[0110] According to step 1, the statistical characteristics of the extracted features at each level are analyzed, and the statistical characteristics related to quality are spliced and fused:
[0111] [...,Eng (i) ,Norm (i) ,Distr (i,i+1) ,...]=Extract([S (1) ,...,S (L) ])
[0112] FT=Merge([...,Eng (i) ,Norm (i) ,Distr (i,i+1) ,...])
[0113] Extract(·) and Merge(·) represent Figure 3 and Figure 4 Feature extraction and concatenation operations are shown.
[0114] Step 2.4: Feed FT Get quality prediction value Combine the subjective quality score q corresponding to the input image x to the model parameters To optimize:
[0115]
[0116] In the above formula represents the optimized model parameters, CE(·) represents the cross entropy, and for the slightly damaged video sequence set The corresponding one-hot encoding label is [1,0]. The corresponding one-hot encoding label is [0,1]. As can be seen from the above formula, the present invention is model-independent, that is, the difference between the damage degree and its predicted value is only used to optimize the subsequent, smaller-scale model. Parameters in Instead of using the original deep vision model f θ (·) parameters to make any changes.
[0117] The above process traverses the entire collected data set, and after the training is completed, a model-independent and scenario-independent quality detection model is obtained.
[0118] Step 3: Apply the trained model to compression damage detection in virtual-reality fusion scenes. After the training process is completed, the optimized model parameters are obtained. Given new virtual-real fusion content, it is processed frame by frame.
[0119] Step 3.1: Let each frame be {y i |i∈[1,N]}, then for y i FT is obtained by the compression damage related feature extraction method described in step 1. i .
[0120] Step 3.2: Pass Get y i The probability α of "minor compression damage" i and the probability of "severe compression damage" β i .
[0121] Step 3.3: By:
[0122]
[0123] Get y i The compression damage degree score of all frames is i On average, the overall compression damage degree of the virtual-real fusion video sequence is obtained.
[0124] Specifically, a highly robust virtual-reality fusion production content compression damage detection system based on statistical feature analysis (implementation in Pytorch) is specifically implemented including the following steps:
[0125] Step 1. Under the premise that the parameters of the backbone network remain unchanged, the energy distribution, normalization coefficient, cross-level energy distribution and other information related to the perceptual quality in the deep vision model are extracted through statistical analysis.
[0126] Step 1.1. Load the model pre-trained on the semantic recognition task as the backbone network. This method is model-independent and can be adapted to different model architectures. For the convenience of description, this embodiment takes the ResNet50 pre-trained on ImageNet2012 as an example as the backbone network. Through the register_forward_hook function of pytorch, a forward callback function is registered for the output layer before the RELU function of each convolution block, so that the input image x is automatically saved when it is sent to the backbone network for forward propagation. In this embodiment, a total of 16 callback functions are registered, respectively facing the following layers of ResNet50: layer.1.0.bn3,...,layer.1.2.bn3,layer.2.0.bn3,...,layer.2.3.bn3,layer.3.0.bn3,...,layer.3.5.bn3,layer.4.0.bn3,...,layer.4.3.bn3. The depth features recorded by the above layers during the forward propagation process are recorded as Where L = 16, C i ,H i ,W i Represents the semantic features S of the i-th layer respectively (i) The number of channels, height, and width of the
[0127] Step 1.2. Extract S (i) Energy distribution (i) In the implementation process, after the input image x is sent to the ResNet50 deep vision model (denoted as f), the scale features S (i) Saved as a pytorch Tensor array.
[0128] Step 1.2.1: Use the torch.view method to convert the dimension [C i ,H i ,W i Features of S (i) Rearrange into dimensions [C i ,H i W i ] and swap its two dimensions to get the sample size N i =H i W i , dimension is C i The characteristics of Seek Covariance matrix in each dimension The calculation process is:
[0129]
[0130] in Indicates that the mean value in the sample size dimension is completed by the torch.mean method, x T Represents the transpose of the given matrix x, implemented by the torch.permute method.
[0131] Step 1.2.2: Each scale feature S (i) Each channel of is convolved with a two-dimensional convolution kernel of size 3, mean 0, and variance 1.5. That is, first construct a two-dimensional tensor array conv_kernel and assign it to: [[0.0947, 0.1183, 0.0947], [0.1183, 0.1478, 0.1183], [0.0947, 0.1183, 0.0947]], then define a torch.nn.Conv2d convolution layer with input and output channels of 1, convolution kernel size of 3, and padding size of 1, and use conv_kernel to initialize the parameter weight. After initialization, set the training mode of the convolution layer to False. After that, the convolution layer can be used to perform Gaussian convolution on the given feature. (i) The channels of the are reconnected to obtain a high-dimensional feature reflecting the neighborhood average. Because it has been convolved or filtered, it is recorded as F (i) Since the convolution process fills the input signal and does not change the signal dimension, Through a similar rearrangement method as described in step 1.2.1, the neighborhood average high-dimensional feature after rearrangement is obtained, which is recorded as
[0132] Step 1.2.3: Use the covariance matrix to transform the high-dimensional Use the following formula to merge into height H i , Width W i A two-dimensional image of:
[0133]
[0134]
[0135] During the implementation, in order to speed up the above calculation process, the torch.einsum library is used to perform calculations on the GPU. For example, if you want to calculate the quadratic form of three arrays X, Y, and E: X T (E+I) -1 Y, then first calculate (E+I) through cov_inv=torch.inverse(E+torch.eye(E.size(1)).cuda()) -1Then, we get the first half of the quadratic form by temp = torch.einsum('bik,bkj->bij',X.transpose(1,2),cov_inv), and finally get the quadratic form matrix by (torch.einsum('bik,bkj->bij',temp,Y)).diagonal(dim1=-2,dim2=-1), and get the diagonal elements by torch.diagonal function. Finally, we rearrange the extracted diagonal elements into [H i ,W i ] dimension.
[0136] The extracted The different positions reflect The degree of fluctuation on all channels at a certain spatial location.
[0137] Step 1.2.4: Use the torch.einsum-based quadratic calculation method similar to step 1.2.3 to convert the high-dimensional Fusion to height H i , Width W i A two-dimensional image of:
[0138]
[0139]
[0140] Step 1.2.5: Pass Get a two-dimensional image that integrates all channel fluctuations and reflects high-dimensional features The degree of change between the mean of the neighborhood at different positions is then convolved using the same convolutional layer as described in step 1.2.2 to obtain the mean of the degree of change, denoted as Eng (i) , indicating its energy distribution.
[0141] Step 1.3. Extraction of S (i) The normalization coefficient Norm (i) .
[0142] Step 1.3.1: Use a quadratic calculation method based on torch.einsum similar to that described in step 1.2.3 to calculate the degree of deviation between the high-dimensional feature and the average vector of its neighborhood.
[0143]
[0144]
[0145] Step 1.3.2: Use the torch.einsum-based quadratic calculation method similar to step 1.2.3 to calculate the symbolic coefficient Sign (i) :
[0146]
[0147] Sign (i) = reshape(Sign' (i) ,[H i ,W i ])
[0148] Among them, sign() represents the sign function, that is, less than zero is -1, and greater than zero is +1. In the specific implementation process, this method is implemented by torch.sign.
[0149] Step 1.3.3: Combine the three two-dimensional tensors calculated in the above steps: Energy distribution Eng (i) , the degree of deviation Sign (i) , calculate the two-dimensional, high-dimensional features that integrate all channel fluctuations The normalization coefficient Norm (i) :
[0150]
[0151] Because theoretically Always greater than 0, step 1.3.3 of the invention content directly passes Calculate the normalization coefficient. However, in this implementation, in order to prevent calculation errors caused by numerical errors, After taking the absolute value, the above normalization operation is performed. In addition, the constant K in the normalization process is set to 1 in this implementation. Represents position-by-position element multiplication, implemented by the default multiplication symbol of PyTorch.
[0152] Step: 1.4: Extract S (i) The cross-level energy distribution Distr (i,j) .
[0153] Step 1.4.1: The cross-level energy extraction process described in the invention needs to be performed on two levels with the same dimension. Combined with the ResNet50 network structure and its scale change law, for the [1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 14, 15] level, the cross-level energy distribution Distr is calculated. (i,j)When j is the next level, that is, j = i + 1, because the scale of the subsequent level is the same as that of the next level. When the level is [3, 7, 13, 16], because the scale of the subsequent features is different or there is no subsequent feature, the first feature with the same scale as the feature of level i is selected as the input for calculating the cross-level energy distribution. At this time, j is [1, 4, 8, 14] respectively.
[0154] Step 1.4.2: Distribute S by similar operations as described in step 1.2.1 (i) and S (j) Rearrange to and And calculate Covariance matrix in each dimension
[0155] Then, we use the torch.einsum matrix quadratic form calculation method described in step 1.2.3 to calculate The position of each airspace relative to E (i) Energy distribution characteristics of Distr (i) :
[0156]
[0157] At the same time, E (i) is the covariance matrix, calculate The position of each airspace relative to E (i) Energy distribution characteristics of Distr (j) :
[0158]
[0159] Step 1.4.3: Calculate Distr (j) Relative to Distr (i) The degree of change of is used to describe their mutual relationship and is recorded as the cross-level energy distribution Distr (i,j) :
[0160]
[0161] Where K = 1 is a constant to prevent numerical errors from increasing. (i) and Distr (j) All are positive values, but in order to prevent numerical anomalies, in this embodiment, the sign function and the absolute value removal operation are combined to ensure that the denominator is relatively stable during the calculation process.
[0162] Step 1.5: Connect the extracted features of various levels and types.
[0163] Step 1.5.1: Concatenation and storage of quality features at level i. Assume that the deep features S at level i (i) The scale is [C i ,H i ,W i ], then the energy distribution Eng extracted in step 1 (i) , Normalization coefficient Norm (i) , cross-level energy distribution Distr (i,j) The height is H i , width is W i The three two-dimensional arrays are respectively converted into [1,H i ,W i ], and concatenate them through the torch.cat function, and each layer gets [3,H i ,W i ]’s quality-related characteristics.
[0164] In order to reduce the complexity of the training process and prevent the same image content from repeatedly extracting the same features through the deep visual model when using different architectures for training, this example separates the quality feature extraction process from the quality feature training and testing process. That is, in the feature extraction stage, the images in the data set are extracted by the method shown in step 1 to obtain statistical information at each level, and then spliced into [3,H i ,W i ] quality-related features. Then, they are stored as feature files fname_L01.mat, fname_L02.mat, ... fname_L16.mat in sequence, where fname represents the original file name of the input image. In the subsequent training process, if the fname image content is to be read for training, the feature extraction stage is skipped, and fname_L{01..16}.mat is directly loaded in sequence to obtain quality-related data for splicing and input into the quality prediction module.
[0165] 1.5.2: For a given image to be analyzed, find the corresponding stored features by its file name, and load them into memory using the scipy.loadmat function in turn. Then, find the feature level with the smallest spatial domain size. For ResNet50, which is used as the backbone network in this embodiment, its resolution decreases with each level, so the level with the smallest resolution is L min = L = 16. For the level i to be spliced, let The feature extracted at level i is Q, and its dimension is [C = 3, H i ,W i ], based on the torch.view method, through Qr =Q.view(C,H i / B i ,B i ,W i / B i , B i) Rearrange the original 3D features. Then, through Q r =Q r .permute(0,1,3,2,4) exchanges dimensions to make the high-dimensional Tensor array Q r The dimension becomes [C=3,H i / B i ,W i / B i ,B i ,B i ], which is actually divided into non-overlapping vertical Horizontal direction The size is [B i ,B i ] is a Tensor array.
[0166] Step 1.5.3: Similarly, based on the torch.view method, compress the last two dimensions into B i B i dimensions, and compress the first and second dimensions into Dimension, that is, high-dimensional Tensor array Q r Rearranged to Then, by exchanging dimensions, we get By this method, the first dimension of the features at any level i is the same regardless of the spatial resolution.
[0167] Step 1.5.4: Use the torch.cat method to convert the dimensions of each layer to The feature matrix of is concatenated to get the dimension The fusion feature FT.
[0168] Step 2: Train the compression damage detection model on natural scenes. The reasons for using natural scene datasets for model training are: 1. The cost of establishing a natural scene content database is much lower than producing virtual content one by one. 2. The compression damage feature proposed in the present invention has excellent performance of being model-independent and scene-independent. Therefore, after natural scene training, it can be directly applied to the compression damage degree detection of virtual scenes and virtual-reality fusion scenes, and achieve good results.
[0169] Step 2.1: Construct a compression damage dataset of natural scene content. Through the collection of open source datasets and filming by cooperative units, 500 lossless video sequences with a duration of 10 seconds and a frame rate of 50fps are constructed. Screenshots of some video sequences are shown below. Figure 5 Then, HEVC and H.264 compression coding standards are used to compress the video at different levels, and a slightly damaged video sequence set is obtained by compressing the video with a CRF of 5 or less. , using CRF 41 and above to compress the severely damaged video sequence set Since two compression coding standards are used, statistics are performed in video frames. and Each contains 50,000 training samples. and The one-hot encodings are [0,1] and [1,0] respectively.
[0170] Step 2.2: Figure 6 The block diagram of the lightweight compression damage prediction model described in step 2.2 of the embodiment is given; Multi-HeadAttention represents the multi-head attention mechanism module, and the number of feature heads in this embodiment is 8. Add&Norm represents hierarchical normalization after feature addition. MLP represents multi-layer perceptron, and its specific dimensions are given in the figure. A batch normalization module is added between each MLP layer. A more lightweight compression damage detection model is constructed compared to the existing deep vision model, which can only contain a small number of convolutional layers or Transformer encoders. Before the input feature FT is sent to the transformer, it needs to pass through the feature embedding layer to convert its feature dimension from 1 to 2. Scaled to 512. The final output layer of the model contains two units, which respectively reflect the probability of the tested content belonging to "severe compression damage" and "slight compression damage". The original deep vision model is denoted as f θ (·) and the smaller learning model is
[0171] Step 2.3: For the dataset and In pytorch, we construct the torch.Dataset class to read the compression damage features of each level corresponding to the video frame file name, and construct a preprocessing function according to step 1.5 in the content of the invention to splice the read multi-level compression damage related features FT into
[0172] Step 2.4: Feed FT Get quality prediction value Combine the subjective quality score q corresponding to the input image x to the model parameters To optimize:
[0173]
[0174] In the above formula represents the optimized model parameters, CE(·) represents the cross entropy, and the loss obtained by the cross entropy is used to calculate the Adam optimizer with a learning rate of 0.001 The parameter gradients in are updated. It should be noted that the resulting cross entropy loss does not affect the original deep vision model f θ (·) parameters to make any changes.
[0175] The above process traverses the entire collected data set, and after the training is completed, a model-independent and scenario-independent quality detection model is obtained.
[0176] Step 3: Apply the trained model to compression damage detection in virtual-reality fusion scenes. After the training process is completed, the optimized model parameters are obtained. Given new virtual-reality fusion content, it is processed frame by frame.
[0177] Step 3.1: For the video sequence of the virtual-reality fusion scene or virtual scene to be tested, use the ffmpeg tool to split it into continuous video frames in the form of ffmpeg-i INPUT img-%03d.png, where INPUT represents the video sequence path to be tested. Each frame is {y i |i∈[1,N]}, then for y i FT is obtained by the compression damage related feature extraction method described in step 1. i .
[0178] Step 3.2: Set to training mode, that is, its train attribute is set to false. Get y i The probability α of "minor compression damage" i and the probability of "severe compression damage" β i .
[0179] Step 3.3: By:
[0180]
[0181] Get y i The compression damage degree score of all frames is i On average, the overall compression damage degree of the virtual-real fusion video sequence is obtained.
[0182] Step 4: Experimental results example.
[0183] Step 4.1: To verify the effect of the present invention, this embodiment collects 4 virtual scene videos and performs H.264 compression encoding to different degrees. The CRF is divided into 4 levels. The first level CRF is randomly selected from 0 to 4; the second level is randomly selected from 25 to 30; the third level is randomly selected from 35 to 40; and the fourth level is selected from 45 or more. According to step 3, the prediction model trained in the natural scene is used. Evaluate the damage degree of each compressed video sequence, and the results are as follows Figure 7 As shown in the figure, each row represents a virtual-real fusion video sequence with different contents, and each column represents a different compression level, and the compression level becomes more and more serious from left to right. The corresponding value below each screenshot is the predicted value of the compression damage degree. In order to be positively correlated with the perceived quality, the predicted value below each screenshot is The higher the score, the less compression damage the image suffers and the better the perceived quality. Figure 7 It can be seen that after training in natural scenes, the present invention can accurately evaluate the degree of compression damage of virtual-real fusion content.
[0184] Step 4.2: To further demonstrate that the features extracted by the present invention are highly robust and scene-independent, this embodiment performs statistical analysis on the high-dimensional features output by the layer.1.0.bn3 layer in the ResNet50 backbone network for natural scene images and virtual-real fusion scene images, and visualizes the extracted energy distribution and cross-layer energy distribution, as shown in FIG. Figure 8 shown. Figure 8 An intuitive display of the extracted statistical characteristics (energy distribution, cross-level energy distribution) is given. It can be seen that whether it is a natural scene or a virtual-real fusion scene, for high-quality images, the extracted features contain certain texture information; when the image is a low-quality distorted image, the texture regularity of the above statistical information is broken. It can be seen that, whether it is a natural scene or a virtual-real fusion scene, the statistical information extracted by this method can effectively reflect the difference between images with slight compression damage and images with severe compression damage. Specifically, for images with slight compression damage, their statistical information is closer to the outline of the original content, while for images with severe compression damage, the statistical information extracts more features such as block effects and pseudo contours.
[0185] Step 4.3: To further illustrate that the features extracted by the present invention are highly robust and model-independent, this embodiment applies the proposed method to the KonIQ dataset, taking 80% of it for training and 20% for testing, and using the parameters given in the Pytorch source as the ResNet50 backbone network parameters. After training, the linear correlation coefficient of this method on the test set is 0.851, and the rank correlation coefficient is 0.869. In comparison, the traditional method of fixing the backbone network and optimizing the subsequent MLP layer has a linear correlation coefficient of 0.839 and a rank correlation coefficient of 0.842. This shows that this method is not only suitable for compression damage assessment, but can also be well extended to assess other damage types such as image overexposure, underexposure, motion blur, etc. On this basis, we fixed our sequenced model, but replaced the pre-trained parameters of the backbone network from the Pytorch source to the HuggingFace source. This method can still achieve a good linear correlation coefficient of 0.572 and a rank correlation coefficient of 0.601. In contrast, the traditional method of fixing the backbone network and optimizing the subsequent MLP layer has a linear correlation coefficient of 0.092 and a rank correlation coefficient of 0.114 after the backbone network parameters are changed. This shows that this method is more stable than the traditional method.
Claims
1. A highly robust method for detecting compression damage of virtual-real fusion content based on statistical feature analysis, characterized in that: The following steps are involved: Step 1: Preprocess each frame of the video to be processed to obtain the scale feature S (i) Set {S (i) } and rescaling features gather According to the rearrangement feature set Calculate the covariance matrix E (i) Set {E (i) }; Step 2: Obtain the scale feature S through channel-by-channel Gaussian convolution (i) The neighborhood average features are rearranged to obtain the high-dimensional features of the rearranged neighborhood average Through matrix quadratic analysis and The energy difference is obtained by (i) ; Step 3: Analysis by matrix quadratic form and The local deviation degree and local deviation direction of the deviation degree matrix are obtained and symbol matrix Sign (i) , combined with the energy distribution Eng calculated in step 2 (i) , get the normalization coefficient Norm (i) ; Step 4: By analyzing the scale feature set {S (i) } in which two adjacent scale features S (i) and S (j) Energy difference, calculate the cross-level energy distribution Distr (i,j) ; Step 5: The energy distribution Eng obtained in step 2 (i) , the normalization coefficient Norm obtained in step 3 (i) , the cross-level energy distribution Distr obtained in step 4 (i,j) Perform multi-level feature concatenation to obtain fused feature FT; Step 6: Construct a prediction model, use the fusion feature FT obtained in step 5 as input, and use the natural scene training set to train the prediction model to obtain a compression damage degree prediction model; Step 7: Input each frame of the video to be detected in the virtual-reality fusion scene into the compression damage degree prediction model to obtain the overall compression damage degree of the virtual-reality fusion video sequence.
2. The highly robust virtual-real fusion content compression damage detection system based on statistical feature analysis according to claim 1 is characterized in that: In step 1, each frame of the video to be processed is preprocessed to obtain a set of scale features {S (i) } and the rearranged feature set at each scale Specifically include: Each frame of the video to be processed is used as input and sent to the deep visual model to extract the deep visual features of each level. The deep visual features at the i-th level are recorded as the scale feature S (i) , Among them C i ,H i ,W i Respectively represent S (i) The number of channels, height, width, and scale features at all levels constitute the scale feature set The scale feature S (i) According to the sample size N i =H i W i and dimension C i Rearrange and get the rearranged features The rearranged features at all levels constitute the rearranged feature set of each scale 3. The highly robust virtual-real fusion content compression damage detection system based on statistical feature analysis according to claim 1 is characterized in that: In step 2, through matrix quadratic analysis and The energy difference is obtained by (i) , specifically including: Through matrix quadratic calculation, we can respectively and Fusion to height H i , Width W i Two-dimensional array of and I represents the covariance matrix E (i) The unit matrix is equal in size. diag means taking the diagonal elements, and reshape means rearranging the vector after extracting the diagonal elements to a height of H. i , Width W i A two-dimensional array: make Use the two-dimensional Gaussian convolution kernel to perform convolution and obtain the energy distribution 4. The highly robust virtual-real fusion content compression damage detection system based on statistical feature analysis according to claim 1 is characterized in that: In step 3, through matrix quadratic analysis and The local deviation degree and local deviation direction of the deviation degree matrix are obtained and symbol matrix Sign (i) , combined with the energy distribution Eng calculated in step 2 (i) , get the normalization coefficient Norm (i) , including: The degree of deviation between each spatial position and its neighborhood mean is calculated by matrix quadratic calculation, which is recorded as the degree of deviation matrix Then, the symbol matrix defining the degree of deviation Where sign() represents the sign function, that is, less than zero is -1, and greater than zero is +1: Sign (i) =reshape(Sign’ (i) ,[H i ,W i ]) Finally, by: Get the normalized coefficient K is a constant.
5. The highly robust virtual-real fusion content compression damage detection system based on statistical feature analysis according to claim 1 is characterized in that: In step 4, by analyzing the scale feature set {S (i) } in which two adjacent scale features S (i) and S (j) Energy difference, calculate the cross-level energy distribution Distr (i,j) , specifically including: Through matrix quadratic calculation, the scale feature set {S (i) } in which two adjacent scale features S (i) and S (j) The second-order signal strength on all channels at each spatial position is recorded as energy distribution Distr (i) and Distr (j) ,in and Respectively by S (i) and S (j) Rearrange to get: Calculate Distr (j) Relative to Distr (i) The degree of change is recorded as the cross-level energy distribution K is a constant:
6. The highly robust virtual-real fusion content compression damage detection system based on statistical feature analysis according to claim 1 is characterized in that: In step 7, the overall compression damage degree of the virtual-real fusion video sequence is calculated by the following formula: The compression damage prediction model outputs the probability α that the compression damage is slight i and the probability of severe compression damage β i The overall compression damage degree s of the virtual-real fusion video sequence is calculated by the formula i :
