No-reference synthetic video quality assessment method based on multimodal learning
Through multimodal learning method, the depth and sparse features of virtual viewpoint video are extracted using optical flow estimation, convolutional neural network and sparse dictionary model, and the problem of universality and robustness of virtual viewpoint video quality evaluation in the prior art is solved. Achieve more efficient and accurate video quality assessment.
Patent Information
- Application Number
- CN202211058103.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-08-30
AI Technical Summary
The existing virtual viewpoint quality evaluation method for video without reference relies on manual feature extraction, which is low in versatility and robustness, making it difficult to effectively evaluate the visual comfort and commercial standards of immersive video in practical applications.
Using a multimodal learning method, the virtual viewpoint video is decomposed into space-time images through optical flow estimation, combined with pre-trained convolutional neural network and sparse dictionary learning model, deep features and sparse features are extracted, and weighted regression is performed to calculate virtual Quality evaluation score of viewpoint video.
It improves the versatility and robustness of virtual viewpoint video quality evaluation, can more accurately reflect the distortion characteristics of video in various time and space domains, reduces redundant calculations and storage, and improves evaluation efficiency and accuracy.
Smart Images

Figure CN115423769B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video quality assessment, and in particular to a reference-free synthetic video quality assessment method based on multimodal learning. Background Art
[0002] At present, immersive video systems such as free viewpoint video, 3-degree-of-freedom (DoF) panoramic video and windowed-6DoF video have broad application prospects in many fields due to their good human-computer interactive visual experience, such as intelligent monitoring, VR games, medical diagnosis, etc. Due to the limitations of camera equipment and network transmission bandwidth of immersive video systems, it is necessary to refer to the texture and depth information of adjacent viewpoints, and use depth image based rendering (DIBR) technology to generate a large number of virtual viewpoint videos to fill in the occluded scenes and ensure smooth switching between viewpoints.
[0003] However, the virtual viewpoint video drawn has serious spatiotemporal distortion, including compression distortion in the spatial domain and flicker distortion in the temporal domain. These distortions will lead to the degradation of the quality of the virtual viewpoint video and even seriously affect the user's visual experience. Therefore, it is necessary to establish an effective virtual viewpoint video quality evaluation index to judge whether the immersive video is visually comfortable and whether it meets commercial standards.
[0004] In recent years, many scholars have proposed a variety of quality evaluation models for virtual viewpoint images / videos, which can be divided into three types according to the reference degree of the original information, namely, full-reference (FR) quality evaluation model, reduced-reference (RR) quality evaluation model and no-reference (NR) quality evaluation model. Although the image / video quality evaluation models based on FR and RR have certain performance competitiveness, in real life, the reference video corresponding to the distorted video is basically difficult to obtain, especially for virtual viewpoint videos. Therefore, the FR and RR virtual viewpoint video quality evaluation models have poor practical application value. For this reason, some scholars have proposed a virtual viewpoint image / video quality evaluation model based on NR, but the existing NR quality evaluation method relies heavily on manual feature extraction, which requires manual analysis and feature extraction of each distortion when the distortion is known. Therefore, the existing NR quality evaluation method is generally only effective for specific databases and has low versatility and robustness.
[0005] In summary, how to avoid the difficulty of manually extracting distortion features, make full use of learning methods, and propose a more universal and robust NR virtual viewpoint image / video quality evaluation method is an urgent problem to be solved. Summary of the invention
[0006] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a reference-free synthetic video quality evaluation method based on multimodal learning, which can extract the deep semantic information hidden in the virtual viewpoint video and the sparse features of the subjective perception of the human eye through multimodal learning, and has no sample dependence, thereby improving the versatility and robustness of the reference-free quality evaluation of synthetic videos.
[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0008] The method for evaluating the quality of synthetic videos without reference based on multimodal learning includes:
[0009] S1: Obtain the virtual viewpoint video to be evaluated;
[0010] S2: Decompose the virtual viewpoint video to be evaluated into several images in the spatiotemporal domain based on optical flow estimation;
[0011] S3: Input the images in each spatiotemporal domain into the pre-trained convolutional neural network learning model to extract the deep features of the virtual viewpoint video;
[0012] S4: Input the images in each spatiotemporal domain into the trained sparse dictionary learning model to extract the sparse features of the virtual viewpoint video;
[0013] S5: Perform weighted regression based on the depth features and sparse features of the virtual viewpoint video, and then calculate the corresponding virtual viewpoint quality evaluation score as the quality evaluation result of the virtual viewpoint video to be evaluated.
[0014] Preferably, in step S2, the virtual viewpoint video is decomposed into images in three spatiotemporal domains, namely, the spatial domain, the vertical time domain and the horizontal time domain, based on optical flow estimation;
[0015] The virtual viewpoint video is represented as three-dimensional data with a width of W, a height of H, and a frame length of N;
[0016] The image representation in the spatial domain consists of N images of size W×H;
[0017] The image representation in the vertical time domain is represented by W VD It is composed of an image of size N×H;
[0018] The image in the horizontal time domain is represented by H HD It is composed of an image of size W×N.
[0019] Preferably, in step S2, the image positions and the number of images in the vertical time domain and the horizontal time domain are determined by the following steps:
[0020] S201: Estimate the optical flow map of the 1st frame and the kth frame, the optical flow map of the k+1th frame and the 2kth frame, the optical flow map of the 2k+1th frame and the 3kth frame in the spatial domain, and so on to obtain several optical flow maps; wherein k represents an adaptive frame interval threshold for repetition of previous and next frames;
[0021] S202: Calculate the maximum pixel value in each optical flow map, and locate the image positions of the x-th image in the vertical time domain and the y-th image in the horizontal time domain based on the coordinates (x, y) of the maximum pixel value in each optical flow map;
[0022] S203: Determine the number of images W in the vertical time domain based on the number of images N in the spatial domain and the adaptive frame interval threshold k VD and the number of images in the horizontal time domain H HD ; Among them, W VD =H HD =N / k.
[0023] Preferably, in step S3, a pre-trained ResNet-50 network is used as a pre-trained convolutional neural network learning model;
[0024] The images in the spatial domain, vertical time domain, and horizontal time domain are input into the pre-trained ResNet-50 network to extract the deep features of the spatial domain, the deep features of the vertical time domain, and the deep features of the horizontal time domain, respectively.
[0025] Preferably, the depth feature F in the spatial domain is expressed by the following formula: SD,de , vertical time domain depth feature F VD,de and the depth feature F in the horizontal time domain HD,de :
[0026]
[0027]
[0028]
[0029] Where: N, W VD , H HD Represents the number of images in the spatial domain, vertical time domain, and horizontal time domain respectively; represents the 2048th convolution neuron of the t1th image, where t1 is the index of the number of images in the spatial domain, t1=1,2,...,N; Represents the 2048th convolutional neuron of the t2th image, t2 is the index of the number of images in the vertical time domain, t2=1,2,...,WVD ; Represents the 2048th convolutional neuron of the t3th image, t3 is the index of the number of images in the horizontal time domain, t3=1,2,...,H HD .
[0030] Preferably, in step S4, the sparse dictionary learning model is trained by the following steps:
[0031] S401: Acquire a natural video as a training sample;
[0032] S402: Decomposing the training samples into three spatiotemporal domain training images: spatial domain, vertical time domain and horizontal time domain;
[0033] S403: Divide the training images in the three spatiotemporal domains, namely, the spatial domain, the vertical time domain and the horizontal time domain, into image blocks for dictionary training to obtain a spatial domain dictionary, a vertical time domain dictionary and a horizontal time domain dictionary and form a sparse dictionary learning model.
[0034] Preferably, step S403 specifically includes the following steps:
[0035] S401: Divide the training image corresponding to the spatiotemporal domain into image blocks B, and rearrange each image block B into a one-dimensional column vector {Bi|1≤i≤n}, where i is the pixel index in the column direction and n is the number of pixels in the image block; then form a dictionary D with m columns and n dimensions;
[0036]
[0037] Where: D represents the dictionary corresponding to the spatiotemporal domain, and the dictionary D is limited to Wherein, j is the pixel index in the row direction, j = 1, 2, ..., m; B represents the image block in the corresponding spatiotemporal domain; ||·||1 represents the L1 norm; represents the square of the L2 norm; α represents the sparse coefficient; λ represents the weight for balancing sparsity; D i,j represents the atoms in the dictionary D; m represents the number of columns in the dictionary D; argmin represents when the objective function The value of dictionary D when taking the minimum value;
[0038] S402: fix the image block B and initialize the dictionary D to an n×m random Gaussian matrix;
[0039] S403: Fix the dictionary D and update the sparse coefficient α using the following formula;
[0040]
[0041] Where: When the formula approaches the image block, the sparse coefficient α is determined;
[0042] S404: Fix the sparse coefficient α and update the dictionary D using the following formula;
[0043]
[0044] S405: Repeat steps S402 to S404 until the sparse dictionary learning model converges.
[0045] Preferably, in step S4, the sparse dictionary learning model extracts sparse features of the virtual viewpoint video through the following steps:
[0046] S411: inputting images in three spatiotemporal domains, namely, spatial domain, vertical time domain and horizontal time domain, into a sparse dictionary learning model;
[0047] S412: converting the image in the corresponding spatiotemporal domain into the grayscale domain and dividing the image into image blocks; then calculating the gradient of the image block to obtain the corresponding gradient block;
[0048] S413: extracting sparse coefficients by combining a sparse coding algorithm with a dictionary in the corresponding spatiotemporal domain, and calculating the energy of a gradient block based on the sparse coefficients; then calculating the energy of each frame image in the corresponding spatiotemporal domain based on the energy of each gradient block;
[0049] S414: Calculating image features of each frame of image based on the energy of each frame of image in the corresponding time and space domain;
[0050] S415: Based on the number of images in the spatial domain, vertical time domain and horizontal time domain combined with the image features of each frame image in the corresponding spatiotemporal domain, sparse features in the spatial domain, sparse features in the vertical time domain and sparse features in the horizontal time domain are calculated.
[0051] Preferably, in step S412, the gradient block is obtained by inputting the image block into a gradient template, and the gradient template is represented by the following formula:
[0052]
[0053] Where: g row and g col Respectively represent the row and column gradient templates used for convolution;
[0054] In step S413, the energy of the gradient block is calculated by the following formula:
[0055]
[0056] in,
[0057]
[0058] Where: E Grepresents the energy of the gradient block G; α i represents the sparse coefficient factor corresponding to the spatiotemporal domain; D represents the dictionary corresponding to the spatiotemporal domain; α represents the sparse coefficient corresponding to the spatiotemporal domain; d i,j represents the atoms in the corresponding space-time domain dictionary; <·,·> represents the inner product of the matrix, <d i,j ,d i,j >=1;
[0059] The energy of the I-th frame image is calculated by the following formula:
[0060]
[0061] Where: E I Represents the energy of the I-th frame image; represents the ath gradient block G a The square of the L2 norm, i.e. the gradient block G a energy, a represents the index of the gradient block, a=1,2,...,A, A represents the number of gradient blocks; [·] T Represents a transpose operation;
[0062] In step S414, the image feature of the I-th frame image is calculated by the following formula:
[0063]
[0064] Where: f I represents the image features of the I-th frame image in the time-space domain; E I Represents the energy of the I-th frame image; is the variance of the ath gradient block, a represents the index of the gradient block, a=1,2,...,A, A represents the number of gradient blocks;
[0065] In step S414, the sparse feature f in the spatial domain is calculated by the following formula: SD,di , sparse features in vertical time domain f VD,di and the sparse features f in the horizontal time domain HD,di :
[0066]
[0067]
[0068]
[0069] Where: Represents the image features of the I-th frame image in the spatial domain, vertical time domain, and horizontal time domain; t1 is the image number index in the spatial domain, t1=1,2,...,N; t2 is the image number index in the vertical time domain, t2=1,2,...,W VD; t3 is the image number index in the horizontal time domain, t3 = 1, 2, ..., H HD ; N, W VD , H HD Represents the number of images in the spatial domain, vertical temporal domain, and horizontal temporal domain.
[0070] Preferably, in step S5, the deep features and the sparse features are respectively input into respective random forest regression models, and the nonlinear mapping between the feature vector and the subjective score of the video is learned to obtain the corresponding deep quality score and the sparse quality score; then the corresponding virtual viewpoint quality evaluation score is calculated based on the deep quality score and the sparse quality score in combination with the following formula;
[0071] Q s =ω·Q de +(1-ω)·Q di ;
[0072] Where: Q s represents the virtual viewpoint quality evaluation score; Q de Indicates the depth quality score; Q di represents the sparse quality score; ω represents the set weight coefficient.
[0073] The non-reference synthetic video quality assessment method based on multimodal learning in the present invention has the following beneficial effects:
[0074] The present invention proposes a reference-free quality evaluation model based on convolutional neural network and sparse dictionary multimodal learning for virtual viewpoint video for the first time. On the one hand, the deep features of virtual viewpoint video are extracted by pre-trained convolutional neural network learning model. Compared with the method of manually extracting features, some hidden deep semantic information can be better extracted, and compared with other neural network models, there is no sample dependence, which can improve the versatility of the quality evaluation model; on the other hand, the sparse features of virtual viewpoint video are extracted by sparse dictionary learning model. Sparse dictionary learning is proposed based on the sparse representation principle of complex stimuli perceived by the visual cortex of the brain. It can quickly, accurately and low-costly represent the sparse features of images / videos, and the sparse features after representation are more in line with the subjective perception of the human eye, which can improve the robustness of the quality evaluation model. Therefore, the present invention extracts the hidden deep semantic information of virtual viewpoint video and the sparse features of subjective perception of the human eye by multimodal learning, and there is no sample dependence, so as to improve the versatility and robustness of reference-free quality evaluation of synthetic video.
[0075] Secondly, the present invention decomposes the virtual viewpoint video into several images in the spatiotemporal domains based on optical flow estimation. On the one hand, it can accurately and comprehensively reflect the distortion characteristics of the virtual viewpoint video in each spatiotemporal domain, which is helpful for feature extraction of subsequent multimodal learning. On the other hand, by accurately decomposing the virtual viewpoint video into images in each spatiotemporal domain, it can effectively reflect the distortion characteristics while reducing the storage and calculation time of redundant images, thereby improving the efficiency of quality evaluation of reference-free synthetic videos.
[0076] Finally, the present invention performs weighted regression on the deep features and sparse features extracted by multimodal learning, thereby weighting the impact of the deep features and sparse features on the quality of the virtual viewpoint video, thereby improving the accuracy of the quality evaluation of the reference-free synthetic video. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0078] Figure 1 This is a logical block diagram of a no-reference synthetic video quality assessment method based on multimodal learning;
[0079] Figure 2 A schematic diagram of decomposing a virtual viewpoint video into three spatiotemporal domains;
[0080] Figure 3 This is the network structure diagram of ResNet-50;
[0081] Figure 4 Examples of natural videos for training dictionaries;
[0082] Figure 5 It is a spatial domain dictionary for natural videos. DETAILED DESCRIPTION
[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0084] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0085] The following is a further detailed description through specific implementation methods:
[0086] Example:
[0087] This embodiment discloses a method for evaluating quality of synthetic videos without reference based on multimodal learning.
[0088] like Figure 1 As shown, the no-reference synthetic video quality evaluation method based on multimodal learning includes:
[0089] S1: Obtain the virtual viewpoint video to be evaluated;
[0090] S2: Decompose the virtual viewpoint video to be evaluated into several images in the spatiotemporal domain based on optical flow estimation;
[0091] S3: Input the images in each spatiotemporal domain into the pre-trained convolutional neural network learning model to extract the deep features of the virtual viewpoint video;
[0092] S4: Input the images in each spatiotemporal domain into the trained sparse dictionary learning model to extract the sparse features of the virtual viewpoint video;
[0093] S5: Perform weighted regression based on the depth features and sparse features of the virtual viewpoint video, and then calculate the corresponding virtual viewpoint quality evaluation score as the quality evaluation result of the virtual viewpoint video to be evaluated.
[0094] The present invention proposes a reference-free quality evaluation model based on convolutional neural network and sparse dictionary multimodal learning for virtual viewpoint video for the first time. On the one hand, the deep features of virtual viewpoint video are extracted by pre-trained convolutional neural network learning model. Compared with the method of manually extracting features, some hidden deep semantic information can be better extracted, and compared with other neural network models, there is no sample dependence, which can improve the versatility of the quality evaluation model; on the other hand, the sparse features of virtual viewpoint video are extracted by sparse dictionary learning model. Sparse dictionary learning is proposed based on the sparse representation principle of complex stimuli perceived by the visual cortex of the brain. It can quickly, accurately and low-costly represent the sparse features of images / videos, and the sparse features after representation are more in line with the subjective perception of the human eye, which can improve the robustness of the quality evaluation model. Therefore, the present invention extracts the hidden deep semantic information of virtual viewpoint video and the sparse features of subjective perception of the human eye by multimodal learning, and there is no sample dependence, so as to improve the versatility and robustness of reference-free quality evaluation of synthetic video.
[0095] Secondly, the present invention decomposes the virtual viewpoint video into several images in the spatiotemporal domains based on optical flow estimation. On the one hand, it can accurately and comprehensively reflect the distortion characteristics of the virtual viewpoint video in each spatiotemporal domain, which is helpful for feature extraction of subsequent multimodal learning. On the other hand, by accurately decomposing the virtual viewpoint video into images in each spatiotemporal domain, it can effectively reflect the distortion characteristics while reducing the storage and calculation time of redundant images, thereby improving the efficiency of quality evaluation of reference-free synthetic videos.
[0096] Finally, the present invention performs weighted regression on the deep features and sparse features extracted by multimodal learning, thereby weighting the impact of the deep features and sparse features on the quality of the virtual viewpoint video, thereby improving the accuracy of the quality evaluation of the reference-free synthetic video.
[0097] In the specific implementation process, the virtual viewpoint video is decomposed into three spatiotemporal domains of images, namely, spatial domain (hereinafter referred to as SD), vertical temporal domain (hereinafter referred to as VD) and horizontal temporal domain (hereinafter referred to as HD) based on optical flow estimation;
[0098] The virtual viewpoint video is represented as three-dimensional data with a width of W, a height of H, and a frame length of N;
[0099] The image representation in the spatial domain consists of N images of size W×H;
[0100] The image representation in the vertical time domain is represented by W VD It is composed of an image of size N×H;
[0101] The image in the horizontal time domain is represented by H HD It is composed of an image of size W×N.
[0102] Since flicker distortion usually occurs locally, and most VD and HD images do not contain flicker distortion, which will lead to serious computational redundancy, it is necessary to accurately determine the number and position of images in VD and HD.
[0103] Figure 2 The decomposition of the lovebird1 sequence into three spatiotemporal domains is shown, as well as the determination of the number of images and image positions in VD and HD. Figure 2 As shown, the image positions and image quantities of VD and HD are determined by the following steps:
[0104] S201: Estimate the optical flow map of the 1st frame and the kth frame, the optical flow map of the k+1th frame and the 2kth frame, the optical flow map of the 2k+1th frame and the 3kth frame in the spatial domain, and so on to obtain several optical flow maps; wherein k represents an adaptive frame interval threshold for repetition of previous and next frames;
[0105] S202: Calculate the maximum pixel value in each optical flow map, and locate the image positions of the x-th image in the vertical time domain and the y-th image in the horizontal time domain based on the coordinates (x, y) of the maximum pixel value in each optical flow map; for example, Figure 2 In the estimated optical flow map, the coordinates of the largest pixel are (598, 561), so the 598th image in VD and the 561st image in HD can be located.
[0106] S203: Determine the number of images W in the vertical time domain based on the number of images N in the spatial domain and the adaptive frame interval threshold k VD and the number of images in the horizontal time domain H HD ; Among them, W VD =H HD =N / k. The present invention proves through experiments that when W VD and H HD When set to 4, the performance and computation time of the model can achieve the best balance. Therefore, for virtual viewpoint videos with different frame lengths, W VD and H HD All are set to 4, the adaptive threshold k = N / 4, and rounded down.
[0107] The present invention decomposes the virtual viewpoint video into images in three spatiotemporal domains, namely, the spatial domain, the vertical time domain, and the horizontal time domain. On the one hand, the present invention can accurately and comprehensively reflect the distortion characteristics of the virtual viewpoint video from the three spatiotemporal domains, namely, the spatial domain, the vertical time domain, and the horizontal time domain, which is helpful for feature extraction of subsequent multimodal learning. On the other hand, by accurately decomposing the virtual viewpoint video into images in each spatiotemporal domain, the present invention can effectively reflect the distortion characteristics while reducing the storage and calculation time of redundant images, thereby improving the efficiency of quality evaluation of reference-free synthetic videos.
[0108] In order to solve the problem that deep learning methods rely on training samples, the present invention uses a pre-trained convolutional neural network model specially built for image recognition tasks to extract deep features of images. Its main advantages are: (1) The pre-trained convolutional neural network model is trained on a large database with millions of samples, while the self-trained convolutional neural network model is trained based on a small virtual viewpoint database with hundreds of samples. Therefore, the former has fewer training overfitting problems. (2) Since the training samples are independent of the virtual viewpoint video, the quality evaluation model using the pre-trained convolutional neural network model has no sample dependence and is more general.
[0109] The present invention uses the pre-trained ResNet-50 network as the pre-trained convolutional neural network learning model, which has the advantages of low computational complexity, strong ability to solve network degradation, and sensitivity to data fluctuations.
[0110] Figure 3 The six main stages of the ResNet-50 network structure are given. "W×W×C" means that the size of the input image is W×W and the number of convolution channels is C. In stage 1, "Conv" represents the convolution kernel, the size is 7×7, and the number of convolution kernels is 64; "BN" represents batch normalization, "ReLU" represents the activation function, and "Maxpool" represents the maximum pooling operation. Stages 2 to 5 are composed of two structures, bottleneck1 and bottleneck2, as shown in Figure 3 As shown in the right sub-figure. In the sixth stage, the model performs average pooling and full connection (FC), and obtains the final output through the Softmax activation function.
[0111] Based on the above description, the present invention inputs images in three spatiotemporal domains into the ResNet-50 network structure and performs feature extraction. As input, the image needs to be distorted to 224×224×3. Since the FC layer is used for probabilistic classification, the present invention removes the last FC layer from the pre-trained ResNet-50 model. By inputting SD, VD and HD images into ResNet-50, each image can obtain the features of 2048-dimensional neurons after the average pooling layer in the 6th stage.
[0112] The images in the spatial domain, vertical time domain, and horizontal time domain are input into the pre-trained ResNet-50 network to extract the deep features F in the spatial domain. SD,de , vertical time domain depth feature F VD,de And the depth feature F in the horizontal time domain HD,de .
[0113]
[0114]
[0115]
[0116] Where: N, W VD , H HD Represents the number of images in the spatial domain, vertical time domain, and horizontal time domain respectively; represents the 2048th convolution neuron of the t1th image, where t1 is the index of the number of images in the spatial domain, t1=1,2,...,N; Represents the 2048th convolutional neuron of the t2th image, t2 is the index of the number of images in the vertical time domain, t2=1,2,...,W VD ; Represents the 2048th convolutional neuron of the t3th image, t3 is the index of the number of images in the horizontal time domain, t3=1,2,...,H HD .
[0117] The present invention extracts the depth features of virtual viewpoint videos through a pre-trained convolutional neural network learning model. Compared with the manual feature extraction method, it can better extract some hidden depth semantic information, and compared with other neural network models, it has no sample dependence, thereby improving the versatility of the quality evaluation model.
[0118] In the specific implementation process, the sparse dictionary learning model mainly consists of a dictionary training stage and a sparse representation stage.
[0119] 1) In the dictionary training phase, the sparse dictionary learning model is trained through the following steps:
[0120] S401: Acquire a natural video as a training sample;
[0121] In this embodiment, natural videos are selected from the LIVE-VQC database, such as Figure 4 shown.
[0122] S402: Decomposing the training samples into three spatiotemporal domain training images: spatial domain, vertical time domain and horizontal time domain;
[0123] S403: Divide the training images in the three spatiotemporal domains, namely, the spatial domain, the vertical time domain and the horizontal time domain, into image blocks for dictionary training to obtain a spatial domain dictionary, a vertical time domain dictionary and a horizontal time domain dictionary and form a sparse dictionary learning model.
[0124] Step S403 specifically includes the following steps:
[0125] S401: Divide the training image corresponding to the spatiotemporal domain into image blocks B, and rearrange each image block B into a one-dimensional column vector {Bi|1≤i≤n}, where i is the pixel index in the column direction and n is the number of pixels in the image block; then form a dictionary D with m columns and n dimensions;
[0126]
[0127] Where: D represents the dictionary corresponding to the spatiotemporal domain, and the dictionary D is limited to Wherein, j is the pixel index in the row direction, j = 1, 2, ..., m; B represents the image block in the corresponding spatiotemporal domain; ||·||1 represents the L1 norm; represents the square of the L2 norm; α represents the sparse coefficient; λ represents the weight for balancing sparsity; D i,j represents the atoms in the dictionary D; m represents the number of columns in the dictionary D; argmin represents when the objective function The value of dictionary D when taking the minimum value;
[0128] S402: fix the image block B and initialize the dictionary D to an n×m random Gaussian matrix;
[0129] S403: Fix the dictionary D and update the sparse coefficient α using the following formula;
[0130]
[0131] Where: When the formula approaches the image block, the sparse coefficient α is determined;
[0132] S404: Fix the sparse coefficient α and update the dictionary D using the following formula;
[0133]
[0134] S405: Repeat steps S402 to S404 until the sparse dictionary learning model converges.
[0135] In this embodiment, the size of each image block is set to 8×8, and finally a dictionary with a size of 64×256 is obtained.
[0136] Figure 5The spatial domain dictionary D1 obtained by training with spatial domain images of natural videos is shown. Similarly, the vertical time domain dictionary D2 and the horizontal time domain dictionary D3 can be obtained by training with vertical time domain images and horizontal time domain images of natural videos, respectively.
[0137] 2) Sparse representation stage: sparse features of virtual viewpoint video are extracted through the following steps:
[0138] S411: inputting images in three spatiotemporal domains, namely, spatial domain, vertical time domain and horizontal time domain, into a sparse dictionary learning model;
[0139] S412: converting the image in the corresponding spatiotemporal domain into the grayscale domain and dividing the image into image blocks; then calculating the gradient of the image block to obtain the corresponding gradient block; the gradient of the image block is calculated to enhance edge information.
[0140] S413: extracting sparse coefficients by combining a sparse coding algorithm with a dictionary in the corresponding spatiotemporal domain, and calculating the energy of a gradient block based on the sparse coefficients; then calculating the energy of each frame image in the corresponding spatiotemporal domain based on the energy of each gradient block;
[0141] In this embodiment, an orthogonal matching pursuit algorithm is used as the sparse coding algorithm.
[0142] S414: Calculating image features of each frame of image based on the energy of each frame of image in the corresponding time and space domain;
[0143] S415: Based on the number of images in the spatial domain, vertical time domain and horizontal time domain combined with the image features of each frame image in the corresponding spatiotemporal domain, sparse features in the spatial domain, sparse features in the vertical time domain and sparse features in the horizontal time domain are calculated.
[0144] The gradient block is obtained by inputting the image block into the gradient template, and the gradient template is expressed by the following formula:
[0145]
[0146] Where: g row and g col Respectively represent the row and column gradient templates used for convolution;
[0147] The energy of the gradient block is calculated by the following formula:
[0148]
[0149] in,
[0150]
[0151] Where: E G represents the energy of the gradient block G; α irepresents the sparse coefficient factor corresponding to the spatiotemporal domain; D represents the dictionary corresponding to the spatiotemporal domain; α represents the sparse coefficient corresponding to the spatiotemporal domain; d i,j represents the atoms in the corresponding spatiotemporal dictionary; <·,·> represents the inner product of the matrix, and due to the orthogonal normalization in dictionary training, <d i,j ,d i,j >=1;
[0152] The energy of the I-th frame image is calculated by the following formula:
[0153]
[0154] Where: E I Represents the energy of the I-th frame image; represents the ath gradient block G a The square of the L2 norm, i.e. the gradient block G a energy, a represents the index of the gradient block, a=1,2,...,A, A represents the number of gradient blocks; [·] T Represents a transpose operation;
[0155] The dimension of the sparse coefficient is determined by the number of image blocks A. However, the resolution of different sequences is inconsistent, which usually leads to different numbers of image blocks, that is, the sparse coefficients generated by different sequences usually have different dimensions, which brings inconvenience to subsequent feature extraction experiments. Therefore, the present invention considers removing block variance to eliminate the influence of scene content and unify feature dimensions.
[0156] The image features of the I-th frame image are calculated by the following formula:
[0157]
[0158] Where: f I represents the image features of the I-th frame image in the time-space domain; E I Represents the energy of the I-th frame image; is the variance of the ath gradient block, a represents the index of the gradient block, a=1,2,...,A, A represents the number of gradient blocks;
[0159] The sparse feature f in the spatial domain is calculated by the following formula: SD,di , sparse features in vertical time domain f VD,di and the sparse features f in the horizontal time domain HD,di :
[0160]
[0161]
[0162]
[0163] Where: Represents the image features of the I-th frame image in the spatial domain, vertical time domain, and horizontal time domain; t1 is the image number index in the spatial domain, t1=1,2,...,N; t2 is the image number index in the vertical time domain, t2=1,2,...,W VD ; t3 is the image number index in the horizontal time domain, t3 = 1, 2, ..., H HD ; N, W VD , H HD Represents the number of images in the spatial domain, vertical temporal domain, and horizontal temporal domain.
[0164] The present invention extracts sparse features of virtual viewpoint videos through a sparse dictionary learning model. Sparse dictionary learning is proposed based on the sparse representation principle of the brain's visual cortex perceiving complex stimuli. It can represent the sparse features of images / videos quickly, accurately, and at low cost. The represented sparse features are more consistent with the subjective perception of the human eye, thereby improving the robustness of the quality evaluation model.
[0165] In the specific implementation process, the deep features and sparse features are respectively input into their respective random forest regression models to learn the nonlinear mapping between the feature vector and the subjective score of the video, and the corresponding deep quality score and sparse quality score are obtained; then, the corresponding virtual viewpoint quality evaluation score is calculated based on the deep quality score and the sparse quality score in combination with the following formula;
[0166] Q s =ω·Q de +(1-ω)·Q di ;
[0167] Where: Q s represents the virtual viewpoint quality evaluation score; Q de Indicates the depth quality score; Q di represents the sparse quality fraction; ω represents the set weight coefficient. In this embodiment, ω is set to 0.6.
[0168] In this embodiment, the random forest regression model is a learner that fits multidimensional features into one-dimensional scores. The feature data obtained by random sampling is input into a large number of weak learners (decision trees). The final decision result is determined by voting or taking the average. The purpose is to make the final result obtained by the model have higher accuracy and generalization performance.
[0169] Based on the random forest regression model, the nonlinear mapping between the feature vector and the subjective score of the video is learned, and the corresponding deep quality score and sparse quality score are obtained using existing mature means. For the specific principle description, please refer to the content recorded in Zhou Zhihua's "Machine Learning".
[0170] The present invention inputs the deep features and sparse features extracted by multimodal learning into a random forest regression model for weighted regression, thereby being able to weight the influence of the deep features and sparse features on the quality of virtual viewpoint videos, thereby being able to further improve the accuracy of quality evaluation of reference-free synthetic videos.
[0171] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.
[0172] 1. Experimental Preparation
[0173] The present invention is compared with the existing Image Quality Assessment (IQA) method and Video Quality Assessment (VQA) method. In addition, the experiment will be verified on three virtual viewpoint video databases: IRCCyN / IVC, SIAT and windowed-6DoF. Based on the differences in the videos in the three databases, the diversity of virtual viewpoint videos is guaranteed, and a total of 370 virtual viewpoint videos are used to evaluate the performance of the model of the present invention.
[0174] In order to evaluate the performance of the model of the present invention, the test database was randomly divided into two parts, i.e., 80% of the samples were used for training and 20% of the samples were used for testing. Then, the Spearman rank correlation coefficient (SROCC), Pearson linear correlation coefficient (PLCC) and root mean square error (RMSE) were used as criteria to judge the model performance. Among them, SROCC has predictive monotonicity, and PLCC and RMSE reflect the prediction accuracy. Since PLCC and RMSE need to perform nonlinear logistic functions, the function is intended to map the predicted scores into the subjective score space, as shown in the following formula.
[0175]
[0176] Where: Q′ represents the score after mapping; Q s represents the model prediction score, that is, the virtual viewpoint quality evaluation score; δ1, δ2, δ3, δ4, and δ5 represent fitting parameters.
[0177] In this embodiment, 1000 training and testing experiments were performed, and the average value of the 1000 experiments was calculated as the final experimental result. The higher the PLCC and SROCC, the lower the RMSE, which means the better the performance of the model.
[0178] 2. Performance Comparison
[0179] Table 1 lists the performance of the proposed model and 24 existing models on two public IRCCyN and SIAT virtual viewpoint video databases, where the top two methods with the best performance are shown in black bold. For the convenience of statistics, the traditional image quality assessment model, the traditional video quality assessment model, the virtual viewpoint image quality assessment model, and the virtual viewpoint video quality assessment model are represented as T-IQA, T-VQA, S-IQA, and S-VQA, respectively. In addition, the subjective score of the SIAT database uses the Differential Mean Opinion Score (DMOS), while the subjective score of the IRCCyN / IVC database uses both DMOS and the Mean Opinion Score (MOS).
[0180] Table 1 Performance of the methods in IRCCYN / IVC and SIAT virtual viewpoint video databases
[0181]
[0182] Combined with Table 1, it can be seen that for T-IQA and T-VQA methods, the V-BLINDS method achieved the highest performance on the IRCCyN / IVC database with a PLCC of 0.7825, and the SpEED method achieved the highest performance on the SIAT database with a PLCC of 0.7370. However, on the IRCCyN and SIAT databases, the PLCC of the proposed model reached 0.8914 and 0.8819, respectively. The results show that the performance of traditional methods in evaluating the quality of virtual viewpoint videos is limited because they do not consider the impact of special rendering distortion on video quality.
[0183] For both S-IQA and S-VQA methods, the proposed model achieved the top two performances on both IRCCyN / IVC and SIAT databases. Among them, in both DMO and MOS cases of IRCCyN / IVC database, the proposed model achieved the highest PLCC, SROCC and the lowest RMSE. It is worth noting that the PLCC, SROCC and RMSE of the second-ranked EM-VQA method are similar to our method. However, the EM-VQA method only evaluated the distortion caused by drawing during the experiment, excluding 9 compression distortions in the database, so its ability to detect distortion diversity is very limited. It can be concluded that the proposed method is superior to other quality assessment methods in terms of both versatility and performance. On the SIAT database, the performance of the proposed model is second only to the SR-3DVQA method. Similar to the proposed model, SR-3DVQA also uses learning theory for feature acquisition, which makes it more competitive in performance than traditional methods. However, the difference is that the proposed method is an NR method, while SR-3DVQA is an RF method. In practical applications, it is difficult and unrealistic to obtain reference videos. Therefore, the practicality and versatility of the method of the present invention are higher than those of the SR-3DVQA method. In addition, some methods perform well on one database but poorly on another. For example, the SVQM method obtains a PLCC of 0.7630 on the SIAT database, but only obtains a PLCC of 0.5446 on the IRCCyN / IVC database. The reason is that the model of manually extracted features is subject to the solidified cognition of the brain, so the model advantage is only reflected in a specific database. Therefore, for databases with different types and degrees of distortion, the performance of the model of manually extracted features is unstable. On the contrary, the learning-based model can automatically learn deeper semantic features and has higher stability and robustness.
[0184] Table 2 lists the experimental results of PLCC, SROCC and RMSE of the NR method on the Windowed-6DoF database, and the two best performing methods are marked in bold. According to the experimental data, the method proposed in the present invention also achieved the best performance compared with other NR indicators. Even the V-BLINDS method, which performs better, is far lower than the method of the present invention. The PLCC of the V-BLINDS method is 0.8652, while the PLCC of the method of the present invention is 0.9208. In general, the method of the present invention performs well on multiple databases and has good versatility and robustness in evaluating the quality of virtual viewpoint videos. This also confirms that quality evaluation based on convolutional neural network learning models and sparse dictionary learning models is a successful strategy.
[0185] Table 2 Performance of NR method in Windowed-6DoF virtual viewpoint video database
[0186]
[0187] 3. Parameter performance
[0188] Table 3 shows that when W VD and H HD PLCC performance and computation time when all are set to 1, 2, 3, 4, 5, 6, and 10, where k is equal to N divided by the number of images.
[0189] From the experimental data, the performance of PLCC generally increases with W, as the increase in the number of images leads to more accurate optical flow estimation and more flicker distortion details. VD H HD The increase of W and the decrease of k show an upward trend. For example, when W VD / H HD =1, PLCC is 0.8899, when W VD / H HD =10, PLCC is 0.9253. However, W VD / H HD The increase in W also brings a lot of computing time. For example, VD / H HD = 10, the calculation time is almost VD / H HD = 1. Fortunately, we found that when W VD / H HD =4, the PLCC performance of the model reaches 0.9208, and the PLCC growth rate becomes slow as the number of images further increases. Therefore, we conclude that when W VD / H HD =4, the performance of the model is close to reaching the stable boundary point, and the corresponding calculation time is relatively acceptable. Therefore, in order to facilitate the establishment of the quality assessment model, the method of the present invention uniformly sets W for sequences of different frame lengths. VD =H HD =4, and k=N / 4.
[0190] Table 3 Performance and computation time of different VD and HD image numbers on Windowed-6DOF database
[0191]
[0192] Table 4 shows the performance of the method of the present invention when using different ResNet structures in the convolutional neural network learning model. Here, we use three ResNet structures for experiments, namely ResNet-50, ResNet-101 and ResNet-152. The larger the number in the ResNet name, the more convolutional layers the neural network has and the more complex the model calculation is. All ResNet structures were tested on the IRCCyN / IVC, SIAT and Windpwed-6DoF databases. The experimental results show that the PLCC, SROCC and RMSE of different ResNet structures are only slightly different, which means that the performance of ResNet-50, ResNet-101 and ResNet-152 structures is similar. It can be concluded that more convolutional layers are difficult to improve the model performance, and Resnet-50 is sufficient to capture the key features in the virtual viewpoint video. Therefore, the present invention finally selects Resnet-50 as the pre-training model structure to extract video features.
[0193] Table 4 Performance of different ResNet structures on three virtual viewpoint video databases
[0194]
[0195] In order to verify the impact of different natural video training samples on dictionary performance. The present invention designs the following four dictionaries. DI: Randomly select 10 videos from the LIVE-VQC database, extract one frame from each video, and obtain 10,000 image blocks for training. D-II: Randomly select 10 videos from the LIVE-VQC database, extract one frame from each video, and obtain 20,000 image blocks for training. D-III: Randomly select 7 videos from the LIVE-VQC database, extract one frame from each video, and obtain 10,000 image blocks for training. D-IV: Randomly select 20 videos from the LIVE-VQC database, extract one frame from each video, and obtain 10,000 image blocks for training. Table 5 lists the performance of the four dictionaries on the IRCCyN / IVC, SIAT and Windowed-6DoF virtual viewpoint video databases. It can be observed that when four different dictionaries are used to predict video scores, the performance data obtained are very similar. Compared with D-IV, the other three dictionaries lack video diversity during training, but the performance does not decrease. Compared with D-II, the other three dictionaries have a limited number of image blocks, but the performance remains basically unchanged. The above results show that the method of the present invention is not sensitive to the diversity and number of training videos. Therefore, the trained dictionary is universal and can be used to predict the quality of various types of virtual viewpoint videos, which makes the method of the present invention more robust in practical applications. For convenience, this experiment uses DI as a sparse dictionary to evaluate the quality of virtual viewpoints.
[0196] Table 5 Performance of different dictionary types on three virtual viewpoint video databases
[0197]
[0198] In order to test the impact of different regression models and weighting strategies on the performance of the quality assessment model, the present invention experiments with two regression models, namely RF regression and support vector regression (SVR), and the weight parameter ω is set to 0.1 to 0.9 with an interval of 0.1. Table 6 lists the performance of different regression models and weight parameters on the IRCCyN / IVC, SIAT and Windowed-6DoF virtual viewpoint video databases. The best performance is marked in bold.
[0199] Table 6 Performance of different feature combination models on three virtual viewpoint video databases
[0200]
[0201] As shown in Table 6:
[0202] 1) The SVR-based model performs well on the IRCCyN / IVC and Windowed-6DoF databases, but performs poorly on the SIAT database. In addition, the stability of the SVR-based model is also unsatisfactory. For example, on the Windowed-6DoF database, when ω increases from 0.1 to 0.9, the SROCC based on the SVR model drops sharply from 0.8651 to 0.6237, indicating that it has poor convergence to the convolutional neural network learning model. In short, under the same weight parameter strategy, the stability and superiority of the SVR model are lower than those of the RF model. Therefore, the present invention uses RF-based regression to generate the final prediction score.
[0203] 2) In the IRCCyN database, the highest performance is obtained when ω=0.7, that is, the SROCC is 0.8760. In the SIAT database, the best performance is achieved when ω=0.6, and the SROCC is 0.8730. Similarly, in the Windowed-6DoF database, the best performance is achieved when ω=0.6, and the SROCC is 0.8931. Since the best performance is obtained on the SIAT and Windowed-6DoF databases when ω is equal to 0.6, and the optimal weight of the IRCCyN database is also close to it, the present invention uniformly sets ω to 0.6 as the final weight parameter.
[0204] In order to test the performance of the model composed of different feature components in evaluating the quality of virtual viewpoint video, the present invention conducted an ablation experiment. Table 7 lists the performance of the model composed of different feature components on the IRCCyN / IVC, SIAT and Windowed-6DoF virtual viewpoint video databases. Specifically, the present invention uses the F SD,de 、F VD,de and F HD,de features, and f obtained in the sparse dictionary learning model SD,di 、f VD,di and f HD,di The features were arranged and combined, and a total of 15 models were obtained.
[0205] Table 7 Performance of models composed of different feature components on three virtual viewpoint video databases
[0206]
[0207] By observing the experimental data, the present invention can draw the following two conclusions:
[0208] 1) Models 1 to 6 are all composed of one feature component. Among them, models 1 to 3 are composed of deep feature components, and models 4 to 6 are composed of sparse feature components. In comparison, the overall performance of models 1 to 3 is higher than that of models 4 to 6. For example, on the Windowed-6DoF database, the SROCC values of models 1 to 3 are all higher than 0.8516, while the highest SROCC value of models 4 to 6 is only 0.5237. This shows that deep features perform better in the case of a single component.
[0209] 2) Models 7 to 12 are composed of two feature components. Models 13 and 14 are composed of three feature components. Models 10 to 12, which are composed of sparse feature components, have greatly improved performance compared with the corresponding single-component models. For example, on the SIAT database, the maximum PLCC of the sparse feature single-component model is 0.6125, and the maximum PLCC of the two-component model is 0.8069, an increase of about 0.1944. The performance of models 7 to 9, which are composed of deep feature components, has also improved, but the upward trend is relatively unobvious compared with the models composed of sparse features. For example, on the SIAT database, the maximum PLCC of the deep feature single-component model is 0.7601, while the maximum PLCC of the two-component model is 0.8073, an increase of 0.0472. Overall, with the increase of feature components, the performance of the model gradually improves and stabilizes. Finally, when all feature components are used, that is, model 15, the quality evaluation model achieves the best performance.
[0210] In summary, the experimental results on three virtual viewpoint video databases show that compared with other classic and state-of-the-art quality evaluation methods, the quality evaluation model proposed in this invention has superior performance in evaluating various virtual viewpoint videos, is more in line with the subjective perception of the human eye, and has stronger versatility and robustness.
[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. A no-reference synthetic video quality assessment method based on multimodal learning, characterized in that: include: S1: Obtain the virtual viewpoint video to be evaluated; S2: Decompose the virtual viewpoint video to be evaluated into several images in the spatiotemporal domain based on optical flow estimation; In step S2, the virtual viewpoint video is decomposed into images in three spatiotemporal domains, namely, the spatial domain, the vertical time domain and the horizontal time domain, based on the optical flow estimation; The virtual viewpoint video is represented as three-dimensional data with a width of W, a height of H, and a frame length of N; The image representation in the spatial domain consists of N images of size W×H; The image representation in the vertical time domain is represented by W VD It is composed of an image of size N×H; The image in the horizontal time domain is represented by H HD It is composed of an image of size W×N; The image positions and number of images in the vertical and horizontal time domains are determined by the following steps: S201: Estimate the optical flow map of the 1st frame and the kth frame, the optical flow map of the k+1th frame and the 2kth frame, the optical flow map of the 2k+1th frame and the 3kth frame in the spatial domain, and so on to obtain several optical flow maps; wherein k represents an adaptive frame interval threshold for repetition of previous and next frames; S202: Calculate the maximum pixel value in each optical flow map, and locate the image positions of the x-th image in the vertical time domain and the y-th image in the horizontal time domain based on the coordinates (x, y) of the maximum pixel value in each optical flow map; S203: Determine the number of images W in the vertical time domain based on the number of images N in the spatial domain and the adaptive frame interval threshold k VD and the number of images in the horizontal time domain H HD ; Among them, W VD =H HD =N / k; S3: Input the images in each spatiotemporal domain into the pre-trained convolutional neural network learning model to extract the deep features of the virtual viewpoint video; S4: Input the images in each spatiotemporal domain into the trained sparse dictionary learning model to extract the sparse features of the virtual viewpoint video; S5: performing weighted regression based on the depth features and sparse features of the virtual viewpoint video, and then calculating the corresponding virtual viewpoint quality evaluation score as the quality evaluation result of the virtual viewpoint video to be evaluated; In step S5, the deep features and the sparse features are respectively input into their respective random forest regression models, and the nonlinear mapping between the feature vector and the subjective score of the video is learned to obtain the corresponding deep quality score and the sparse quality score; then the corresponding virtual viewpoint quality evaluation score is calculated based on the deep quality score and the sparse quality score in combination with the following formula; Q s =ω·Q de +(1-ω)·Q di ; Where: Q s represents the virtual viewpoint quality evaluation score; Q de Indicates the depth quality score; Q di represents the sparse quality score; ω represents the set weight coefficient.
2. The method for evaluating quality of a non-reference synthetic video based on multimodal learning according to claim 1, characterized in that: In step S3, the pre-trained ResNet-50 network is used as a pre-trained convolutional neural network learning model; The images in the spatial domain, vertical time domain, and horizontal time domain are input into the pre-trained ResNet-50 network to extract the deep features of the spatial domain, the deep features of the vertical time domain, and the deep features of the horizontal time domain, respectively.
3. The method for evaluating quality of a non-reference synthetic video based on multimodal learning as claimed in claim 2, characterized in that: The deep feature F in the spatial domain is expressed by the following formula: SD,de , vertical time domain depth feature F VD,de And the depth feature F in the horizontal time domain HD,de : Where: N, W VD , H HD Represents the number of images in the spatial domain, vertical time domain, and horizontal time domain respectively; Represents the 2048th convolution neuron of the t1th image, where t1 is the index of the number of images in the spatial domain, t1=1,2,...,N; Represents the 2048th convolutional neuron of the t2th image, t2 is the index of the number of images in the vertical time domain, t2=1,2,...,W VD ; Represents the 2048th convolutional neuron of the t3th image, t3 is the index of the number of images in the horizontal time domain, t3=1,2,...,H HD .
4. The method for evaluating quality of a non-reference synthetic video based on multimodal learning according to claim 1, characterized in that: In step S4, the sparse dictionary learning model is trained by the following steps: S401: Acquire a natural video as a training sample; S402: Decomposing the training samples into three spatiotemporal domain training images: spatial domain, vertical time domain and horizontal time domain; S403: Divide the training images in the three spatiotemporal domains, namely, the spatial domain, the vertical time domain and the horizontal time domain, into image blocks for dictionary training to obtain a spatial domain dictionary, a vertical time domain dictionary and a horizontal time domain dictionary and form a sparse dictionary learning model.
5. The method for evaluating quality of synthetic videos without reference based on multimodal learning as claimed in claim 4, characterized in that: Step S403 specifically includes the following steps: S401: Divide the training image corresponding to the spatiotemporal domain into image blocks B, and rearrange each image block B into a one-dimensional column vector {Bi|1≤i≤n}, where i is the pixel index in the column direction and n is the number of pixels in the image block; then form a dictionary D with m columns and n dimensions; Where: D represents the dictionary corresponding to the spatiotemporal domain, and the dictionary D is limited to Wherein, j is the pixel index in the row direction, j = 1, 2, ..., m; B represents the image block in the corresponding spatiotemporal domain; ||·||1 represents the L1 norm; represents the square of the L2 norm; α represents the sparse coefficient; λ represents the weight for balancing sparsity; D i,j represents the atoms in the dictionary D; m represents the number of columns in the dictionary D; argmin represents when the objective function The value of dictionary D when taking the minimum value; S402: fix the image block B and initialize the dictionary D to an n×m random Gaussian matrix; S403: Fix the dictionary D and update the sparse coefficient α using the following formula; Where: When the formula approaches the image block, the sparse coefficient α is determined; S404: Fix the sparse coefficient α and update the dictionary D using the following formula; S405: Repeat steps S402 to S404 until the sparse dictionary learning model converges.
6. The method for evaluating quality of synthetic videos without reference based on multimodal learning according to claim 4, characterized in that: In step S4, the sparse dictionary learning model extracts sparse features of the virtual viewpoint video through the following steps: S411: inputting images in three spatiotemporal domains, namely, spatial domain, vertical time domain and horizontal time domain, into a sparse dictionary learning model; S412: converting the image in the corresponding spatiotemporal domain into the grayscale domain and dividing the image into image blocks; then calculating the gradient of the image block to obtain the corresponding gradient block; S413: extracting sparse coefficients by combining a sparse coding algorithm with a dictionary corresponding to the spatiotemporal domain, and calculating the energy of the gradient block based on the sparse coefficients; Then, the energy of each frame image in the corresponding spatiotemporal domain is calculated based on the energy of each gradient block; S414: Calculating image features of each frame of image based on the energy of each frame of image in the corresponding time and space domain; S415: Based on the number of images in the spatial domain, vertical time domain and horizontal time domain combined with the image features of each frame image in the corresponding spatiotemporal domain, sparse features in the spatial domain, sparse features in the vertical time domain and sparse features in the horizontal time domain are calculated.
7. The method for evaluating quality of synthetic videos without reference based on multimodal learning according to claim 5, characterized in that: In step S412, the gradient block is obtained by inputting the image block into the gradient template, and the gradient template is represented by the following formula: Where: g row and g col Respectively represent the row and column gradient templates used for convolution; In step S413, the energy of the gradient block is calculated by the following formula: in, Where: E G represents the energy of the gradient block G; α i represents the sparse coefficient factor corresponding to the spatiotemporal domain; D represents the dictionary corresponding to the spatiotemporal domain; α represents the sparse coefficient corresponding to the spatiotemporal domain; d i,j represents the atoms in the corresponding space-time domain dictionary; <·,·> represents the inner product of the matrix, <d i,j ,d i,j >=1; The energy of the I-th frame image is calculated by the following formula: Where: E I Represents the energy of the I-th frame image; represents the ath gradient block G a The square of the L2 norm, i.e. the gradient block G a energy, a represents the index of the gradient block, a=1,2,...,A, A represents the number of gradient blocks; [·] T Represents a transpose operation; In step S414, the image feature of the I-th frame image is calculated by the following formula: Where: f I represents the image features of the I-th frame image in the time-space domain; E I Represents the energy of the I-th frame image; is the variance of the ath gradient block, a represents the index of the gradient block, a=1,2,…,A, A represents the number of gradient blocks; In step S414, the sparse feature f in the spatial domain is calculated by the following formula: SD,di , sparse features in vertical time domain f VD,di and the sparse features f in the horizontal time domain HD,di : Where: Represents the image features of the I-th frame image in the spatial domain, vertical time domain, and horizontal time domain; t1 is the image number index in the spatial domain, t1=1,2,...,N; t2 is the image number index in the vertical time domain, t2=1,2,...,W VD ; t3 is the image number index in the horizontal time domain, t3 = 1, 2, ..., H HD ; N, W VD , H HD Represents the number of images in the spatial domain, vertical temporal domain, and horizontal temporal domain.
