A cross-modal multi-quality evaluation dataset joint training method and system

Abstract

The application provides a cross-modal multi-quality evaluation dataset joint training method and system, comprising: constructing a cross-modal multi-quality evaluation model, including two parts of feature extraction and feature fusion; the training process comprises three steps: in the first step, according to the number M of the quality evaluation datasets to be trained, the feature fusion part is expanded to M, and after the feature extraction part is used to extract features and input into the corresponding feature fusion part of each dataset, training is performed; in the second step, the feature extraction part inherits the model parameters after the first step of training, and the feature fusion part is expanded according to the number N of modes and then trained; in the third step, the feature extraction part inherits the model parameters after the second step of training, the model parameters of the feature extraction part are frozen, the feature extraction part is restored to one, and then training is performed. The application effectively utilizes cross-modal audio, image, video and audio-video quality evaluation datasets to jointly train a multi-modal cross-modal multi-quality evaluation model.

Description

Technical Field

[0001] This invention relates to the field of multimodal media quality assessment technology, and more specifically, to a method and system for joint training of cross-modal multi-quality assessment datasets. Background Technology

[0002] In recent years, multimedia data on the internet has developed rapidly, especially user-generated content (UGC) such as audio and video content uploaded by users in various social media and streaming applications. However, many factors can introduce distortion into user-generated audio and video content, such as non-professional shooting equipment, amateur user shooting skills, compression techniques, and transmission systems. More and more platform operators are paying attention to the quality evaluation of multimedia data, which is crucial for multimedia processing and recommendation systems. This includes filtering extremely low-quality multimedia data and recommending high-quality data to users, as well as guiding compression and transmission algorithms to strike a trade-off between quality and bitrate.

[0003] Based on whether or not reference information is utilized, quality assessment can be categorized into three types: full-reference (FR), semi-reference (RR), and no-reference (NR). Unlike full-reference and semi-reference quality assessments, no-reference quality assessment does not rely on reference information and is particularly suitable for assessing the quality of user-generated content. To predict the quality of multimodal media, researchers have proposed different types of no-reference quality assessment methods, including audio quality assessment (AQA), image quality assessment (IQA), video quality assessment (VQA), and audio-visual quality assessment (AVQA). Most of these focus on single-modal quality assessment. Despite achieving state-of-the-art performance, a unified quality assessment model is still lacking to simultaneously predict the quality of audio, image, video, and audio-visual content. Since audio, image, video, and audio-visual quality assessments all aim to model the human perception system, they can be compatible and mutually beneficial within a unified quality assessment model.

[0004] Training a single quality assessment model simultaneously across multiple quality assessment datasets across different modalities is a challenge. Joint training directly on multiple quality assessment datasets with different modalities is the most straightforward approach to training a single model. Different quality assessment datasets exhibit different perceptual scales due to varying subjective testing methods, environments, and subjects. For example, the BID dataset (A. Ciancio, E.A. Silva, A. Said, R. Samadani, P. Oprador et al., “No-reference blur assessment of digital pictures based on multifeatured classifiers,” IEEE Transactions on Image Processing, vol.20, no.1, pp.64–75, 2010.) underwent subjective testing in a controlled laboratory setting, with subjective ratings ranging from [0,5]. The CLIVE dataset (D. Ghadiyaram and ACBovik, “Massive online crowdsourced study of subjective and objective picture quality,” IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372–387, 2015.) was tested on an unconstrained crowdsourcing platform with subjective ratings ranging from [0, 100]. This implies that separate subjective experiments are needed to readjust the perceptual scale of images or videos in each dataset.

[0005] Because subjective experiments require significant human and financial resources, researchers have considered pairwise learning on multiple datasets to address the issue of varying perceptual scales across different datasets. They utilized the relative quality scores of each pair of samples, rather than absolute subjective quality scores, as training labels. Different loss functions have been proposed for pairwise learning. For example, Yang et al., in "D. Yang, V.-T. Peltoketo, and J.-K. Kamarainen, 'CNN-based cross-dataset no-reference image quality assessment,'" in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019, pp. 0–0, utilized margin ranking loss and Euclidean loss. Zhang et al., in "W. Zhang, K. Ma, G. Zhai, and X. Yang, 'Uncertainty-aware blind image quality assessment in the laboratory and wild,'" IEEE Transactions on Image Processing, vol. 30, pp. 3474–3486, 2021, utilized fidelity loss to train the image quality assessment model. However, pairwise learning increases training costs. Li et al., in "D.Li, T.Jiang, and M.Jiang, 'Unified quality assessment of in-the-wild videos with mixed datasetstraining,' International Journal of Computer Vision, vol. 129, pp. 1238–1257, 2021," proposed a dataset-specific perceptual scale alignment method for joint training on multiple video quality assessment datasets without incurring additional training costs. However, this study only attempted joint training on a single modality quality assessment dataset. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for joint training of cross-modal multi-quality evaluation datasets.

[0007] According to one aspect of the present invention, a joint training method for cross-modal multi-quality assessment datasets is provided for training video, audio, image, and audio-visual datasets, comprising:

[0008] Construct a cross-modal multi-quality assessment model, including feature extraction and feature fusion components;

[0009] Training the cross-modal multi-quality assessment model includes:

[0010] The first step is to expand the feature fusion part to M based on the number M of the quality assessment datasets to be trained. In the first step of training, the feature extraction part has a common model parameter and structure for all quality assessment datasets. After extracting features, M feature fusion parts corresponding to each quality assessment dataset are input to predict scores. The feature fusion parts corresponding to each dataset are independent of each other and do not interfere with each other.

[0011] The second step involves expanding the feature fusion part to N based on the number of modalities N in the quality assessment dataset to be trained. During the training process in the second step, the feature extraction part inherits the model parameters trained in the first step and continues to train. All quality assessment datasets share a single feature extraction model parameter and structure. After feature extraction, N feature fusion parts corresponding to each modality are input to predict scores, and the same modality shares a single feature fusion part. The quality assessment dataset is at least two of the following: video, audio, image, and audio-visual datasets.

[0012] The third step is to restore the feature fusion part to a single part. During the training process in the third step, the feature extraction part inherits the model parameters after the training in the second step. After freezing the model parameters of the feature extraction part, the feature fusion part is trained. All quality assessment datasets share a single feature fusion part.

[0013] Preferably, the cross-modal multi-quality assessment model is constructed based on deep learning methods. First, the feature extraction part extracts quality-related features from the input data, and then the feature fusion part fuses all the extracted quality-related features to predict the quality score.

[0014] Preferably, in the first step, the subjective testing methods and environments of different quality assessment datasets are different, and the subjective score perception scales in different quality assessment datasets are different. With the help of M feature fusion parts corresponding to the quality assessment datasets, the perception scales of different quality assessment datasets can be automatically adjusted and unified.

[0015] Preferably, in the first step, the loss function is defined as:

[0016] L = L MAE +L rank ,

[0017] Where L MAE L can make predicted scores closer to subjective scores. rank This allows the ranking of predicted scores to more closely approximate subjective scores; LMAE and L rank Defined as:

[0018]

[0019]

[0020]

[0021]

[0022] Where B is the batch size, s i and o i It represents the subjective quality score and prediction score of the i-th image in a batch.

[0023] Preferably, in both the second and third steps, the Spearman rank correlation coefficient (SRCC) loss function is used to transform the quality assessment problem into a learning ranking problem, thus bypassing the perceptual scale problem.

[0024] The SRCC loss function is defined as:

[0025] L SRCC =1-SRCC,

[0026]

[0027] in and Sorting of predicted scores And the sorting of truth value quality scores

[0028] Preferably, in the first, second, and third steps, for each optimization step, a small batch is extracted from the same quality evaluation dataset, the gradient is calculated, and then the parameters are updated.

[0029] The sampling frequency is determined by a weighted sampling method, which means that each quality assessment dataset is sampled according to the number of training steps in each dataset.

[0030] Preferably, in the third step, the feature vectors extracted by different modalities using the same feature extraction part have different lengths, and the feature vector lengths are padded to the same length by zero-padding.

[0031] According to a second aspect of the present invention, a joint training system for cross-modal multi-quality assessment datasets is provided, comprising:

[0032] Building modules: Constructing a cross-modal multi-quality assessment model, including feature extraction and feature fusion components;

[0033] Training module: This module trains the cross-modal multi-quality assessment model; it includes:

[0034] The first training submodule expands the feature fusion part to M based on the number M of the quality evaluation datasets to be trained. In the first training step, the feature extraction part has a common model parameter and structure for all datasets. After extracting features, M feature fusion parts corresponding to each dataset are input to predict scores. The feature fusion parts corresponding to each dataset are independent of each other and do not interfere with each other.

[0035] The second training submodule expands the feature fusion part to N based on the number of modalities N in the quality evaluation dataset to be trained. In the second training step, the feature extraction part inherits the model parameters trained in the first step and continues to train. All datasets share a feature extraction model parameter and structure. After extracting features, N feature fusion parts corresponding to each modality are input to predict scores. The same modality shares a feature fusion part.

[0036] The third training submodule restores the feature fusion part to one; in the third training step, the feature extraction part inherits the model parameters after the second training step, the model parameters of the feature extraction part are frozen, and the feature fusion part is trained, so that all quality assessment datasets share a single feature fusion part.

[0037] According to a third aspect of the present invention, a multimodal quality assessment method is provided for quality assessment of video, audio, image, and audio-visual datasets. The method includes a training unit and an application unit, comprising:

[0038] The training unit uses the aforementioned cross-modal multi-quality assessment dataset joint training method to obtain a well-trained cross-modal multi-quality assessment model.

[0039] The application unit uses the pre-trained cross-modal multi-quality assessment model to assess the quality of video, audio, image, and audio-visual datasets, and obtains quality assessment scores.

[0040] According to a fourth aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the cross-modal multi-quality assessment dataset joint training method, or to run the cross-modal multi-quality assessment dataset joint training system.

[0041] According to a fifth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the aforementioned cross-modal multi-quality assessment dataset joint training method, or to run the aforementioned cross-modal multi-quality assessment dataset joint training system.

[0042] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0043] The cross-modal multi-quality assessment dataset joint training method and system in this invention embodiment can effectively utilize cross-modal audio, image, video and audio-visual quality assessment datasets to jointly train a multi-modal cross-modal multi-quality assessment model, and the obtained cross-modal multi-quality assessment model performs well in performance tests. Attached Figure Description

[0044] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 This is a flowchart of a cross-modal multi-quality assessment dataset joint training method according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating the training process of a cross-modal multi-quality assessment model according to a preferred embodiment of the present invention.

[0047] Figure 3 This is a flowchart of weighted task sampling in a preferred embodiment of the present invention;

[0048] Figure 4 This is a framework diagram of a cross-modal multi-quality assessment dataset joint training system according to an embodiment of the present invention. Detailed Implementation

[0049] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0050] This invention provides a method for joint training of cross-modal multi-quality assessment datasets, used for training video, audio, graphics, and audio-visual datasets. (Refer to...) Figure 1 and Figure 2 The specific steps are as follows:

[0051] S100, construct a cross-modal multi-quality evaluation model, including feature extraction and feature fusion;

[0052] S200, training a cross-modal multi-quality assessment model, including:

[0053] S201, First step: Based on the number M of the quality evaluation datasets to be trained, expand the feature fusion part to M; During the first step of training, the feature extraction part has a common model parameter and structure for all datasets. After extracting features, input M feature fusion parts corresponding to each dataset to predict scores. The feature fusion parts corresponding to each dataset are independent of each other and do not interfere with each other.

[0054] S202, the second step, expands the feature fusion part to N based on the number of modalities N in the quality assessment dataset to be trained; during the training process in the second step, the feature extraction part inherits the model parameters trained in the first step and continues to train. All datasets share a feature extraction model parameter and structure. After extracting features, N feature fusion parts corresponding to each modality are input to predict scores. The same modality shares a feature fusion part; the quality assessment dataset is at least two of the following: video, audio, image, and audio-visual datasets, i.e., M and N are natural numbers not less than 2.

[0055] S203, the third step, restores the feature fusion part to one; during the training process in the third step, the feature extraction part inherits the model parameters after the training in the second step. After freezing the model parameters of the feature extraction part, the feature fusion part is trained, and all quality assessment datasets share one feature fusion part.

[0056] This embodiment proposes a cross-modal joint training method, which jointly trains a unified cross-modal multi-quality assessment model across quality assessment datasets of different modalities without increasing the overall training cost.

[0057] Since common quality assessment dataset types include audio quality assessment datasets, image quality assessment datasets, video quality assessment datasets, and audio-visual quality assessment datasets, in a preferred embodiment of the present invention, the specific steps of the cross-modal multi-quality assessment dataset joint training method are introduced as an example of joint training on 3 audio quality assessment datasets, 4 image quality assessment datasets, 3 video quality assessment datasets, and 2 audio-visual quality assessment datasets.

[0058] In a preferred embodiment, S100 is implemented to divide the cross-modal multi-quality assessment model into two parts: feature extraction and feature fusion. In the feature extraction part, audio features are extracted using a pre-trained NISQA audio quality assessment network, image features are extracted using a ConvNeXt V2 neural network pre-trained on ImagNet, and video features are extracted using a SlowFast R50 action recognition network pre-trained on the Kinetics 400 dataset. The combined features are then input into the feature fusion part, which consists of fully connected layers, to predict the final quality score.

[0059] In a preferred embodiment, step S201 is implemented where, during the first training step, the feature extraction part remains unchanged, while the feature fusion part is expanded to 12 based on the number of quality assessment datasets to be trained. Since the subjective testing methods and environments differ across quality assessment datasets, the perceived scale of subjective scores varies. By utilizing the 12 feature fusion parts corresponding to each dataset, the perceived scale of different datasets can be automatically adjusted and unified. During joint training, all datasets share a common feature extraction part to extract features, which are then input into the corresponding feature fusion part to predict the quality score.

[0060] Furthermore, for each optimization step, a small batch is drawn from the same dataset, the gradient is calculated, and then the parameters are updated. A weighted task sampling method is used to determine the sampling frequency, such as... Figure 3 As shown, sampling is performed on each dataset according to the number of training steps required for each dataset. For example, if the data ratio of the datasets is a:b:c, then the ratio of the number of training steps required for the three datasets is also a:b:c. The weighted task sampling method can balance the number of training steps for each dataset, thereby averaging the number of training steps for each sample in each dataset. This prevents excessive training steps for samples in small datasets and insufficient training steps for samples in large datasets, ultimately enabling the model to achieve good performance on each dataset.

[0061] Furthermore, the loss function is defined as:

[0062] L = L MAE +L rank ,

[0063] Where L MAE This allows predicted scores to more closely approximate subjective scores, L rank This allows the ranking of predicted scores to more closely approximate subjective scores. MAE and L rank It can be defined as:

[0064]

[0065]

[0066]

[0067]

[0068] Where B is the batch size, s i and o i It represents the subjective quality score and prediction score of the i-th image in a batch.

[0069] In a preferred embodiment, in step S202, during the second training step, the 12 feature fusion parts divided according to the dataset are replaced with 4 feature fusion parts divided according to the input modality, namely audio, image, and video (audio-video feature fusion parts). During training, the feature extraction part inherits the model parameters trained in the first step and predicts the final quality score through the feature fusion parts of the corresponding modality.

[0070] Furthermore, similar to the first step of the training process, a weighted task sampling method is also used, and each optimization step involves sampling a small batch from the same dataset.

[0071] Furthermore, since datasets with the same modality but different perceptual scales will utilize the same feature fusion component to predict the final score, the Spearman Rank Correlation Coefficient (SRCC) loss function is used in the second training step to transform the quality assessment problem into a ranking problem, thus bypassing the perceptual scale issue. The SRCC loss function can be defined as:

[0072] L SRCC =1-SRCC,

[0073]

[0074] in and Sorting of predicted scores And the sorting of truth value quality scores

[0075] In a preferred embodiment, S203 is implemented. In the third training step, the four feature fusion parts divided according to the input modality in the second training step are merged into one feature fusion part. During the training, the feature extraction part inherits the parameters after the second training step. The feature extraction part is fixed, and only the feature fusion module is trained. Since the feature vectors extracted by the feature extraction module for different modalities are not the same length, the feature vector length is padded to the same length by zero-padding. The weighted task sampling method and Spearman rank correlation coefficient (SRCC) are also used as the loss function.

[0076] Based on the same inventive concept, in other embodiments of the present invention, a multimodal quality assessment method is provided for evaluating the quality of datasets of various modalities, such as video, audio, images, and audio-visual media. This multimodal quality assessment method includes a training unit and an application unit, comprising:

[0077] The training unit uses the cross-modal multi-quality assessment dataset joint training method or system in the above embodiments to obtain a well-trained cross-modal multi-quality assessment model;

[0078] The application unit uses the pre-trained cross-modal multi-quality assessment model to evaluate the quality of various modal datasets, such as video, audio, image, and audio-visual data, and obtains quality assessment scores.

[0079] The multimodal quality assessment method described above allows for a direct understanding of the quality level of the corresponding dataset, determining whether the dataset meets the requirements of various social media and streaming media, and further improving the shooting, compression, and transmission processes.

[0080] Based on the same inventive concept, a joint training system for cross-modal multi-quality assessment datasets is provided. See [link to relevant documentation]. Figure 4 ,include:

[0081] Building modules: Constructing a cross-modal multi-quality assessment model, including feature extraction and feature fusion components;

[0082] Training module: This module trains the cross-modal multi-quality assessment model; it includes:

[0083] The first training submodule expands the feature fusion part to M based on the number M of the quality evaluation datasets to be trained. In the first training step, the feature extraction part has a common model parameter and structure for all datasets. After extracting features, M feature fusion parts corresponding to each dataset are input to predict scores. The feature fusion parts corresponding to each dataset are independent of each other and do not interfere with each other.

[0084] The second training submodule expands the feature fusion part to N based on the number of modalities N in the quality evaluation dataset to be trained. In the second training step, the feature extraction part inherits the model parameters trained in the first step and continues to train. All datasets share a feature extraction model parameter and structure. After extracting features, N feature fusion parts corresponding to each modality are input to predict scores. The same modality shares a feature fusion part.

[0085] The third training submodule restores the feature fusion part to one; in the third training step, the feature extraction part inherits the model parameters after the second training step, the model parameters of the feature extraction part are frozen, and the feature fusion part is trained, so that all quality assessment datasets share a single feature fusion part.

[0086] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the cross-modal multi-quality assessment dataset joint training method in the above embodiments, and will not be repeated here.

[0087] Based on the same inventive concept, in other embodiments, a terminal is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the cross-modal multi-quality assessment dataset joint training method, or to run the cross-modal multi-quality assessment dataset joint training system.

[0088] Based on the same inventive concept, in other embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the cross-modal multi-quality assessment dataset joint training method, or to run the cross-modal multi-quality assessment dataset joint training system.

[0089] Implementation results:

[0090] To verify the effectiveness of the cross-modal multi-quality assessment dataset joint training method provided in the above embodiments of the present invention, experiments were conducted on three audio quality assessment datasets, four image quality assessment datasets, three video quality assessment datasets, and two audio-visual quality assessment datasets:

[0091] - Audio quality assessment datasets: ITU-T dataset (“ITU-T coded-speechdatabase,” Series P, Supplement, vol. 23, 1998.), NOIZEUS dataset (“Subjective comparison of speech enhancement algorithms,” in Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, 2006, pp. I–I.), and TCDVoIP dataset (“TCD-voip, a research database of degraded speech for assessing quality in voip applications,” in Proceedings of the International Workshop on Quality of Multimedia Experience, 2015, pp. 1–6.).

[0092] -Image quality evaluation datasets: BID dataset (A. Ciancio, E. A. da Silva, A. Said, R. Samadani, P. Obrador et al., "No-reference blur assessment of digital pictures based on multifeatured classifiers," IEEE Transactions on Image Processing, vol. 20, no. 1, pp. 64–75, 2010.), CLIVE dataset (D. Ghadiyaram and A. C. Bovik, "Massive online crowdsourced study of subjective and objective picture quality," IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372–387, 2015.), KonIQ-10K dataset (V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, "Koniq-10k: an ecologically valid database for deep learning of blind image quality assessment," IEEE Transactions on Image Processing, vol. 29, pp. 4041–4056, 2020.) and SPAQ dataset (Y. Fang, H. Zhu, Y. Zeng, K. Ma, and Z. Wang, "Perceptual quality assessment of smartphone photography," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 3677–3686.).

[0093] - Video quality assessment datasets: YouTube-UGC dataset (Y. Wang, S. Inguva, and B. Adsumilli, “YouTube UGC dataset for video compression research,” in Proceedings of the International Workshop on Multimedia Signal Processing, 2019, pp. 1–5), KoNViD-1k dataset (V. Hosu, F. Hahn, M. Jenadeleeh, H. Lin, H. Men, T. Sziranyi, S. Li, and D. Saupe, “The konstanz natural video database (konvid-1k),” in Proceedings of the International Conference on Quality of Multimedia Experience, 2017, pp. 1–6.) and LIVE-VQC dataset (Z. Sinno and A.C. Bovik, “Large-scale study of perceptual video quality,” IEEE Transactions on Image Processing, vol. 28, no. 2, pp. 612–627, 2018.)

[0094] - Audio-visual quality assessment datasets: SJTU-UAV dataset (“Y.Cao, X.Min, W.Sun, X.Zhang, and G.Zhai, “Audio-visual quality assessment for user-generated content: database and method,” ArXiv PreprintArXiv:2303.02392, 2023.) and LIVE-SJTU dataset (“X.Min, G.Zhai, J.Zhou, MCFarias, and ACBovik, “Study of subjective and objective quality assessment of audio-visual signals,” IEEE Transactions on Image Processing, vol.29, pp.6054–6068, 2020.)

[0095] Table 1

[0096]

[0097] Table 2

[0098]

[0099] Table 3

[0100]

[0101] Table 4

[0102]

[0103] Direct joint training involves normalizing the subjective scores from different datasets to 0-1 and then directly inputting them into the model for joint training. The performance comparison of the proposed method and direct joint training on audio quality evaluation datasets, image quality evaluation datasets, video quality evaluation datasets, and audio-visual quality evaluation datasets are shown in Tables 1, 2, 3, and 4, respectively. As can be seen from the tables, the method proposed in the above embodiments of the present invention can effectively improve the cross-modal joint training effect on multiple audio, image, video, and audio-visual datasets.

[0104] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A joint training method for cross-modal multi-quality assessment datasets, used for training on datasets of video, audio, images, and audio-visual data, characterized in that, include: Construct a cross-modal multi-quality assessment model. It includes a feature extraction part and a feature fusion part; Training the cross-modal multi-quality assessment model includes: The first step is to determine the number of quality assessment datasets to be trained. Expanding the feature fusion part to In the first step of training, the feature extraction part shares a common model parameter and structure across all quality assessment datasets. This is input after feature extraction. Each feature fusion component predicts a score corresponding to each quality assessment dataset, and the feature fusion components corresponding to each dataset are independent of each other and do not interfere with each other. The second step is to determine the number of modalities in the quality assessment dataset to be trained. Expanding the feature fusion part to In the second training step, the feature extraction part inherits the model parameters trained in the first step and continues training. All quality assessment datasets share a common feature extraction model parameter and structure, which is input after feature extraction. Each modality has a predicted score for a feature fusion component, with the same modality sharing a single feature fusion component; the quality assessment dataset is at least two of the following: video, audio, image, and audio-visual datasets; The third step is to restore the feature fusion part to a single part. During the training process in the third step, the feature extraction part inherits the model parameters after the training in the second step. After freezing the model parameters of the feature extraction part, the feature fusion part is trained. All quality assessment datasets share a single feature fusion part.

2. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, The cross-modal multi-quality assessment model is constructed based on deep learning methods. First, the feature extraction part extracts quality-related features from the input data. Then, the feature fusion part fuses all the extracted quality-related features to predict the quality score.

3. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, In the first step, the methods and environments for subjective testing differ across different quality assessment datasets, and the subjective score perception scales also differ. M A feature fusion component corresponding to a quality assessment dataset automatically adjusts and unifies the perception scale of different quality assessment datasets.

4. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, In the first step, the loss function is defined as: ； in This allows predicted scores to more closely approximate subjective scores. It can make the ranking of predicted scores closer to subjective scores; and Defined as: ； ； ； ； in It refers to the batch size. and It is the first in a batch The subjective quality score and prediction score of each image.

5. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, In both the second and third steps, the Spearman rank correlation coefficient (SRCC) loss function is used to transform the quality assessment problem into a learning ranking problem, thus bypassing the perceptual scale problem. The SRCC loss function is defined as: ； ； in and Sorting of predicted scores And the sorting of truth value quality scores .

6. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, In the first, second, and third steps, for each optimization step, a small batch is extracted from the same quality assessment dataset, the gradient is calculated, and then the parameters are updated. The sampling frequency is determined by a weighted sampling method, which means that each quality assessment dataset is sampled according to the number of training steps in each dataset.

7. The method for joint training of cross-modal multi-quality assessment datasets according to claim 1, characterized in that, In the third step, the feature vectors extracted by different modalities using the same feature extraction part have different lengths. The feature vector lengths are padded to the same length by zero-padding.

8. A joint training system for cross-modal multi-quality assessment datasets, used for training video, audio, image, and audio-visual datasets, characterized in that, include: Module building: Constructing a cross-modal, multi-quality assessment model. It includes a feature extraction part and a feature fusion part; Training module: Trains the cross-modal multi-quality assessment model; It includes: The first training submodule, based on the number of quality evaluation datasets to be trained... Expanding the feature fusion part to In the first step of training, the feature extraction part shares a common model parameter and structure across all quality assessment datasets. This is input after feature extraction. Each feature fusion component predicts a score corresponding to each quality assessment dataset, and the feature fusion components corresponding to each dataset are independent of each other and do not interfere with each other. The second training submodule determines the number of modes in the quality evaluation dataset to be trained. Expanding the feature fusion part to In the second training step, the feature extraction part inherits the model parameters trained in the first step and continues training. All quality assessment datasets share a common feature extraction model parameter and structure, which is input after feature extraction. Each modality has a predicted score for a feature fusion component, with the same modality sharing a single feature fusion component; the quality assessment dataset is at least two of the following: video, audio, image, and audio-visual datasets; The third training submodule restores the feature fusion part to one; in the third training step, the feature extraction part inherits the model parameters after the second training step, the model parameters of the feature extraction part are frozen, and the feature fusion part is trained, so that all quality assessment datasets share a single feature fusion part.

9. A multimodal quality assessment method for evaluating the quality of video, audio, image, and audio-visual datasets, characterized in that, The method includes a training unit and an application unit, wherein: The training unit uses the joint training method for cross-modal multi-quality assessment datasets as described in any one of claims 1-7 to obtain a well-trained cross-modal multi-quality assessment model. The application unit uses the pre-trained cross-modal multi-quality assessment model to assess the quality of video, audio, image, and audio-visual datasets, and obtains quality assessment scores.

10. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

Images