Quality evaluation method, network training method and device for video content

Abstract

The disclosure provides a quality evaluation method and device of video content and a network training method, relates to the technical field of artificial intelligence, and particularly relates to the technical fields of big data, deep learning and intelligent recommendation and the like. The specific implementation scheme is as follows: acquiring a multi-modal feature of a target video; acquiring an information feature of reference information of the target video; the reference information comprises account information of creating the target video and / or operation records of the target video; and determining a content quality of the target video based on the multi-modal feature and the information feature. The disclosure uses the multi-modal feature of the video signal and the information feature of the discrete information of the video to perform quality evaluation, realizes comprehensive evaluation of the content quality of the video from multiple angles, and can obtain an accurate video quality evaluation result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of big data, deep learning, and intelligent recommendation. Background Technology

[0002] With the rapid development of internet technology, massive amounts of video resources have been uploaded to the network. Filtering high-quality videos from this vast pool and recommending them to users to improve their viewing experience can increase user retention. Therefore, a method for evaluating video content quality is needed to select high-quality video resources. Summary of the Invention

[0003] This disclosure provides a method for evaluating the quality of video content, a network training method, and an apparatus.

[0004] According to one aspect of this disclosure, a method for evaluating the quality of video content is provided, comprising:

[0005] Obtain the multimodal features of the target video; and;

[0006] Information characteristics for obtaining reference information about the target video; the reference information includes the account information that created the target video and / or the operation records of the target video;

[0007] The content quality of the target video is determined based on multimodal features and information features.

[0008] According to another aspect of this disclosure, a neural network training method is provided, comprising:

[0009] The sample image is input into the image-based mask autoencoder network (MAE) to obtain the initial image features of the sample image;

[0010] The initial image features of the sample image are input into the feature extraction module of the first fully connected network to obtain the image features of the sample image;

[0011] The classification module of the first fully connected network is used to classify the image features of the sample images to obtain the content quality of the sample images;

[0012] Loss is determined based on the content quality and quality labels of the sample images;

[0013] The network parameters of the feature extraction module and classification module of the first fully connected network are adjusted based on the loss;

[0014] Among them, the image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0015] According to another aspect of this disclosure, a neural network training method is provided, comprising:

[0016] The sample video is input into the video-based mask autoencoder network (MAE) to obtain the initial video features of the sample video.

[0017] The initial video features of the sample video are input into the feature extraction module of the second fully connected network to obtain the video features of the sample video.

[0018] The classification module of the second fully connected network is used to classify the video features of the sample video to obtain the content quality of the sample video;

[0019] Loss is determined based on the content quality and quality labels of the sample videos;

[0020] The network parameters of the feature extraction module and classification module of the second fully connected network are adjusted based on the loss;

[0021] Among them, the video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0022] According to another aspect of this disclosure, a video content quality evaluation apparatus is provided, comprising:

[0023] The acquisition module is used to acquire the multimodal features of the target video; and to acquire the information features of the reference information of the target video; the reference information includes the account information of the creator of the target video and / or the operation records of the target video;

[0024] The evaluation module is used to determine the content quality of the target video based on multimodal features and information features.

[0025] According to another aspect of this disclosure, a neural network training apparatus is provided, comprising:

[0026] The initial image feature extraction module is used to input the sample image into the image-based mask autoencoder network (MAE) to obtain the initial image features of the sample image.

[0027] The image feature extraction module is used to input the initial image features of the sample image into the feature extraction module of the first fully connected network to obtain the image features of the sample image;

[0028] The first quality determination module is used to classify the image features of the sample image using the classification module of the first fully connected network to obtain the content quality of the sample image.

[0029] The first loss determination module is used to determine the loss based on the content quality and quality label of the sample image;

[0030] The first adjustment module is used to adjust the network parameters of the feature extraction module and the classification module of the first fully connected network based on the loss.

[0031] Among them, the image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0032] According to another aspect of this disclosure, a neural network training apparatus is provided, comprising:

[0033] The initial video feature determination module is used to input the sample video into the video-based mask autoencoder network (MAE) to obtain the initial video features of the sample video.

[0034] The video feature determination module is used to input the initial video features of the sample video into the feature extraction module of the second fully connected network to obtain the video features of the sample video.

[0035] The second quality determination module is used to classify the video features of the sample video using the classification module of the second fully connected network to obtain the content quality of the sample video.

[0036] The second loss determination module is used to determine the loss based on the content quality and quality label of the sample video.

[0037] The second adjustment module is used to adjust the network parameters of the feature extraction module and the classification module of the second fully connected network based on the loss.

[0038] Among them, the video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0039] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0040] At least one processor; and

[0041] The memory is communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0043] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0044] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0045] The video content quality assessment method provided in this disclosure uses modal features and information features to evaluate videos, enabling a comprehensive evaluation of video content from multiple perspectives.

[0046] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0047] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0048] Figure 1 This is a flowchart illustrating a video content quality evaluation method according to an embodiment of the present disclosure;

[0049] Figure 2 This is a flowchart illustrating a method for training a divine general network according to an embodiment of the present disclosure;

[0050] Figure 3 This is a flowchart illustrating yet another neural network training method provided according to yet another embodiment of the present disclosure;

[0051] Figure 4 This is a structural diagram of video content evaluation according to an embodiment of the present disclosure;

[0052] Figure 5 This is a schematic diagram of a MAE training method provided according to an embodiment of the present disclosure;

[0053] Figure 6 This is a framework diagram for video content evaluation provided according to an embodiment of the present disclosure;

[0054] Figure 7 This is a schematic diagram of a video content quality evaluation device according to an embodiment of the present disclosure;

[0055] Figure 8 This is a schematic diagram of a neural network training apparatus provided according to an embodiment of the present disclosure;

[0056] Figure 9 This is a schematic diagram of yet another neural network training apparatus provided according to yet another embodiment of the present disclosure;

[0057] Figure 10 This is a block diagram of an electronic device used to implement the video content evaluation method or network training method of the embodiments of this disclosure. Detailed Implementation

[0058] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] In the information age, the internet contains a vast amount of information, among which video, with its intuitive and efficient advantages, has become the mainstream method of information dissemination, and various video platforms have a large number of users. Users obtain video resources from video platforms mainly in two ways: one is by directly searching using keywords to purposefully find a specific type or category of video; the other is by the video platform directly recommending video resources to users.

[0060] When video platforms recommend video resources to users, recommending low-quality videos, such as those containing vulgar elements or plagiarized videos, will significantly impact the user's viewing experience and lower their evaluation of the platform. Conversely, recommending high-quality videos will improve the user's viewing experience and increase user retention. The processing resources used for recommending high-quality videos will also be used effectively, avoiding waste. Therefore, this disclosure provides a method for evaluating the quality of video content, assessing the target video's quality from multiple perspectives, such as... Figure 1 As shown, the method includes:

[0061] S101. Obtain the multimodal features of the target video.

[0062] Multimodal features, as the name suggests, refer to features extracted from multiple modalities of information. Multimodal features are beneficial for comprehensively describing video content, thereby improving the accuracy of video content quality assessment. Given that the target video contains multiple modalities of information, including image information, video information, audio information, and text information, in practice, features can be extracted from at least two of these types of information to obtain multimodal features. For example, image features of each frame in the target video, video features of the target video, and text features of the text information in the target video can be obtained. Then, at least two of these features are selected as features to be fused, and the obtained at least two features to be fused are fused to obtain the multimodal features of the target video.

[0063] Utilizing multimodal features for video quality assessment allows for more comprehensive use of information from multiple modalities, including video, audio, and text. Compared to single-modal video assessment, it possesses stronger representational capabilities, leading to more accurate evaluation of video content quality. Furthermore, the multiple features extracted in this disclosure are only used for fusion to generate multimodal features and are not required to perform any specific task. Therefore, unsupervised training models can be used to extract multiple features to be fused, making the extracted features more generalizable.

[0064] For ease of understanding, the following examples illustrate how image features, video features, and text features are obtained.

[0065] 1) Extracting image features:

[0066] In some embodiments, image features of each frame can be extracted using an image-based MAE (Masked Autoencoder) network.

[0067] Furthermore, in order to extract image features that are more suitable for video content evaluation, embodiments of this disclosure employ a combination of image-based MAE and a first fully connected network to extract image features for each frame.

[0068] First, an image-based image quality assessment (MAE) can be trained, and then a first fully connected network can be trained based on the trained MAE. The first fully connected network is trained to perform image quality assessment. This network includes a feature extraction module and a classification module. The feature extraction module extracts image features, and the classification module performs quality assessment based on these features, i.e., assigns a quality level. The classification module assists in training the feature extraction module. In one possible embodiment, the fully connected network includes an input layer I, hidden layers H, and an output layer O. The input layer I and hidden layers H can be considered as the feature extraction module, and the output layer O as the classification module. Figure 2 The diagram shown illustrates the training process of the first fully connected network, including:

[0069] S201. Input the sample image into the image-based MAE to obtain the initial image features of the sample image.

[0070] S202. Input the initial image features of the sample image into the feature extraction module of the first fully connected network to obtain the image features of the sample image.

[0071] S203. The classification module of the first fully connected network is used to classify the image features of the sample image to obtain the content quality of the sample image.

[0072] S204. Determine the loss based on the content quality and quality label of the sample image.

[0073] S205. Adjust the network parameters of the feature extraction module and classification module of the first fully connected network based on the loss.

[0074] Among them, the image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0075] This training method employs image-based MAE to extract initial image features, ensuring that the extracted features are not affected by occlusion and possess greater stability and expressiveness. Furthermore, by training a first fully connected network, the network not only performs video quality assessment but also learns more relevant image features from the initial image features, resulting in more expressive extracted features and improved accuracy of video quality assessment.

[0076] Based on the above description, after training the image-based MAE and the first fully connected network, the image features of each frame in the target video can be obtained using the following methods:

[0077] Step A1: Extract multiple frames of images from the target video.

[0078] In practice, the multi-frame images can include each frame of the target video, or keyframe images can be extracted as multi-frame images for extracting image features.

[0079] Step A2: Input each frame of image into the image-based MAE to obtain the initial image features corresponding to each frame.

[0080] Image-based MAE is an unsupervised training model. When an occluded image is input into MAE, MAE restores the occluded image to its unoccluded state.

[0081] Step A3: Input the initial image features of each frame into the feature extraction module of the first fully connected network to obtain the image features of each frame.

[0082] Using unsupervised training models allows for more generalized tasks, resulting in more universal image features. Simultaneously, MAE can restore occluded images, enabling relatively accurate feature extraction even when a frame of the target video is occluded. Furthermore, in this embodiment, in conjunction with video content quality assessment requirements, a first fully connected network meeting video content assessment needs is employed to further extract features based on the features extracted by image-based MAE, thereby obtaining more expressive image features and improving the accuracy of video content quality assessment.

[0083] 2) Extract video features:

[0084] In this disclosure, since it is necessary to evaluate the quality of the target video content, and the video content is inseparable from the temporal sequence of the images, it is necessary to consider the relationship between the video images and the time series when obtaining the video features of the target video.

[0085] Similar to image feature extraction, embodiments of this disclosure can employ video-based MAE to extract video features. To accurately assess image content quality and obtain video features more suitable for quality evaluation, embodiments of this disclosure combine video-based MAE with a second fully connected network to extract video features.

[0086] First, a video-based content quality assessment (MAE) can be trained, and then a second fully connected network can be trained based on this trained MAE. The second fully connected network is trained to perform video content quality assessment. This second fully connected network includes a feature extraction module and a classification module. The feature extraction module extracts video features, and the classification module performs quality assessment based on these features, i.e., assigns a quality level. The classification module assists in training the feature extraction module. In one possible embodiment, the fully connected network includes an input layer I, hidden layers H, and an output layer O. The input layer I and hidden layers H can be considered as the feature extraction module, and the output layer O as the classification module. Figure 3 The diagram shown illustrates the training process of the second fully connected network, including:

[0087] S301. Input the sample video into the video-based MAE to obtain the initial video features of the sample video;

[0088] S302. Input the initial video features of the sample video into the feature extraction module of the second fully connected network to obtain the video features of the sample video;

[0089] S303. The classification module of the second fully connected network is used to classify the video features of the sample video to obtain the content quality of the sample video.

[0090] S304. Determine the loss based on the content quality and quality labels of the sample videos;

[0091] S305. Adjust the network parameters of the feature extraction module and classification module of the second fully connected network based on the loss;

[0092] Among them, the video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0093] This training method uses video-based MAE to extract initial video features, which contain temporal information. Then, by training a second fully connected network, the second fully connected network can learn more video features from the initial video features while simultaneously evaluating video quality. This makes the extracted video features more expressive and improves the accuracy of video quality evaluation.

[0094] Therefore, after training the video-based MAE and the second fully connected network, the video features of the target video can be obtained using the following method:

[0095] Step B1: Input the frame sequence of the target video into the video-based MAE in sequence to obtain the initial video features of the target video.

[0096] Among them, the video-based Autoencoder (MAE) network, similar to the image-based MAE network, is an unsupervised training model capable of restoring occluded parts of an image. Furthermore, when extracting video features, the video-based MAE network effectively considers the temporal signals of the video, identifying actions in consecutive frames, thus ensuring that the obtained video features reflect the temporal order of the target video.

[0097] Step B2: Input the initial video features into the feature extraction module of the second fully connected network to obtain the video features of the target video.

[0098] Video-based MAE can restore occluded image patches and their temporal information, enabling relatively accurate extraction of temporal video features even when the target video is occluded. Using temporally-considered MAE also improves video content evaluation. Furthermore, in this embodiment, in addition to meeting video content quality evaluation requirements, a first fully connected network satisfying video content evaluation needs is used to further extract features based on the features extracted by video-based MAE, resulting in more expressive video features and thus improving the accuracy of video content quality evaluation.

[0099] 3) Extract text features:

[0100] In some embodiments, text information related to the target video can be used to extract text features for video content quality evaluation. For example, text features can be extracted based on the audio of the target video.

[0101] To extract more expressive text features, this embodiment extracts text features from various text information. For example, the following method can be used to obtain text features from the text information of a target video:

[0102] Step C1: Extract the first text information contained in the video frames of the target video; and obtain the second text information in the audio of the target video.

[0103] The initial text information included in the video can be obtained from each frame of the video using OCR (optical character recognition) technology.

[0104] Step C2: Extract features from the first and second text information to obtain text features.

[0105] When extracting text features from the first text information, the unsupervised training model used can be BERT (Bidirectional Encoder Representation from Transformers), a bidirectional language representation training model. BERT can generate deep bidirectional language representations that incorporate contextual information. When obtaining the second text information from the audio of the target video and performing feature extraction, the unsupervised training model used can be HuBERT (Hidden-unitBERT). HuBERT uses clustering to provide labels for BERT, and then uses a masking loss similar to BERT to allow the model to learn the acoustic and language models in continuous speech data.

[0106] Extracting the first text information from the target video and the second text information from the audio, and performing feature extraction on both types of text information separately, can more comprehensively extract the text features of the target video, thereby improving the accuracy of video content quality evaluation.

[0107] In some embodiments, the fusion process can be performed using the concat method. Concat is a method that combines multiple features based on their dimensions. For example, consider two feature vectors e1 = ([[1,2],[3,4]]) and e2 = ([[5,6]]). Fusing the row dimensions of e1 and e2 yields the following result:

[0108] ([[1,2],

[0109] [3,4],

[0110] [5,6]])

[0111] When using concat for multimodal feature fusion, the dimensions of each feature vector must be the same.

[0112] It should be noted that `concat` is only one method to achieve multimodal feature fusion. The `add` method can also be used to directly fuse multiple features together, or an attention mechanism can be used to fuse the features. This disclosure does not impose specific limitations on the method of using multiple features for fusion.

[0113] S102. Information features of the reference information of the target video; the reference information includes the account information of the creator of the target video and / or the operation record of the target video.

[0114] In some embodiments, account information includes at least one of the following: number of followers, number of works, and whether the creator is a high-quality creator. Operation records include at least one of the following: number of likes, number of dislikes, number of shares, and number of comments on the video. The more followers and works a target video creator has, the more likely the target video is to be a high-quality work. Creators with a large number of high-quality videos are considered high-quality creators, and some official accounts can also be certified as high-quality creators due to the authority of their published works. Likes and shares indicate user appreciation of the video, so more likes and shares mean higher video quality. Similarly, more dislikes indicate less user dislike of the video, thus lower video quality. More comments indicate higher video popularity; however, since comments can be positive or negative, they need to be reviewed. More positive comments generally indicate higher video quality.

[0115] Vectorizing the obtained account information yields the information features of the reference information. Vectorization methods can include word2vec (word to vector) or distributed representation. The resulting discrete vectors after vectorizing the account information represent the desired information features.

[0116] In summary, information such as the target video creator's account information and the target video's activity logs are highly valuable for evaluating video content. Utilizing this information can significantly improve the accuracy of quality assessments when evaluating the quality of target video content.

[0117] S103. Determine the content quality of the target video based on multimodal features and information features.

[0118] In some embodiments, a transformer can be used to fuse the multimodal features and information features described above, and the quality of the video can be evaluated based on the fusion result.

[0119] The structure of a transformer is as follows: Figure 4 As shown, it mainly consists of an encoder and a decoder. When using a transformer to implement feature fusion and quality assessment, the multimodal features and information features are first input into the transformer's encoder. The encoder contains a feature extraction module and a feature enhancement module, which can aggregate and enhance the features. The features processed by the encoder are then input into the decoder, where the quality assessment of the video is performed. In some embodiments, the quality assessment result can be different quality levels. For example, there are five quality levels from 1 to 5, where 1 is the highest quality video, 5 is the lowest quality video, and the quality of the remaining levels decreases sequentially.

[0120] Using multimodal features allows for the evaluation of video from multiple modal information perspectives. Furthermore, by combining reference information, the discrete information of the target video can be comprehensively used to evaluate the video content. Therefore, the embodiments of this disclosure can comprehensively evaluate video content from multiple perspectives.

[0121] For users, the attractiveness of a video cover image significantly influences whether they click to watch the video. Therefore, in some embodiments, when evaluating the quality of a target video, in addition to extracting the multimodal features and information features of the target video, the cover quality features can also be extracted. In practice, the video cover image can be input into an aesthetic model for image quality aesthetic model recognition to obtain the cover quality features of the target video. It should be noted that this disclosure does not specifically limit the type of aesthetic model used. For example, NIMA (Neural Image Assessment) or ILGNet (Inception Local Global Net, a deep convolutional neural network) can be used. Using an aesthetic model, the aesthetic aspects of the target video cover image can be scored, including image sharpness, color richness, and artistic quality. The cover image of the target video is used to obtain the cover quality features of the target video through the aesthetic model. After fusing the multimodal features, information features, and cover quality features, a more accurate content quality of the target video is obtained based on the fusion result.

[0122] Based on the same technical concept, this disclosure also provides a method for training neural networks to extract image features from sample images and video features from sample videos.

[0123] For specific training methods, please refer to Figure 2 and Figure 3The training methods described earlier will not be elaborated upon here. Both methods utilize a pre-trained MAE to extract image or video features from the target video, facilitating the extraction of features suitable for video content quality assessment. The following will combine... Figure 5 This section provides a detailed introduction to the training process of image-based MAE.

[0124] First, the sample image is divided into multiple image blocks, and then a portion of the image is masked. Figure 5 In the image, the image was divided into 16 blocks, numbered 1-16 in sequence. Blocks 2, 3, 7, 10, and 15 were not obscured, while the remaining blocks were obscured.

[0125] For unmasked image patches, they are first arranged according to their original order in the sample image to obtain the first stitching information. For example... Figure 5 As shown, the first stitching information consists of image blocks 2, 3, 7, 10, and 15. The first stitching information is input into the MAE encoder to obtain the initial image features of the unmasked image blocks.

[0126] The initial image features of the undisturbed image patch and the undisturbed image patch are concatenated according to their positions in the sample image to obtain the second concatenation information. This second concatenation information is then input into the decoder of the first autoencoder network to obtain the decoded image. Based on the loss between the decoded image and the sample image, the autoencoder network is adjusted to obtain an image-based autoencoder network. For example... Figure 5 The final image patch 5 does not match the sample image, resulting in a loss. The automatic masking network will adjust based on this loss.

[0127] The training method for video-based MAE is similar to that for image-based mask autoencoder networks (MAE). However, because videos have preceding and following frames, in many cases, occluded image patches can be found in the preceding and following frames, which reduces the training effect and makes it impossible for the resulting video-based MAE to accurately extract video features.

[0128] Therefore, during video-based MAE training, on the one hand, the occlusion rate of image patches was increased to enhance MAE's ability to extract video features. On the other hand, a time-manipulation mechanism was used to extend the occlusion mechanism along the time axis, ensuring that occluded image patches are always occluded within a short time frame. This reduces the interference of preceding and following frames on training and increases the consideration of time series, enabling the extracted video features to more accurately evaluate video content.

[0129] In summary, using MAE ensures that the extracted image and video features are not affected by occlusion. Using MAE for feature extraction also makes the results more reliable. Furthermore, because video-based MAE considers time series data, the extracted video features can more accurately assess the final video content quality.

[0130] To improve the accuracy of video content quality assessment, this disclosure also includes pre-training the process of determining the content quality of a target video based on multimodal features and information features. The pre-trained model is then used for video content quality assessment.

[0131] This disclosure relates to pre-trained models (such as image-based MAE, video-based MAE, etc.) for extracting multimodal features, information features of reference information, and cover aesthetic features, as well as evaluation models for completing video content quality assessment. The following will combine... Figure 6 The diagram shown illustrates the training method for an evaluation model used to assess the quality of a target video. Figure 6 The evaluation model shown uses a transformer.

[0132] First, the target video, which has undergone pre-quality evaluation, is categorized into multiple information states, such as image information, video information, audio information, and text information. Then, the multimodal information is input into a pre-trained model for feature extraction, yielding image features for each frame of the target video, video features, and text features. These extracted features are then fused using concat to obtain multimodal features. The cover image of the target video is obtained, and its cover quality features are extracted using an aesthetic model. The author account information from the reference information and the operation records of the target video are discretized, and their information features are extracted. The multimodal features, cover quality features, and information features are fused using a transformer. The fused result is used for quality evaluation and compared with the pre-evaluated quality assessment. The loss is calculated, and the transformer and concat are adjusted based on the loss until convergence.

[0133] Based on the same technical concept, this disclosure also provides a device for evaluating the quality of video content, such as... Figure 7 As shown, the device includes:

[0134] The acquisition module 701 is used to acquire the multimodal features of the target video; and to acquire the information features of the reference information of the target video; the reference information includes the account information of the creator of the target video and / or the operation record of the target video;

[0135] Evaluation module 702 is used to determine the content quality of the target video based on multimodal features and information features.

[0136] In some embodiments, the multimodal features of the target video are acquired, and the acquisition module is used to:

[0137] At least two of the following features are obtained as features to be fused: image features of each of the multiple frames in the target video, video features of the target video, and text features of the text information in the target video;

[0138] At least two types of features to be fused are fused to obtain the multimodal features of the target video.

[0139] In some embodiments, the process of acquiring image features of multiple frames in a target video is performed, and the acquisition module is used to:

[0140] Extract multiple frames of images from the target video;

[0141] Each frame of the image is input into an image-based mask autoencoder network (MAE) to obtain the initial image features corresponding to each frame.

[0142] The initial image features of each frame are input into the feature extraction module of the first fully connected network to obtain the image features of each frame. The first fully connected network is trained with the task of image quality evaluation as the training task.

[0143] In some embodiments, the video features of the target video are acquired, and the acquisition module is used to:

[0144] The frame sequence of the target video is sequentially input into a video-based mask autoencoder (MAE) network to obtain the initial video features of the target video.

[0145] The initial video features are input into the feature extraction module of the second fully connected network to obtain the video features of the target video; the second fully connected network is trained with the task of evaluating video content quality.

[0146] In some embodiments, the text feature acquisition module is used to acquire text information of the target video by performing text feature acquisition.

[0147] Extract the first text information included in the video frames of the target video; and,

[0148] Obtain the second text information from the audio of the target video;

[0149] Feature extraction is performed on the first and second text information to obtain text features.

[0150] In some embodiments, the device further includes:

[0151] The extraction module is used to extract the cover quality features of the target video;

[0152] The system performs content quality assessment based on multimodal features and information features to determine the content quality of the target video. The assessment module is used to input multimodal features, information features, and cover quality features into an attention-based quality assessment network to obtain the content quality of the target video.

[0153] Based on the same technical concept, this disclosure also provides a neural network training device, such as... Figure 8 As shown, the device includes:

[0154] The initial image feature extraction module 801 is used to input the sample image into the image-based mask autoencoder network (MAE) to obtain the initial image features of the sample image.

[0155] The image feature extraction module 802 is used to input the initial image features of the sample image into the feature extraction module of the first fully connected network to obtain the image features of the sample image;

[0156] The first quality determination module 803 is used to classify the image features of the sample image using the classification module of the first fully connected network to obtain the content quality of the sample image;

[0157] The first loss determination module 804 is used to determine the loss based on the content quality and quality label of the sample image;

[0158] The first adjustment module 805 is used to adjust the network parameters of the feature extraction module and the classification module of the first fully connected network based on the loss.

[0159] Among them, the image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0160] Based on the same technical concept, this disclosure also provides another neural network training device, such as Figure 9 As shown, the device includes:

[0161] The initial video feature determination module 901 is used to input the sample video into the video-based mask autoencoder network (MAE) to obtain the initial video features of the sample video.

[0162] The video feature determination module 902 is used to input the initial video features of the sample video into the feature extraction module of the second fully connected network to obtain the video features of the sample video;

[0163] The second quality determination module 903 is used to classify the video features of the sample video using the classification module of the second fully connected network to obtain the content quality of the sample video.

[0164] The second loss determination module 904 is used to determine the loss based on the content quality and quality label of the sample video.

[0165] The second adjustment module 905 is used to adjust the network parameters of the feature extraction module and the classification module of the second fully connected network based on the loss.

[0166] Among them, the video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

[0167] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0168] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0172] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as video content quality assessment methods. For example, in some embodiments, the video content quality assessment method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the video content quality assessment method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a quality assessment method for video content by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for evaluating the quality of video content, comprising: Obtain multimodal features from the target video; Information features for obtaining reference information of the target video; The reference information includes the account information that created the target video and / or the operation records of the target video; Determining the content quality of the target video based on the multimodal features and the information features includes: using a transformer composed of an encoder and a decoder to fuse the multimodal features and the information features, and evaluating the content quality of the target video based on the fusion result; inputting the features processed by the encoder into the decoder, and directly evaluating the content quality of the target video in the decoder, with the evaluation result being different quality levels; The acquisition of multimodal features of the target video includes: At least two of the following features are obtained as features to be fused: the image features of each of the multiple frames in the target video, the video features of the target video, and the text features of the text information of the target video; At least two features to be fused are extracted using an unsupervised training model. The obtained at least two features to be fused are then subjected to multimodal fusion processing by concatenating or adding them to obtain the multimodal features of the target video.

2. The method according to claim 1, wherein, Obtain the image features of each of the multiple frames in the target video, including: Extract multiple frames of images from the target video; Each frame of the image is input into an image-based mask autoencoder network (MAE) to obtain the initial image features corresponding to each frame. The initial image features of each frame are input into the feature extraction module of the first fully connected network to obtain the image features of each frame. The first fully connected network is trained with the task of image quality evaluation as the training task.

3. The method according to claim 1 or 2, wherein, Obtaining the video features of the target video includes: The frame sequence of the target video is sequentially input into a video-based mask autoencoder network (MAE) to obtain the initial video features of the target video. The initial video features are input into the feature extraction module of the second fully connected network to obtain the video features of the target video; wherein, the second fully connected network is trained with the task of evaluating video content quality.

4. The method according to any one of claims 1-3, wherein, The text features for obtaining the text information of the target video include: Extract the first text information included in the video frames of the target video; and, Obtain the second text information from the audio of the target video; Feature extraction is performed on the first text information and the second text information to obtain the text features.

5. The method according to any one of claims 1-4, wherein the account information includes at least one of the following: number of followers, number of works, and whether the author is a high-quality author; The operation records include at least one of the following: the number of likes, dislikes, shares, and comments on the video.

6. The method according to any one of claims 1-5, further comprising: Extract the cover quality features of the target video; Determining the content quality of the target video based on the multimodal features and the information features includes: The multimodal features, information features, and cover quality features are input into an attention-based quality assessment network to obtain the content quality of the target video.

7. A neural network training method, comprising: The sample image is input into an image-based mask autoencoder network (MAE) to obtain the initial image features of the sample image; the image-based MAE is an unsupervised training model. The initial image features of the sample image are input into the feature extraction module of the first fully connected network to obtain the image features of the sample image; wherein, the first fully connected network is a network trained with image quality evaluation as the training task; The classification module of the first fully connected network is used to classify the image features of the sample image to obtain the content quality of the sample image; Loss is determined based on the content quality and quality labels of the sample images; Adjust the network parameters of the feature extraction module and classification module of the first fully connected network based on the loss; The image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

8. A neural network training method, comprising: The sample video is input into a video-based mask autoencoder network (MAE) to obtain the initial video features of the sample video. The video-based MAE is an unsupervised training model; The initial video features of the sample video are input into the feature extraction module of the second fully connected network to obtain the video features of the sample video; wherein, the second fully connected network is a network trained with the task of completing video content quality evaluation; The classification module of the second fully connected network is used to classify the video features of the sample video to obtain the content quality of the sample video; Loss is determined based on the content quality and quality labels of the sample videos; Adjust the network parameters of the feature extraction module and classification module of the second fully connected network based on the loss; The video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

9. A video content quality evaluation device, comprising: The acquisition module is used to acquire at least two of the following features as features to be fused during the process of acquiring the multimodal features of the target video: image features of each of the multiple frames in the target video, video features of the target video, and text features of the text information of the target video; extract at least two features to be fused using an unsupervised training model, and perform multimodal fusion processing on the acquired at least two features to be fused by concatenation or addition to obtain the multimodal features of the target video; and acquire information features of reference information of the target video; the reference information includes account information of the creator of the target video and / or operation records of the target video; The evaluation module is used to determine the content quality of the target video based on the multimodal features and the information features. It uses a transformer composed of an encoder and a decoder to fuse the multimodal features and the information features, and evaluates the content quality of the target video based on the fusion result. In this case, the features processed by the encoder are input into the decoder, and the content quality evaluation of the target video is completed directly in the decoder. The quality evaluation result is a different quality level.

10. The apparatus according to claim 9, wherein, The module is used to acquire the image features of multiple frames in a target video. Extract multiple frames of images from the target video; Each frame of the image is input into an image-based mask autoencoder network (MAE) to obtain the initial image features corresponding to each frame. The initial image features of each frame are input into the feature extraction module of the first fully connected network to obtain the image features of each frame. The first fully connected network is trained with the task of image quality evaluation as the training task.

11. The apparatus according to claim 9 or 10, wherein, The module is used to acquire video features of the target video. The frame sequence of the target video is sequentially input into a video-based mask autoencoder network (MAE) to obtain the initial video features of the target video. The initial video features are input into the feature extraction module of the second fully connected network to obtain the video features of the target video; wherein, the second fully connected network is trained with the task of evaluating video content quality.

12. The apparatus according to any one of claims 9-11, wherein, The text feature acquisition module is used to obtain text information from the target video. Extract the first text information included in the video frames of the target video; and, Obtain the second text information from the audio of the target video; Feature extraction is performed on the first text information and the second text information to obtain the text features.

13. The apparatus according to any one of claims 9-12, wherein the account information includes at least one of the following: number of followers, number of works, and whether the author is a high-quality author; The operation records include at least one of the following: the number of likes, dislikes, shares, and comments on the video.

14. The apparatus according to any one of claims 9-13, further comprising: An extraction module is used to extract the cover quality features of the target video; The evaluation module is used to input the multimodal features, the information features, and the cover quality features into an attention-based quality evaluation network to determine the content quality of the target video based on the multimodal features and the information features.

15. A neural network training device, comprising: An initial image feature extraction module is used to input the sample image into an image-based mask autoencoder network (MAE) to obtain the initial image features of the sample image; the image-based MAE is an unsupervised training model. The image feature extraction module is used to input the initial image features of the sample image into the feature extraction module of the first fully connected network to obtain the image features of the sample image; wherein, the first fully connected network is a network trained with image quality evaluation as the training task; The first quality determination module is used to classify the image features of the sample image using the classification module of the first fully connected network to obtain the content quality of the sample image. The first loss determination module is used to determine the loss based on the content quality and quality label of the sample image; The first adjustment module is used to adjust the network parameters of the feature extraction module and the classification module of the first fully connected network based on the loss. The image features of the target video extracted by the first fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

16. A neural network training device, comprising: The initial video feature determination module is used to input the sample video into a video-based mask autoencoder network (MAE) to obtain the initial video features of the sample video. The video-based MAE is an unsupervised training model; The video feature determination module is used to input the initial video features of the sample video into the feature extraction module of the second fully connected network to obtain the video features of the sample video; wherein, the second fully connected network is a network trained with the task of completing video content quality evaluation; The second quality determination module is used to classify the video features of the sample video using the classification module of the second fully connected network to obtain the content quality of the sample video. The second loss determination module is used to determine the loss based on the content quality and quality label of the sample video. The second adjustment module is used to adjust the network parameters of the feature extraction module and the classification module of the second fully connected network based on the loss. The video features extracted by the second fully connected network are used to construct the multimodal features of the target video, and the multimodal features are used to evaluate the content quality of the target video.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images