Video clarity assessment model training method, device and storage medium

Abstract

The present application discloses a video clarity assessment model training method, device and storage medium. In the scheme, the parameters of the pre-trained feature extraction layer are kept unchanged, and only the fully connected layer and the feature mapping layer in the initial model are updated using the training video set, and the fully connected layer in the initial model is replaced by the updated fully connected layer, and the feature mapping layer in the initial model is replaced by the updated feature mapping layer, so that the updated initial model can be obtained; if the fully connected layer and the feature mapping layer in the updated initial model converge, the training video set is used to train the entire updated initial model, so as to obtain a video clarity assessment model. The present application can extract video features with higher precision, so as to improve the accuracy of the video clarity assessment results, thereby improving the model assessment accuracy, and is conducive to model convergence, and reducing the model training cost.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video clarity assessment model training method, device and storage medium. Background Art

[0002] At present, convolutional neural networks are generally used to evaluate the clarity of live videos. However, due to the limited ability of convolutional neural networks to extract features, the accuracy of video clarity evaluation results is limited. At the same time, convolutional neural networks have many network parameters, which is not conducive to model convergence when training convolutional neural networks. Therefore, the training cost of convolutional neural networks is also high. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a video clarity assessment model training method, device and storage medium to reduce the model training cost and improve the model assessment accuracy. The specific scheme is as follows:

[0004] To achieve the above objectives, on the one hand, the present application provides a video clarity assessment model training method, comprising:

[0005] Label the clarity score of each video sample to obtain a training video set;

[0006] Inputting the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set;

[0007] The initial model includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features;

[0008] Keeping the parameters of the feature extraction layer unchanged, and updating the parameters of the fully connected layer and the feature mapping layer using the loss value between the first clarity prediction result and the clarity score;

[0009] Replacing the fully connected layer in the initial model with the updated fully connected layer, and replacing the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model;

[0010] If the fully connected layer and the feature mapping layer in the updated initial model converge, the updated initial model is trained using the training video set to obtain a video clarity assessment model.

[0011] On the other hand, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned video clarity assessment model training method.

[0012] On the other hand, the present application also provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, the aforementioned video clarity assessment model training method is implemented.

[0013] The initial model in the present application includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features. First, keep the parameters of the pre-trained feature extraction layer unchanged, and use the training video set to train the fully connected layer and the feature mapping layer, that is, only update the fully connected layer and the feature mapping layer in the initial model, and then replace the fully connected layer in the initial model with the updated fully connected layer, and replace the feature mapping layer in the initial model with the updated feature mapping layer, so as to obtain an updated initial model; if the fully connected layer and the feature mapping layer in the updated initial model converge, then use the training video set to train the entire updated initial model to obtain a video clarity evaluation model. Among them, since the feature extraction layer in the initial model has been pre-trained, the subsequent training cost can be reduced; compared with the entire model, the fully connected layer and the feature mapping layer have fewer parameters, so training them to converge first can also reduce the subsequent training cost; compared with the existing convolutional neural network, the video clarity assessment model in this application has a feature extraction layer that has been trained more times, so it can extract more accurate video features, thereby improving the accuracy of the video clarity assessment results, thereby improving the model evaluation accuracy.

[0014] Correspondingly, the video clarity assessment model training device and storage medium provided in this application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a physical architecture applicable to the present application provided for the present application;

[0017] Figure 2A flow chart of the first video clarity assessment model training method provided for this application;

[0018] Figure 3 A flow chart of the second video clarity assessment model training method provided in this application;

[0019] Figure 4 A flowchart of the third video clarity assessment model training method provided for this application;

[0020] Figure 5 A video clarity assessment model application flow chart provided in this application;

[0021] Figure 6 for Figure 5 The corresponding system framework diagram;

[0022] Figure 7 A schematic diagram of the training process of a video clarity assessment model provided in this application;

[0023] Figure 8 A schematic diagram of a video clarity assessment model training device provided in this application;

[0024] Fig. 9 A server structure diagram provided for this application;

[0025] Fig.10 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0026] Existing convolutional neural networks evaluate the clarity of live videos, but due to the limited ability of convolutional neural networks to extract features, the accuracy of video clarity evaluation results is limited. At the same time, convolutional neural networks have many network parameters, which is not conducive to model convergence when training convolutional neural networks, so the training cost of convolutional neural networks is also high.

[0027] In view of the above-mentioned problems currently existing, this application proposes a video clarity assessment model training scheme, which can extract more accurate video features, thereby improving the accuracy of video clarity assessment results, thereby improving model evaluation accuracy, and is conducive to model convergence and reducing model training costs.

[0028] For ease of understanding, the physical framework to which this application applies is first introduced.

[0029] It should be understood that the video clarity assessment model training method provided in this application can be applied to a system or program with a model training function. Specifically, the system or program with a model training function can be run in a server, a personal computer, or other device.

[0030] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture applicable to this application. Figure 1 In the invention, a system or program with a model training function can be run on a server, which obtains a training video set from other terminal devices through a network, and then stores an initial model including: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classification features into the server. The server annotates the clarity score of each video sample to obtain a training video set; inputs the training video set into the initial model to obtain a first clarity prediction result of each video sample in the training video set; keeps the parameters of the feature extraction layer unchanged, and updates the parameters of the fully connected layer and the feature mapping layer using the loss value between the first clarity prediction result and the clarity score; replaces the fully connected layer in the initial model with the updated fully connected layer, and replaces the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model; if the fully connected layer and the feature mapping layer in the updated initial model converge, the updated initial model (i.e., the feature extraction layer, the feature mapping layer, and the fully connected layer in the updated initial model) are trained using the training video set, thereby obtaining a video clarity evaluation model. The video clarity assessment model can be used to assess parameters such as the clarity of live or recorded videos.

[0031] As shown in the figure, the server can establish communication connections with multiple devices, and the server obtains live videos or recorded videos that meet the training conditions from these devices. The server trains the initial model by collecting live videos or recorded videos uploaded by these devices to obtain a video clarity assessment model. Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the model training process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0032] It should be noted that the video clarity assessment model training method provided in this embodiment can be performed offline, that is, the server locally stores a training video set, which can directly use the solution provided in this application to train and obtain the desired model.

[0033] It is understandable that the above-mentioned system or program with model training function can also be run on a personal mobile terminal or as a cloud service program. The specific operation mode depends on the actual scenario and is not limited here.

[0034] Specifically, after the model training is completed, the obtained video clarity assessment model can be applied to video live broadcast or recording, so that the clarity of live video or recorded video can be evaluated in real time. If the video clarity is low, relevant personnel can adjust the relevant equipment in time to optimize the video clarity.

[0035] Combined with the above commonalities, please see Figure 2 , Figure 2 The first video definition assessment model training method flow chart provided in the present application embodiment. Figure 2 As shown, the video clarity assessment model training method may include the following steps:

[0036] S201. Label the clarity score of each video sample to obtain a training video set.

[0037] The training video set includes videos in different scenes (including different live scenes and / or different recording scenes), and the clarity score of a video sample in the training video set is considered to be the real clarity of the video. For any scene, there are videos with different clarity, so the training video set covers a variety of scenes, which can improve the model's evaluation effect on fine-grained scenes.

[0038] The amount of videos corresponding to different scenes is roughly balanced, and the amount of videos of different resolutions in any scene is also roughly balanced, so that the model can learn the ability to evaluate the resolution of different scenes and resolutions in a balanced manner. The live broadcast scene can be divided into indoor scenes, outdoor scenes, game scenes, etc. Different recording scenes include: indoor / outdoor interview program recording, theater performance, stage performance recording, etc.

[0039] In a specific implementation, the training video set includes: videos of different resolutions corresponding to at least two scenes, each video is annotated with a resolution score and recording prompt information. The recording prompt information, for example, provides suggestions on which equipment or parameters to adjust to improve the resolution if the resolution score is low.

[0040] It should be noted that in order to ensure the accuracy of the clarity score of each video sample in the training video set, multiple people can label each video sample so that each video sample corresponds to multiple clarity labels, and then calculate the average of these clarity labels as the clarity score of the video sample. Since manual labeling may have errors and omissions, before calculating the average value, the multiple manually annotated clarity labels can be subjected to consistency checks and / or outlier checks to remove individual clarity labels that are obviously erroneous.

[0041] In a specific implementation, labeling the clarity score of each video sample to obtain a training video set includes: collecting multiple video samples corresponding to at least two scenes; obtaining multiple clarity labels for each video sample, calculating the average of the multiple clarity labels; labeling the average as the clarity score of each video sample to obtain the training video set. Before calculating the average of the multiple clarity labels, it also includes: performing consistency check and / or outlier check on the multiple clarity labels.

[0042] S202: Input the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set.

[0043] Among them, the initial model includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features. It should be noted that the feature extraction layer in the initial model can use the feature extractor in Inception-Resnet V2, which has fewer parameters, extracts more features, and has higher extraction efficiency. The feature extractor is trained using a public video set (different from the training video set in this embodiment), so that a pre-trained feature extraction layer can be obtained. The fully connected layer and feature mapping layer in the initial model are both in the initial state. The initial model also includes a feature intersection and convergence layer, which has no parameters that need to be updated, and is used to perform bilinear pooling and aggregation operations on the features. The first clarity prediction result is: the training video set passes through the feature extraction layer, feature mapping layer, feature intersection and convergence layer and fully connected layer in sequence, and the clarity prediction result output is output.

[0044] Among them, the feature extraction layer can extract more informative video features with better extraction efficiency. The feature mapping layer can map video features from low-dimensional features to high-dimensional features. The feature intersection and convergence layer can perform bilinear pooling operations on high-dimensional features and aggregate them into aggregated features including high-order features (detailed features in the video) and low-order features (contour features in the video), thereby containing more information in the video and avoiding information loss. The fully connected layer is equivalent to a classifier, which is used to classify aggregated features to evaluate the clarity score of the aggregated features, thereby obtaining a video clarity evaluation result.

[0045] S203 , keeping the parameters of the feature extraction layer unchanged, and using the loss value between the first clarity prediction result and the clarity score to update the parameters of the fully connected layer and the feature mapping layer.

[0046] S204, replacing the fully connected layer in the initial model with the updated fully connected layer, and replacing the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model.

[0047] This embodiment first keeps the parameters of the feature extraction layer unchanged, and uses the training video set to train the fully connected layer and the feature mapping layer, that is: using the loss value between the first clarity prediction result and the clarity score, update the parameters of the fully connected layer and the feature mapping layer, replace the fully connected layer in the initial model with the updated fully connected layer, and replace the feature mapping layer in the initial model with the updated feature mapping layer, so as to obtain an updated initial model. If the fully connected layer and the feature mapping layer in the updated initial model meet the convergence conditions, the updated initial model is trained using the training video set, that is, the current fully connected layer, feature intersection and convergence layer, feature mapping layer, and feature extraction layer are regarded as the updated initial model, and the entire model is trained. Compared with the entire model, the fully connected layer and the feature mapping layer have fewer parameters, so training them to converge first can reduce the subsequent training costs.

[0048] S205, determining whether the fully connected layer and the feature mapping layer in the updated initial model have converged; if so, executing S206; if not, executing S207.

[0049] S206: Use the training video set to train the updated initial model to obtain a video clarity assessment model.

[0050] S207: Replace the initial model with the updated initial model, and execute S202.

[0051] Of course, if after a certain update of the parameters of the fully connected layer and the feature mapping layer, the fully connected layer and the feature mapping layer in the updated initial model have not converged, the initial model is replaced with the updated initial model, and S202 is executed to iteratively train the fully connected layer and the feature mapping layer until the fully connected layer and the feature mapping layer converge. The convergence conditions of the fully connected layer and the feature mapping layer can use the commonly used model convergence conditions, such as: after calculating the loss value between the first clarity prediction result and the clarity score using the relevant loss function, detect whether the loss value is less than a certain threshold. If only one of the fully connected layer and the feature mapping layer reaches convergence, the fully connected layer and the feature mapping layer are continuously iterated until both the fully connected layer and the feature mapping layer converge.

[0052] In a specific embodiment, the updated initial model is trained using the training video set to obtain a video clarity evaluation model, including: extracting video features of each video sample in the training video set using the feature extraction layer in the updated initial model; mapping the video features from low-dimensional features to high-dimensional features using the feature mapping layer in the updated initial model; performing bilinear pooling and aggregation operations on the high-dimensional features using the feature intersection and convergence layer in the updated initial model to obtain aggregate features including high-order features and low-order features; the updated initial model also includes a feature intersection and convergence layer; using the fully connected layer in the updated initial model to classify and aggregate features to obtain a second clarity prediction result for each video sample in the training video set; using the second clarity prediction result and the loss value of the clarity score to update the parameters of the updated initial model; if the model convergence condition is met, the current model is used as the video clarity evaluation model. If the model convergence condition is not met, the updated initial model is replaced with the current model, and the step of extracting video features of each video sample in the training video set using the feature extraction layer in the updated initial model is performed. This training process can refer to the existing related technology, and this embodiment will not be repeated. Among them, the feature intersection and convergence layer includes a feature intersection module and a feature aggregation module. The feature intersection module is used to perform bilinear pooling operations on features, and the feature aggregation module is used to aggregate features. The second definition prediction result is: the training video set passes through the feature extraction layer, feature mapping layer, feature intersection and convergence layer and fully connected layer in sequence, and the output definition prediction result.

[0053] After the video clarity assessment model is trained, in a specific implementation, it also includes: using the video clarity assessment model to evaluate the clarity of the live video or recorded video in real time, and displaying the corresponding video clarity assessment results and recording prompt information, so that relevant personnel can adjust relevant equipment in time to optimize the video clarity.

[0054] In this embodiment, since the feature extraction layer in the initial model has been pre-trained, the subsequent training cost can be reduced; the fully connected layer and the feature mapping layer have fewer parameters compared to the entire model, so they are trained first to converge, which can also reduce the subsequent training cost.

[0055] It can be seen that compared with the existing convolutional neural network, the feature extraction layer of the video clarity assessment model in this embodiment has undergone more training times, so it can extract more accurate video features, thereby improving the accuracy of the video clarity assessment results, thereby improving the model evaluation accuracy.

[0056] See also Figure 3 , Figure 3 The flowchart of the second video definition assessment model training method provided in the embodiment of the present application is as follows. Figure 3 As shown, the video clarity assessment model training method may include the following steps:

[0057] S301, marking the clarity score of each video sample to obtain a training video set;

[0058] S302, inputting the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set;

[0059] S303, keeping the parameters of the feature extraction layer unchanged, and using the loss value between the first clarity prediction result and the clarity score to update the parameters of the full connection layer and the feature mapping layer;

[0060] S304, replacing the fully connected layer in the initial model with the updated fully connected layer, and replacing the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model;

[0061] S305, determining whether the fully connected layer and the feature mapping layer in the updated initial model have converged; if so, executing S306; if not, replacing the initial model with the updated initial model, and executing S302;

[0062] S306, training the updated initial model using the training video set to obtain a video clarity assessment model;

[0063] S307, counting the total parameters of the video clarity evaluation model;

[0064] S308: If the total parameter amount is greater than a preset threshold, reducing the parameters of the fully connected layer in the video definition evaluation model to obtain a lightweight video definition evaluation model;

[0065] S309: Train a lightweight video clarity assessment model using the training video set to obtain an updated video clarity assessment model.

[0066] In this embodiment, after the video clarity assessment model is trained using the training video set, the total number of parameters of the video clarity assessment model can be counted; if the total number of parameters is greater than a preset threshold, it indicates that the total number of parameters of the video clarity assessment model is large, and the model training cost is still relatively large, so the parameters of the fully connected layer in the video clarity assessment model can be reduced to obtain a lightweight video clarity assessment model, and then the lightweight video clarity assessment model is trained using the training video set to obtain an updated video clarity assessment model, thereby achieving further optimization of the video clarity assessment model.

[0067] Of course, if the total parameter amount is not greater than the preset threshold, it means that the total parameter amount of the video clarity assessment model is not large, and the model training cost is relatively acceptable, that is, a relatively ideal video clarity assessment model is obtained. Therefore, it is generally not necessary to reduce the parameters of the fully connected layer in the video clarity assessment model. The preset threshold can be the number of network parameters of the convolutional neural network in the prior art, or it can be an empirical value set by a technician.

[0068] Among them, reducing the parameters of the fully connected layer in the video clarity assessment model includes: reducing the number of weight parameters of the convolution in the fully connected layer. The specific method can refer to the existing related technology.

[0069] It can be seen that the initial model in this embodiment includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features. First, keep the parameters of the pre-trained feature extraction layer unchanged, and use the training video set to train the fully connected layer and the feature mapping layer, that is, only update the fully connected layer and the feature mapping layer in the initial model, and then replace the fully connected layer in the initial model with the updated fully connected layer, and replace the feature mapping layer in the initial model with the updated feature mapping layer, so as to obtain an updated initial model; if the fully connected layer and the feature mapping layer in the updated initial model converge, then use the training video set to train the entire updated initial model, so as to obtain a video clarity evaluation model. Among them, since the feature extraction layer in the initial model has been pre-trained, the subsequent training cost can be reduced; compared with the entire model, the fully connected layer and the feature mapping layer have fewer parameters, so training them to converge first can also reduce the subsequent training cost; compared with the existing convolutional neural network, the video clarity assessment model in this application has a feature extraction layer that has been trained more times, so it can extract more accurate video features, thereby improving the accuracy of the video clarity assessment results, thereby improving the model evaluation accuracy.

[0070] See also Figure 4 , Figure 4 The flowchart of the third video definition assessment model training method provided in the embodiment of the present application is as follows. Figure 4 As shown, the video clarity assessment model training method may include the following steps:

[0071] S401, marking the clarity score of each video sample to obtain a training video set;

[0072] S402, inputting the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set;

[0073] S403, keeping the parameters of the feature extraction layer unchanged, and using the loss value between the first clarity prediction result and the clarity score to update the parameters of the full connection layer and the feature mapping layer;

[0074] S404, replacing the fully connected layer in the initial model with the updated fully connected layer, and replacing the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model;

[0075] S405, determining whether the fully connected layer and the feature mapping layer in the updated initial model have converged; if so, executing S406; if not, replacing the initial model with the updated initial model, and executing S402;

[0076] S406, training the updated initial model using the training video set to obtain a video clarity assessment model;

[0077] S407, sampling and compressing video skipping frames in the test video set to obtain a test sample;

[0078] S408, inputting the test sample into the video clarity evaluation model to obtain a clarity test result;

[0079] S409: Calculate the accuracy of the clarity test result using PLCC or SROCC, and determine the accuracy of the video clarity evaluation model based on the accuracy.

[0080] In this embodiment, after the video clarity assessment model is obtained by training with the training video set, the accuracy of the video clarity assessment model can be tested with the test video set. Specifically, the video frames in the test video set are first sampled and compressed to obtain a test sample with a smaller data volume to improve the test efficiency. The test sample is input into the video clarity assessment model to obtain a clarity test result, and then the accuracy of the clarity test result is calculated using PLCC (Pearson Correlation Coefficient) or SROCC (Spearman Rankorder Correlation Coefficient), and the accuracy of the video clarity assessment model is determined based on the accuracy. The test video set can be a subset of the training video set.

[0081] Among them, the higher the accuracy of the clarity test result, the higher the accuracy of the video clarity assessment model. The accuracy of the clarity test result can be determined based on the difference between the clarity test result and the actual result (such as the clarity score of the video sample). The greater the difference, the lower the accuracy of the clarity test result. For example: for video A, the video clarity assessment model's assessment score is X, and the actual manual annotation score of video A is Y. Then the accuracy of X can be determined based on the difference between X and Y. The closer X is to Y, the higher the accuracy of X.

[0082] If the accuracy of the video clarity assessment model is low, the video clarity assessment model may be retrained to further optimize the video clarity assessment model.

[0083] It can be seen that the initial model in this embodiment includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features. First, keep the parameters of the pre-trained feature extraction layer unchanged, and use the training video set to train the fully connected layer and the feature mapping layer, that is, only update the fully connected layer and the feature mapping layer in the initial model, and then replace the fully connected layer in the initial model with the updated fully connected layer, and replace the feature mapping layer in the initial model with the updated feature mapping layer, so as to obtain an updated initial model; if the fully connected layer and the feature mapping layer in the updated initial model converge, then use the training video set to train the entire updated initial model, so as to obtain a video clarity evaluation model. Among them, since the feature extraction layer in the initial model has been pre-trained, the subsequent training cost can be reduced; compared with the entire model, the fully connected layer and the feature mapping layer have fewer parameters, so training them to converge first can also reduce the subsequent training cost; compared with the existing convolutional neural network, the video clarity assessment model in this application has a feature extraction layer that has been trained more times, so it can extract more accurate video features, thereby improving the accuracy of the video clarity assessment results, thereby improving the model evaluation accuracy.

[0084] The following is an introduction to the model application scheme provided by this application through a specific application scenario example description, namely: a specific scheme for evaluating video clarity using a video clarity evaluation model. This scheme can evaluate the clarity of live or recorded videos in real time. If the video clarity is low, relevant personnel can adjust the relevant equipment in time to optimize the video clarity, so the following model application process can be used.

[0085] Set up a video clarity assessment model on the live broadcast platform and use it to assess the clarity of live broadcast videos in real time. Figure 5 The specific process includes:

[0086] S501, the live broadcast terminal requests the live broadcast platform to run the live broadcast service;

[0087] S502, the live broadcast platform feeds back a response message to the live broadcast terminal;

[0088] S503, the live broadcast terminal starts live broadcast based on the response message;

[0089] S504, the live broadcast platform obtains the live broadcast video from the live broadcast terminal in real time;

[0090] S505, the live broadcast platform inputs the live broadcast video into the video clarity evaluation model in real time, so that the video clarity evaluation model outputs a corresponding video clarity score and recording prompt information;

[0091] S506, the live broadcast platform sends the video clarity score and recording prompt information to the live broadcast terminal;

[0092] S507: The live broadcast terminal displays the video clarity score and recording prompt information.

[0093] Among them, the live broadcast platform can be deployed in the server, and the live broadcast terminal can be a terminal capable of live broadcast, such as a smart phone, tablet computer, laptop computer or desktop computer. Figure 5 The corresponding system framework diagram can be found in Figure 6 , Figure 6 The server in the video is the live broadcast platform. Video preprocessing generally includes: video reduction, cropping, etc. Video decoding is: restoring and decoding the encoded live video. Frame skipping is to extract some frames from the video and input the extracted frames into the video clarity assessment model.

[0094] In this embodiment, the training process of the video clarity evaluation model can be found in Figure 7 .

[0095] exist Figure 7 In the data set, the live video data set is the training video set, which collects videos of different clarity in various scenes, and makes the number of videos in various scenes evenly distributed, and the number of videos of different clarity in each scene evenly distributed, and then cleans each video to remove useless or erroneous videos. Each video is labeled according to different clarity (at least including clear, general, and blurred), and the clarity score and recording prompt information are also labeled. For the same video, after being labeled by multiple people, the average annotation score is calculated as the final score of the video; of course, the score of the same video labeled by multiple people can be subjected to consistency test and outlier test to determine whether the annotation is accurate. The labeled videos are divided into training video set and test set. Of course, the labeled videos can also be segmented to increase the number of videos.

[0096] exist Figure 7 In the example, the selected basic network includes Inception-Resnet V2, a pre-trained feature extractor, an initial feature intersection and convergence layer, an initial feature mapping layer, and an initial fully connected layer; the initial feature intersection and convergence layer includes a feature intersection module and a feature aggregation module. Transfer learning means that the pre-trained feature extractor is used as the feature extraction layer to be used in this solution. The ranking learning method is used in the fully connected layer to classify the aggregated features. Two-step fine-tuning means that the parameters of the feature extractor are kept unchanged, and the fully connected layer and feature mapping layer are trained with the training video set to obtain the fully connected layer and feature mapping layer that meet the convergence conditions; then the current entire model is trained with the training video set to adjust the model's hyperparameters until a video clarity evaluation model is obtained.

[0097] exist Figure 7 In the process of training the video clarity evaluation model, the accuracy of the model is evaluated by PLCC or SROCC, and the training video set and the test set are crossed to test the model. The total number of model parameters can also be analyzed to determine whether the total number of model parameters meets the actual requirements. The test set can also be subjected to frame skipping and video compression to obtain test samples, so as to use the test samples to verify the performance of the model.

[0098] It can be seen that the video clarity evaluation model in this embodiment can evaluate the clarity of live video in real time, and provide real-time optimization suggestions for broadcast conditions during the host's broadcast, thereby improving the clarity of live video and bringing users a better product experience.

[0099] See also Figure 8 , Figure 8 A schematic diagram of a video clarity assessment model training device provided in an embodiment of the present application includes:

[0100] The labeling module 801 is used to label the clarity score of each video sample to obtain a training video set;

[0101] An input module 802 is used to input the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set;

[0102] The initial model includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features;

[0103] A partial training module 803 is used to keep the parameters of the feature extraction layer unchanged, and update the parameters of the full connection layer and the feature mapping layer using the loss value between the first clarity prediction result and the clarity score;

[0104] A replacement module 804 is used to replace the fully connected layer in the initial model with the updated fully connected layer, and replace the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model;

[0105] The model training module 805 is used to train the updated initial model using the training video set to obtain a video clarity assessment model if the fully connected layer and the feature mapping layer in the updated initial model converge.

[0106] In a specific embodiment, it also includes:

[0107] The iterative module is used to replace the initial model with the updated initial model if the fully connected layer and / or the feature mapping layer in the updated initial model has not converged, and execute the step of inputting the training video set into the initial model to obtain the first definition prediction result of each video sample in the training video set.

[0108] In a specific implementation, the model training module includes:

[0109] A feature extraction unit, used to extract video features of each video sample in the training video set using the feature extraction layer in the updated initial model;

[0110] A feature mapping unit, used to map video features from low-dimensional features to high-dimensional features using the feature mapping layer in the updated initial model;

[0111] A feature aggregation unit, used to perform bilinear pooling and aggregation operations on high-dimensional features using the feature intersection and convergence layer in the updated initial model to obtain aggregated features including high-order features and low-order features; the updated initial model also includes a feature intersection and convergence layer;

[0112] A prediction unit, configured to use the fully connected layer classification aggregation features in the updated initial model to obtain a second definition prediction result for each video sample in the training video set;

[0113] a loss value calculation unit, configured to update the parameters of the updated initial model using the second clarity prediction result and the loss value of the clarity score;

[0114] The output unit is used to use the current model as a video clarity evaluation model if the model convergence condition is met.

[0115] In a specific implementation, the model training module further includes:

[0116] The iteration unit is used to replace the updated initial model with the current model if the model convergence condition is not met, and execute the step of extracting video features of each video sample in the training video set using the feature extraction layer in the updated initial model.

[0117] In a specific embodiment, it also includes:

[0118] A statistical module, used to calculate the total number of parameters of the video clarity evaluation model;

[0119] A parameter reduction module, used for reducing the parameters of the fully connected layer in the video definition evaluation model if the total parameter amount is greater than a preset threshold, so as to obtain a lightweight video definition evaluation model;

[0120] The optimization module is used to train a lightweight video clarity assessment model using a training video set to obtain an updated video clarity assessment model.

[0121] In a specific embodiment, it also includes:

[0122] A sampling module, used for sampling and compressing video frames in a test video set to obtain a test sample;

[0123] A testing module, used for inputting a test sample into a video clarity evaluation model to obtain a clarity test result;

[0124] The determination module is used to calculate the accuracy of the clarity test result by using PLCC or SROCC, and determine the accuracy of the video clarity evaluation model according to the accuracy.

[0125] In a specific embodiment, it also includes:

[0126] The application module is used to use the video clarity evaluation model to evaluate the clarity of live video or recorded video in real time, and display the corresponding video clarity evaluation results and recording prompt information.

[0127] In a specific implementation, the annotation module includes:

[0128] A collection unit, used for collecting a plurality of video samples corresponding to at least two scenes;

[0129] An acquisition unit, used for acquiring multiple definition labels of each video sample, and calculating an average value of the multiple definition labels;

[0130] The labeling unit is used to label the average value as the clarity score of each video sample to obtain a training video set.

[0131] In a specific implementation, the marking module further includes:

[0132] A checking unit is used to perform consistency checking and / or outlier checking on a plurality of clarity labels.

[0133] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0134] It can be seen that this embodiment provides a video clarity assessment model training device, which can extract video features with higher precision, thereby improving the accuracy of video clarity assessment results, thereby improving model evaluation accuracy, and is conducive to model convergence and reducing model training costs.

[0135] Furthermore, the present application also provides an electronic device. The electronic device can be Fig. 9 The server 50 shown may also be Fig.10 Terminal 60 is shown. Fig. 9 and Fig.10 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0136] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the video clarity assessment model training disclosed in any of the aforementioned embodiments.

[0137] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0138] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0139] Among them, the operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50 to realize the operation and processing of the data 523 in the memory 52 by the processor 51, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the video clarity assessment model training method disclosed in any of the aforementioned embodiments, the computer program 522 can further include a computer program that can be used to complete other specific tasks. In addition to data such as live video, the data 523 can also include data such as developer information of the application.

[0140] Fig.10 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0141] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0142] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0143] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the video clarity assessment model training method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include but is not limited to video data.

[0144] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0145] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0146] Furthermore, the embodiment of the present application also discloses a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by the processor, the video clarity assessment model training method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0147] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0148] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0149] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video clarity assessment model training method, characterized in that: include: Label the clarity score of each video sample to obtain a training video set; Inputting the training video set into the initial model to obtain a first definition prediction result of each video sample in the training video set; The initial model includes: a pre-trained feature extraction layer for extracting video features, a feature mapping layer for mapping video features from low-dimensional features to high-dimensional features, and a fully connected layer for classifying features; Keeping the parameters of the feature extraction layer unchanged, and updating the parameters of the fully connected layer and the feature mapping layer using the loss value between the first clarity prediction result and the clarity score; Replacing the fully connected layer in the initial model with the updated fully connected layer, and replacing the feature mapping layer in the initial model with the updated feature mapping layer to obtain an updated initial model; If the fully connected layer and the feature mapping layer in the updated initial model converge, training the updated initial model using the training video set to obtain a video clarity assessment model; The step of training the updated initial model using the training video set to obtain a video clarity assessment model includes: Utilizing the feature extraction layer in the updated initial model, extracting video features of each video sample in the training video set; Mapping the video features from low-dimensional features to high-dimensional features using the feature mapping layer in the updated initial model; Using the feature intersection and convergence layer in the updated initial model, bilinear pooling and aggregation operations are performed on the high-dimensional features to obtain aggregated features including high-order features and low-order features; the updated initial model also includes the feature intersection and convergence layer; Classifying the aggregated features using the fully connected layer in the updated initial model to obtain a second definition prediction result for each video sample in the training video set; Using the second clarity prediction result and the loss value of the clarity score, updating the parameters of the updated initial model; If the model convergence condition is met, the current model is used as the video clarity evaluation model.

2. The video clarity assessment model training method according to claim 1, characterized in that: If the fully connected layer and / or feature mapping layer in the updated initial model has not converged, the initial model is replaced by the updated initial model, and the step of inputting the training video set into the initial model to obtain the first clarity prediction result of each video sample in the training video set is performed.

3. The video clarity assessment model training method according to claim 1, characterized in that: If the model convergence condition is not met, the updated initial model is replaced by the current model, and the step of extracting video features of each video sample in the training video set by using the feature extraction layer in the updated initial model is performed.

4. The video clarity assessment model training method according to any one of claims 1 to 3, characterized in that: Also includes: Counting the total number of parameters of the video clarity assessment model; If the total parameter amount is greater than a preset threshold, reducing the parameters of the fully connected layer in the video definition evaluation model to obtain a lightweight video definition evaluation model; The lightweight video clarity assessment model is trained using the training video set to obtain an updated video clarity assessment model.

5. The video clarity assessment model training method according to any one of claims 1 to 3, characterized in that: Also includes: Skip frames of the video in the test video set and compress them to obtain test samples; Inputting the test sample into the video clarity evaluation model to obtain a clarity test result; The accuracy of the clarity test result is calculated using PLCC or SROCC, and the accuracy of the video clarity evaluation model is determined based on the accuracy.

6. The video clarity assessment model training method according to claim 1, characterized in that: The step of labeling the clarity score of each video sample to obtain a training video set includes: Collecting multiple video samples corresponding to at least two scenes; Obtain multiple definition labels for each video sample, and calculate an average value of the multiple definition labels; The average value is marked as the clarity score of each video sample to obtain the training video set.

7. The video clarity assessment model training method according to claim 6, characterized in that: Before calculating the average value of the plurality of clarity labels, the method further includes: The plurality of clarity labels are subjected to consistency check and / or outlier check.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the video clarity assessment model training method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the video clarity assessment model training method as described in any one of claims 1 to 7 is implemented.

Images